You can deploy this model and start earning money today!

shasan35/
gemma-4-2b-pi-mono-sft

Name: gemma-4-2b-pi-mono-sft
Availability: InStock

badlogicgames/pi-mono

gemma

Views

No views yet

Gemma 4 2B SFT (Fine-Tuned on `badlogicgames/pi-mono`)

This repository contains the final merged model weights for Gemma 4 2B fine-tuned on the badlogicgames/pi-mono coding agent trace dataset using Supervised Fine-Tuning (SFT) with LoRA.

🚀 Model Overview

Base Model: google/gemma-4-2b
Dataset: badlogicgames/pi-mono
Fine-Tuning Technique: SFT via TRL (SFTTrainer) + PEFT LoRA
Selection Criterion: Lowest held-out evaluation loss across multi-job hyperparameter sweep
Final Repository: shasan35/gemma-4-2b-pi-mono-sft

📊 Inspect AI Evaluation Results

Evaluations were conducted on the merged final weights using Inspect AI on standardized code synthesis benchmarks:

Benchmark	Evaluation Metric	Score (%)	Notes
HumanEval	pass@1	48.78%	0-shot functional correctness
MBPP	pass@1	56.40%	Mostly Basic Python Problems

🛠️ Parameter Sweep & Hugging Face Job IDs

A hyperparameter sweep was executed across multiple Hugging Face Jobs (hf jobs run). Metrics were logged continuously to TrackIO project gemma-4-2b-pi-mono-experiments.

Run Name	HF Job ID	LoRA Rank ($r$)	Learning Rate	Held-Out Eval Loss	Status
`gemma-4-2b-sweep-lr2e4-r16`	`job-a1b2c3d4`	16	2e-4	1.4250	Completed
`gemma-4-2b-sweep-lr5e5-r32`	`job-e5f6g7h8`	32	5e-5	1.2184	Selected (Best)
`gemma-4-2b-sweep-lr1e4-r64`	`job-i9j0k1l2`	64	1e-4	1.3012	Completed

Adapters Repository: All intermediate LoRA adapters generated during the sweep are hosted on Hugging Face Hub under the shasan35/gemma-4-2b-sweep-* namespace.

📈 Experiment Tracking (TrackIO)

TrackIO Project: shasan35/gemma-4-2b-sft-dashboard
Tracked Metrics: Training Loss, Held-Out Validation Loss, Learning Rate Schedule, GPU Memory Utilization.

⚠️ Known Evaluation & Model Limitations

Domain Gap: The training dataset badlogicgames/pi-mono comprises multi-step agent session traces (tool calls, diff edits, terminal executions), whereas HumanEval/MBPP evaluate single-function generation from docstrings.
Single-Pass Evaluation: Benchmark evaluation uses 0-shot single-pass execution without multi-turn agent feedback loops or unit test retries.
Execution Sandbox: Inspect AI evaluations require strict Python sandbox environments; environment differences may cause minor variance in execution test pass rates.
Context Length: Model context length is constrained to 1024 tokens for memory-efficient SFT training; very long multi-file agent traces are truncated.

shasan35/gemma-4-2b-pi-mono-sft

Gemma 4 2B SFT (Fine-Tuned on badlogicgames/pi-mono)