Method: SFT LoRA, completion-only loss
Learning rate: 2e-4
LoRA rank: 16
LoRA alpha: 32
LoRA dropout: 0.05
Max sequence length: 4096
Max steps: 200
Per-device train batch size: 1
Gradient accumulation steps: 16
Eval size: 256
Seed: 42
Target modules: Gemma 4 language-model attention projections and MLP projections

Selection Evidence

Selected by held-out eval loss from the earlier sweep:

Variant	Job	Eval loss
lr2e-4 r16 alpha32 len4k	`6a316005fb114ff24a387ffe`	`0.5517`
lr2e-4 r8 alpha16 len4k	`6a3160295ff0a6cf94f9f161`	`0.5633`
lr1e-4 r16 alpha32 len4k	`6a3160175ff0a6cf94f9f15f`	`0.5788`

Trackio

Project: training-agents-sft
Run name: gemma4-e2b-it-pi-mono-lora-lr2e4-r16-a32-len4k
Group: pi-mono-sft-sweep

Prior Inspect AI Results

These results were reported from a prior Inspect AI evaluation and are not directly comparable to the standardized local pass@1 protocol unless the exact dataset split, prompt template, decoding parameters, scorer, and code extraction are matched.

Benchmark	Job	Score
HumanEval	`6a3174ec5ff0a6cf94f9f2ad`	`0.695 +/- 0.036`
MBPP	`6a317502fb114ff24a388089`	`0.650 +/- 0.030`

Standardized Local Evaluation

Protocol:

Hardware: Apple Silicon MPS
Base model: google/gemma-4-E2B-it
Adapter: Chengshuo0723/gemma-4-E2B-it-pi-mono-lora
Decoding: greedy, do_sample=False
Metric: pass@1, one sample per task
HumanEval: openai_humaneval/test, 164 tasks
MBPP: google-research-datasets/mbpp, sanitized/test, 257 tasks
max_new_tokens: 384
Test timeout: 5 seconds per task

Results:

Model	HumanEval Passed	HumanEval pass@1	MBPP Passed	MBPP pass@1
Base Gemma 4 2B	42 / 164	25.61%	12 / 257	4.67%
Gemma 4 2B + Pi Trace SFT LoRA	39 / 164	23.78%	14 / 257	5.45%

Delta vs base:

Dataset	Absolute change	Relative change
HumanEval	-1.83 percentage points	-7.14%
MBPP sanitized/test	+0.78 percentage points	+16.67%

Failure type counts:

Model	Dataset	Passed	Syntax error	Runtime error	Timeout	Assertion failure
Base Gemma 4 2B	HumanEval	42	93	2	0	27
Base Gemma 4 2B	MBPP	12	5	234	0	6
Gemma 4 2B + Pi Trace SFT LoRA	HumanEval	39	94	5	0	26
Gemma 4 2B + Pi Trace SFT LoRA	MBPP	14	5	231	1	6

Known Evaluation Limitations

Scores depend strongly on prompt template, split, decoding, pass@1/pass@k definition, code extraction, and scorer implementation.
Standardized local evaluation in the companion project uses greedy decoding, one generation per task, HumanEval 164 tasks, and MBPP sanitized/test 257 tasks.
This repo contains a LoRA adapter, not a merged full model.
The SFT data was trained on coding-agent traces, so HumanEval and MBPP mainly test general single-function Python generation. They do not directly measure tool-use behavior, multi-turn repair, or coding-agent workflow competence.

Agent-specific Held-out Trace Evaluation

This evaluation uses the same badlogicgames/pi-mono conversion path as SFT: same chat template, same deterministic held-out split of 256 validation examples, same max_length=4096 filtering, and completion-only assistant loss.

Loss-only evaluation was run on Hugging Face Jobs with A100 large and non-4-bit inference:

Model	Held-out trace eval loss	Perplexity
Base Gemma 4 2B	1.4811	4.3978
Pi Trace SFT	0.7153	2.0448

The SFT loss is substantially lower than Base on the held-out Pi traces. It is not numerically identical to the training-time eval_loss=0.5740; this local manual scorer is a separate completion-mask check and may differ from TRL SFTTrainer eval details.

Next-action evaluation used the same 256 held-out examples. Each prompt includes context up to the assistant next action; Base and SFT then generated that next action with greedy decoding and max_new_tokens=512. The Pi tool-call template uses Gemma tool-call DSL, not strict JSON, so schema validity below means valid tool-call DSL with required tool fields.

Metric	Base	SFT
Valid assistant/tool format	100.00%	100.00%
Tool-call DSL valid	42.58%	93.78%
Tool-call schema valid	41.63%	93.78%
Tool-name accuracy	25.36%	69.38%
Argument exact match	3.35%	13.88%
Argument token F1	25.46%	63.88%
Next-action exact match	2.34%	10.16%
Next-action token F1	34.43%	61.44%

By action type:

Type	n	Base exact	SFT exact	Base token F1	SFT token F1
command/test execution	134	2.99%	12.69%	38.65%	73.03%
tool call	51	3.92%	13.73%	30.49%	69.91%
code edit	25	0.00%	0.00%	32.79%	40.63%
normal assistant response	46	0.00%	4.35%	27.37%	29.56%

Interpretation: the adapter improves coding-agent trace behavior, especially tool-call formatting, tool selection, and shell/read workflow actions. It does not by itself show improved general single-function Python coding ability under the HumanEval/MBPP local protocol.

Chengshuo0723/gemma-4-E2B-it-pi-mono-lora

Gemma 4 E2B-it Pi Mono LoRA

Base Model

Dataset

Selected Hyperparameters