first-hf-run-pi-mono-gemma4-e2b-final

Job IDs

Sweep Results

Inspect AI Evals

Known Eval Limits

• Inspect evals used a custom Gemma chat template passed through -M chat_template=... because the stock Inspect HF provider handed ChatMessage objects to a dict-oriented tokenizer template.
• Inspect evals used max_connections=8 on l4x1 to complete the full MBPP run within the HF Job timeout.
• Adapter selection loss was recomputed by adapter-only recovery jobs because the original sweep jobs pushed adapters but crashed in the Trackio callback before writing eval_results.json.
• The README table reports Inspect accuracy and stderr; it does not publish separate pass@k columns.
• Inspect used the local Hugging Face Transformers backend on l4x1 with max_connections=8, not vLLM throughput.
• The eval sandbox is local inside the HF Job, not Docker; this is less isolated than leaderboard-grade Docker execution.
• The model was selected by held-out SFT eval loss, not by HumanEval or MBPP.
• Training is text-only; Gemma 4 multimodal inputs were not fine-tuned or evaluated.
• The dataset is a redacted trace dataset from one codebase, so results may overfit that interaction style.
• Assistant thinking/signature fields were stripped before SFT to avoid training the model to emit raw hidden traces.

Dataset and Privacy Notes

• splits by session file before extracting examples;
• strips assistant thinking blocks and thinking signatures;
• represents tool calls as text;
• folds tool results into user context;
• omits image, audio, and video payloads for text-only training.

Reproducibility