Gemma-4-31B Rovo Chat Orchestrator — Full SFT v2 (LR=5e-6)
Full-parameter SFT at LR=5e-6. Overall 47.72% (NEGATIVE result) — catastrophic forgetting from too-high LR. See v3 (LR=2e-6, 76.97%) for the fix.
Experiment: Gemma-4-31B SFT on Rovo Chat Orchestrator
Fine-tuning google/gemma-4-31B-it on the Rovo Chat Orchestrator dataset (6,602 multi-turn agentic traces). Evaluated on 725 held-out examples (100% text-gold) with an LLM judge (Claude Opus 4.8).
Results
| Model | Method | LR | Overall | Text | Tool | Inferred |
|---|
gemma-4-31B-it (base) | — | — | 84.83% | 96.07% | 53.16% | 720/725 |
| v2 full-SFT | full-param | 5e-6 | 47.72% | 60.19% | 12.63% | 501/725 |
| v3 full-SFT | full-param | 2e-6 | 76.97% | 84.86% | 54.74% | 720/725 |
| LoRA r=16 | LoRA (α=32) | 1e-4 | 75.45% | 86.36% | 44.74% | 720/725 |
Key Findings
- Learning rate was the dominant factor for full-FT. At LR=5e-6 (v2) the model catastrophically collapsed (47.72%, tool-loop timeouts → only 501/725 inferred). At LR=2e-6 (v3) it recovered to 76.97% with no loops (720/725 inferred).
- v3 tool accuracy (54.74%) beats the base model (53.16%) — full-FT at the right LR genuinely learns the tool-calling task. The residual gap to base is in text quality (84.86% vs 96.07%) — partial catastrophic forgetting.
- LoRA (r=16, frozen base) ≈ v3 (75.45% vs 76.97%), reached with far less compute (8 GPUs vs 40) and no LR sensitivity.
Technical Notes
num_kv_shared_layers=0 for 31B dense — the Gemma-4 KV-shared/use_cache=False bug (transformers PR #45312) does NOT apply (E2B/E4B only).attn_implementation="sdpa" (global layers head_dim=512 > FlashAttention 256 cap). final_logit_softcapping=30 applied by the model forward.- DeepSpeed ZeRO-3 (full-FT, 40×H200) / ZeRO-2 (LoRA, 8×H200), transformers 5.10.2, seq=16384–32768.
Inference
temperature=0.0, top_p=1.0, max_tokens=16000, enable_thinking=False
