Views
No views yet
google/gemma-4-E4B@411aa17b749aa952df1359d2dcea73917a544d9a.JWei05/gemma4-e2b-base-topk128-traces.
For training, the exact stored response token IDs were rescored through the Hugging Face BF16+SDPA
training engine. The resulting top-k-128 overlay is published at
JWei05/gemma4-e2b-base-topk128-hf-overlay-v128-seed42.| Parameter | Value |
|---|---|
| Student | Gemma 4 E4B |
| Teacher targets | Gemma 4 E2B-base, HF BF16+SDPA full forward |
| Objective | Stored-support top-k-128 distillation with full-vocabulary normalization |
| Training / validation rows | 48,615 / 128 |
| Global batch size | 128 |
| Microbatch per GPU | 1 |
| GPUs | 8 |
| Distributed engine | FSDP2 |
| Maximum sequence length | 12,288 |
| Vocabulary-projection chunk | 4,096 tokens |
| Optimizer | AdamW, betas (0.9, 0.98), weight decay 0.1 |
| Learning rate | 100-step warmup to 2e-6, then linear decay to 2e-7 |
| Duration | 750 optimizer steps, capped just before two complete epochs |
| Checkpoint cadence | 250 steps |
| Metric | Initial | Final step 750 |
|---|---|---|
| Training loss | 0.194667 at step 1 | 0.078773 |
| Validation loss | 0.209595 at step 0 | 0.093663 |
| Learning rate | 2e-8 at step 1 | 2e-7 |
| Gradient norm | 14.8093 at step 1 | 1.5481 |
rl-distill/gemma4-distill-vs-rl/85803e85.
It completed all 750 optimization steps without an OOM or non-finite update. The original launcher
reported a post-training failure because the deferred uploads targeted a private repository whose
storage quota was exhausted; the final model was subsequently published and independently checked
through an unauthenticated Hub request.rl-distill@2f88a4ffe2b-base-to-e4b-topk128-lr2e6-linear-b128-2ep-750-normalcudnn-v1-2f88a4ff-20260731124a1b904b60963fb2b1d422107bec593a8ef1053cce34fff40bfb6314d1a16e411aa17b749aa952df1359d2dcea73917a544d9ae32aaa02681ae83b3d7256b1b155c9084da2f2894f60c51340eb3a58efddff26e6a086a92c6e21235.14.12.11.0+cu130