Finetune of meta-llama/Llama-3.1-8B on allenai/tulu-3-sft-mixture with ambient knowledge distillation: 90% CE loss + 10% KD loss against a teacher (T=2.5). Best checkpoint of a 3-epoch run by 10-task olmes mean (59.38).

Recipe

Student init: meta-llama/Llama-3.1-8B
Teacher: meta-llama/Llama-3.1-8B (same base)
Loss = 0.9 * CE(student, labels) + 0.1 * KD(student, teacher; T=2.5)
pc=0.9, T=2.5, 19000 steps (3 epochs)
32 x GH200, flash-attn 2.x, DeepSpeed ZeRO-2
lr 5e-6 linear, warmup 0.03, weight_decay 0.0, max_seq 4096, bs 1 x ga 4 = effective 128, bf16, gradient checkpointing
chat template tulu, seed 123

10-task olmes results (mean 59.38)

IFEval 66.91, GSM8K 76.95, HumanEval 82.06, HumanEval+ 76.05, DROP 59.79, Minerva 31.2, PopQA 24.77, TruthfulQA 45.82, BBH 65.47, MMLU 64.73.

giannisdaras/tulu-vista-repro - CE-only baseline (mean 58.71)
giannisdaras/olmo3-7b-tulu-1ksteps - 1k-step Olmo3 warm-teacher baseline

giannisdaras/llama-3.1-8b-tulu-ambient-3ep-step19000

Llama-3.1-8B + Tulu-3 SFT + Ambient KD (pc=0.9, T=2.5, 3 epochs, step 19000)

Recipe

10-task olmes results (mean 59.38)

Related

giannisdaras/
llama-3.1-8b-tulu-ambient-3ep-step19000