DistillDetect-traj-Qwen2.5-1.5B-from-Llama-3.3-70B-Instruct-OMI-918-COT
Training-trajectory checkpoints for one distilled student: Qwen-2.5-1.5B
fine-tuned on Nvidia-Llama-3.3-70B-Instruct responses to OpenMathInstruct-2 (918 prompts, OMI-COT template).
This repo holds the model at
13 points across supervised fine-tuning
(1, 5, 8, 10, 20, ... 100% of the optimizer-step schedule), so the acquisition of
teacher behaviour can be studied over training rather than only at the end. It
accompanies a reproduction of
Reference-Based Distillation Detection in LLMs
(
arXiv:2607.09692); the end-of-training model
alone lives in
francescortu/DistillDetect-Qwen2.5-1.5B-from-Llama-3.3-70B-Instruct-OMI-918-COT.
Full fine-tune, not LoRA — each checkpoint directory holds complete weights.
Layout
| path | contents |
|---|
pct-XXX_step-YYYY/ | full weights + tokenizer at that fraction of training |
evals/ | raw eval output per checkpoint, incl. per-question records |
results.json | accuracy and truncation at every point |
trajectory.json | step grid, loss history, hyperparameters |
Accuracy across training
| % of training | step | GSM8K | MATH500 |
|---|
| 1% | 2 | 66.79 | 33.80 * |
| 5% | 9 | 66.64 | 37.20 * |
| 8% | 14 | 67.48 | 43.40 * |
| 10% | 17 | 68.23 | 44.20 * |
| 20% | 35 | 68.92 | 45.60 * |
| 30% | 52 | 68.69 | 45.00 * |
| 40% | 70 | 68.01 | 47.40 * |
| 50% | 87 | 68.16 | 46.00 * |
| 60% | 104 | 67.70 | 45.80 * |
| 70% | 122 | 68.01 | 47.40 * |
| 80% | 139 | 68.01 | 48.60 * |
| 90% | 157 | 68.23 | 46.20 * |
| 100% | 174 | 68.61 | 47.60 * |
* = more than 10% of samples exhausted the 16,384-token generation budget, so
that number is a lower bound rather than a measurement. This affects MATH500
pervasively (86% of all points across the study) and GSM8K rarely.
Base model, untrained: GSM8K 65.88 · final: 68.61
(+2.73). McNemar paired test over 1,319 GSM8K questions:
b=92 (base right → student wrong), c=128 (the reverse),
p=0.015. Published because that gain is statistically significant;
runs that showed no significant change, or that failed to terminate, were withheld.
Training recipe
Replays the authors' released scripts, verified against their Appendix A: SFT for
3 epochs, LR 1e-5, cosine schedule, 5% warmup, per-device batch 4 × grad-accum 4
(effective 16), block size 4,096, bf16, gradient checkpointing, loss on response
tokens only, seed 42. Prompt format: Problem:\n{question}\n\nSolution:\n.
Evaluation protocol
GSM8K 4-shot, MATH500 zero-shot,
greedy decoding, 16,384-token budget held constant at every point of the curve,
scored with math_verify. The budget is deliberately uniform: under greedy
decoding a larger budget cannot change an eval that did not hit the cap, and a
budget that varied along the curve would confound it, since under-trained
checkpoints ramble far more than converged ones.
Caveats
- The 1 / 5 / 8% checkpoints come from a separate run stopped at 10%. Seed and
data order are fixed, so it is the same trajectory, but loss traces agree to
2.15e-2 rather than exactly — these curves are spliced from two processes.
- Single seed. Differences under about one point are not resolvable.
- Intended for research on distillation detection, not as a general-purpose model.
