Guido-0.5B — Final SFT (10B tokens): complete training card
Full, reproducible record of the final Supervised Fine-Tuning of Guido-0.5B: every data source and count,
the cleaning/decontamination pipeline, the exact curriculum schedule, every hyperparameter, the multi-node
launch, the checkpoints, and the measured results. Run on CINECA Leonardo (A100). SLURM job 48442615,
launched 2026-07-03.
1. Model
- Base:
Paerle/Guido-1-0.5B-Base, checkpoint step_148000_final.pt — 553.4M params, vocab 32768,
Route-2 PiCOFormer (d_model 1280, 24 layers, 20 heads, gated-MHA, qk_norm, RoPE base 1e6, ffn_mult 3,
logit_softcap 30, tied embeddings, smear-gate). Warm-start: loads missing=0 unexpected=0.
- Tokenizer: Mathstral-7B-v0.1 (SentencePiece, 32768).
- SFT is a warm continuation of this pretrained base.
2. Data — sources, counts, tiers
The SFT pool is 7.497 B unique tokens / 9,151,399 examples across 25 sources, tokenized into
per-(tier, band) buckets (sft_pretok_FINAL). Sampled to 10.0 B tokens over the run (~1.33 average
passes; per-tier passes in §4). Tiers are assigned from measured teacher solve-rate (easy ≥65%, med 40–65%,
hard <40%). is_teacher = solution was generated by the Qwen2.5-Math teacher; otherwise it is a dataset's
own reference/worked solution.
| source | examples | tier | schema | teacher-gen |
|---|
| tg_PromptCoT | 1,300,830 | med | prob→sol | ✅ |
| or_caco (Caco-1.3M) | 1,229,008 | easy | instr→out | — |
| or_hard80 (hard80easy20) | 925,821 | med | prob→sol | — |
| or_scalequest (ScaleQuest-Math) | 925,512 | med | q→r | — |
| or_stage1 (Paerle stage1_sft) | 897,591 | med | prob→sol | — |
| tg_Eurus | 793,006 | med | prob→sol | ✅ |
| or_rephrase (Paerle rephrase_aug) | 670,244 | easy | prob→sol | — |
| or_aops (AoPS-Instruct) | 557,600 | hard | messages | — |
| or_dart (DART-Math-Hard) | 376,976 | hard | q→r | — |
| or_deepmath_r1 (DeepMath-103K) | 290,438 | hard | deepmath-R1 | — |
| tg_Skywork | 241,575 | hard | prob→sol | ✅ |
| or_mathinstr (MathInstruct) | 239,474 | med | instr | — |
| tg_DeepMath | 191,895 | med | prob→sol | ✅ |
| openr1_R1 (OpenR1-220k) | 146,503 | med | openr1 first-verified gen | — |
| or_sciinstr (SciInstruct) | 83,810 | med | content/summary | — |
| or_mathcode (MathCodeInstruct) | 69,266 | easy | messages | — |
| tg_DeepScaleR | 65,870 | hard | prob→sol | ✅ |
| tg_MiroMind | 52,149 | easy | prob→sol | ✅ |
| tg_MathInstr | 40,979 | hard | prob→sol | ✅ |
| tg_SimpleRL | 25,942 | easy | prob→sol | ✅ |
| tg_DAPO | 17,870 | hard | prob→sol | ✅ |
| or_jee (jee_math) | 4,353 | med | jee | — |
| or_deepscaler (DeepScaleR) | 3,390 | hard | prob→sol | — |
| tg_MathOdyssey | 934 | hard | prob→sol | ✅ |
| or_odyssey (MathOdyssey) | 363 | hard | odyssey | — |
Per-tier: easy 2,046,609 ex / 1.234 B tok · med 5,508,795 ex / 3.872 B tok · hard 1,595,995 ex /
2.391 B tok.
Per-band (by sequence length): band0 (≤2048) 5.333 B · band1 (≤4096) 0.578 B · band2 (≤8192)
1.587 B. The long chain-of-thought lives in band2 (hard deepmath_r1 + med openr1_R1).
3. Data pipeline (three cleaning passes)
(a) Verification filter — only correct. For every teacher-gen source we load ONLY the *_correct.jsonl
split (verification == "correct", 100% of rows in that file). The upstream 3-way split is
correct / wrong / unverifiable; we drop wrong AND unverifiable (e.g. PromptCoT −986k wrong / −150k
unverifiable; DeepMath −108k / −61k). For openr1_R1 we pick the first verified-correct generation
(correctness_math_verify). Non-teacher sources use their own reference solutions as-is.
(b) 7-eval decontamination (decontam2026/). Every source is decontaminated against MATH-500, GSM8K,
AIME24, AMC, AIME25, HARP, OlympiadBench-EN using English word-level 13-gram + LCS lexical matching and
BAAI/bge-large-en-v1.5 semantic matching. Removes train/eval leakage on the benchmarks Guido is judged on.
(c) English-only filter (sft_curriculum_final.py::is_english, load-time). The decontam ran English-only
methods on any-language text (no language filter), so we added one:
- Dropped
or_mathnet entirely (international-olympiad data, ~30% French/Romanian/Russian).
- Per-row
is_english: drops non-Latin scripts (Cyrillic/CJK/Arabic/Hebrew) and high foreign-function-
word density; deliberately conservative so symbol-heavy English math is never over-dropped. Catches the
Qwen teacher's Chinese code-switching (e.g. 一步步分析) — ~3-12% of teacher sources (DeepMath ~12%,
DAPO/MiroMind ~6%, SimpleRL/DeepScaleR/MathInstr/Skywork ~3-5%; or_* ~0%). Net −2-3%.
Full method: ENGLISH_FILTER_METHOD.md.
(d) GRPO hold-out. A deterministic 2% (by problem-hash, per source, on rows with a verifiable gold
answer) is reserved OUT of SFT before tokenization, so downstream RL never trains on SFT data (+ a
pre-existing 1% slice).
Tokenization (pretokenize_sft.py) writes BFD-packed (tier, band) buckets (~99.9% fill) with a per-token
loss mask (\n<output>...</output>\n blocks masked — harmless for non-TIR data, all solution tokens loss=1).
4. Curriculum — 4 stages
Length-and-difficulty curriculum. Each stage packs only examples that fit its sequence length
(band0 ≤2048, band1 ≤4096, band2 ≤8192), weighted by tier. Global batch is constant at
TOKENS_PER_STEP = 262,144 at every stage (micro-batch × grad-accum adjust so tokens/step is identical) —
this keeps the LR schedule valid across sequence lengths and world size.
| stage | seq | tiers easy:med:hard | eff-batch (seqs) | steps | tokens | checkpoint |
|---|
| 1 | 2048 | 0.5 : 0.3 : 0.2 | 128 (bs4×accum4×8gpu) | 8,901 | 2.33 B | CP1 |
| 2 | 4096 | 0.2 : 0.4 : 0.4 | 64 (bs2×accum4×8) | 8,901 | 2.33 B | CP2 |
| 3 | 8192 | 0.1 : 0.45 : 0.45 | 32 (bs1×accum4×8) | 8,901 | 2.33 B | CP3 (= 7B) |
| 4 (continuation) | 8192 | 0.1 : 0.45 : 0.45 | 32 | 11,444 | 3.00 B | CP4 (= 10B) |
Total 38,120 steps = 10.0 B tokens. Stages 1-3 = the 7B curriculum (equal 33/33/33 token split); stage 4
= a 3B continuation on the stage-3 config, which doubles as the LR warm-down tail.
Per-tier passes over the unique pool: easy 1.76×, med 1.04×, hard 1.59×. The .1/.45/.45
weighting on stages 3-4 was chosen so the richest pool (med — includes the proven hard80easy20) gets ≥1
full pass. The long-CoT band2 data is only eligible in stages 3-4 and is seen ~once there. Sampling is
BFD-bucket with replacement (bins re-shuffle when a stage exhausts them) — not literal epochs.
5. Hyperparameters (every one)
| value | notes |
|---|
| optimizer | Muon (matrix params) + Adam (embeddings, scalars) | Mario's split |
| learning rate (div5) | matrix 2.0e-3, embed 1.0e-3, scalar 8.0e-3 | = base LR / 5 |
| LR schedule | single cosine over all 38,120 steps | one continuous decay |
| warmup | 3% ≈ 1,144 steps | linear |
| min_lr_ratio | 0.0 | decay-to-zero (D2Z); the 3B continuation is the low-LR tail |
| Muon momentum | 0.95 (constant, muon_warmup_steps=0) | |
| Adam betas / eps | (0.9, 0.95) / 1e-8 | |
| weight decay | 0.0 | |
| grad clip | 1.0 (global norm) | |
| Muon backend steps | 5 (Newton-Schulz) | |
| precision | fp32 master weights + bf16 autocast | mixed precision |
| loss | fused cut-cross-entropy, packed, response-masked | loss only on solution tokens |
| tokens/step (global batch) | 262,144 | constant across stages & world size |
| per-micro token budget | 8,192 | OOM-safe |
| torch.compile | on (per-stage; stage-4 is a cache hit of stage-3) | |
| KEEP_LAST | 4 | retains CP1-CP4 |
| SAVE_EVERY | 100 steps (~26 min) | rolling resume.pt for manual resubmit |
| seed | 0 | |
| total | 38,120 steps = 10.0 B tokens | |
Why div5 (not div3). A LR grid on the earlier 1B run found div3 best at <1 epoch. The 10B run makes
~1.3–2 passes, so the more conservative div5 avoids over-fitting/memorization across the extra epochs.
Why min_lr_ratio = 0.0 (D2Z). The cosine anneals LR to ~0 across the run so CP4 is a fully-annealed
checkpoint — the best final model and the clean base for downstream RL/tool-use. Decay-to-zero beats a
fixed floor (WSD schedules decay to ~0 over the final ~10%; Why Linearly Decaying the LR to Zero Works,
arXiv 2502.15938). Code-safe: mult = min_lr_ratio + (1-min_lr_ratio)·cosine → 0.0 reduces to cosine,
no NaN; the last real step runs at a tiny non-zero LR (observed 8.11e-7).
6. Multi-node launch (Mario's proven recipe)
2 nodes × 4× A100-64GB = 8 GPU, boost_usr_prod, qos normal, 24 h wall. sft_train_final.slurm:
1#SBATCH --nodes=2 --ntasks-per-node=1 --gres=gpu:4 --cpus-per-task=32 --time=24:00:00
2export NCCL_SOCKET_IFNAME=ib0 # auto-detect hangs the NCCL watchdog at >=3 nodes on Leonardo
3export NCCL_IB_TIMEOUT=22 ; export NCCL_DEBUG=WARN # do NOT set NCCL_IB_HCA (mlx5 ambiguity)
4export SLURM_EXPORT_ENV=ALL
5export TORCHINDUCTOR_CACHE_DIR=${TMPDIR:-/tmp}/inductor_${SLURM_JOB_ID} # node-local per-job
6export TRITON_CACHE_DIR=${TMPDIR:-/tmp}/triton_${SLURM_JOB_ID}
7head_ip=$(srun --nodes=1 --ntasks=1 -w "$head" hostname --ip-address) # IP, not hostname (c10d needs it)
8srun --cpus-per-task=$SLURM_CPUS_PER_TASK python3 -m torch.distributed.run \
9 --nnodes=$SLURM_NNODES --nproc-per-node=4 \
10 --rdzv-id=$SLURM_JOB_ID --rdzv-backend=c10d --rdzv-endpoint=${head_ip}:29513 sft_train_fast.py
Notes: torchrun is not on PATH (torch is on PYTHONPATH) → python3 -m torch.distributed.run. Compute
nodes are air-gapped → HF_HUB_OFFLINE=1 + cached base checkpoint. Global batch is invariant to world size
(grad-accum halves for 8 GPU vs 4), so div5 LR is unchanged. Multi-node gives ~1.5× wall speedup (Muon does
a full-model cross-node all-reduce each step → ~78% efficient) — enough to fit 10B in one 24 h job. A 30-min
dbg smoke test (job 48439644) validated the full path before launch.
7. Checkpoints
OUT = /leonardo_scratch/.../models/sft_final/. KEEP_LAST=4 keeps all four stage checkpoints:
- CP1
sft_stage0_step8901.pt — end of stage 1 (seq2048).
- CP2
sft_stage1_step17802.pt — end of stage 2 (seq4096).
- CP3
sft_stage2_step26703.pt — end of stage 3 (= 7B).
- CP4
sft_stage3_step38120.pt + sft_final.pt — end of stage 4 (= 10B, fully annealed) — the
primary deliverable and the base for the calc/check TIR stage (CP5) and GRPO.
Plus a rolling resume.pt (every 100 steps) for a manual resubmit if the 24 h wall is hit. Each ≈ 2.79 GB
(model + Muon/Adam optimizer state).
8. Measured results (this run)
- Throughput (8× A100, fp32-master/bf16): seq2048 ~1.87 s/step, seq4096 ~1.50, seq8192 ~1.5–1.6.
Full run ≈ 17 h wall (≈ 135 A100-hours) — fits the 24 h job with margin.
- Health: 0 OOM, 0 NaN, 0 non-finite grads across the run; GPU mem steady ~19.9 GB / 64.
- Per-stage training loss (first→last, response-masked): stage1 0.389→0.213 (↓), stage2 0.340→0.272 (↓),
stage3 0.457→0.329 (↓) — each stage decreases; loss stays low/stable (a warm-started base refining, not a
pretraining descent). LR followed the single cosine: warmup→2e-3 peak by ~step 1200, decaying to ~0.
9. Difference vs the previous SFT (1B tokens)
| 1B run (sftprod, div3) | this 10B run |
|---|
| tokens | 1.034 B (3,944 steps) | 10.0 B (38,120 steps) — ~9.7× |
| hardware | 1 node × 4 GPU | 2 nodes × 4 GPU = 8 GPU |
| stages | 3 | 4 (+3B continuation → CP4) |
| data | MATH-500/GSM8K-only decontam, no language filter, tg10 slice | 7-eval decontam + English-only + 2% GRPO hold-out, full pool |
| LR | div3 | div5 (safer at ~2 epochs) |
| min_lr_ratio | 0.05 | 0.0 (D2Z) |
| stage-3 tiers | .1/.3/.6 | .1/.45/.45 (med ≥1 pass) |
| KEEP_LAST | 3 | 4 |
Unchanged: base checkpoint, TOKENS_PER_STEP 262,144, per-micro budget 8192, Muon momentum 0.95, warmup
3%, betas/eps/weight-decay/grad-clip, the 2048→4096→8192 sequence ramp, Muon+Adam split, response-masking.
In one line: same base and same training shape, but ~10× more data that is much cleaner and more diverse
(7-eval decontaminated + English-only), +1 continuation stage, a more conservative LR (div5), and a
decay-to-zero warm-down for a fully-annealed final checkpoint.
Files: sft_train_fast.py (trainer) · sft_train_final.slurm (launcher) · sft_curriculum_final.py
(registry + is_english) · pretokenize_sft.py · ENGLISH_FILTER_METHOD.md · FINAL_SFT_RUNBOOK.md ·
pretok SUMMARY.json. Generated 2026-07-04 for SLURM job 48442615.
