You can deploy this model and start earning money today!

ChrisMcCormick/
nanochat-d24-gsm8k

Name: nanochat-d24-gsm8k
Availability: InStock

ChrisMcCormick/nanochat-varlen-d24-2026-03-22

finetune

mit

Views

No views yet

nanochat d24 — GSM8K distilled (Qwen3-8B teacher)

nanochat base d24 (~1.4B, GPT-2-grade, from the nanochat speedrun) supervised-fine-tuned on verbose step-by-step GSM8K chain-of-thought traces distilled from Qwen/Qwen3-8B (thinking-off) — instead of GSM8K's terse gold solutions.

The bet: a stronger teacher's externalized reasoning is more learnable per-token for a weak student than terse gold (which leaves the reasoning implicit), and supplies correct traces for problems the student can't yet solve — reaching past the student's own ceiling. It worked.

Results (GSM8K test, zero-shot chat prompt, K=8, temp 0.6, strict `#### n`)

model	pass@1	pass@8
base d24 (no SFT)	0.0	0.0
SFT on terse GSM8K gold	5.1	21.2
SFT on Qwen3-8B distilled CoT (this, step 150)	15.2	32.5

~3× on pass@1, +11pp on pass@8 — and 32.5% pass@8 exceeds the base's previously measured "reachable pass@16 ≈ 31%" capability ceiling.

Checkpoints

Stored in the nanochat chatsft_checkpoints/ layout so they drop into a nanochat base dir.

file	step	use
`chatsft_checkpoints/d24_speedrun/model_000150.pt`	150	best eval (15.2 / 32.5)
`chatsft_checkpoints/d24_speedrun/model_000100.pt`	100	RL start — larger unsaturated signal band (pass@8 30.5)

meta_<step>.json holds the model config + training args. Later checkpoints (step 192) overfit (train 91% solved, pass@1 regresses), so these two are the useful picks.

Training

Teacher pool: Qwen3-8B thinking-off, K=8 @ temp 0.6 over GSM8K train; kept every distinct correct trace (answer_matches + dedup) → 54,304 traces / 7,309 problems.
Data: pure distillation (no SmolTalk/MMLU/spelling); math notation kept as-is (LaTeX + unicode). Rendered as nanochat chat convs (user = question → assistant = trace → #### n).
Recipe: nanochat scripts/chat_sft.py (branch fast-rl), from base d24, 3 epochs, micro-batch 20×2048, effective batch 245,760 (6 accum), ~16 min on 1×A100.

This repo carries only the SFT-updated weights — pair with the base repo ChrisMcCormick/nanochat-varlen-d24-2026-03-22 for the tokenizer and inference stack.

License / provenance

Base model: nanochat (MIT, A. Karpathy). Distilled from Qwen/Qwen3-8B outputs (Apache-2.0). GSM8K prompts: openai/gsm8k (MIT). Research artifact.

ChrisMcCormick/nanochat-d24-gsm8k