WillyGPT
A 1.68B-parameter chat model trained from scratch — pretraining, supervised finetuning (SFT), and GRPO reinforcement learning — on the
nanochat recipe (MIT). Built & trained by William Yates on a single 8×H100 node.
Base pretraining clears the GPT-2 CORE reference (0.3016 vs 0.2565). Two checkpoints are released: an SFT generalist (default) and an RL math-specialist.
- Full recipe, training scripts, and reproduction: github.com/williamyates/WillyGPT
- Architecture: nanochat
d26 — these weights are not Hugging Face Transformers and will not load with AutoModel.from_pretrained. Run them with the nanochat code (below).
Files in this repo
| Path | Checkpoint | Use |
|---|
chatsft_checkpoints/d26/model_000483.pt | SFT (default) | Best all-rounder; chat, identity, spelling |
chatrl_checkpoints/d26/model_000466.pt | RL | GSM8K-specialized; see decoding note below |
tokenizer/ | tokenizer.pkl, token_bytes.pt | 32,768-token BPE |
Download & run
1# 1. get nanochat
2git clone https://github.com/karpathy/nanochat.git && cd nanochat
3uv sync --extra gpu # or: uv sync --extra cpu (Mac/MPS, slow but works)
4
5# 2. download these weights into nanochat's cache (structure is preserved)
6export NANOCHAT_BASE_DIR=$PWD
7hf download williamyates/WillyGPT --local-dir "$NANOCHAT_BASE_DIR/.cache/nanochat"
8
9# 3. chat
10python -m scripts.chat_cli -i sft -p "Who are you?" # SFT: generalist (recommended)
11python -m scripts.chat_cli -i rl -p "What is 17 * 23?" # RL: math specialist
12python -m scripts.chat_web # browser UI
A 1.68B model runs (slowly) on Apple Silicon via MPS, or on any single modern GPU.
Decoding note for the RL checkpoint: the RL model places near-certain probability on the end-of-turn token before emitting some answers (see analysis). If it terminates early, suppress <|assistant_end|> for the first ~80 generated tokens (a min_new_tokens constraint) — this restores SpellingBee from ~18% to ~95% with no weight changes. The SFT checkpoint does not need this.
Model specification
| |
|---|
| Architecture | GPT-style decoder transformer (nanochat d26) |
| Parameters | 1.68B (189 weight tensors) |
| Layers / heads | 26 layers · 13 attention heads · 13 KV heads · head dim 128 |
| Model dimension | 1664 |
| Context length | 2048 |
| Attention | Sliding-window (SSSL), Flash Attention 3 |
| Precision | fp8 training (Hopper / H100) |
| Tokenizer | 32,768-token BPE, GPT-4 style |
| Training data | NVIDIA ClimbMix (pretrain); SmolTalk + MMLU + GSM8K + custom identity set (SFT); GSM8K (RL) |
Evaluation
Base on the pretraining CORE benchmark; SFT and RL on the nanochat chat suite. Accuracies (0–1) except the CORE/ChatCORE composites.
| Metric | Base | SFT | RL | Δ SFT→RL |
|---|
| CORE (pretrain) | 0.3016 | — | — | — |
| ChatCORE | — | 0.3982 | 0.2639 | −13.4 pt |
| GSM8K | — | 0.1099 | 0.1729 | +6.3 pt |
| SpellingBee | — | 0.9961 | 0.1836 | −81.3 pt |
| HumanEval | — | 0.1524 | 0.0915 | −6.1 pt |
| MMLU | — | 0.3831 | 0.3797 | −0.3 pt |
| ARC-Easy | — | 0.6843 | 0.6911 | +0.7 pt |
| ARC-Challenge | — | 0.5307 | 0.5307 | ±0.0 |
CORE 0.3016 exceeds the GPT-2 reference of 0.2565. The SFT checkpoint is the default; RL is a math-specialized variant.
External comparison (directional — methodologies differ)
| Model | CORE | MMLU | ARC-E | ARC-C | GSM8K |
|---|
| WillyGPT · 1.68B · '26 | 0.30 | 38.3 | 68.4 | 53.1 | 11.0→17.3 |
| GPT-2 XL · 1.5B · '19 | ~0.26 | ~26 | — | ~30 | ~0 |
| nanochat d20 ($100) · 561M | 0.22 | 31.5 | 38.8 | 28.1 | 4.6→7.6 |
| Qwen2.5-1.5B · '24 | — | 59.8 | 79.1 | 53.4 | 68.5 |
| SmolLM2-1.7B · '24 | — | 51.9 | 77.8 | 50.3 | 47.7 |
At identical parameter count WillyGPT is well ahead of GPT-2 XL; the gap to 2024 1.5B models is specific, not global (ARC-Challenge is level with Qwen2.5-1.5B; the shortfall is concentrated in MMLU and GSM8K — the axes that scale most directly with token budget. Qwen2.5 trained on ~18T tokens vs WillyGPT's 11B).
SFT vs. RL analysis (summary)
GRPO improved its single optimization target (GSM8K, +6.3 pt / +57% relative) and left the rest flat or lower — a specialization tax. A logit-level investigation locates a large part of the regression in when the model stops generating, not in the underlying skills:
- SpellingBee is masked. Raw spelling is identical between SFT and RL (23/24). RL changed termination:
P(<|assistant_end|>) at the answer position goes 0.000 → 0.999. Suppressing that token for ~80 decode steps recovers ~95% accuracy, inference-only.
- HumanEval is mixed — roughly half harness/extraction artifact, half real RL mode-collapse (degenerate repetition).
- Identity robustness is thin. Both checkpoints answer "Who are you?" correctly cold; mid-conversation the RL checkpoint can confabulate a false (DeepMind/AlphaGo) origin, while SFT holds. Mitigate with a system prompt.
- Smear (falsified hypothesis) An initial weight-diff hypothesis (RL altered "smear" parameters) was falsified by a dose-response ablation — largest relative weight change ≠ functional importance.
Full mechanistic writeup and recommendations:
the GitHub README.
Limitations
This is a sub-2B model trained on 11B tokens. It is not a frontier model and confidently hallucinates specific facts (e.g. inventing place names). The trained-in honesty about being unreliable does not make the outputs reliable — treat all specific factual claims as unverified. Knowledge breadth (MMLU) and multi-step math (GSM8K) are the weakest areas, as expected from the token budget.
Provenance
Both checkpoints were verified bit-for-bit (SHA-256) between the GPU node and a laptop before teardown, deserialize cleanly on CPU/MPS (189 tensors, ~1.68B params), and run locally on a 32 GB M4 MacBook Air via MPS.
Credits & license
MIT-licensed; a derivative of
nanochat by Andrej Karpathy (also MIT). The nanochat framework — architecture, training scripts, tokenizer, and eval harness — is Karpathy's work. This release adds the d26 configuration, the identity layer, the GRPO stage, and the accompanying analysis. Pretraining data: NVIDIA ClimbMix. SFT data: SmolTalk, MMLU, GSM8K, plus a custom identity set.
