Scaling Ladder — d12 (286M total parameters), seed 1
Research artifact. One of 24 base language models (3 sizes x 8 random
seeds) trained to study seed-to-seed variance in language-model pretraining.
Every model is a plain next-token predictor trained on the same data for
200 tokens per parameter. No instruction tuning, no safety training.
Tokens per parameter (TPP) is computed over scaling parameters throughout,
not total parameters — see Parameter counts below.
This repository holds size d12, seed 1. The seed sets both the
weight initialization and the data order; everything else is identical
across the eight seeds of a size.
This revision (main) mirrors TPP_200 — the completed ladder.
validation loss across tokens per parameter
Validation loss in bits per byte (lower is better) against realized tokens
per parameter, measured at each checkpoint save on the training run's
held-out validation split. One series is the annealed models (the TPP_X
revisions); the other is the un-annealed main-run (constant learning rate)
checkpoints they forked from (the TPP_X_preanneal revisions). A thin
connector joins each annealed model to the fork checkpoint its anneal
started from. This repository's single run only — no averaging across
seeds.
The ladder
repos
layers
hidden size
total parameters
scaling parameters
d12_optimized_286m_seed1..8
12
768
286M
110M
d16_optimized_537m_seed1..8
16
1024
537M
235M
d20_optimized_897m_seed1..8
20
1280
897M
435M
"optimized" in every repo name marks the architecture family: the full
nanochat GPT with all of its architecture mechanisms enabled (see
Architecture). It distinguishes this ladder from possible future ladders
trained with a plainer ("clean") architecture.
Parameter counts
The size in each repository name is the total parameter count of the
checkpoint — everything model.safetensors holds, including the token
embedding and the value-embedding tables (6 at this size; 6/8/10
across d12/d16/d20): 286,261,730 / 536,871,738 / 896,533,746 for
d12 / d16 / d20. The scaling-parameter count — the weight matrices
plus the output head only, and the basis for tokens per parameter and
the scaling-law fits — is smaller: 110,100,912 / 234,881,792 /
435,160,240. The gap is mostly the vocabulary-sized value-embedding
tables (see Architecture).
Revisions: checkpoints along training
Each model's main run trains with a constant learning rate for 200
tokens per parameter, saving checkpoints on a fixed step cadence; the main
run's learning rate is never decayed. Every annealed checkpoint comes from
a separate anneal run: it forks the main run at the saved checkpoint
closest to (mark minus 0.75B tokens) — before the mark — then
trains a fixed 0.75B tokens (for this size) while the learning
rate decays linearly to 5% of the constant value, landing at the mark.
The annealed model's total token count is therefore the mark itself, not
the mark plus the anneal. Because the fork snaps to the nearest saved
checkpoint, the realized total can deviate from the nominal mark; the
table below records it exactly, and the deviation is largest at the
lowest marks. Both stages are published as git revisions (branches) of
this repository:
TPP_X (X = 10, 20, ..., 200): the annealed model at the
X-tokens-per-parameter mark. Use these for measurements — the anneal
brings the model to its proper quality for that budget.
TPP_X_preanneal: the constant-learning-rate checkpoint of the
main run that the TPP_X anneal forked from. Nominally the fork point
sits about 6.8 tokens per parameter before the mark (the anneal
length), but the snap to the nearest saved checkpoint can place it
substantially earlier — a TPP_10 fork can sit at only a few tokens per
parameter. The table below records every pre-anneal position exactly.
main: identical to TPP_200 once it exists; while the ladder is
still training, main holds the latest available TPP_X.
Marks are added incrementally while training continues, so a missing
revision only means it has not landed yet.
Currently available marks in this repository:
mark
annealed step
annealed tokens/param
pre-anneal step
pre-anneal tokens/param
TPP_10
1933
9.20
500
2.38
TPP_20
4433
21.11
3000
14.29
TPP_30
6433
30.63
5000
23.81
TPP_40
8433
40.16
7000
33.33
TPP_50
10433
49.68
9000
42.86
TPP_60
12433
59.20
11000
52.38
TPP_70
14933
71.11
13500
64.29
TPP_80
16933
80.63
15500
73.81
TPP_90
18897
89.99
17464
83.16
TPP_100
20933
99.68
19500
92.86
TPP_110
23199
110.47
21766
103.65
TPP_120
24933
118.73
23500
111.90
TPP_130
26933
128.25
25500
121.43
TPP_140
28933
137.78
27500
130.95
TPP_150
31433
149.68
30000
142.86
TPP_160
33433
159.20
32000
152.38
TPP_170
35433
168.73
34000
161.90
TPP_180
37933
180.63
36500
173.81
TPP_190
39433
187.78
38000
180.95
TPP_200
41933
199.68
40500
192.86
"Annealed tokens/param" counts every token the annealed model saw, the
anneal's own tokens included. "Pre-anneal tokens/param" is the fork
point's position in the main run: fork step x tokens per step / scaling
parameters.
trust_remote_code=True is required: the architecture matches no stock
transformers class, so the modeling code ships in the repository
(modeling_nanochat_gpt.py, plain PyTorch). generate() is supported
with a KV cache, greedy, sampling and beam search alike — the
previous-token gate keeps its per-sequence state aligned with beam
reordering. Assisted decoding (an assistant model) is refused: it
requires cache cropping, which that state does not support.
Architecture
Full nanochat GPT architecture —
the "optimized" family in the repo name: depth 12, hidden size
768, 6 attention heads (head dimension 128), sequence length
2048, vocabulary 32,768. All of nanochat's architecture mechanisms are
active: value embeddings on alternating layers
(the vocabulary-sized tables behind the scaling-vs-total parameter gap),
re-injection of the input embedding at every layer, per-layer residual
scaling, a learned gate that mixes each token's embedding with the previous
token's, a mid-network subtraction of the stored input contribution,
query-key sharpening, logit softcap 15, and attention that alternates
short sliding windows with full-context layers (pattern "SSSL"). Also:
parameter-free RMSNorm, rotary embeddings (base 100,000) with query-key
RMS normalization after rotation, relu(x)^2 MLP, no biases, untied
embeddings.
Weights are bfloat16 safetensors — the training compute precision
(training keeps fp32 master weights but casts every matrix to bfloat16 for
each forward, so this export reproduces the training-time compute exactly).
Every revision's upload is byte-verified against the converted
checkpoint (file sizes and content hashes of the hub listing). The
conversion itself is verified by bitwise logit comparison against the
original training code on at least one revision per repository; a
revision that was logit-verified carries the record verify_results.json.
Tokenizer
nanochat byte-pair encoding, 32,768 tokens (32,759 learned + 9 special;
only <|bos|>, id 32759, appears in pretraining). Trained once on
ClimbMix and pinned across every model of the study. Load it with
trust_remote_code=True as in the snippet (the config carries an
auto_map, and resolving it without the flag triggers an interactive
prompt).
Training data
ClimbMix (NVIDIA,
filtered English web text), pinned snapshot climbmix_1201, single pass,
sequences of 2048 tokens. The base data carries a CC BY-NC 4.0,
research-and-development-only license, which this model mirrors.
License
Model weights: cc-by-nc-4.0 (mirrors the ClimbMix training data;
research use).
Modeling/configuration code: MIT (derived from karpathy/nanochat; see
the bundled LICENSE file).
