vanilla-small-20B — dense baselines (12L / d=1024, 202.53M)
Dense (sparsity_mode=none) counterparts to the cayley-small-*-mlp_in-20B
ParityTransformer siblings. One 20B-token run, five checkpoints: three
val-matched early stops plus the token-matched end of the run.
These are the dense comparators for the interpretability claim: if a
ParityTransformer is more interpretable than a dense model that is at least as
good on val, the comparison cannot be dismissed as a handicapped baseline.
Checkpoints
Files are named by iter. They were previously named by the --save-at-val
threshold that fired; those names misstated the actual loss, so they were renamed
(ckpt_val_3.1584.pt → ckpt_val_1900.pt, 3.1484 → 2100, 3.1330 → 2300).
| file | iter | tokens | measured val_loss | in-run trigger |
|---|
ckpt_val_1900.pt | 1900 | 2.99B | 3.1477 | 3.1536 |
ckpt_val_2100.pt | 2100 | 3.30B | 3.1285 | 3.1354 |
ckpt_val_2300.pt | 2300 | 3.62B | 3.1116 | 3.1162 |
ckpt_best.pt | 12200 | 19.19B | 2.8559 | 2.8586 |
ckpt_final_20B.pt | 12716 | 20.001B | 2.8530 | 2.8664 |
Siblings, measured in the same pass on the same batches:
| reference | measured val_loss | published (in-run) |
|---|
cayley-small-2L-mlp_in-20B | 3.1671 | 3.1584 |
cayley-small-3L-mlp_in-20B | 3.1425 | 3.1330 |
Paired differences vs the siblings
Negative = dense is better. Every model saw byte-identical batches, so these are
exact paired comparisons, not error bars added in quadrature (paired SEM ±0.0004).
| dense ckpt | vs cayley-small-2L (3.1671) | vs cayley-small-3L (3.1425) |
|---|
ckpt_val_1900 | −0.0195 (45σ better) | +0.0052 (13σ worse) |
ckpt_val_2100 | −0.0387 (86σ better) | −0.0140 (33σ better) |
ckpt_val_2300 | −0.0556 (111σ better) | −0.0309 (65σ better) |
ckpt_final_20B | −0.3141 (245σ better) | −0.2894 (231σ better) |
Note ckpt_val_1900 clears 2L but does not clear 3L. Pick the checkpoint your
comparison requires from this table.
ckpt_final_20B.pt is also better than ckpt_best.pt by 0.0029 (110σ)
despite the name: ckpt_best was selected by a noisy 4.9M-token in-run eval that
happened to dip at iter 12200. Prefer ckpt_final_20B.pt for everything;
ckpt_best.pt is kept only for provenance.
How these numbers were measured
evals/language_modeling/eval_val_loss.py over the entire FineWeb-Edu val
split — contiguous non-overlapping windows, bf16 autocast, batch 16 × 1024,
fineweb-edu-25B, seed 0.
| quantity | value |
|---|
| tokens per model | 25,161,728 (whole val split) |
| batches per model | 1,536, byte-identical across all models |
| absolute SEM (single model) | ±0.0045 |
| paired SEM (difference of two models) | ±0.0004 |
The paired SEM is ~13× tighter because shared passage difficulty cancels. Quote
the paired differences.
Two reasons the run log disagrees with this table:
| source of disagreement | effect |
|---|
In-run evals are master-rank only, eval_iters × bs × seq_len = 4.9M tokens regardless of world size; adjacent evals wobble ~0.005 nats. | --save-at-val triggers inherit that noise. |
best_val_loss inside a checkpoint is the run's best, not that checkpoint's val. | Meaningless to difference against a reference ckpt.pt saved at a different iter. |
Measured on this protocol, dense sits ~0.0025 nats below its in-run trigger,
while the cayley siblings sit ~0.011 above their published numbers. The
cayley siblings' published numbers are in-run values too, so use the measured
column for any cross-model claim and do not mix the two — a ~0.01 nat protocol
artifact otherwise lands inside the very gap being resolved.
One caveat we cannot yet separate: the cayley path must run bf16 through this
loader (it builds the backbone fp32 but forces the SAE to bf16 + triton), so its
offset could be protocol or genuine bf16 sensitivity in top-k selection. Dense
is bf16-insensitive here — its fp32 and autocast runs agree to 0.0002.
Alignment tax
Equal budget (20B tokens both sides):
| pair | dense | cayley | gap |
|---|
| vs 2L | 2.8530 | 3.1671 | 0.314 nats |
| vs 3L | 2.8530 | 3.1425 | 0.289 nats |
Token efficiency — where dense first reaches each sibling's measured val,
interpolated on the in-run eval curve shifted by the measured −0.0025 protocol
offset:
| sibling | its measured val | dense crossing | tokens | tax |
|---|
| 2L | 3.1671 | iter ~1683 | 2.65B | ~7.6× |
| 3L | 3.1425 | iter ~2021 | 3.18B | ~6.3× |
Both inside the <15× target. Eval spacing (100 iters = 157M tokens) caps
resolution, so treat as two significant figures. Note the large chassis gives
~3.5× on the same construction (
markhenry/vanilla-large-20B) —
the ratio is not a constant across scale.
Corrections to earlier versions of this card / report.md
| claim (earlier) | corrected |
|---|
| iter 1900 beats 2L by 0.0048 — "a statistical tie" | beats by 0.0195 at 45σ — not a tie |
| iter 2300 beats 2L by 0.042, 3L by 0.017 | 0.0556 and 0.0309 |
| iter 2300 is the 3L match | iter 2100 is the first to clear 3L |
| tax 6.9× (2L) / 5.7× (3L) | 7.6× / 6.3× |
ckpt_best is the run's best model | ckpt_final_20B is better by 0.0029 (108σ) |
All five corrections move the same way: measured consistently, the dense
baselines are stronger than the in-run numbers implied, so the alignment tax
is larger. That is the conservative direction for the interpretability claim.
Architecture and training
12 layers, 8 heads, d=1024, block_size 1024, learned positional encoding, RMSNorm
(affine), no bias, vocab 50304, 202.53M params — identical to the cayley-small
siblings, which add buffers but no nn.Parameters.
Muon + AdamW lockstep 1.2e-2 (min 1.5e-4),
linear_warmdown wf=0.9, warmup 200,
tokens/iter = 32 × 1024 × 48 = 1,572,864 → 12,716 iters. FineWeb-Edu
sample-100BT, GPT-2 tokenizer. 2× H200, 8h32m, ~695k tok/s.
Wandb
9iezcto0.
--n-head 8 must be passed explicitly: train.py arch defaults are
mode-dependent, so sparsity_mode=none at d=1024 otherwise resolves n_head=12
and 1024 % 12 != 0 asserts. The cayley siblings hit the cayley default of 8.
Also in this repo: report.md, eval_curve.csv (full in-run eval trajectory),
train.log, and the training script.
Loading
1import torch
2ck = torch.load("ckpt_val_2100.pt", map_location="cpu", weights_only=False)
3ck["iter_num"] # 2100
4cfg, sd = ck["model_config"], ck["model"] # GPTConfig fields, state dict
Checkpoints carry optimizer_states, so they are resumable, not inference-only.
Siblings
Re-evaluation and card by Claude Code. Training run and report.md predate it;
where they disagree, the corrections table above states why.
