Base-model runs from the nano-math branch of a nanochat fork: depth-12,
packed-varlen training with per-document attention isolation. One directory per
run under base_checkpoints/, each with the final model weights, full optimizer
state (resumable), and the run's config in meta_001680.json.
Both runs use an identical architecture and identical hyperparameters — same
LRs, weight decay, warmup, batch size, and data order (pq_idx 18 / rg_idx 59).
Both start from the same init (step-0 loss 10.397434, step-0 val bpb
3.176160).
The two differ only in code refactors that are numerically inert (verified
bitwise-identical forward and backward, at full width). The 0.000223 bpb gap
and the CORE gap should be read as run-to-run trajectory divergence, not as an
effect of the refactor — see Caveat below.
286,261,730 total parameters. Trained on 1x A100-SXM4-80GB, bf16, FlashAttention 3.
Caveat on comparing these two runs
d12-rotary looks slightly better on both metrics, but the code changes between
the runs were verified numerically inert — bitwise-identical outputs and
gradients, including at the full T = 131,072 training width. The runs
nonetheless diverge from step 1 onward, so the gap reflects a chaotic split
between two trajectories rather than a measured improvement. Treat a 0.000223
bpb difference as within noise unless a repeat run establishes the noise floor.
Layout
base_checkpoints/<tag>/model_001680.pt # weights
base_checkpoints/<tag>/optim_001680_rank0.pt # optimizer state (resume)
base_checkpoints/<tag>/meta_001680.json # config + val_bpb + loop state
logs/ # full training logs
tokenizer/ # tokenizer.pkl, token_bytes.pt
Note on d12-smear-0s: the directory is named for the run
(A100_d12_bs64_flat_smear_0s.log). Its meta_001680.json still records the
original model_tag of d12-flat-forward, which is what the training script
actually wrote — the metadata is left unedited so it remains a faithful record
of the run.
