100M English LM (overtrained + instruct SFT)
Instruct SFT of the overtrained base. Validation loss 1.7337 — the winning arm of the A/B below.
Experiment: does overtraining a tiny LM pay off?
Two ~100M-parameter from-scratch English language models, identical architecture
and identical instruct SFT, differing only in pretraining token budget:
| model | pretrain tokens | instruct val loss |
|---|
| overtrained | ~26B (~260 tok/param) | 1.7337 |
| chinchilla (compute-optimal floor) | ~2B (~20 tok/param) | 2.0180 |
After identical SFT, the heavily overtrained base yields a measurably more
instruction-shaped model: ~0.28 nats lower validation loss. Trained end to end
on a single consumer RTX 3070 Ti (8GB).
Architecture & data
~100M-parameter vanilla Llama (RMSNorm, SwiGLU, RoPE, grouped-query attention,
tied embeddings), 32,000-token ByteLevel BPE, 2048 context. English-only corpus
(cosmopedia-v2 + fineweb-edu-dedup + wikipedia-en), trained from scratch.
Custom architecture: load with the included model.py + config.py, not the HF
AutoModel API. Instruct models use ChatML with assistant-only loss masking.
