babylm-2026-strict-small-childes-heavy

• Architecture: GPT-2-style decoder-only transformer, from-scratch init (no pretrained weights): 6 layers, 512 hidden, 8 heads, 512-token context, 27.4M parameters (tied embeddings).
• Objective: causal language modeling.
• Tokenizer: byte-level BPE, 16k vocab, trained only on this run's ≤10M-word training mixture.
• Training data: 9,999,223 words (custom mixture resampled from the official 100M Strict pool). Word counts use whitespace tokenization matching the official BabyLM 2026 dataset cards. Manifest and datasheet in the GitHub repo.
• Training: 10 epochs (~100M words exposure), packed 512-token sequences, effective batch 128, lr 5e-4 cosine (3% warmup), weight decay 0.1, bf16, seed 42, single A100-80GB.

Intermediate checkpoints

Evaluation

Mixture (strict_small_childes_heavy)

Limitations

Citation