instanton-baseline

• Architecture: GPT-2-style decoder-only transformer, from-scratch init (no pretrained weights): 6 layers, 512 hidden, 8 heads, 512-token context, 27.4M parameters (tied embeddings).
• Objective: causal language modeling.
• Tokenizer: byte-level BPE, 16k vocab, trained only on this run's <=10M-word training mixture.
• Training (v2 recipe): 10 epochs (~100M words exposure), packed 512-token sequences, batch 32 (no gradient accumulation, ~9.9k steps), lr 1e-3 cosine (5% warmup), weight decay 0.1, bf16, seed 42, ~12 minutes on one A100-80GB. The recipe was selected by a controlled sweep: 4x more optimizer steps at higher LR beat both a LLaMA-style architecture swap and a 97M-parameter capacity increase at this word budget.
• Training data: official BabyLM 2026 Strict-Small corpus (10,000,000 words), unmodified. Word counts use whitespace tokenization matching the official BabyLM 2026 dataset cards.

Intermediate checkpoints

Evaluation

Lineage