morpheus-10M-v3 — BabyLM 2026 Strict-Small (10M) submission
150M-parameter Llama-architecture LM trained from scratch on the BabyLM 10M strict-small corpus with a
morphological tokenizer (every token a linguistic morpheme; MorphyNet + hand-curated analyses; lowercased;
▁ marks word-initial morphemes; OOV words back off to character pieces).
Recipe (the overnight stack, BLiMP 70.45 on our strict reproduction of the 2025 pipeline, vs 68.0 for the
v2 baseline model): hybrid causal + masked-next-token-prediction objective (50% of steps masked at 15%,
bidirectional attention on masked batches, GPT-BERT-style output shift), context 256, attention dropout 0.1,
AdamW lr 1e-3 (betas 0.9/0.95), weight decay 0.3, cosine schedule 2% warmup, 10 epochs (BabyLM cap), bf16.
- Vocab: 48,728 morpheme tokens (
our_vocab.json), char-backoff pieces included
- Best-val checkpoint (step 4500, val 3.4322287875196964); milestone branches
chck_1M..chck_100M
- The tokenizer loads with
trust_remote_code=True, needs no extra pip deps, and returns word-granularity
offset_mapping as the BabyLM strict harness requires.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2tok = AutoTokenizer.from_pretrained("juand-r/morpheus-10M-v3", trust_remote_code=True)
3model = AutoModelForCausalLM.from_pretrained("juand-r/morpheus-10M-v3")
Tokenizer provenance: resources snapshot verified to reproduce the training token stream exactly
(14,446/14,446 documents, 2026-07-20). Sibling model: juand-r/morpheus-10M-v2 (pure-causal baseline, ctx 512).
