morpheus-10M-v5
BabyLM 2026 Strict-Small (10M-word) submission: a ~135M-parameter Llama decoder trained
from scratch with a curated morphological tokenizer.
Model
- Architecture:
LlamaForCausalLM — 12 layers, hidden size 768, 12 attention heads, intermediate size 3072, rotary position embeddings, tied input/output embeddings
- Context length: 256 tokens
- Parameters: 135,050,496 (~135M)
- Vocabulary: 28,366 tokens (pruned morpheme inventory + character backoff)
Tokenizer
A curated, lemma-based morphological analyzer (built on MorphyNet + hand curation) that splits
words into semantically-meaningful morphemes — e.g. unhappiness → ▁un happy ness, while keeping
lexicalized wholes like business intact. The production vocabulary prunes morphemes attested
fewer than three times and routes them through character backoff for out-of-vocabulary pieces.
Self-contained: the analyzer, its resolved resource tables, and the vocabulary ship with the model,
and it loads with trust_remote_code=True — no external dependencies at load time.
Training
- Objective: masked-next-token prediction (MNTP) at a 20% masking fraction (GPT-BERT-style 80/10/10 corruption; mask rate decays 0.30 → 0.15)
- Optimizer: Muon (lr 0.04) on 2D weight matrices + an AdamW auxiliary optimizer (lr 1e-3, weight decay 0.3) on embeddings and normalization parameters
- Schedule: cosine with 2% warmup; bf16 precision; 10 epochs over the 10M-word BabyLM corpus
Results (official babylm-eval pipeline)
- BLiMP: 72.52 on the full benchmark (72.92 on the
fast_eval subset)
- BLiMP supplement: 58.3
Usage
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3tok = AutoTokenizer.from_pretrained("juand-r/morpheus-10M-v5", trust_remote_code=True)
4model = AutoModelForCausalLM.from_pretrained("juand-r/morpheus-10M-v5", trust_remote_code=True)
Data statement (datasheet)
Training corpus. We train on a cleaned version of the official BabyLM 2026 strict-small (10M) corpus —
no other text corpus is used. Cleaning is deterministic and reproducible (scripts/clean_corpus.py --drop-childes-comments in the accompanying code release):
| Words (wc -w) |
|---|
Official raw strict-small | 10,000,019 |
| Cleaned (what this model trained on) | 9,732,593 |
| Removed (≈2.7%) | 267,426 |
What cleaning removes is transcription/annotation junk, not child-directed speech: 40,073 CHILDES
transcriber-comment lines (the dominant removal), OpenSubtitles mojibake/OCR/ASS-tag/watermark artifacts
(~2.5k lines), and Simple-Wikipedia HTML entities. Lines kept per source: bnc_spoken 65,220 · childes
491,232 · gutenberg 59,934 · open_subtitles 385,975 · simple_wiki 58,709 · switchboard 2,892.
Word-budget accounting (strict-small ≤ 10M). Training text = 9,732,593 words < 10M. Ten epochs =
~97.3M cumulative words seen < the 100M exposure cap. No synthetic data, no external model, and no
distillation were used for this submission.
Tokenizer resources. The morphological analyzer is a lookup/rule system — nothing in it is trained on
external text. It consults: (a) hand-curated morpheme tables written by the authors (28,544 entries);
(b) the MorphyNet English inflectional/derivational lexicon, used in full (no frequency truncation in the
submission pipeline): the shipped analyzer tables contain 187,132 form→segmentation entries plus a
617,042-item known-word list of single-word spellings (every lemma/form spelling in MorphyNet, consulted
for is-this-a-word decisions during segmentation). These are dictionary-style resources consulted at tokenization
time, fully documented in the accompanying paper; the only corpus statistics used to build the vocabulary
come from the training corpus itself.
Effective lexicon usage. During training, the analyzer can only ever consult entries for words that occur
in the training corpus (135,085 word types); all other entries are inert. The corpus-intersecting slice is:
known-word list 59,285 (of 617,042), inflection entries 16,968 (of 187,132), curated entries 21,535 (of
28,544). Counted maximally strictly (every key and every segmentation-value word), this slice is 168,757
words, giving 9,732,593 + 168,757 = 9,901,350 < 10M even if lexicon annotations are charged against the
word budget. (The intersecting key words are themselves corpus words, already counted once in the corpus
total, so this is an upper bound.)
Intermediate checkpoints. Revisions chck_1M … chck_10M (every 1M words seen) and chck_20M …
chck_100M (every 10M words seen) — the 19 checkpoints required for strict-small — are available as
branches of this repository and load with revision="chck_<N>M".
