A 213M-parameter GPT-2-style causal language model submitted to the multilingual track of the BabyLM 2026 Challenge (English, Dutch, Chinese), trained from scratch within the 100M English-equivalent-word budget.

This is the overall submission. It obtained a leaderboard-reported Multilingual Average of 50.68.

Model details


Architecture	GPT-2-style decoder with multi-token prediction (MTP)
Parameters	212,790,272 (213M)
Layers / heads / hidden	14 / 16 / 1024
Context length	512
Tokenizer	byte-level BPE, 32,000 vocab
Training objective	next-token + MTP auxiliary heads ($k{=}4$, ramped $1\to4$ over epochs; only the first head is used at inference)
Optimizer	AdamW, peak LR $4.5\times10^{-4}$, 10 epochs
Track	BabyLM 2026 multilingual
Languages	English, Dutch, Chinese

Training data

Trained within the fixed multilingual budget on a combination of:

the official BabyBabelLM corpora for English, Dutch, and Chinese;
parallel-corpus text from Tatoeba and OPUS-100;
task-family-targeted synthetic text generated offline by an approved teacher model (Qwen3.5-4B).

Published training data: 1ghub/babylm-2026-mtl-data

Note on budget: language shares were balanced with the track's Byte Premium; the corpus construction used a Chinese premium of 0.93 rather than the official 0.9894, so budget figures should be treated as provisional pending a recount with the official 2026 premiums.

Results

Development-set scores (percentages):

Metric	Score
Official Multilingual Average	50.68
Zero-shot (16-task mean, ZS16)	57.82
Fine-tuning (22-task mean, FT22)	41.25
ZS — English / Dutch / Chinese	59.89 / 59.77 / 53.41
FT — English / Dutch / Chinese	43.86 / 39.37 / 40.61

Intended use and limitations

Research model for data-efficient multilingual language modeling under the BabyLM setting. It is a small model trained on a developmentally plausible amount of text and is not intended for production use. Scores are on development sets; evaluation reuse across selection stages introduces selection bias.

Teacher model

Synthetic training text was generated (not distilled) by Qwen3.5-4B, an approved teacher for the challenge. Generation prompts used task-level descriptions.

Citation

Companion paper: Language Allocation and Targeted Data Design for a Multilingual BabyLM (BabyLM 2026 Workshop, EMNLP 2026).

1ghub/babylm-2026-mtl-bigckpt

babylm-2026-mtl-bigckpt