A trilingual causal language model (English / Dutch / Chinese) pretrained from scratch on developmentally plausible text data, using the OLMo2 architecture.

Architecture: Olmo2ForCausalLM
Parameters: ~15M (512h / 8L / 8H / 1376ff)
Tokenizer: BPE, vocab=32000
Sequence length: 512 tokens
Languages: English, Dutch, Chinese (equal data mix)
Data: BabyLM 2026 multilingual corpus (100M tokens total)

Results

Multilingual Track leaderboard (as of submission):

Model	Multilingual Avg	EN Avg	NL Avg	ZH Avg
This model	~45-47	-	-	-
Best baseline (gpt2-en_nld_zho_equal)	38.86	40.04	40.02	36.51

Selected task scores:

MultiBLiMP: EN 84.7%, NL 92.4%
ZhoBLiMP: 78.5%
SIB-200: EN 81.0%, NL 77.5%, ZH 77.0%
MNLI: EN 57.8%, NL 57.3%, ZH 57.1%

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("exusiaiw/emnlp-babylm-2026")
tokenizer = AutoTokenizer.from_pretrained("exusiaiw/emnlp-babylm-2026")

Citation

BabyLM 2026 Challenge — EMNLP 2026 Workshop, Budapest