Baby Curriculum Multi
This repository contains our submission for the BabyLM Challenge 2026 Multilingual Track.
The model is a GPT-2 language model trained under the strict BabyLM setting using only 100 million adjusted training tokens across English, Dutch, and Chinese.
Model Details
- Architecture: GPT-2
- Training objective: Causal Language Modeling (CLM)
- Model configuration:
- Vocabulary size: 32,000
- Hidden size: 768
- Layers: 12
- Attention heads: 12
Training Details
The model is trained under the BabyLM 2026 multilingual track using:
- A custom 32k multilingual BPE tokenizer
- Three-stage curriculum learning
- Pack-Wrapped sequence packing
- Languages: English, Dutch, Chinese
- Training Hyperparameters:
- Learning rate: 1e-4
- Sequence length: 256
- Token budget: 100M adjusted tokens
Dataset
Training follows the BabyLM 2026 multilingual track using the official multilingual BabyLM datasets.
No external corpora are used.
Evaluation
The model is evaluated using the official BabyLM multilingual evaluation pipeline, including:
- Zero-shot evaluation
- Fine-tuning evaluation
