This is the final checkpoint from the multitask_ewok_semantic_cloze_v1
experiment for BabyLM 2026 Strict-Small.
The model is a compact BERT-style masked language model trained from scratch on the
BabyLM 2026 Strict-Small corpus with a small set of auxiliary training tasks. The
main added objective is an MLM-head semantic cloze ranking task: each cloze example
contains a single [MASK], a good single-token completion, and a bad single-token
completion. The model scores the good and bad completions with the standard MLM head.
Intended Use
This checkpoint is intended for BabyLM 2026 shared-task evaluation and for
reproducing the accompanying multi-task semantic cloze experiment. It is not intended
as a general-purpose language model.
Training Setup
Track: BabyLM 2026 Strict-Small
Training data: BabyLM-community/BabyLM-2026-Strict-Small
Tokenizer: bert-base-uncased
Architecture: compact BERT masked language model
Hidden size: 256
Layers: 4
Attention heads: 4
Intermediate size: 1024
Parameters: 11,201,338
Maximum sequence length: 128
Batch size: 96
Maximum steps: 10,000
Actual stopped step: 9,500
Learning rate: 5e-4
Weight decay: 0.01
Warmup fraction: 0.06
Precision: fp32
Seed: 1
Device used for training: NVIDIA RTX A6000
Early stopping monitored validation MLM loss with patience 2, minimum delta 0.01,
and start-after step 9000.
Task Mixture
Task
Sampling probability
MLM
0.600
Replaced-token detection
0.100
Connective prediction
0.075
Definiteness prediction
0.075
Collocation
0.050
Grammar minimal pair
0.050
Semantic cloze ranking
0.050
The semantic cloze ranking loss is:
softplus(-(score_good - score_bad))
where score_good and score_bad are MLM log-probabilities at the masked position.
No classifier head or CLS pooling is used for this objective.
The semantic cloze examples are synthetic, single-mask examples over affordance,
animate agency, part-whole, physical-property, and typical-location templates. No
official EWoK or BLiMP evaluation items were used to generate the auxiliary data.
Official Fast Evaluation Snapshot
These local official fast-eval scores are from the experiment records in the
submission repository.
System
BLiMP
Supplement
EWoK
Entity
Eye
SPR
MLM baseline
64.33
54.00
50.55
15.56
2.47
2.42
BLiMP-pair repair best
54.80
51.60
51.00
27.54
8.02
3.05
Semantic cloze best
54.10
57.60
50.73
36.61
7.56
3.11
Semantic cloze final
55.26
54.40
52.00
27.44
8.18
3.71
The final checkpoint is the recommended checkpoint for EWoK-focused reporting. It
improves EWoK over the MLM baseline in this local comparison, but trades off against
the best-validation checkpoint on Entity Tracking.
This is a single-seed research checkpoint. The result should be presented cautiously:
semantic cloze produced the best EWoK score in the isolated experiment series, while
weight sweeps and relational cloze follow-ups did not improve the EWoK result further.
Full official leaderboard submission artifacts still need to be generated with the
BabyLM evaluation pipeline.
