This model is a ModernBERT-Small masked language model (RoPE, GeGLU, alternating local/global attention, 384 hidden size, 16 layers, 6 attention heads) trained from scratch on the BabyLM 2026 Strict-Small 10M-word corpus, as part of an ablation study on parameter-efficient token embedding layers for developmentally-plausible pretraining under the BabyLM Challenge's strict-small compute and data budget.

Standard transformer token embedding tables scale as vocab_size x hidden_size, which for this model's 30,522-token vocabulary and 384 hidden size would be an 11.7M-parameter dense lookup table -- a large fraction of the model's total parameter budget under the strict-small constraint. Like this sweep's other factorized-linear checkpoint, this model uses an ALBERT-style linear factorization: tokens are first embedded into a smaller 128-dimensional bottleneck space, then linearly projected up to the model's 384-dimensional hidden size, replacing one large vocab_size x hidden_size matrix with two much smaller ones (vocab_size x 128 and 128 x 384).

This checkpoint is the tied ("dual-use") variant: tie_word_embeddings=true, so the same factorized embedding weights are reused in reverse for the MLM output projection, rather than training a separate untied decoder as its sibling checkpoint in this sweep does. This roughly halves the embedding-related parameter count relative to the untied variant, at the cost of constraining the input and output representations to share the same factorized subspace -- a direct ablation of whether tying helps or hurts under a fixed factorized-embedding budget.

This is one variant in a broader comparison of embedding-layer parameterizations (dense, linear and MLP-style factorization, tensor-train decomposition, deterministic Fourier expansion, and compositional/frequency-adaptive variants) evaluated under identical data, tokenizer, optimizer, and training budget, to isolate the effect of the embedding layer's parameterization on downstream BabyLM evaluation performance.

BabyLM 2026 Strict-Small corpus (~10M words), tokenized with a byte-level BPE tokenizer trained on the same corpus (vocab size 30,522). No external data, synthetic augmentation, or human annotation beyond the corpus as officially released.

Each chck_{N}M branch of this repository corresponds to a BabyLM-Challenge compliance checkpoint (one per N million words of training data seen); main points at the final, fully-trained checkpoint. Note: this run's W&B logging only captured checkpoints from chck_20M through chck_100M -- earlier milestones (chck_1M-chck_10M) were not preserved, so this repository has fewer checkpoint branches than this sweep's other models.

Evaluated with the official babylm-eval harness (BLiMP, EWoK, entity tracking, COMPS, Global PIQA, reading-time correlation, GLUE/SuperGLUE fine-tuning, and Age-of-Acquisition word-surprisal correlation) under the strict-small track. Age-of-Acquisition results are unavailable for this checkpoint since that metric requires the full chck_1M-chck_100M checkpoint set, which this run does not have.