This model is a ModernBERT-Small masked language model (RoPE, GeGLU, alternating local/global attention, 384 hidden size, 16 layers, 6 attention heads) trained from scratch on the BabyLM 2026 Strict-Small 10M-word corpus, as part of an ablation study on parameter-efficient token embedding layers for developmentally-plausible pretraining under the BabyLM Challenge's strict-small compute and data budget.

Standard transformer token embedding tables scale as vocab_size x hidden_size, which for this model's 30,522-token vocabulary and 384 hidden size would be an 11.7M-parameter dense lookup table. This checkpoint replaces that table entirely with a compositional byte-n-gram embedding, inspired by fastText's subword hashing trick: each vocabulary token is decomposed into its raw UTF-8 byte sequence, every byte n-gram (n = 1 to 4) is extracted from it and hashed into one of 8,192 shared buckets, and the token's representation is the mean of its buckets' embeddings (a 128-dimensional EmbeddingBag table, ~1.0M parameters total) rather than a dedicated per-token row. This composed vector is then projected up to the model's 384-dimensional hidden size. The same shared n-gram bucket table is used to produce the MLM output head's logits (tie_word_embeddings=true is required for this embedding type), so the entire vocabulary's token representations are generated algorithmically from ~1M shared subword-hash parameters instead of ~11.7M independent per-token ones. This particular checkpoint uses no token-specific residual on top of the composed representation (compositional_residual_mode=none) -- it is the "composition-only" control in a broader sweep that also tests frequency-gated residual variants.

This is one variant in a broader comparison of embedding-layer parameterizations (dense, linear and MLP-style factorization, tensor-train decomposition, deterministic Fourier expansion, and compositional/frequency-adaptive variants) evaluated under identical data, tokenizer, optimizer, and training budget, to isolate the effect of the embedding layer's parameterization on downstream BabyLM evaluation performance.

BabyLM 2026 Strict-Small corpus (~10M words), tokenized with a byte-level BPE tokenizer trained on the same corpus (vocab size 30,522). No external data, synthetic augmentation, or human annotation beyond the corpus as officially released.

Each chck_{N}M branch of this repository corresponds to a BabyLM-Challenge compliance checkpoint (one per N million words of training data seen); main points at the final, fully-trained checkpoint.

Evaluated with the official babylm-eval harness (BLiMP, EWoK, entity tracking, COMPS, Global PIQA, reading-time correlation, GLUE/SuperGLUE fine-tuning, and Age-of-Acquisition word-surprisal correlation) under the strict-small track.