This model is a ModernBERT-Small masked language model (RoPE, GeGLU, alternating local/global attention, 384 hidden size, 16 layers, 6 attention heads) trained from scratch on the BabyLM 2026 Strict-Small 10M-word corpus, as part of an ablation study on parameter-efficient token embedding layers for developmentally-plausible pretraining under the BabyLM Challenge's strict-small compute and data budget.

Standard transformer token embedding tables scale as vocab_size x hidden_size, which for this model's 30,522-token vocabulary and 384 hidden size would be an 11.7M-parameter dense lookup table. This checkpoint's base representation is the same compositional byte-n-gram embedding used across this ablation sweep: each vocabulary token is decomposed into its raw UTF-8 byte sequence, every byte n-gram (n = 1 to 4) is extracted and hashed into one of 8,192 shared buckets, and the token's representation is the mean of its buckets' embeddings (a 128-dimensional EmbeddingBag table, ~1.0M parameters) rather than a dedicated per-token row.

On top of that shared composed representation, this variant adds a frequency-gated per-token residual -- a learned vocab_size x 128 embedding added to the composed vector, scaled per token by frequency / (frequency + tau) (tau = 100) so common tokens lean more on their own residual and rare tokens rely mostly on the composition (using each token's real training-corpus frequency, unlike the shuffled-frequency control variant in this same sweep).

This checkpoint additionally applies a Zipfian-centering geometry penalty during training: at each step, a batch of token representations is sampled with probability proportional to training-corpus frequency (Zipfian sampling, so the penalty reflects the distribution of tokens the model actually sees), and the squared norm of that sample's mean vector is added to the loss (weight 0.01). This regularizes the embedding space to be centered near the origin under realistic token-usage statistics, rather than under a naive uniform-over-vocabulary assumption -- targeting the anisotropy/off-center-embedding-space problem common in learned token representations.

This is one variant in a broader comparison of embedding-layer parameterizations (dense, linear and MLP-style factorization, tensor-train decomposition, deterministic Fourier expansion, and compositional/frequency-adaptive/geometry-regularized variants) evaluated under identical data, tokenizer, optimizer, and training budget, to isolate the effect of the embedding layer's parameterization on downstream BabyLM evaluation performance.

BabyLM 2026 Strict-Small corpus (~10M words), tokenized with a byte-level BPE tokenizer trained on the same corpus (vocab size 30,522). No external data, synthetic augmentation, or human annotation beyond the corpus as officially released.

Each chck_{N}M branch of this repository corresponds to a BabyLM-Challenge compliance checkpoint (one per N million words of training data seen); main points at the final, fully-trained checkpoint.

Evaluated with the official babylm-eval harness (BLiMP, EWoK, entity tracking, COMPS, Global PIQA, reading-time correlation, GLUE/SuperGLUE fine-tuning, and Age-of-Acquisition word-surprisal correlation) under the strict-small track.