DAT Strict NextLat Final is a causal Dual Attention Transformer language model trained from scratch for the BabyLM 2026 Strict track. It combines self-attention, relational attention, position-relative symbols, and a NextLat auxiliary training objective. The repository contains the tokenizer and custom HuggingFace Transformers code required by the Auto classes.

Architecture

16 transformer layers
Hidden dimension 1,024
12 self-attention heads and 4 relational-attention heads
Rotary token positional encoding
Shared position-relative symbol retrieval without relative-symbol RoPE
RCA relational attention
SwiGLU feed-forward layers
Vocabulary size 16,384
Maximum model sequence length 514
Untied token embedding and language-model head

Training

The model was trained for ten passes through the BabyLM 2026 Strict corpus under the 100 million word data limit. Training used batches of 42 sequences of 512 tokens, seed 1, Muon for hidden matrix parameters, and LambW for the remaining parameters. The Muon learning rate was 0.02, the auxiliary learning rate was 0.001, weight decay was 0.01, and the schedule used one percent warmup followed by cosine decay to ten percent of the initial rate.

NextLat used horizon 1 with cross-entropy weight 0.0, KL weight 0.5, and MSE weight 1.0. An exponential moving average with decay 0.999 was maintained during training. The published final model and checkpoint revisions use the EMA weights. No teacher model, synthetic data augmentation, or multimodal input was used.

Checkpoints

The final EMA model is stored on main. Intermediate EMA checkpoints use the official BabyLM Strict revision names from chck_1M through chck_1000M. These names represent cumulative words processed across repeated passes through the corpus.

Loading

python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "abe123/babylm-dat-strict-nextlat-final"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)

Use an explicit immutable commit SHA for reproducible evaluation.

Intended use and limitations

This checkpoint is intended for BabyLM evaluation and language-model research. It uses custom model code and therefore requires trust_remote_code=True when loaded through HuggingFace Auto classes.