emg_mts1_mopbf_en_nl_zh_equal is a small, multilingual, linguistically motivated causal language model trained for the BabyLM 2026 multilingual track (English, Dutch, Mandarin Chinese) on a developmentally plausible word budget. It is a research model whose purpose is to test whether a cognitively grounded memory mechanism can match a strong opaque baseline on syntactic benchmarks, not a general-purpose chat or production model.

S1 is the fast-only member of the eMG-MTS family (one retention tier). Despite that, it already recovers long-range filler–gap dependencies at a level at or above the Hawk baseline (see Evaluation).

What it is

eMG-MTS realizes memory as multi-timescale, content-addressable retrieval. Each "selective state" is a decayed linear-attention retention tier: the read at position t is a competition over cue·key similarity weighted by a base-level recency term γ^(t−i) —

o_t = Σ_{i≤t} γ^(t−i) (q_t · k_i) v_i / Σ_{i≤t} γ^(t−i) (q_t · k_i)

which is the Lewis & Vasishth (2005) retrieval equation made architectural. The per-tier decay γ is fixed, not learned, so the timescale is a structural constant rather than something the model must discover. Because the underlying recurrence S_t = γ S_{t−1} + k_t v_tᵀ is linear with a scalar decay, it has an exact parallel form (loop-free, torch.compile-friendly), and the state is a bounded d_k × d_v fast-weight matrix (this is a linear RNN with a content-addressable state, not softmax attention: no KV-cache, no lookback, no softmax). Long-range information travels only through retention reads: the residual stream carries a strictly local causal-conv path, so depth composes range without an escape hatch.

For numerical stability the queries/keys are RMS-normalized before the feature map and the read is computed in fp32 (a normalized linear-attention read is otherwise prone to divergence).

Architecture: 14 blocks, hidden size 704, one fast retention tier (γ = 0.85), depthwise causal conv (kernel 4) + gated MLP local path.
Parameters: ≈103M total (≈75M compute core; the rest is the tied embedding over a ~39.9K vocab).
Tokenizer: native MorPiece (MoP-16K, byte-fallback), shipped with the model as morpiece_native.json + tokenization_morpiece.py + tokenizer_MorPiece.py. The byte-fallback keeps rare Chinese characters distinct, which is why ZhoBLiMP does not collapse.
Heads: causal-LM plus sequence- and token-classification heads (for GLUE-style and UD-style fine-tuning) share one backbone.

How to use

The model and tokenizer use custom code, so trust_remote_code=True is required.

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

name = "NeTSlab/emg_mts1_mopbf_en_nl_zh_equal"
tok = AutoTokenizer.from_pretrained(name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(name, trust_remote_code=True).eval()

# minimal-pair scoring (lower loss = more grammatical)
def sentence_logprob(text):
    ids = tok(text, return_tensors="pt").input_ids
    with torch.no_grad():
        out = model(input_ids=ids, labels=ids)
    return -out.loss.item() * ids.shape[1]

print(sentence_logprob("The keys to the cabinet are on the table."))
print(sentence_logprob("The keys to the cabinet is on the table."))

For fine-tuning (e.g. the BabyLM classification / UD tasks), the same checkpoint loads under the classification auto-classes with a freshly initialized head:

python

from transformers import AutoModelForSequenceClassification, AutoModelForTokenClassification
seq = AutoModelForSequenceClassification.from_pretrained(name, num_labels=3, trust_remote_code=True)
tok_clf = AutoModelForTokenClassification.from_pretrained(name, num_labels=17, trust_remote_code=True)

Training data & procedure

Data: the BabyLM 2026 multilingual selection (English / Dutch / Chinese), with per-language exposure balanced by byte-premium-adjusted English-equivalent words (_equal). The three languages contribute roughly equal English-equivalent word counts.
Objective: plain next-token cross-entropy (no auxiliary or supervision losses in this S1 model).
Optimizer: AdamW, bf16 mixed precision, cosine LR schedule with warmup, gradient clipping 1.0, weight decay 0.01.
Tokenizer: native MorPiece MoP-16K (byte-fallback), loaded and re-emitted natively so training and evaluation segmentation match exactly.

Evaluation

Zero-shot minimal-pair accuracy on the BabyLM 2026 suite. Aggregate group scores: English 0.6425 · Dutch 0.5667 · Chinese 0.6497.

Benchmark	Metric	S1
English — BLiMP (BabyLM filtered)	acc	0.7029
English — MultiBLiMP	acc	0.8792
Dutch — BLiMP-NL	acc	0.7975
Dutch — MultiBLiMP	acc	0.9086
Chinese — ZhoBLiMP	acc	0.8090
Chinese — xcomps_zh	acc	0.5676
Dutch — xcomps_nl	acc	0.5704

Selected syntactic phenomena (English BLiMP):

Phenomenon	acc
wh_vs_that_with_gap	0.343
wh_vs_that_with_gap (long distance)	0.135
wh_vs_that_no_gap	0.961
wh_questions_subject_gap	0.891
wh_questions_subject_gap_long_distance	0.876
wh_questions_object_gap	0.754
wh_island	0.596
adjunct_island	0.767
determiner_noun_agreement (mean)	~0.90
anaphor_number_agreement	0.969
principle_A_case_1	1.000
distractor_agreement (mean)	0.453

Two findings are worth highlighting because they are the point of the model:

Filler–gap is genuinely acquired. wh_vs_that_with_gap reaches 0.343 (with no_gap at 0.96), a level comparable to or above the Griffin/Hawk baseline, and a large jump over cue-based predecessors that were mechanistically inert on this dependency. The residual short-vs-long gap (0.343 → 0.135) is the expected limitation of a single fast tier and is the target of the slow-tier sibling (S2).
Chinese is not degraded by tokenization. ZhoBLiMP is 0.809, with no wholesale collapse — a direct benefit of the native byte-fallback MorPiece tokenizer.

Model family

Model	Tiers	Switchboard	Note
emg_mts1 (this repo)	fast (γ=0.85)	off	the fast-only baseline
emg_mts2	fast + slow (γ=0.995)	off	adds the slow tier for long-distance preservation
emg_mts1par / emg_mts2par	as above	on (film)	+ typological switchboard (Roberts-grounded FiLM)

Limitations & intended use

Research model. Intended for syntactic evaluation and analysis, not for text generation quality, factual reliability, or downstream deployment. Commonsense/NLI-style tasks (HellaSwag, WinoGrande, XStoryCloze) sit near chance, as expected at this scale and budget.
Rare constructions are under-learned. Dutch parasitic_gaps (0.251) and a few Chinese subtasks are low — these are constructions that are rare in the training diet; the limitation is one of recruitment (data frequency), not of the architecture's capacity.
Long-distance object extraction is the known weak point of the fast-only S1 (see the 0.343 → 0.135 drop); use S2 where long-range preservation matters.
Trained on ~3 languages only; any typological interpretation of internal representations is correspondingly limited.

References

De et al. (2024), Griffin / Hawk: Mixing Gated Linear Recurrences with Local Attention (baseline family).
Lewis & Vasishth (2005), An Activation-Based Model of Sentence Processing as Skilled Memory Retrieval (the retrieval equation realized here).
Roberts (2019), typological parameters (used by the optional switchboard variants).
BabyLM 2026 multilingual shared task.

Developed at IUSS Pavia / NeTS-lab with crucial optimizations and bug-fixes by Claude Opus 4.8