bind2_0

⚠️ The official benchmark badge is NOT the point of this repo

On the official BabyLM 2026 strict-small zero-shot surface this model is statistically tied with its matched monolithic and bind1 controls (mean-4 excl. entity: 56.92 vs 56.60/56.63 at 23.9M; slightly above the GPT-2 baseline on BLiMP, 66.11 vs 65.08). It wins nothing there, and that is part of the finding.

What this stage actually shows (three results, all kept):

1. The mechanism is trainable: with direct-task training on a purpose-built synthetic swap-tracking task (n=800 per eval, 5-way, chance 0.20), the forced-bottleneck architecture reached 0.9988 accuracy — with a sharp grokking transition between 5M and 10M training tokens (0.179 → 0.969 → 0.996) — while matched monolithic, bind1-style, and no-binding controls stayed at chance (0.2125 / 0.1938 / 0.1938).
2. It does not emerge from plain LM pretraining: after standard BabyLM strict-small pretraining, zero-shot give-verb state-tracking probes (n=60, chance 0.50) show no model above chance; this build (23.9M) scored 0.200 — significantly below chance, i.e. strong recency capture. The synthetic grok did not transfer.
3. The architecture costs ~nothing on general language ("no tax, no win"): official zero-shot scores tied across architectures; raw LM perplexity is slightly worse than the monolithic control (11.0 vs 9.7 at 23.9M), as expected for a forced bottleneck.

The full falsification-timeline context (what came before this stage and what it forced next) lives at the hub: SecludedCorner/bind-evolution.

Model description (family level)

• Delta-rule fast-weight memory: gated delta-rule recurrent value dynamics (GatedDeltaNet), using the third-party MIT-licensed flash-linear-attention (fla) implementation as the core recurrent layer. The delta-rule/fast-weight design follows Yang, Kautz & Hatamizadeh, Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025, arXiv:2412.06464); only the permissively licensed fla implementation is used here.
• A forced bottleneck: attention is chunk-local; information can cross chunk boundaries only through the recurrent state S. This makes the recurrent state the sole carrier of long-range bindings — the design hypothesis under test at this stage.

Training data

Results (official pipeline, strict-small zero-shot, single seed)

Honest limitations — what this stage cannot do

• It does not track state zero-shot. After plain LM pretraining, give-verb state-tracking probes are at or below chance (0.200 at 23.9M = strong recency capture). Do not use this model expecting emergent entity/state tracking.
• It does not beat its controls on the official benchmark. Tied within noise; that is the honest reading, not modesty.
• The synthetic grok required direct-task training — it is evidence the bottleneck can force state into the recurrent path, not evidence of a general capability.
• Single seed per build; raw LM perplexity pays a small bottleneck tax (11.0 vs 9.7).

What this stage forced next

How to load

• trust_remote_code=True is required for the model (the config's auto_map points at the inlined modeling_babylm.py shipped in this repo). The tokenizer loads without it.
• A CUDA GPU with a working Triton is required for inference. The fla 0.5.1 GatedDeltaNet path dispatches to Triton kernels; on CPU-only environments the forward pass fails at kernel launch (RuntimeError: 0 active drivers). The checkpoint itself loads fine on CPU; only the forward pass needs a GPU.

Dependency pins

torch==2.12.1+cu126
transformers==5.13.0
triton-windows==3.7.1.post27
flash-linear-attention==0.5.1
fla-core==0.5.1
safetensors==0.8.0
numpy==2.4.6

• fla is required at runtime: the inlined modeling code lazily imports fla.layers.GatedDeltaNet, which this architecture instantiates. Installed from PyPI as release 0.5.1 (no commit pin — the environment records the plain PyPI release; fla-core 0.5.1 comes with it).
• triton: the training/export environment is Windows and uses the triton-windows fork (3.7.1.post27); on Linux use the upstream triton matching your torch build.
• torch build is CUDA 12.6 (+cu126); pick the equivalent build for your platform.

Export fidelity & known-defect disclosure

• At grid-evaluation time (2026-07-12) the HF export was verified logit-identical to the training model (wrapper-vs-original logit diff = 0.00e+00).
• On 2026-07-15 a dedicated round-trip validation re-checked both exports, as they sit on disk, against the original training checkpoints: all 186 weight tensors bitwise identical (max abs diff 0.0, no missing/extra keys) and logits bitwise identical (max abs diff 0.00e+00 across 4 deterministic batches of 8×128 tokens, fp32, passing at both atol 1e-4 and atol 1e-5) for both the 23.9M and 27M builds. Caveat, disclosed: that re-check ran on CPU, where fla's Triton kernels cannot execute, so three fla components were replaced by math-equivalent pure-PyTorch implementations applied identically to both sides. It therefore validates export fidelity (weights and module wiring survive .pt → safetensors → AutoModelForCausalLM exactly), not Triton-kernel numerics; a GPU re-run with stock kernels remains the gold check.

• The exported wrapper accepts attention_mask in forward() and never uses it — on the causal-LM path and, for this architecture, on the AutoModel (sequence-classification) path as well. Empirically, attention_mask=ones, =zeros, and omitted all produce bitwise-identical logits on both builds.
• Consequence: in a batch, right-padding is silently attended over as real tokens — batched padded inference gives wrong results. Run unbatched, or length-sorted/unpadded. Per-example inference is unaffected; the published zero-shot numbers above came from the per-example evaluation setting and are unaffected by this defect.

How to cite this model

Card freeze policy

Branches

• main — 23.9M-parameter build (the primary artifact)
• 27m — 27M-parameter build (same architecture and recipe, wider)