Delta Slot-Stack 236M — a language model built to be read, not to win benchmarks
A 236M-parameter LM whose residual pathway is not an additive stream. Every
attention layer writes to its own slot, and that slot is never overwritten.
Ten layers, ten slots, plus one for the token embedding.
The point is not accuracy. The point is that after training you can still open the
model and see which head put what where — and then change it.
⚠️ Read the Limitations section before drawing conclusions.
Most importantly: no same-parameter baseline was trained at this scale.
How this was built. The architecture, the probe-token design, and the
experimental questions are the author's. The author does not read or write
English and does not write code — the training script, the six analysis scripts,
and this model card were all written by Claude (Anthropic) from the author's
specifications in Chinese, over roughly one day of back-and-forth.
The author works in a factory and trains on rented GPUs as a hobby. Several
results in this card exist because the author pushed back on claims Claude had
overstated; the "Things we got wrong" section is not decoration.
中文说明见文末。
The one structural difference
Standard transformer — a whiteboard. Layer 1 writes on it, layer 2 writes on the
same board, and so on. What layer 3 wrote is physically gone by the end; it has been
overwritten and rescaled a dozen times.
This model — a row of boxes. Each layer writes into its own box and the box is
sealed. At the end, box 3 still contains exactly what attention layer 3 produced.
Everything below follows from that.
Supporting choices (each is a real trade-off, not free):
No W_o. Head h permanently occupies dimensions [h·128, (h+1)·128) of its
slot. Head mixing is deferred to the downstream gate. Attention is 3d² per layer
instead of 4d².
Per-head normalization. Each head's value vector is normalized within its own
128 dims. Zeroing one head does not rescale the other nine — this is what makes
slots separable, and it is the technical content of the whole design.
Cosine attention scores, √hd · cos(q,k). Since q/k are unit-normalized, the
standard 1/√hd becomes a multiply, not a divide.
Bilinear FFN, no activation function anywhere.Wd(gate ⊙ up), where gate
reads the history slots and up reads the newest slot. History selects, newest
supplies.
Embeddings on the unit sphere (hard-projected every optimizer step) with a
separate head bias carrying the unigram prior.
105d² vs 120d² for a same-shape standard transformer (14% fewer)
What this model lets you do
All numbers below are measured on the final checkpoint (step 38146) with the scripts
in this repo. Every one is reproducible in minutes.
1. Rewrite the answer by patching 3 heads — with zero collateral damage
Minimal pairs differing in exactly one token (answer "cat" vs "dog"). Run the "dog"
sequence, but replace three head outputs with their values from the "cat" run:
heads patched
says "cat"
says "dog"
says something else
clean run
0%
100%
0%
+ L8h5
31%
69%
0%
+ L8h7
88%
12%
0%
+ L1h9
100%
0%
0%
The "something else" column staying at 0% is the result. This is targeted rewriting,
not damage. (Activation patching itself is standard practice — see
Limitations for what is and isn't novel here.)
2. Establish what is not involved
Of 100 head positions, ~90 produce exactly 0.000 effect — not small, zero. A head
that doesn't attend to the answer position produces bit-identical output on both
sequences, so patching it is a no-op.
Being able to rule heads out is what makes attribution trustworthy in the first place.
3. Recompute only the last FFN — bit-exact
Slot 8 is still sitting there at the end of the forward pass. Change it, and only
ffn10 → norm → unembed needs to rerun. Verified: recomputing from clean slots
reproduces full-forward logits with max deviation 0.00e+00.
This also cleanly separates direct effect (straight to logits) from indirect
(via later attention layers):
head
total effect
direct
direct share
L8h5
0.359
0.364
1.01
L1h9
0.181
0.147
0.81
L3h8
0.011
0.003
0.29
Aggregate direct share is 0.93 (prev) and 0.84 (b128) — depth here builds
slots rather than transforming them layer by layer.
4. Zero out "layer i reading layer j" and measure it
Each FFN's gate is k physically separate weight blocks; block j reads slot j.
So "is layer 7 actually using layer 3?" is answered by zeroing that block:
This also falsified a metric we had been logging: the norm-share percentages in the
training log (correlation) do not match causal ablation. f8's gate blocks all show
0.003–0.016 individually, yet zeroing them together costs 0.571 — the blocks are
mutually redundant, individually removable but collectively necessary.
5. Three end-to-end circuits, four independent methods agreeing
circuit
answer carriers
transport edge
previous-token
L8h5, L8h7, L1h9
f10 ← slot8 (+5.71)
retrieve-128-back
L1h9, L2h8, L8h7
f10 ← slot1 (+6.06)
identity (self)
— (margin 11.75, 0% error)
f10 ← slot0 (+6.55)
The two retrieval circuits share no components except the final FFN. Ablation
(necessity), patching (sufficiency), weight zeroing (transport), and direct/indirect
decomposition all point at the same heads.
6. Double dissociation between syntax and part-of-speech
The model has real contextual ability: on ambiguous tokens (same token, multiple POS
tags — 272 types, 17% of positions) probing accuracy goes 50.3% → 62.0%. Dependency
tree depth, which is essentially unavailable from the embedding, goes R² 0.111 → 0.479.
Both peak at slots 3–5 and fall off at the last slot — the same profile reported for
BERT-base.
And it localizes to individual heads:
ablated
dependency depth R²
POS (ambiguous)
baseline
0.479
60.5
slot4-head2
0.434 (−0.045)
60.5 (−0.0)
slot3-head3
0.458 (−0.010)
58.4 (−2.4)
One head carries structure, the other carries POS, and removing either leaves the
other intact. Downstream propagation checks out too (ablating slot4-head2 also costs
slot5 −0.031, while slot3 is untouched).
The part worth pausing on: ablating either head costs only ~0.02 LM loss —
the median across all 100 heads. They are functionally specific yet invisible in the
aggregate metric. That combination is exactly what interpretability work is usually
hunting for.
7. The residual read-out is not additively expressible
An additive residual forces every consumer to read all sources through one shared
matrix — formally, the gate must be rank-1 along the block dimension. Projecting the
trained model onto that class:
deletion (matched Frobenius energy)
ΔVAL
block-rank-1 (= additive-expressible)
+14.18
random subspace, same energy removed
+3.41
smallest-singular-value truncation
+3.15
4.2× worse than an energy-matched random subspace deletion. Block permutation
(each block reads a different slot) costs +9.67, i.e. worse than uniform.
This gap widened over training (3.3× at step 2000 → 4.2× at 38146): the longer it
trains, the more it relies on structure an additive stream cannot express.
Quickstart
python
1import torch
2from delta_d1280_h10_l10 import DeltaLM
34# 2.84 GB checkpoint — still carries optimizer state; weights are under "model"5ck = torch.load("delta_d1280_h128_l10_latest.pt", map_location="cpu",6 weights_only=False)7model = DeltaLM()8model.load_state_dict(ck["model"])9model.eval()1011# slots stay separable all the way to the output12logits, slots = model.run(idx, keep_slots=True)# slots[0]=embedding, slots[k]=att_k1314# zero layer 5 head 3, or substitute any tensor15logits = model.intervene(idx,[(5,3,None)])
Deliberately not wrapped in a transformers interface — the architecture doesn't
fit the abstraction, and pretending otherwise would add glue code and hide the parts
that matter.
Analysis toolkit
script
what it answers
head_test.py
head subspace independence; 10×10 ablation heat map; probe × head
patch_test.py
targeted rewriting; direct vs indirect; greedy joint patching
additive expressibility, with energy-matched controls
ling_probe.py
POS / dependency-depth probes, per-slot and per-head, with ablation
margin_test.py
float64 probe loss; margin as headroom; error rate in the tail
Each carries its reasoning and its caveats in the docstring, including which controls
are required and which metrics turned out to be misleading.
Also included: log_d1280_h10_l10.txt — the complete training log with all 38
diagnostic dumps; and test_log.txt — raw output of every analysis quoted above.
Nothing was cherry-picked; the logs are there so you can check.
Limitations
Trained probe tokens. Seven special tokens (self, prev, first, dup,
b16/b128/b512) are inserted at 0.2% during training. They were designed years ago
for probing standard transformers — the original motivation was finding a clean
scratchpad inside an additive residual stream — so they are not tailored to this
architecture. But they are trained-in, and a model without them would behave
differently.
No same-scale baseline. This is the big one. On a 48M controlled run (same params,
same data, same steps) the slot stack came out 0.07 nats worse than a standard
transformer. At 236M it was not measured. Every "cleaner" claim in this card is an
absolute measurement with no comparison — a standard transformer might well
produce a similarly sparse effect matrix. Untested.
Depth is a hard ceiling. Gate parameters grow as O(L²) — 52% of matrix
parameters at L=10, 71% at L=24, 79% at L=37. At L=24, training becomes unstable:
gradient norms spike to 10³ while a parameter-matched 35-layer standard transformer
sits at 0.2–0.9, and the loss gap widens during training (+1.38 → +1.70). L=10 is
comfortably inside the safe zone; do not assume this design scales in depth.
Gradients are structurally large. Slot 0 feeds every gate directly, so its gradient
is a sum over L paths. Hyperparameters cannot be copied from standard transformers, and
the mismatch grows with depth.
Head subspaces degrade over training. Mean pairwise overlap of the value subspaces
rose from 0.108 (step 4000) to 0.259 (final, L2) against a random baseline of 0.100.
"Each head reads its own directions" holds, but weakens.
Not a useful general model. 236M with a 3.16 val loss is not competitive. Use it
to study the architecture; don't use it to generate text.
Things we got wrong
Kept deliberately, because the failed hypotheses cost real GPU hours:
Scaling Wd by 1/√L made everything worse (VAL 6.48 → 6.97, spikes 1704 → 3203).
The standard-transformer depth correction addresses forward variance accumulation;
this architecture replaces rather than accumulates, so the correction is misapplied.
Loosening gradient clipping was wrong. The large gradient norms looked like
collateral damage from clipping; releasing the threshold triggered spike → damage →
more spikes. clip=1.0 is load-bearing.
A "rotated block rank-1" control was void. It amounts to additive residual with
scrambled inputs — strictly worse than additive, so it proves nothing. Additive
expressibility and block-rank-1 are the same set; there is no way to hold rank
fixed while toggling additivity. Use the random-subspace control instead.
Norm-share ≠ causal dependency. The gate-dependency percentages in the training
log are correlational and do not survive causal ablation.
Ratio metrics with near-zero denominators rank the worst items first. A
"syntactic specificity = POS/identity" score put a head with 22.1% POS accuracy
(majority baseline: 15.4%) at the top, purely because its denominator was smaller.
Citation
bibtex
1@misc{delta_slotstack_2026,
2 title = {Delta Slot-Stack: a non-additive residual pathway for interpretable transformers},
3 year = {2026},
4 note = {236M parameter model + interpretability toolkit},
5 url = {https://huggingface.co/Aurov/delta-slotstack-236m}
6}
Apache-2.0. Issues and reproductions welcome — especially the missing baseline.
