You can deploy this model and start earning money today!

nambor/
refpred-operators

transformers

interpretability

mechanistic-interpretability

residual-stream

other

endpoints_compatible

Views

No views yet

refpred-operators — fitted reference operators for transformer residual streams

Per-layer affine transition operators A_l, b_l fitted to the residual streams of Gemma-3-1b and Llama-3.2-1B, by streaming ridge regression over a corpus. They give you a reference for what each layer usually does to the state, so you can score a single trajectory against it instead of against zero:

h_{l+1} ≈ A_l · h_l + b_l          (the reference prediction)
ε_l     =  h_{l+1} − (A_l · h_l + b_l)   (the innovation — what it does NOT predict)

This is the tool for the write-up A Fitted Reference Predictor for Transformer Residual-Stream Trajectories. You do not need to re-run the fitting pipeline — torch.load these and subtract from your own hidden states.

Files

file	contents
`operators_gemma.pt`	Gemma-3-1b, d=1152, 26 layers (~790 MB)
`operators_llama.pt`	Llama-3.2-1B, d=2048, 16 layers (~1.6 GB)
`load_operators.py`	tiny loader: `innovation(ops, h_l, h_next, layer)`

Each .pt is a self-describing dict:

python

{
  "meta": { family, d, n_layers, hf_id_pt, hf_id_it, corpus, ridge_lambda, ... },
  "PT-COMP": { "raw": {"A": [nL,d,d], "b": [nL,d]}, "normed": {...} },
  "IT-COMP": { "raw": {...}, "normed": {...} },
  "IT-CHAT": { "raw": {...}, "normed": {...} },
}

Conditions — PT-COMP (base weights, no template), IT-COMP (instruct weights, no template — the usual "deploy on the weights you'll use" default), IT-CHAT (instruct weights, chat-templated).
Variants — raw fits hidden_states as returned; normed fits input_layernorm_l(h_l), i.e. what block l actually reads. The two diverge strongly on Gemma because of its (1+γ) RMSNorm — use normed if you feed normalised states.

Usage

python

import torch
from load_operators import load_operators, innovation

ops = load_operators("operators_gemma.pt")          # or operators_llama.pt

# your own hidden states for a prompt: hs[l] is [seq, d] (output_hidden_states=True)
layer = 12
eps = innovation(ops, hs[layer], hs[layer + 1], layer,
                 condition="IT-COMP", variant="raw")   # [seq, d]

predict_next(ops, h, layer, ...) gives the reference prediction alone.

Fit details

Ridge (λ_rel = 1e-6), fp32 extraction / fp64 solve, on fineweb-edu (sample-10BT, seq_len 128, ~1400 fit docs). Operators are stored fp32. The BOS position is excluded (attention sink); the chat scaffold is excluded by position for IT-CHAT.

What these are (and are not)

They are regression coefficients fitted on activations — not the model weights, and not a runnable model. To produce hidden states you must load the base model yourself (both are gated):

Gemma-3-1b — google/gemma-3-1b-pt / google/gemma-3-1b-it, Gemma Terms of Use (https://ai.google.dev/gemma/terms).
Llama-3.2-1B — meta-llama/Llama-3.2-1B / meta-llama/Llama-3.2-1B-Instruct, Llama 3.2 Community License.

Your use of these operators is subject to the respective base-model licenses.

Caveats (read the write-up)

The reference is relative, not absolute: an operator fitted on one domain can be worse than useless on another (prose-vs-arithmetic differ ~58% on shared support). Fit-distribution coverage is a real precondition.
A large ε means "large," not "unusual" — turning it into an anomaly score needs local calibration that these operators alone do not provide.
Everything here is in the model's native basis; the geometry is coordinate-dependent.

Links & citation

Write-up: A Fitted Reference Predictor for Transformer Residual-Stream Trajectories — https://robman.fyi
Code (fitting pipeline + analyses): see the accompanying repository.

bibtex

@misc{manson_refpred_operators,
  title  = {Fitted reference operators for transformer residual streams},
  author = {Manson, Rob},
  year   = {2026},
  url    = {https://robman.fyi}
}

nambor/refpred-operators