Two fitted template lenses for one model: a 12-word pilot (the method proven,
Fig-62 replicated) and a 175-word deception probe panel with a calibrated
reader. Contents are described per release below.
Release 1 — pilot (12 words)
A fitted template lens for Qwen/Qwen3.6-27B,
implementing the method from the appendix "Extending the Jacobian lens to
multi-token concepts" of Anthropic's
Verbalizable Workspace paper.
To our knowledge this is the first public template-lens artifact for any model
(survey 2026-08-19: no implementation in the companion repo, the jlens HF
ecosystem, or GitHub).
For a word w, the template vector is the ridge-regularized linear
discriminant for "the model is about to say w":
where mu_w(l) is the mean residual-stream activation at the final position
of passages written so that w is their natural continuation (and never
contains w), and mu, Sigma are the mean/covariance over all training
passages at layer l. Read a concept by projecting a residual onto t_w;
steer by adding/subtracting; no training loop, no gradients.
Contents
file
what
out/templates.safetensors
templates [17, 12, 5120] fp32 (layers 0,4,…,60,64 in hidden_states indexing: index l = output of block l−1), plus mu [17,5120], mu_w [17,12,5120]
out/sigma.safetensors
per-layer covariance [17,5120,5120] fp32 — mint templates for new words without refitting
Vocabulary (leading-space token counts verified on this model's tokenizer):
fire, Paris, mirror, blackmail, consciousness (1 tok); photosynthesis,
Beethoven, New Zealand (2); self-awareness, Golden Gate Bridge (3);
Tchaikovsky, eavesdropping (4).
Pilot results (all numbers reproducible from this repo)
Held-out readout (gate 1). Top-1 word identification among 12
(chance 0.083) climbs monotonically with depth: 0.31 (embeddings) → 0.63
(L12) → 0.83 (L36) → 0.979 (final layer).
Functional comparison vs an Anthropic-fit J-lens (gate 3). Using the
Neuronpedia-hosted J-lens
for this model (n=1000, credited to Anthropic Interpretability), top-hit rate
by concept token length on the same held-out residuals, layer 60:
lens
1 tok
2 tok
3 tok
4 tok
template (top-1 of 12)
1.00
1.00
0.92
0.92
J-lens (first token in top-10 of 248,320)
0.92
0.67
0.33
0.00
The J-lens is healthy on single tokens and degrades sharply with concept
length; the template lens stays flat — the paper's core claim, replicated.
Note the ranking universes differ (12-way vs full vocab); the shape
contrast is the result, not the absolute gap.
Geometric alignment (gate 2). Median cosine between single-token
templates and J-lens directions J_l^T u_t is 0.115 (per-word maxes
0.18–0.33). Strongly non-random in 5120 dims (~8–24σ) but far from vector
identity — consistent with the paper's footnoted deviations (different
averaging corpora; logit- vs probability-gradients; ridge).
Swap steering (gate 4) — partial, reported honestly. Using a simplified
intervention (h += gamma*(mu_B − mu_A) at a layer band; NOT the paper's
projective coefficient-matched swap): Tchaikovsky→Beethoven reproducibly flips
"The composer of Swan Lake… was born in the country of" from Russia to
Germany (two band/scale combos) or Austria/"Ludwig". The reverse direction
injects Tchaikovsky content ("…the composer of the Nutcr…") without flipping
the answer, and a fire→mirror null control contaminates generation — so this
translation is blunt concept injection, a causal handle but not a clean
swap. The projective swap is the natural next step.
A bucketed probe panel: 62 deception + 12 AI-deception + 25 honesty (the
opposite pole) + 48 neutral controls + the 28 pilot words as a regression set.
Controls and the honesty pole are what make readings interpretable.
Full report: RESULTS_panel_20260820.md.
file
what
out_panel/templates_13k.safetensors
templates [17, 175, 5120] fp16 (+ mu, mu_w); same layer convention as release 1
out_panel/sigma_13k.safetensors
per-layer covariance from 9,800 passages
out_panel/null_calibration.safetensors
per-word null mean/std over 3,936 wikitext positions → z-scores
out_panel/null_calibration_ab.safetensors
position-matched null for forced-A/B pre-answer reads
out_panel/metrics_13k.json
held-out top-1/top-10 per layer + naturalness audit (subject's rank of each target at each passage endpoint)
out_panel/bucket_eval.json
bucket-level held-out accuracy
passages_panel/shard_000.jsonl
11,200 passages (64/word), written by gpt-5.4 (reasoning: none) via the Responses API, validated by assemble_passages.py
wordlist_deception.json
panel with buckets and sense hints
panel_chat.py, panel_battery*.py, latent_trace.py
the reader, battery runner, and per-position tracer
Held-out (1,400 passages): word top-1 0.31 / top-10 0.52 at L60 (175-way,
chance 0.006); bucket top-1 0.65 (chance ≈ 0.25). A coarse instrument by
design — synonyms blur; families separate.
Whole-prompt readout (the paper's multi-hop result, on this panel): reading
every prompt position, all six calibration latents rank 1/175 at their cue
tokens, and four abstract latents never named in the prompt read at z 10–16
(procrastination, deja vu, eavesdropping, vertigo 16.0); controls show nothing
in the pilot bucket. Reading only the four pre-answer template positions was a
null — read where the concept is inferred, not where the answer is typed,
and mask position 0 / special tokens.
Audit-driven caveats: 13 templates are weak (the subject rarely ranks the
target in its top-50 at their passage endpoints: infiltrate, scheming,
duplicity, …) and fire idiosyncratically; readers filter them. Filtering the
training passages to subject-endorsed ones and refitting made accuracy worse
(too few passages left) — kept as a negative result. Polysemy leaks (bluff
fires at "cliff"). Nothing here yet separates talking about deception from
being deceptive; that requires matched content with varied intent.
Each reply prints the panel words that fired as word(max-z × tokens≥z) by
bucket. panel_battery_trace.py --prompts file.txt reads every position of
prompt+reply at chosen layers and writes a transcript-style report.
Usage (release 1 artifacts)
python
1import json, torch
2from safetensors import safe_open
3from safetensors.torch import load_file
45d = load_file("out/templates.safetensors")6with safe_open("out/templates.safetensors","pt")as f:7 meta = f.metadata()8layers = json.loads(meta["layers"]); words = json.loads(meta["words"])910# score a residual-stream activation h (torch [5120], hidden_states index l)11li = layers.index(44)12scores = d["templates"][li] @ h # [12] — one score per word13print(sorted(zip(words, scores.tolist()), key=lambda x:-x[1])[:3])
To mint a template for a new word: generate passages the same way
(passages/ shows the format), collect final-position residuals, and solve
against out/sigma.safetensors — no model fitting required beyond forward
passes.
Provenance
Release 1: fit 2026-08-19 on 1× RTX PRO 6000 Blackwell 96 GB (vast.ai), bf16,
624 training passages, 134 s; passages written by Claude (Fable 5).
Release 2: fit 2026-08-20, same hardware class, 9,800 training passages,
~13 min; passages written by OpenAI gpt-5.4 (reasoning none) and audited
against the subject's own next-token ranks. Details in the fit_provenance*.json
files.
Method: Anthropic, Verbalizable Representations Form a Global Workspace in
Language Models (2026), appendix. Cross-check lens: neuronpedia/jacobian-lens
(@mntss). This artifact: ec75hash, with Claude Code.