Six JumpReLU sparse autoencoders trained on the residual stream of a Nemotron 3.5 30B-A3B
agent, plus the full evidence set for a home-repair tool-selection scenario: an inference
audit over a 36-cell factorial prompt grid, and a safety tripwire that failed two
gates — the second of them pre-registered — and is published as a negative result.
The SAEs were trained on
575-lab/kiji-inspector-demo-pairs,
a public dataset of contrastive agent prompts, so both the dictionaries and the evidence
below are reproducible from public inputs.
Results
There is no first-tool decision to explain in this scenario
The agent's first tool follows the explicit ask with p ≈ 1. A sweep of 743 anchor/contrast
pairs from the scenario's own training data found zero pairs whose first tool flips
(evidence/sweep/training_pairs_flips.json), and the audit grid reproduces it: with a
quote, video, or no-ask request the model picks the same tool with p ≥ 0.95 in 27 of 27
cells, whatever the stated fuel or appliance age (evidence/audit/audit_report.json →
audit.askSummary).
One exception. Asking for a replacement part is the only non-saturated ask: ordering the
part competes with checking the manual first. Every stated-12-year-old cell leans to the
manual (ManualCheck 0.59–0.68), while three cells are near-exact ties at ≈ 0.498
(audit.softRegion).
What the features read into a request, and whether it survives a test
Each cell's top features are binned stated / inferred / bleed / ambient. A feature only
counts as inferred when its label carries content the request never states and exactly
one grid axis carries its activation (dominant axis range ≥ 0.5 and ≥ 2× the runner-up).
Over the 36 cells: 107 inferred, 75 ambient, 17 stated, 17 bleed (audit.binCounts).
Twenty distinct features pass the inference test, and in this run every one of them is
validated by the ask axis, not by fuel or age (audit.axisFindings) — including
gas-labelled features such as "Gas appliance repair video search" (video-cell mean 6.73 vs
≤ 0.85 elsewhere). The label names an appliance; the grid shows the ask is what moves it.
This is the central caution of the scenario: a label is a hypothesis, the axis is the
evidence.
The safety tripwire: tested twice, failed twice
The idea was an alarm — sum activation on the hazardous side of the training contrastive map
and fire when a hazard-present request with a compliant DIY ask clearly exceeds its matched
hazard-free control (≥ 1.5×, strict majority of pairs).
Round
Layer
Result
Verdict
Selection pairs (metric designed on these)
27
1 / 7 pairs
NO-GO
Held-out pairs, authored blind, pre-registered
34
4 / 9 pairs
NO-GO
Layer 27 also fails as an absolute detector: hazard and control masses overlap (weakest
hazard 4.26 < loudest control 5.72), so no fixed threshold separates them.
A post-hoc sweep of all six layers found layer 34 passing 5/7 on the same ten prompts.
Because that is six attempts reported as one result, it was demoted to a hypothesis and
tested once: evidence/holdout_prereg.md freezes the layer, score, threshold and pass rule,
and nine fresh hazard/control pairs were authored before any capture. The gate returned
4 / 9 (nearest miss 1.49×), and the pre-registered side-prediction — that the two
electrical pairs would miss — came back refuted: they were the strongest of the round at
2.74× and 2.37×, while the gas/thermal pairs that had produced the selection-round pass went
2/7. Full per-pair numbers in evidence/audit/holdout/holdout_gate_result.json.
What did replicate is behavioural, not feature-level: on two prompts the model changed
tool unasked, both electrical hazards — dw_breaker (PartsSearch 0.91 → ManualCheck 0.60)
and, on the blind set, wa_tingle (PartsSearch 0.90 → ManualCheck 0.59). No gas hazard ever
flipped the tool; the strongest gas response was verbal ("this is a serious safety concern")
while still fetching the requested video.
The honest summary: these features can read a hazard; they could not be made into an
alarm.
Causal ablation: which features actually carry the tool choice
For each contrast type the intervention zeros the top 10 ranked features whose mean
activation is greater on the anchor prompts, and evaluates every retained prompt under four
paired conditions: the unmodified model; SAE reconstruction with nothing ablated;
reconstruction with the 10 selected features ablated; and reconstruction with 10
frequency-matched random features ablated. The intervention happens in SAE latent space —
z = encode(x), zero the selected coordinates to get z_-S, feed decode(z_-S) back.
so the prompt-specific SAE round-trip effect cancels algebraically. Flip-rate intervals are
exact Clopper–Pearson; CATE intervals are 2,000-resample percentile bootstraps. Pooled
over the five home-repair contrast types, per layer:
Layer
Types
Baseline-correct prompts
Targeted flip [95% CI]
Random flip [95% CI]
Reconstruction flip [95% CI]
Paired CATE [95% CI]
6
5
158
0.0127 [0.0015, 0.0450]
0.0063 [0.0002, 0.0348]
0.0063 [0.0002, 0.0348]
−0.00051 [−0.00151, +0.00022]
13
4
148
0.0135 [0.0016, 0.0480]
0.0068 [0.0002, 0.0371]
0.0000 [0.0000, 0.0246]
+0.00142 [+0.00003, +0.00312]
20
5
158
0.0190 [0.0039, 0.0545]
0.0316 [0.0104, 0.0723]
0.0253 [0.0069, 0.0635]
+0.00061 [−0.00030, +0.00191]
27
5
158
0.0759 [0.0399, 0.1289]
0.0190 [0.0039, 0.0545]
0.0127 [0.0015, 0.0450]
+0.00332 [+0.00125, +0.00599]
34
4
148
0.0405 [0.0150, 0.0861]
0.0000 [0.0000, 0.0246]
0.0000 [0.0000, 0.0246]
+0.00413 [+0.00073, +0.00847]
43
4
148
0.1081 [0.0631, 0.1696]
0.0000 [0.0000, 0.0246]
0.0000 [0.0000, 0.0246]
+0.00680 [+0.00190, +0.01297]
Causal leverage arrives late. From layer 20 upward the paired CATE grows monotonically
(+0.0006 → +0.0033 → +0.0041 → +0.0068) and by layer 43 the targeted arm flips 10.8 % of
prompts while both control arms flip zero. Layers 6–20 are indistinguishable from their
controls; at layer 20 the random arm actually flips more often than the targeted one.
But it is not spread evenly across the features. The same battery, split by contrast
type, separates two families cleanly:
Layer
safe_vs_hazardous flip [95% CI]
CATE
warranty_covered_vs_out_of_pocket flip [95% CI]
CATE
paired p
6
0.000 [0.000, 0.218]
+0.0000
0.095 [0.012, 0.304]
−0.0047
0.50
13
0.000 [0.000, 0.176]
+0.0002
0.118 [0.015, 0.364]
+0.0131
0.25
20
0.000 [0.000, 0.218]
+0.0003
0.095 [0.012, 0.304]
−0.0001
1.00
27
0.000 [0.000, 0.218]
+0.0006
0.429 [0.218, 0.660]
+0.0130
0.0039
34
0.000 [0.000, 0.176]
+0.0001
0.353 [0.142, 0.617]
+0.0360
0.0156
43
0.000 [0.000, 0.176]
−0.0002
0.412 [0.184, 0.671]
+0.0570
0.0078
The hazard map is causally inert at every trained depth — flip rate exactly 0.000 at all
six layers, CATE never exceeding ±0.0006, paired exact test p = 1.00 throughout. The warranty
map, measured the same way on comparable prompt counts, reaches a 41 % flip rate and a CATE
of +0.057.
This matters for the tripwire above. Its gate failed because hazard-side mass does not
separate hazardous requests from matched controls; the ablation shows, independently, that
those same features do not move the decision at any layer. The failure was not a bad
threshold, a wrong layer, or an unlucky prompt set — the features the tripwire was built on
are not load-bearing.
Read the prompt counts before the effect sizes. The battery only evaluates prompts where
the unmodified model already predicts the pair's expected tool, and in this scenario it
usually does not: 158 of 500 attempted prompts survive that filter at layer 27, and only 4
survive for diy_vs_professional. That is the "the ask decides the tool" result appearing
inside the ablation's own filter, and it is why several intervals here are wide. Per-type
counts, drivers and all three arms are in layer_<N>/ablation/ablation_report.json; the
pooled view is evidence/ablation_summary.json.
Experimental setup
Subject model: Nemotron 3.5 30B-A3B BF16 (d_model 2688). Activations were extracted
from a local -no-mtp conversion via a modified vLLM hidden-states connector, at the
decision token — the position where the agent commits to a tool name.
Layers: 6, 13, 20, 27, 34, 43.
SAE: JumpReLU, d_sae 10752 (4× expansion), adaptive-L1 to a target L0 of 75, with
threshold auto-calibration.
Training (layer_*/sae_checkpoints/config.json): batch 256, lr 3e-4, 10 epochs,
gradient clip 1.0, l1_max 0.1, dead-feature resampling off, seed 42.
Extraction scale (layer_*/activations/metadata.json): 95,386 decision-token
activations from 47,693 contrast pairs, one vector per prompt, spanning 18 contrast
types across 2 scenarios — home_repair (24,796 pairs, 4 tools) and tool_selection
(22,897 pairs, 8 tools). Those 95,386 vectors cover only 17,883 unique prompt strings;
the generator repeats itself ~5×. See Limitations before quoting the scale.
Training data is public:
575-lab/kiji-inspector-demo-pairs
— 50,076 generated pairs, of which the 47,693 whose anchor and contrast map to different
tools were kept for extraction. The dataset is content-identical to the local pairs used
here; only two label conventions differ (contrast_type carries a repair_/selection_
prefix there, and scenario_name is home/tool rather than
home_repair/tool_selection). Every prompt, tool, intent and similarity score matches
row for row, so the dictionaries are reproducible end to end from public inputs. Fetch it
into the layout the training pipeline expects with:
uv run python -m kiji_inspector.huggingface.download_pairs 575-lab/kiji-inspector-demo-pairs
which writes output/pairs/shard_*.parquet plus scenarios_meta.json (override the
destination with --pairs-dir).
SAE health
Reconstruction is measured on this scenario's own vLLM decision-token activations
(evidence/ companion run); L0 is the mean active count on those prompts.
Layer
Alive
Dead
L0 (mean)
Explained var.
Cosine
6
18.0 %
67.8 %
180.5
0.836
0.918
13
21.2 %
55.3 %
128.2
0.849
0.922
20
21.4 %
54.3 %
145.1
0.828
0.913
27
21.4 %
57.3 %
140.4
0.867
0.933
34
17.8 %
63.6 %
128.8
0.915
0.954
43
13.7 %
68.6 %
135.9
0.935
0.966
Alive/dead from layer_*/sae_checkpoints/feature_health.json; reconstruction from the
six-layer vLLM evaluation. On the training distribution the achieved L0 is 57.7–65.8,
undershooting the target of 75; on these live prompts it roughly doubles (128–181), i.e. the
dictionaries are being pushed off-distribution and say so rather than failing silently.
Caveat: these 24 evaluation prompts come from the same scenario family the SAEs were
trained on, so the explained-variance column is in-distribution, not held out.
Feature interpretation and validation
Labels are generated per feature and then independently fuzz-tested
(layer_*/activations/fuzzing_summary.json): a scorer must identify which examples actually
activate the feature given only its label.
Layer
Features scored
Combined score
Token-level accuracy
6
1975
0.972
0.965
13
2318
0.960
0.954
20
2341
0.958
0.950
27
2319
0.962
0.953
34
1964
0.966
0.959
43
1582
0.967
0.962
All layers reject the baseline at p < 0.001. A high fuzzing score means the label predicts
the activating examples — not that the label names the feature's cause, which is exactly
what the evidence/ audit shows can differ.
Repository contents
layer_<N>/sae_checkpoints/sae_final.pt trained SAE (encoder, decoder, thresholds)
layer_<N>/sae_checkpoints/config.json training hyperparameters (incl. seed)
layer_<N>/sae_checkpoints/feature_health.json alive/dead/ultra-rare counts
layer_<N>/sae_checkpoints/firing_rates.npy per-feature firing rate
layer_<N>/sae_checkpoints/metrics.jsonl per-step training curve
layer_<N>/activations/feature_descriptions.json generated feature labels
layer_<N>/activations/contrastive_features.json per-contrast-type feature maps (Cohen's d)
layer_<N>/activations/fuzzing_results.json per-feature label validation
layer_<N>/activations/fuzzing_summary.json aggregate label validation
layer_<N>/activations/decision_report.json decision-token analysis
layer_<N>/activations/metadata.json extraction provenance
layer_<N>/ablation/ablation_report.json paired-CATE ablation: all three arms,
per-prompt deltas, CP + bootstrap intervals
evidence/holdout_prereg.md pre-registration of the second gate
evidence/audit/audit_capture.json 36 grid cells + 10 tripwire prompts, readouts
evidence/audit/audit_activations.npz their decision-token activations, all layers
evidence/audit/audit_report.json binned features, saturation, soft region, gates
evidence/audit/holdout/* the nine blind pairs and their gate result
evidence/sweep/training_pairs_flips.json the 0-of-743 first-tool sweep
evidence/ui_data.json the rendered demo payload
evidence/ablation_summary.json the ablation pooled over home-repair types
Deliberately not uploaded: raw activation shards (shard_*.npy, ~978 MiB per layer),
intermediate optimiser checkpoints (step_*.pt, ~331 MiB each) and per-layer prompts.json.
normalize_input applies the exact transform the SAE was trained under —
(x - mean_vec) / rms_scale. Raw activations must go through it or the JumpReLU thresholds
are meaningless.
To reproduce the audit numbers without a GPU, encode the captured activations directly:
Use the checkpoint's native thresholds with no offset for vLLM-captured activations.
Limitations
The dictionaries are small-data. 95,386 training activations for 10,752 features is
~8.9 vectors per feature — and far fewer distinct ones, since only 17,883 prompt strings
are unique — against ~28.9 M decoder parameters. Production SAEs train on orders of
magnitude more. It shows in the numbers above: 54–69 % of features are dead and the
achieved L0 undershoots its target. The pair dataset is public, so this is checkable
rather than taken on trust.
Every training vector is the same token position of the same family of agent prompts.
A dictionary that has only read appliance-support requests has little material to learn
"danger" as something separable from "gas appliance" — the most likely reason the
tripwire's hazard side behaves topically.
Feature identity does not survive retraining. Retraining on the same data with a
different seed, mean best decoder cosine to any feature of the other run is 0.246 at
layer 27 / 0.263 at layer 43, with only 0.21 % / 0.11 % of features exceeding 0.7 —
against a Gaussian null of ~0.065. Functionally it is kinder: mean best activation
correlation 0.664 / 0.752, with 43 % / 63 % finding a counterpart at r ≥ 0.7. What the
dictionary measures is partly reproducible; which index measures it is not.
Features are not the best predictor of the decision. At layer 27 a held-out
bag-of-words probe predicts the tool at 86.3 % and the raw residual at 87.0 %, versus
76.8 % for SAE features — consistent with features that describe the request rather than
carry the choice.
The tripwire result is negative and should not be re-tuned on this data. Its decision
rule was frozen before the held-out capture; re-selecting the layer, threshold or feature
set against the same captures would be selection, not evidence.
Concept labels are auto-generated. Fuzzing validates that a label predicts activating
examples, not that it names the cause; the grid audit exists because those differ.
Scenario scope. All results are for one home-repair agent with four tools. The same
training dataset also contains a tool_selection scenario (eight tools), where the first
tool choice genuinely does move — these null results are specific to home repair, not a
claim that SAE features are never causal.
Revision history
r1 (2026-08-21) — initial release: six-layer run plus the home-repair audit and the
pre-registered tripwire post-mortem.
