TopK (k=32) sparse autoencoders trained on residual-stream activations of meta-llama/Llama-3.2-1B under the V4 sandbagging-concealment prompt set (neutral, V0 threat, V3 threat).

Part of the ongoing research at deception-nanochat-sae-research.

Model specs

Property	Value
Base model	`meta-llama/Llama-3.2-1B`
Parameters	1.24B
Hidden dim (d_model)	2048
Total layers	16
Architecture	TopK SAE
Expansion factor	4 (d_sae = 8192)
Top-k	32
Dtype	float32

Available layers

Layer	Tag	MSE	Explained var	L0	Alive features
L8	`llama_topk_k32_L8_v4mixed`	0.0001	1.000	32	562
L12	`llama_topk_k32_L12_v4mixed`	0.0002	1.000	32	537

Research context

Llama-3 family. Trained as a companion to the §64/§68 directional-steering investigation in the parent repo. When used as the decoder for V4-prompt (genuine − threat) directions, achieves cos(raw, SAE) = 0.97 and SAE-space steering ≈ 92% of raw effect — refuting the earlier §64 impression (cos 0.44, 62% of raw) that was an artifact of using a SAE trained on broader activation distribution. See the parent repo's RESULTS_INDEX.md §63–§79 and the 2026-04-18 deep-work session log.

These SAEs were trained as part of investigating whether directional steering and sparse-feature interventions can rescue trivia-answering capability under explicit sandbagging/threat prompts. See the parent repo's RESULTS_INDEX.md §63–§79 for the directional-steering analysis, and the papers/specificity_gap/manuscript.md for the full writeup.

Training data

Prompts: 50 trivia questions × 3 templates (neutral, V0 threat, V3 'Lie about the answer')
Activations: Residual-stream post-block output at all token positions (~3000–6000 vectors per layer)
Epochs: 200 with Adam lr=3e-4, batch=128, decoder-row normalization per step

Loading

python

from huggingface_hub import hf_hub_download
import torch, json

repo_id = 'Solshine/deception-v4-saes-llama-3-2-1b-v4mixed'
tag = 'llama_topk_k32_L8_v4mixed'  # example

state_path = hf_hub_download(repo_id, f'{tag}.pt')
cfg_path = hf_hub_download(repo_id, f'{tag}_cfg.json')
cfg = json.load(open(cfg_path))
state_dict = torch.load(state_path, weights_only=True)

The companion *_cfg.json files are SAELens-compatible — keys include model_name, hook_name, hook_layer, d_in, d_sae, architecture, k, normalize_activations.

Caveats

Trained on a small activation set (~3–6k vectors per layer). These SAEs are optimized for the v4 sandbagging research pipeline and may not transfer cleanly to generic text without further training.
MSE is evaluated on the training activations; held-out reconstruction quality was not separately measured.
Not evaluated with Neuronpedia tooling — companion cfg is SAELens-standard but feature-level Cohen's-d statistics were not computed (unlike the earlier 393-SAE cohort).

License

Apache-2.0. See parent research repo for attribution.

Citation

If you use these SAEs, please cite:

@misc{deleeuw2026specificitygap,
  author = {DeLeeuw, Caleb},
  title = {Deception Research: Specificity Gap Between Circuit-Level and Feature-Level Interventions},
  year = {2026},
  url = {https://github.com/SolshineCode/deception-nanochat-sae-research}
}