TopK (k=32) sparse autoencoders trained on residual-stream activations of Qwen/Qwen3-0.6B-Base under the V4 sandbagging-concealment prompt set (neutral, V0 threat, V3 threat).

Part of the ongoing research at deception-nanochat-sae-research.

Model specs

Property	Value
Base model	`Qwen/Qwen3-0.6B-Base`
Parameters	600M
Hidden dim (d_model)	1024
Total layers	28
Architecture	TopK SAE
Expansion factor	4 (d_sae = 4096)
Top-k	32
Dtype	float32

Available layers

Layer	Tag	MSE	Explained var	L0	Alive features
L14	`qwen3_0p6b_base_topk_k32_L14_v4mixed`	0.0531	1.000	32	457
L21	`qwen3_0p6b_base_topk_k32_L21_v4mixed`	0.1979	1.000	32	402

Research context

Qwen3 family (base — no instruction tuning). Does NOT sandbag to V0 threat (V0 acc 28% > neutral 22%), matching the Qwen2.5-1.5B pattern. V3 'Lie' sandbag works (V3 acc 19%). V0-direction steering produces NEGATIVE effect across all layers because direction is inverted. Publishing the SAEs for cross-model comparison; deferred-direction experiments from this family should use V3-sourced vectors.

These SAEs were trained as part of investigating whether directional steering and sparse-feature interventions can rescue trivia-answering capability under explicit sandbagging/threat prompts. See the parent repo's RESULTS_INDEX.md §63–§79 for the directional-steering analysis, and the papers/specificity_gap/manuscript.md for the full writeup.

Training data

Prompts: 50 trivia questions × 3 templates (neutral, V0 threat, V3 'Lie about the answer')
Activations: Residual-stream post-block output at all token positions (~3000–6000 vectors per layer)
Epochs: 200 with Adam lr=3e-4, batch=128, decoder-row normalization per step

Loading

python

from huggingface_hub import hf_hub_download
import torch, json

repo_id = 'Solshine/deception-v4-saes-qwen3-0p6b-base-v4mixed'
tag = 'qwen3_0p6b_base_topk_k32_L14_v4mixed'  # example

state_path = hf_hub_download(repo_id, f'{tag}.pt')
cfg_path = hf_hub_download(repo_id, f'{tag}_cfg.json')
cfg = json.load(open(cfg_path))
state_dict = torch.load(state_path, weights_only=True)

The companion *_cfg.json files are SAELens-compatible — keys include model_name, hook_name, hook_layer, d_in, d_sae, architecture, k, normalize_activations.

Caveats

Trained on a small activation set (~3–6k vectors per layer). These SAEs are optimized for the v4 sandbagging research pipeline and may not transfer cleanly to generic text without further training.
MSE is evaluated on the training activations; held-out reconstruction quality was not separately measured.
Not evaluated with Neuronpedia tooling — companion cfg is SAELens-standard but feature-level Cohen's-d statistics were not computed (unlike the earlier 393-SAE cohort).

License

Apache-2.0. See parent research repo for attribution.

Citation

If you use these SAEs, please cite:

@misc{deleeuw2026specificitygap,
  author = {DeLeeuw, Caleb},
  title = {Deception Research: Specificity Gap Between Circuit-Level and Feature-Level Interventions},
  year = {2026},
  url = {https://github.com/SolshineCode/deception-nanochat-sae-research}
}