You can deploy this model and start earning money today!

analist/
sae-r1-distill-qwen-1.5b-resid-l20-reasoning

Name: sae-r1-distill-qwen-1.5b-resid-l20-reasoning
Availability: InStock

sparsify

sparse-autoencoder

interpretability

mechanistic-interpretability

reasoning

deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B

finetune

mit

Views

No views yet

SAE for DeepSeek-R1-Distill-Qwen-1.5B — resid_post layer 20

A TopK sparse autoencoder trained from scratch on the residual stream (resid_post) at layer 20 of deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B, over reasoning traces.

Details

Site: resid_post of decoder layer 20 (= output of model.model.layers[20], TransformerLens blocks.20.hook_resid_post)
Latents: 65536 | k (L0): 32 | d_in: 1536
Training data: open-thoughts/OpenThoughts-114k
Tokens: 300,000,000
Held-out reasoning FVU: 0.1236

Usage

python

from sparsify import Sae
sae = Sae.load_from_hub("analist/sae-r1-distill-qwen-1.5b-resid-l20-reasoning", hookpoint="resid_post_layer20", device="cuda")

Steering is intended at this layer (~71% relative depth), a region where reasoning-behavior features are steerable without producing token-level artifacts.

Notes

Trained with AuxK dead-latent resurrection and a unit-norm decoder; b_dec initialized to the resid_post activation mean. See training_log.json for the full config and FVU history.