TopK Sparse Autoencoders trained on Gemma 4 residual-stream activations, part of the
CrimsonRed emotion-vector project.
To our knowledge these are the first general-purpose SAEs published for Gemma 4 E2B / E4B.
(Existing public SAEs cover Gemma 2 (Gemma Scope), Gemma 4 31B, and the Gemma 4 26B MoE — not the
on-device E2B/E4B dense models.)
Available SAEs + emotion vectors
Model
Layer
d_model
d_sae
TopK
Recon cosine
SAE weights
Emotion vectors
Gemma 4 E2B (IT)
17
1536
24576 (16×)
50
0.982
gemma_e2b/layer17.sae.pt
gemma_e2b/emotion_vectors.pt
Gemma 4 E4B (IT)
21
2560
40960 (16×)
50
0.995
gemma_e4b/layer21.sae.pt
gemma_e4b/emotion_vectors.pt
Each emotion_vectors.pt is {layer: {emotion_name: float32[d_sae]}} — 171 contrastive
emotion profiles (emotion SAE profile − neutral SAE profile), drop-in for
crimsonred.sae_scoring.SAEScoringStrategy.
Residualizes a held-out query against the neutral profile (matching how the contrastive vectors are
built), then scores it against all 171 stored emotion vectors by cosine. 171-way classification;
random chance is 0.6%.
Model
cos top-1
cos top-5
Qwen (reference)
0.901
1.000
Gemma E2B
0.865
1.000
Gemma E4B
0.702
0.971
E4B's lower accuracy reflects genuine near-synonym confusions (energized→excited, terrified→uneasy),
not degenerate collapse — its misses are all within-family.
encode: pre = x @ W_enc.T + b_enc
acts = relu(pre)
f = topk(acts, k=50) # keep 50 largest, zero the rest
decode: x_hat = f @ W_dec.T + b_dec
Decoder columns are kept at unit norm during training.
Weights format
Each layer{N}.sae.pt is a torch.load(..., weights_only=True) dict:
Key
Shape
dtype
W_enc
[d_sae, d_model]
float32
W_dec
[d_model, d_sae]
float32
b_enc
[d_sae]
float32
b_dec
[d_model]
float32
This format is drop-in compatible with crimsonred.sae_scoring.load_sae_weights.
Training
Data: ~206,400 mean-pooled residual-stream vectors per model — 171 emotions × 1200 stories
plus 1200 neutral baselines, at the model's optimal scoring layer. Activations extracted with the
CrimsonRed llama.cpp fork (Q4_K_M quants).
Optimizer: Adam, lr 3e-4, batch 4096, 20 epochs (~4 min / model on an RTX 3090).
Loss: pure L2 reconstruction (sparsity is hard-enforced by TopK, no L1 term).
Eval metrics (20K held-out vectors)
Metric
E2B (L17)
E4B (L21)
mean cosine
0.9822
0.9953
median cosine
0.9830
0.9955
MSE
0.1212
0.0257
variance explained
0.4052
0.2676
mean L0
50.0 (exact)
50.0 (exact)
feature utilization
0.88%
0.12%
Feature utilization is low because the SAEs were trained on emotion-domain activations, which occupy
a lower-dimensional manifold than general text — most general-purpose features never fire. The
reconstruction cosines (0.982 / 0.995) confirm both SAEs capture the emotion-relevant subspace well.
Usage
python
1import torch
2from crimsonred.sae_scoring import SAEScoringStrategy, load_sae_weights
34# Load pre-computed emotion vectors (fastest path — no activation files needed)5raw = torch.load("gemma_e2b/emotion_vectors.pt", weights_only=False)6vectors = raw[17]# {emotion_name: float32[24576]}78# Score new hidden states9strategy = SAEScoringStrategy(vectors, sae_dir="gemma_e2b", layer=17)10scores = strategy.score_emotions(hidden_states, layer=17)# list of EmotionScore
