Gemma 2b Residual Stream SAEs.

Resid Post 0

• 16384 Features (expansion factor 8)
• CE Loss score of 99.1% (2.647 without SAE, 2.732 with the SAE)
• Mean L0 54 (in practice L0 is log normal distributed and is heavily right tailed).
• Dead Features: We think this SAE may have ~2.5k dead features.

• This SAE was trained with methods from the Anthropic April Update excepting activation normalization.
• It is likely under-trained.

Resid Post 6

• 16384 Features (expansion factor 8) achieving a CE Loss score of
• CE Loss score of 95.33% (2.647 without SAE, 3.103 with the SAE)
• Mean L0 53 (in practice L0 is log normal distributed and is heavily right tailed).
• Dead Features: We think this SAE may have up to 7k dead features.

• This SAE was trained with methods from the Anthropic April Update
  • Excepting activation normalization.
  • We increased the learning rate here by one order of magnitude in order to explore whether this resulted in faster training (in particular, a lower L0 more quickly)
    • We find in practice that the drop in L0 is accelerated but this results is significantly more dead features (likely causing worse reconstruction)
• As above, it is likely under-trained.

Resid Post 12

• 16384 Features (expansion factor 8) achieving a CE Loss score of
• CE Loss score of 95.99% (2.563 without SAE, 2.96 with the SAE)
• Mean L0 52 (in practice L0 is log normal distributed and is heavily right tailed).
• Dead Features: Less than 200 dead features.

• This SAE was trained with methods from the Anthropic April Update
  • With activation normalization. This means that activations should be multiplied by a constant such that E(|X|) = sqrt(2048)
• As above, it is likely under-trained.