You can deploy this model and start earning money today!

google/
gemma-scope-9b-pt-att

Name: gemma-scope-9b-pt-att
Availability: InStock

saelens

2408.05147

cc-by-4.0

Views

No views yet

1. Gemma Scope

Gemma Scope is a comprehensive, open suite of sparse autoencoders for Gemma 2 9B and 2B. Sparse Autoencoders are a "microscope" of sorts that can help us break down a model’s internal activations into the underlying concepts, just as biologists use microscopes to study the individual cells of plants and animals.

See our landing page for details on the whole suite. This is a specific set of SAEs:

2. What Is `gemma-scope-9b-pt-att`?

gemma-scope-: See 1.
9b-pt-: These SAEs were trained on Gemma v2 9B base model.
att: These SAEs were trained on the model's attention layer output before the linear projection.

3. How can I use these SAEs straight away?

python

from sae_lens import SAE  # pip install sae-lens

sae, cfg_dict, sparsity = SAE.from_pretrained(
    release = "gemma-scope-9b-pt-att-canonical",
    sae_id = "layer_0/width_16k/canonical",
)