Views
No views yet
meta-llama/Llama-3.3-70B-Instruct into soft token embeddings for self-interpretation via patching.Warning: These adapters are trained specifically formeta-llama/Llama-3.3-70B-Instruct(residual stream dim 8192). They will produce garbage results on other models, even if tensor shapes happen to match.
| File | Architecture | Training Data | Params | Val Loss |
|---|---|---|---|---|
goodfire-sae-scalar-affine.safetensors | Scalar affine | Goodfire SAE | 8,193 | 2.023 |
goodfire-sae-sa-lr16.safetensors | SA + Low-rank (r=16) | Goodfire SAE | 270,337 | 1.868 |
1from selfie_adapters import load_adapter
2
3adapter = load_adapter("goodfire-sae-scalar-affine.safetensors", device="cuda")
4soft_tokens = adapter.transform(hidden_state_vectors)<|reserved_special_token_0|> as the injection site for the soft token):<|begin_of_text|><|start_header_id|>user<|end_header_id|>
What is the meaning of "<|reserved_special_token_0|>"?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
The meaning of "<|reserved_special_token_0|>" is ".safetensors file contains the projection weights with full training config embedded in the header metadata. You can inspect the metadata without loading the tensors:1from safetensors import safe_open
2import json
3
4with safe_open("goodfire-sae-scalar-affine.safetensors", framework="pt") as f:
5 meta = f.metadata()
6 print(meta["projection_type"]) # "scalar_affine"
7 print(meta["model_name"]) # "meta-llama/Llama-3.3-70B-Instruct"
8 config = json.loads(meta["config_json"]) # full training config