Views
No views yet
Qwen/Qwen2.5-14B-Instruct into soft token embeddings for self-interpretation via patching. Part of the Qwen 2.5 scaling series (7B, 14B, 32B, 72B).Warning: This adapter is trained specifically forQwen/Qwen2.5-14B-Instruct(residual stream dim 5120). It will produce garbage results on other models, even if tensor shapes happen to match.
| File | Architecture | Training Data | Params | Val Loss |
|---|---|---|---|---|
wikipedia-full-rank.safetensors | Full-rank affine | Wikipedia contrastive vectors | 26,219,520 | 1.486 |
1from selfie_adapters import load_adapter
2
3adapter = load_adapter("wikipedia-full-rank.safetensors", device="cuda")
4soft_tokens = adapter.transform(hidden_state_vectors)<|fim_pad|> as the injection site for the soft token):<|im_start|>user
What is the meaning of "<|fim_pad|>"?<|im_end|>
<|im_start|>assistant
The meaning of "<|fim_pad|>" is ".safetensors file contains the projection weights with full training config embedded in the header metadata. You can inspect the metadata without loading the tensors:1from safetensors import safe_open
2import json
3
4with safe_open("wikipedia-full-rank.safetensors", framework="pt") as f:
5 meta = f.metadata()
6 print(meta["projection_type"]) # "full_rank"
7 print(meta["model_name"]) # "Qwen/Qwen2.5-14B-Instruct"
8 config = json.loads(meta["config_json"]) # full training configwikipedia-full-rank adapter was trained on contrastive hidden-state vectors — raw activations with the per-layer dataset mean subtracted. To use this adapter on new inputs, you need the same mean vectors that were subtracted during training.mean-vectors.safetensors contains one mean vector per layer (24 layers: 12–35).1import json
2from safetensors import safe_open
3from safetensors.torch import load_file
4
5# Load all mean vectors
6mean_vectors = load_file("mean-vectors.safetensors")
7
8# Access a specific layer's mean vector
9mean_vec = mean_vectors["layer_24"] # shape: [5120], dtype: float32
10
11# Given a raw hidden state from that layer:
12contrastive_vec = raw_hidden_state.float() - mean_vec
13soft_tokens = adapter.transform(contrastive_vec)
14
15# To see which layers are available:
16with safe_open("mean-vectors.safetensors", framework="pt") as f:
17 meta = f.metadata()
18 layers = json.loads(meta["layer_indices"])
19 print(layers) # [12, 13, ..., 35]"Tell me about {title}." with the Qwen chat format. Subtracting them ensures the adapter sees zero-centered inputs matching its training distribution.