Views
No views yet
mistralai/Mistral-7B-Instruct-v0.3,
from Anthropic's Verbalizable Representations Form a Global Workspace in Language
Models, fitted with the
reference implementation anthropics/jacobian-lens.mistral7b_v03_lens.pt — a torch.save dict with keys J (dict layer -> [4096, 4096]
fp16 tensor), n_prompts, source_layers, d_model. Load it with
jlens.JacobianLens.load(...).| field | value |
|---|---|
| base model | mistralai/Mistral-7B-Instruct-v0.3 (32 layers, d_model=4096) |
| source layers | 0–30 (all 31 layers below the final) |
| target layer | 31 (final) |
| Jacobian shape | [4096, 4096] per layer, fp16 on disk |
| file size | ~0.97 GiB |
| fitting corpus | 100 WikiText-103 prompts × 128 tokens (first 16 positions skipped as attention sinks) |
| estimator | dim_batch=32, mean over valid source positions |
| fit hardware | 1× NVIDIA H100 80GB, ~45 min |
1import torch, transformers, jlens
2from huggingface_hub import hf_hub_download
3
4hf = transformers.AutoModelForCausalLM.from_pretrained(
5 "mistralai/Mistral-7B-Instruct-v0.3", dtype=torch.bfloat16
6).to("mps") # or "cuda"
7tok = transformers.AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
8model = jlens.from_hf(hf, tok)
9
10path = hf_hub_download(
11 "dormantx/jacobian-lens-mistral-7b-instruct-v0.3", "mistral7b_v03_lens.pt"
12)
13lens = jlens.JacobianLens.load(path)
14
15# per-layer lens logits at the last position, J-lens vs logit-lens baseline
16lens_logits, model_logits, _ = lens.apply(
17 model, "Fact: The currency used in the country shaped like a boot is",
18 positions=[-1], layers=[14, 18, 21], use_jacobian=True,
19)mistralai/Mistral-7B-Instruct-v0.3 is subject to its own license (Apache-2.0).