Views
No views yet
qwen7b_lens.pt — 27 Jacobians (source layers 0–26), each [3584, 3584] in fp16, ≈ 0.65 GiB. Loadable with jlens.JacobianLens.load.| Model | Qwen/Qwen2.5-7B-Instruct (28 layers, d_model 3584, bf16) |
| Corpus | 100 WikiText prompts, seq_len 128 |
| Source layers | 0–26 (all but the last) |
| skip_first | 16 (source positions before 16 skipped — attention sinks) |
| dim_batch | 64 |
| Hardware | 1× H100 80GB (≈ 18–22 s/prompt, ~34 min total) |
| Convergence | running-mean max delta fell 5.1e-1 → 1.2e-2 (~1/n) |
1import torch, transformers, jlens # pip install git+https://github.com/anthropics/jacobian-lens
2
3hf = transformers.AutoModelForCausalLM.from_pretrained(
4 "Qwen/Qwen2.5-7B-Instruct", dtype=torch.bfloat16
5).to("mps") # or "cuda"
6tok = transformers.AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
7model = jlens.from_hf(hf, tok)
8
9from huggingface_hub import hf_hub_download
10lens = jlens.JacobianLens.load(
11 hf_hub_download("dormantx/jacobian-lens-qwen2.5-7b-instruct", "qwen7b_lens.pt")
12)
13
14prompt = "Fact: The currency used in the country shaped like a boot is"
15lens_logits, model_logits, _ = lens.apply(model, prompt, layers=[16, 19, 24], positions=[-1])