Qwen3-8B Jacobian Lens (jlens)
Jacobian lens fitted for
Qwen/Qwen3-8B, following the
Anthropic jlens reference and cot-oracle layer
selection (oracle injection layers 9/18/27 plus AO band 21–25).
Fit configuration
| Setting | Value |
|---|
| Model | Qwen/Qwen3-8B |
| Corpus | HuggingFaceFW/fineweb (sample-10BT) |
| Final fit prompts | 1000 (4-node shard: 250 × 4, --corpus-skip 0 / 250 / 500 / 750) |
| Pilot fit | 100 prompts (4 × 25), used for early read/fingerprint below |
| Source layers | 9, 18, 21, 22, 23, 24, 25, 26, 27 |
| Target layer | 35 (final block, default) |
prompt_batch / dim_batch | 16 / 16 |
max_seq_len | 128 |
skip_first | 16 positions |
| dtype | bf16 |
| Hook point | Decoder block output (forward hook, tuple[0]) |
Compute: CSCS Clariden GH200, 4 nodes × 4 GPUs, device_map=auto per node, torch 2.13.0+cu130.
Full-fit wall time ~1h14m per shard (parallel across nodes).
Known-answer read (sanity check)
Run after pilot fit (N=100). Prompt:
Fact: The currency used in the country shaped like a boot is
At the final token ( is), model logits (logit lens): the, called, Euro, euro, EUR, …
jlens readout at layer 27: top tokens include 欧元 (Euro), currency, euros, Euros — consistent with the expected answer (Italy → Euro).
At mid layers, readout picks up Italy/geography tokens (e.g. layer 18–26: 意大利, Italy, Italian), which matches the “boot-shaped country” cue before the currency slot.
Full per-layer table: Qwen3-8B/known_answer_read.json.
Fingerprint (layer curves)
Also run on pilot lens (N=100), 32 prompts, corpus_skip=100000.
| Layer | top-1 agreement | top-5 agreement | kurtosis |
|---|
| 9 | 1.4% | 5.0% | 1.26 |
| 18 | 1.1% | 3.6% | 1.01 |
| 21 | 1.1% | 3.6% | 0.94 |
| 22 | 2.4% | 6.9% | 0.97 |
| 23 | 5.7% | 12.8% | 1.20 |
| 24 | 7.5% | 17.0% | 1.27 |
| 25 | 9.6% | 21.0% | 1.33 |
| 26 | 10.9% | 23.6% | 1.43 |
| 27 | 14.4% | 28.0% | 1.61 |
Suggested workspace band: layers 9–27 (heuristic: kurtosis ≥ 0.5×peak, top1 ≥ 0.5).
Plot: Qwen3-8B/fingerprint.png. Raw curves: Qwen3-8B/fingerprint.json.
Files in this repo
| Path | Description |
|---|
Qwen3-8B/lens.pt | Merged Jacobian lens (N=1000) |
Qwen3-8B/lens.meta.json | Fit conventions sidecar (required for correct apply) |
Qwen3-8B/known_answer_read.json | Italy/Euro readout (pilot lens, all source layers) |
Qwen3-8B/fingerprint.json | Layer diagnostic curves (pilot lens) |
Qwen3-8B/fingerprint.png | Fingerprint plot |
Qwen3-8B/results_summary.md | This file |
Load
1from jlens.lens import JacobianLens
2
3lens = JacobianLens.from_pretrained(
4 "senku21x/cot-oracle-jlens",
5 filename="Qwen3-8B/lens.pt",
6)
CLI readout example:
1python -m jlens.read_cli \
2 --lens senku21x/cot-oracle-jlens --lens-filename Qwen3-8B/lens.pt \
3 --model Qwen/Qwen3-8B \
4 --prompt "Fact: The currency used in the country shaped like a boot is" \
5 --positions=-2,-1 --layers 9,18,21-27 --logit-lens
Notes
- Read/fingerprint artifacts in this repo were generated from the pilot (N=100) lens during the pipeline; the uploaded
lens.pt is the full N=1000 fit. Re-run read/fingerprint locally against the full lens if you want diagnostics matched to the final artifact.
- Merge:
JacobianLens.merge() over 4 disjoint corpus shards (jlens/merge_cli.py).
