You can deploy this model and start earning money today!

alunxu/
qwen-2.5-7b-tuned-lens-fwedu-262M

Name: qwen-2.5-7b-tuned-lens-fwedu-262M
Availability: InStock

tuned-lens

mechanistic-interpretability

qwen

mit

Views

No views yet

Qwen-2.5-7B Tuned Lens (FineWeb-Edu 262M tokens)

Linear tuned lens for Qwen/Qwen2.5-7B, trained from scratch via the tuned-lens package (Muon-optimizer fork) on 262M tokens of HuggingFaceFW/fineweb-edu (1000 outer steps, 262144 tokens/step, bf16 precision, H100).

Companion to alunxu/qwen-2.5-7b-tuned-lens-105M (wikitext-103 105M). The larger FineWeb-Edu training corpus gives lower per-layer reconstruction loss across the early/mid layers, which matters when the lens is read off non-encyclopedic prompts (instruction-following, MASK, ETHICS-style framings).

Used in the Computational Topography of Truthfulness project (EMNLP 2026 working paper) to extract per-layer P(answer) trajectories without per-task lens specialisation.

Usage

python

from huggingface_hub import snapshot_download
import json
import torch
from tuned_lens import TunedLens
from tuned_lens.nn.lenses import TunedLensConfig
from tuned_lens.nn.unembed import Unembed
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B",
                                              torch_dtype=torch.bfloat16,
                                              device_map="cuda")
lens_dir = snapshot_download("alunxu/qwen-2.5-7b-tuned-lens-fwedu-262M")
with open(f"{lens_dir}/config.json") as f:
    cfg = TunedLensConfig.from_dict(json.load(f))
lens = TunedLens(Unembed(model), cfg)
state = torch.load(f"{lens_dir}/params.pt", map_location="cuda")
lens.layer_translators.load_state_dict(state)
lens = lens.to("cuda").eval()

Patches required

The upstream tuned-lens 0.2.0 does not recognise the Qwen2 architecture in model_surgery.py. Apply the patch from alunxu/relocation-hypothesis/experiments_alun/patch_tuned_lens.py before loading. bf16 inference required — fp16 final-layer projection over the 152k vocab overflows on ~7% of prompts.

Training corpus

Dataset: HuggingFaceFW/fineweb-edu (educational web subset of FineWeb)
Tokens consumed: 1000 outer steps x 262,144 tokens/step ≈ 262M tokens
Optimizer: Muon (https://github.com/AlignmentResearch/tuned-lens PR #141)
Precision: bf16
Hardware: 1 x NVIDIA H100 80GB HBM3