Conditional residual denoiser trained to correct hidden-state
corruptions after layer 16 of Qwen/Qwen2.5-1.5B.
The model was trained without access to the fantasy steering vector.
Training corruptions consist of random unit directions and activation mixup.
The checkpoint is intended for activation-steering experiments with
Qwen2.5-1.5B at layer 16. It is not a standalone language model.
At alpha=60, corpus Dist-2 increased from 0.5983 for naive steering
to 0.6406, while concept similarity remained approximately unchanged:
0.4151 versus 0.4155.
The model was evaluated on one concept vector and one training seed.
Dist-2 is a diversity proxy and does not fully measure fluency.
1import torch
2
3from conditional_resid_denoiser import ConditionalResidualDenoiser
4from huggingface_hub import hf_hub_download
5
6checkpoint_path = hf_hub_download(
7 repo_id="USERNAME/qwen2.5-1.5b-conditional-steering-denoiser",
8 filename="checkpoint.pt",
9)
10
11model = ConditionalResidualDenoiser(
12 d_model=1536,
13 expansion_factor=0.5,
14 cond_dim=64,
15)
16
17checkpoint = torch.load(
18 checkpoint_path,
19 map_location="cpu",
20 weights_only=True,
21)
22
23model.load_state_dict(checkpoint["model_state_dict"])
24model.eval()
