A 127 M parameter Perceiver hypernetwork trained on Qwen/Qwen3.5-2B. Reads a document once, outputs LoRA deltas, and lets the base LLM answer questions without the document ever appearing in the context window.

Based on Doc-to-LoRA (Charakorn et al., 2026). Uses KL context distillation (Cartridges) and token-init.

Results

Metric	Value
Base model	`Qwen/Qwen3.5-2B`
Perceiver params	127 M
LoRA rank / alpha	8 / 8.0
Target module	`down_proj`
Training steps	1,400
Final CE loss	1.3218
Exact-match accuracy (NIAH)	0.0%
Training ctx length	32–256 tokens

Files

File	Description
`hypernet.pt`	Perceiver weights + full config to rebuild the class
`inference_example.py`	Self-contained script (download and run)
`training_config.json`	Training hyperparameters
`curves.png`	Loss and accuracy curves

Quick start

pip install transformers>=5.2.0 huggingface_hub torch

python

from huggingface_hub import hf_hub_download
import torch

ckpt = torch.load(hf_hub_download("farpluto/doc-to-lora-niah-qwen3.5-2B", "hypernet.pt"),
                   map_location="cuda", weights_only=False)
# See inference_example.py for the complete working script.

Qwen3 note

Chain-of-thought thinking is suppressed via /no_think appended to every query. Residual <think> tokens are stripped from generated output. Both techniques are harmless no-ops on non-Qwen3 models.