This repository contains the selected rank-64 rsLoRA adapter from a scaled activation-only future-lens experiment. The student receives only one residual-stream activation, injected at the Karvonen token; it cannot attend to the source text. A frozen Qwen3.6-27B teacher receives the actual FineWeb prefix and scores the student's own sampled continuation.

This is a research checkpoint, not a normal chat model. Loading it requires the custom activation-injection hook in skip-lens.

Training

Base model: Qwen/Qwen3.6-27B
Activations: raw output of decoder block 62 (the penultimate block)
Data: 100,000 activation/context pairs from 20,000 FineWeb documents; five uniformly sampled positions per document; 5% document-disjoint validation
Labels/control data: base-model continuations sampled at temperature 1.0, top-p 1.0, top-k 0, with eight tokens per span
Shared warm start: 358,400 SFT tokens
OPD stage: 1,024,000 tokens, 500 optimizer updates, effective batch 256 (physical 8 × accumulation 32)
Optimizer: AdamW; learning rate 3e-5 cosine-decayed to 3e-6
Adapter: rank 64, alpha 16, rsLoRA on attention modules

For OPD, the student samples exactly eight tokens at temperature 1.0. EOS remains in the sampled distribution but is disabled as a stopping condition, so the OPD and SFT arms are exactly token-matched. The teacher scores every student-sampled token on the real prefix plus the sampled prefix. Training uses the sampled per-token reverse-KL policy-gradient estimator with stored behavior log-probabilities and importance ratios, following Thinking Machines' on-policy distillation recipe.

Selection and evaluation

The uploaded checkpoint is update 500, selected for minimum teacher-forced L62 validation NLL over checkpoints saved every 50 updates. Its comparison control is SFT at the same update and exact token budget.

Fed activation	Arm	Reference NLL ↓	Exact reverse KL ↓	Teacher top-1 ↑	Coherence /5 ↑	Hallucination ↓
L62	OPD	3.012	0.704	66.2%	3.99	24.0%
L62	Matched SFT	2.945	0.737	66.6%	4.11	31.2%
L42	OPD	3.451	1.126	57.9%	3.78	27.7%
L42	Matched SFT	3.386	1.242	58.0%	3.80	33.6%

Paired L62 differences (OPD − matched SFT):

Reference NLL: +0.067 nats/token (95% CI +0.048 to +0.087)
Exact reverse KL: -0.032 (95% CI -0.048 to -0.016)
Sonnet-5 coherence: -0.121/5 (95% CI -0.209 to -0.035)
Sonnet-5 hallucination rate: -7.2 points (95% CI -11.1 to -3.3)

The evals/ directory contains the complete automatic analysis, raw selected-checkpoint L62/L42 evaluations, and Sonnet-5 judgments.

Limitations

One training seed.
Checkpoint selection and the reported validation trajectory use the same held-out split; these are descriptive post-selection estimates rather than an unbiased final test.
Middle-layer L42 evaluation feeds raw block-42 activations into a lens trained at L62; it is a transfer test, not an independently trained L42 lens.
Coherence and hallucination are model-judged.
This adapter requires custom activation injection and is not meaningful without an input activation from the matching base model.

Files

adapter_model.safetensors, adapter_config.json: PEFT adapter
nla_meta.yaml: injection token, neighbor-token, layer, and prompt metadata
opd_meta.json: training budget and stopping metadata
selection.json: selected update, checkpoint paths, and selection criterion
evals/: raw and summarized evaluation artifacts

Repository: ceselder/skip-lens-qwen36-27b-fineweb-opd-scaled.