nanonla-qwen3-8b-L24-av-1slot-affine-singletag
1-slot control arm (status-quo single injection, matched compute).
Continued AV-SFT warm-started from
syvb/nanonla-qwen3-8b-L24-av
(Qwen3-8B, injection layer 24, d_model 4096).
Multi-input affine experiment
Hypothesis: repeating the injection marker N times, each slot getting its own
learned affine A_i·v_norm + b_i over the (normalized) injected activation, gives
the backbone N independently-readable "views" of one activation and improves
verbalization vs. the status-quo single injection.
- Experiment arm: N=16 markers, 16 full d×d+bias affines, identity-init,
full continued AV-SFT (affines and backbone trained).
- Control arm: N=1, no affine — same warm-start, same data, same step budget
(matched compute).
Both arms: 1-slot WITH affine, 950 steps, eff batch 256, backbone 5e-5 + affine 1e-3 (20x), single-tag, injection_scale = sqrt(d_model). Trained with the
standalone (miles-free) trainer tools/train_av_standalone.py.
Held-out val NLL (lower is better)
Evaluated on 4001 doc-disjoint held-out rows (val rows whose doc_id
does not appear in the training split — a true held-out set, not just a row
split), gold activation injected. Both arms scored on the same rows (paired).
| arm | val NLL/token | perplexity |
|---|
| 16-slot affine (experiment) | 1.3234 | 3.756 |
| 1-slot (control) | 1.3638 | 3.911 |
| Δ (control − experiment) | +0.0405 | — |
Paired row-level bootstrap (10k resamples): Δ = +0.0405 nats/token,
95% CI [+0.0394, +0.0416].
✅ The 16-slot affine significantly improves val NLL (Δ=+0.0405, 95% CI [+0.0394, +0.0416] excludes 0).
Caveats
- This is a system-level comparison (16 markers + per-slot affine vs. 1 marker).
It does not isolate the affine from the effect of simply repeating the
injection 16× — a "16 markers, no affine" arm would be needed for that.
- Continued warm-start, not a converged run (1-slot WITH affine, 950 steps, eff batch 256, backbone 5e-5 + affine 1e-3 (20x), single-tag).
For the 16-slot model the per-slot affines are in nla_affine.safetensors
(weight [16,4096,4096], bias [16,4096]); apply them at injection time — see
launch/eval_av_val_loss.py --multi-input-slots 16 --affine-path ....
