Views
No views yet
syvb/nanonla-qwen3-8b-L24-av
(Qwen3-8B, injection layer 24, d_model 4096). Both arms trained with the
standalone (miles-free) trainer at 3000 steps, eff. batch 32, lr 2e-5 cosine→2e-6,
warmup 50, injection_scale = sqrt(d_model), single seed.A_i·v_norm + b_i over the normalized injected activation (identity-init,
fp32, +269M params). Affines and backbone trained.doc_id
does not appear in the training split), gold activation injected, paired (same
rows, both arms), token-NLL on the response.| arm | val NLL/token | perplexity |
|---|---|---|
| 16-slot affine | 1.4141 | 4.113 |
| 1-slot control | 1.4305 | 4.181 |
| Δ (control − experiment) | +0.0165 | — |
‖W_i − W̄‖ is nearly as large as each slot's deviation from identity, i.e. the
16 affines are mostly unique, not a shared shift — partial support for the
"independent views" idea, but not proof it's what helps.nla_affine.safetensors
(weight [16,4096,4096], bias [16,4096]); apply at injection time — see
launch/eval_av_val_loss.py --multi-input-slots 16 --affine-path ....