EasyNLA whitening pilot (2026-07-18)
Half-scale pilot: Qwen3-8B L24 NLA trained on ZCA-whitened activations.
Code: whitening branch of syvb/EasyNLA. Wandb: octahedral-systems/easynla-whitening-pilot.
Data: syvb/easynla-whitening-pilot-data. Raw-model reference: syvb/nanonla-qwen3-8b-L24-{av,ar,rl-lora}.
av_sft_lora/, ar_sft_lora/ — quarter-epoch SFT LoRAs (all saved iters), whitened activationsmerged_av/, merged_ar/ — bf16 merges of iter_0001400 (vLLM RL inputs)rl_lora/ — GRPO RL checkpoints (60 steps, 2xH100) [phase 2]whitening_stats.npz — mu, W=Sigma^-1/2, W_inv, spectra (KEEP: W_inv maps preds back to raw space)eval/ — dump_nla_eval artifacts (free cell + final 2x2 inputs)launchers/, logs/ — exact run scripts (incl. wandb guard) and stdout logs
Pilot results (2026-07-18)
2x2, pinned convention (calibrate to native train-mean gold norm -> affine map -> normalize to sqrt(d) -> FVE
vs raw-variance predict-mean baseline), n~1000 held-out av_sft_val rows, greedy:
| model | raw-space FVE | whitened-space FVE |
|---|
| raw NLA (nanonla ckpts, mature training) | 0.539 | 0.172 |
| whitened SFT (quarter-epoch warm start) | -0.734 | -0.031 |
| whitened RL (50 GRPO steps, 2xH200) | -0.371 | 0.195 |
Headline: the whitened-trained pilot beats the mature raw model on the equal-weighted (whitened)
metric at a small fraction of its compute, while giving up essentially all raw-space FVE -- raw FVE
is dominated by the few huge-variance directions the whitened objective down-weights to 1/4096 each
(and whose errors W^-1 re-amplifies on the map back). In-run whitened FVE was still climbing at
step 50 (22.8% on the RL split eval). Wandb (private): octahedral-systems/easynla-whitening-pilot,
runs cyx306dh (AV SFT), abk8hqvz (AR SFT), 3jnzwt2b (RL, definitive). eval/heldout_examples.md has
30 non-cherry-picked side-by-side explanation examples.
