bsf_071226 — Load-Balanced Block-Sparse Featurizer, Qwen3-8B layer 24
BSF with dead-zone usage-bias selection (G=2048 blocks x b=64 dims, 131k code dims =
32x expansion at d=4096; per-token top-k=16 blocks by ||z_g|| + tau_g). The per-block
selection bias tau is trained by a dead-zone controller (boost blocks firing < 1e-4,
handicap > 50%, decay between; step/clamp calibrated in k-th-place-norm units) and is
STORED IN THE CHECKPOINT (selection_bias buffer) — the same biased ranking runs at
inference, exactly 16 blocks/token. Signed codes, pure MSE, denoising snr=0.1,
~100M tokens (FineFineWeb).
Results
- reconstruction FVE 0.848 (train-final; rank-1 SAE d65k k80 = 0.803)
- population: 1,927/2,048 blocks deployed-alive; 5 generalists >50% / 48 mid-band
(5-50%) / ~270 specialists / ~1,700 rare tail held at ~1e-4; top block 9.3% of energy
- autointerp (delphi, Opus judge; n=1,436 = whole population): detection 0.809 /
fuzzing 0.824 per-block mean; 841 blocks >= 0.8 detection
- usage-weighted (per-token) view: detection 0.665 / fuzzing 0.601 — per-block means
favor rare crisp blocks; both views reported for honesty
Loading
Requires the nla repo's SAE/bsf_sae.py (the selection_bias buffer is new):
BlockSparseSAE.from_pretrained("ae.pt") — group_sizes/k/act_scale/tau load from the
state dict. Trained 2026-07-11, run bsf_stream_L24_G2048_b64_k16_biasdz
(wandb nla-sae k1abto0r).
