bsf_070926 — Block-Sparse Featurizer, Qwen3-8B layer 24
Vanilla BSF (Fel et al. 2026, arXiv:2606.25234): G=4096 blocks x b=64 dims
(262,144 code dims, 64x expansion at d=4096), per-token top-k=16 blocks by code
L2 norm, signed codes, pure MSE, denoising snr=0.1, ~100M tokens (FineFineWeb).
- FVE 0.865 (held-out, rl_shuf 20k acts)
- autointerp (delphi, Opus judge): detection 0.717 / fuzz 0.708, ~22% of scored blocks >= 0.8
- 362 live blocks; firing rate of live blocks p10/p50/p90 = 0.16% / 1.16% / 6.7%
(~8 always-on generalists carry ~half the energy)
Load with SAE/bsf_sae.py::BlockSparseSAE.from_pretrained("ae.pt") from the nla repo.
Trained 2026-07-09, run bsf_stream_L24_G4096_b64_k16 (wandb nla-sae).
