| # bsf_070926 — Block-Sparse Featurizer, Qwen3-8B layer 24 | |
| Vanilla BSF (Fel et al. 2026, arXiv:2606.25234): G=4096 blocks x b=64 dims | |
| (262,144 code dims, 64x expansion at d=4096), per-token top-k=16 blocks by code | |
| L2 norm, signed codes, pure MSE, denoising snr=0.1, ~100M tokens (FineFineWeb). | |
| - FVE 0.865 (held-out, rl_shuf 20k acts) | |
| - autointerp (delphi, Opus judge): detection 0.717 / fuzz 0.708, ~22% of scored blocks >= 0.8 | |
| - 362 live blocks; firing rate of live blocks p10/p50/p90 = 0.16% / 1.16% / 6.7% | |
| (~8 always-on generalists carry ~half the energy) | |
| Load with `SAE/bsf_sae.py::BlockSparseSAE.from_pretrained("ae.pt")` from the nla repo. | |
| Trained 2026-07-09, run `bsf_stream_L24_G4096_b64_k16` (wandb nla-sae). | |