| # bsf_071226 — Load-Balanced Block-Sparse Featurizer, Qwen3-8B layer 24 |
| |
| BSF with dead-zone usage-bias selection (G=2048 blocks x b=64 dims, 131k code dims = |
| 32x expansion at d=4096; per-token top-k=16 blocks by ||z_g|| + tau_g). The per-block |
| selection bias tau is trained by a dead-zone controller (boost blocks firing < 1e-4, |
| handicap > 50%, decay between; step/clamp calibrated in k-th-place-norm units) and is |
| STORED IN THE CHECKPOINT (`selection_bias` buffer) — the same biased ranking runs at |
| inference, exactly 16 blocks/token. Signed codes, pure MSE, denoising snr=0.1, |
| ~100M tokens (FineFineWeb). |
|
|
| ## Results |
| - reconstruction FVE 0.848 (train-final; rank-1 SAE d65k k80 = 0.803) |
| - population: 1,927/2,048 blocks deployed-alive; 5 generalists >50% / 48 mid-band |
| (5-50%) / ~270 specialists / ~1,700 rare tail held at ~1e-4; top block 9.3% of energy |
| - autointerp (delphi, Opus judge; n=1,436 = whole population): detection 0.809 / |
| fuzzing 0.824 per-block mean; 841 blocks >= 0.8 detection |
| - usage-weighted (per-token) view: detection 0.665 / fuzzing 0.601 — per-block means |
| favor rare crisp blocks; both views reported for honesty |
|
|
| ## Loading |
| Requires the nla repo's `SAE/bsf_sae.py` (the `selection_bias` buffer is new): |
| `BlockSparseSAE.from_pretrained("ae.pt")` — group_sizes/k/act_scale/tau load from the |
| state dict. Trained 2026-07-11, run `bsf_stream_L24_G2048_b64_k16_biasdz` |
| (wandb nla-sae k1abto0r). |
|
|
