chanind-goodfire/synthsaebench-full-width-sweep

SynthSAEBench full-width SAE sweep

209 full-budget sparse autoencoders trained on the released 16k SynthSAEBench world. Each SAE has 16,384 latents, 768 input dimensions, and was trained for 200M samples. The collection includes 162 recipe-screening runs, 40 L0-grid runs, and seven MinFire-period follow-ups. Smoke runs are excluded.

Inference uses the public SAELens library; no custom remote code is required. Each model includes its training settings, evaluation metrics, and recorded training history.

Load one model

Install the exact SAELens source revision used in the experiment:

pip install 'sae-lens @ git+https://github.com/decoderesearch/SAELens.git@b5711e34d072846dc112881f4c8d209407a32b56'

from pathlib import Path
from huggingface_hub import snapshot_download
from sae_lens import SAE

repo = 'chanind-goodfire/synthsaebench-full-width-sweep'
run_id = 'r3/jr_ste_c01_resample_decay_k35_s0'
subdir = f'runs/{run_id}/final'
snapshot = snapshot_download(repo, allow_patterns=[f'{subdir}/*'])
sae = SAE.load_from_disk(Path(snapshot) / subdir, device='cpu').eval()
# x is a batch of unnormalized 768-dimensional synthetic-world observations.
# codes = sae.encode(x)
# reconstruction = sae.decode(codes)

This downloads about 101 MB instead of the full 21 GB collection. Pin revision to a Hub commit for a stable snapshot. The three training architectures all export ordinary JumpReLU inference SAEs with native thresholds. Input scaling is already folded into the weights. W_dec has shape [16384, 768]; decoder rows are latents. These models are for the released synthetic world, not language-model activations.

Results and selection

catalog.json lists every run's exact recipe, target/measured L0, metrics, file sizes, and SHA-256 checksums for the published files. headline: true selects the 42 baseline/improved models forming the seven-point L0 curves. results.csv is the flat, full-precision metric table. All settings use seed 0; small numerical differences are not resolved by repeated seeds.

Architecture	L0 35 baseline MCC	Improved MCC	Improved recipe
BatchTopK	0.535	0.804	MinFire gate + final 20% LR decay
Matryoshka BatchTopK	0.466	0.841	Dead-latent resampling, 100-step window + decay
JumpReLU	0.470	0.860	Input STE, L0 coefficient 0.1, resampling + decay

BatchTopK's headline MinFire periods are 5, 3, 2 at target L0 15, 20, 25 and 1 at targets 30–45. Those selected low-L0 checkpoints are in gridp/; the default period runs and other tried periods are also released. Follow the catalog's headline flag instead of inferring the selection from recipe names alone.

Protocol

• World: decoderesearch/synth-sae-bench-16k-v1 revision b2efd8b919ae46d6d487c73d46db5ee52813621d, with scale_children_by_parent=False.
• Training: batch 1,024; Adam (0.9, 0.999); LR 3e-4; gradient norm clip 1; bf16 autocast except explicitly named fp32 ablations; 200M samples per run.
• SAELens 6.51.0 revision b5711e34d072846dc112881f4c8d209407a32b56, with the upstream coefficient autotuner and the gate/resampling settings recorded in each run's configuration.
• Evaluation: unmodified SAELens synthetic scorer, 1M fresh samples, shared seed
  49100. MCC is one-to-one Hungarian matching over all 16,384 decoder rows; F1 is the official macro average. Dead latents are retained. Scores are not directly comparable with the upstream 4,096-latent SAE board.

Layout and provenance

Each runs/<stage>/<run>/ contains:

• final/cfg.json and final/sae_weights.safetensors: inference export.
• run.json and score.json: full settings and evaluation metrics.
• history.json and progress.json: recorded training history and final progress.

Stages r1/r2/r3 contain 162 screening runs, grid contains 40 additional L0-grid runs, and gridp contains seven MinFire-period follow-ups. There are no optimizer checkpoints or smoke runs. All 209 models completed 200M training samples. provenance.json identifies the public benchmark, SAELens, and coefficient autotuner repositories and their exact revisions.