Massive Activations Confound Sparse Autoencoders
Collection
SAEs and results from the paper. • 6 items • Updated
Massive SAEs: controls
SAEs from Massive Activations Confound Sparse Autoencoders. Main SAEs: nathandelisle/massive-saes.
Checkpoints for the paper's control experiments:
beta-gpt2/: GPT-2 TopK SAEs with an added L1 penalty of varying strength.gated-controls/: Gated SAE calibration controls.matched-budget-63+1/: SAEs with 63 learned features plus one PC1 slot.rebuttal-layers/: Pythia-1.4B TopK SAEs at blocks 2, 3, 12 and 23, with the PC1 files for each layer.
Each checkpoint folder has sae.pt and config.json. Load with torch.load(path, map_location="cpu", weights_only=True).
License: CC BY 4.0. The base models' own licenses also apply.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support