Massive Activations Confound Sparse Autoencoders
Collection
SAEs and results from the paper. • 6 items • Updated
Massive SAEs: other layers
SAEs from Massive Activations Confound Sparse Autoencoders. Main SAEs: nathandelisle/massive-saes.
TopK SAEs (k = 64) at two more layers per model, one before the massive activations appear and one after they fade. Eight models, standard and bypass, five seeds.
Files: <model>/topk/block<L>/<standard|bypass>/seed<k>/{sae.pt, config.json}.
The training mean and PC1 for these layers were not saved, so the bypass SAEs here cannot be used for reconstruction as released.
Load a checkpoint with torch.load(path, map_location="cpu", weights_only=True).
License: CC BY 4.0. The base models' own licenses also apply.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support