SAEs for deep-ignorance-unfiltered (expansion factor 8, all layers)
Sparse autoencoders trained on all 32 layers of EleutherAI/deep-ignorance-unfiltered using the WMDP-Bio-Remove-Dataset.
Trained with EleutherAI/sparsify.
Training hyperparameters
| Parameter | Value |
|---|---|
| expansion_factor | 8 |
| k | 32 |
| layers | all 32 (0-31) |
| batch_size | 4 |
| grad_acc_steps | 8 |
| micro_acc_steps | 2 |
| ctx_len | 2048 |
| optimizer | signum |
| loss_fn | fvu |
| activation | topk |
| distributed | distribute_modules across 4x GH200 120GB |
Dataset
Unlearning/WMDP-Bio-Remove-Dataset (24,453 examples)
Usage
from sparsify import Sae
sae = Sae.load_from_hub("EleutherAI/sae-deep-ignorance-unfiltered-ex8", hookpoint="layers.10")
# Or load all layers
saes = Sae.load_many("EleutherAI/sae-deep-ignorance-unfiltered-ex8")
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for EleutherAI/sae-deep-ignorance-unfiltered-ex8
Unable to build the model tree, the base model loops to the model itself. Learn more.