Hugging Face's logo

davidquarel
/
topk.tblock.gpt2.k32.x32

Safetensors
Model card Files
xet
Community
YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

TopK SAE trained bewteen each layer for GPT-2

  • Trained on dataset "HuggingFaceFW/fineweb-edu"
  • K=32
  • Expansion factor = 32 (768 * 32 = 24576)

gpt2_defaults = {
    "data_dir": "data/fineweb_edu_10b",
    "eval_interval": 250,
    "eval_steps": 100,
    "batch_size": 1,
    "gradient_accumulation_steps": 32 // 1,
    "learning_rate": 5e-4,
    "warmup_steps": 750,
    "max_steps": 7500,
    "decay_lr": True,
    "min_lr": 1e-4,
}

    SAEConfig("topk.tblock.gpt2",
        gpt_config = gpt_options['gpt2'],
        n_features=tuple(768 * n for n in (32,)*13),
        sae_variant=SAEVariant.TOPK,
        top_k = (32,) * 13,
        sae_keys=gen_sae_keys(n_features=13, loc="standard"),
    ),
Downloads last month

-

Downloads are not tracked for this model. How to track
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including davidquarel/topk.tblock.gpt2.k32.x32