| --- |
| license: mit |
| tags: |
| - sparse-autoencoder |
| - dictionary-learning |
| - interpretability |
| --- |
| |
| # ConSDL checkpoints |
|
|
| Model checkpoints for the vision and language experiments in *ConSDL: |
| Consistent Sparse Dictionary Learning*. This collection includes ConSDL |
| and six SAE baselines across four backbones, three dictionary sizes, and |
| three sparsity levels, with three training seeds per configuration. |
|
|
| ## Layout |
|
|
| ``` |
| <modality>/<backbone>_layer<L>/<arch>/x<E>/<level>/seed<S>/ |
| config.json |
| model.safetensors |
| stats.<dataset>.split_<split>.seed_no/ per-latent firing statistics |
| <modality>/<backbone>_layer<L>/<arch>/x<E>/<level>/ |
| dict_metrics_step_final.json cross-seed dictionary metrics (DCS, #dead, #dup) |
| code_metrics_step_final.json cross-seed code metrics (MCC, R²) |
| ``` |
|
|
| | field | values | |
| |---|---| |
| | modality / backbone | `vision/clip-vit-b-16_layer10`, `vision/dinov2-b_layer10`, `lm/pythia-160m_layer8`, `lm/gpt2-small_layer8` | |
| | arch | `consdl`, `relu`, `topk`, `batchtopk`, `matryoshka`, `mp`, `isae` | |
| | x\<E\> | dictionary size = E × 768: `x4`, `x8`, `x16` | |
| | level | target L0: `l0_32`, `l0_64`, `l0_128` | |
| | seed | `seed0`, `seed1`, `seed2` | |
|
|
| Vision SAEs are trained on ImageNet patch tokens (one epoch), language SAEs on |
| 500M FineWeb tokens (residual stream after the given block). The exact |
| hyper-parameters of every checkpoint are in its `config.json`; the |
| per-cell values are also listed in the paper's appendix. |
|
|
| ## Usage |
|
|
| ```python |
| from consdl.io.hub import load_pretrained |
| |
| sae = load_pretrained("vision/clip-vit-b-16_layer10/consdl/x16/l0_64/seed0") |
| codes = sae.encode(activations) # (..., 768) -> (..., 12288) |
| recon = sae.decode(codes) |
| ``` |
|
|
| To recompute the cross-seed metrics of a cell, download it and run the |
| metric CLIs on it: |
|
|
| ```python |
| from consdl.io.hub import download |
| cell = download("vision/clip-vit-b-16_layer10/consdl/x16/l0_64") |
| ``` |
| ```bash |
| python -m consdl.cli.dict_metrics --tag <cell> |
| ``` |
|
|
