File size: 1,992 Bytes
005a77d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | ---
license: mit
tags:
- sparse-autoencoder
- dictionary-learning
- interpretability
---
# ConSDL checkpoints
Model checkpoints for the vision and language experiments in *ConSDL:
Consistent Sparse Dictionary Learning*. This collection includes ConSDL
and six SAE baselines across four backbones, three dictionary sizes, and
three sparsity levels, with three training seeds per configuration.
## Layout
```
<modality>/<backbone>_layer<L>/<arch>/x<E>/<level>/seed<S>/
config.json
model.safetensors
stats.<dataset>.split_<split>.seed_no/ per-latent firing statistics
<modality>/<backbone>_layer<L>/<arch>/x<E>/<level>/
dict_metrics_step_final.json cross-seed dictionary metrics (DCS, #dead, #dup)
code_metrics_step_final.json cross-seed code metrics (MCC, R²)
```
| field | values |
|---|---|
| modality / backbone | `vision/clip-vit-b-16_layer10`, `vision/dinov2-b_layer10`, `lm/pythia-160m_layer8`, `lm/gpt2-small_layer8` |
| arch | `consdl`, `relu`, `topk`, `batchtopk`, `matryoshka`, `mp`, `isae` |
| x\<E\> | dictionary size = E × 768: `x4`, `x8`, `x16` |
| level | target L0: `l0_32`, `l0_64`, `l0_128` |
| seed | `seed0`, `seed1`, `seed2` |
Vision SAEs are trained on ImageNet patch tokens (one epoch), language SAEs on
500M FineWeb tokens (residual stream after the given block). The exact
hyper-parameters of every checkpoint are in its `config.json`; the
per-cell values are also listed in the paper's appendix.
## Usage
```python
from consdl.io.hub import load_pretrained
sae = load_pretrained("vision/clip-vit-b-16_layer10/consdl/x16/l0_64/seed0")
codes = sae.encode(activations) # (..., 768) -> (..., 12288)
recon = sae.decode(codes)
```
To recompute the cross-seed metrics of a cell, download it and run the
metric CLIs on it:
```python
from consdl.io.hub import download
cell = download("vision/clip-vit-b-16_layer10/consdl/x16/l0_64")
```
```bash
python -m consdl.cli.dict_metrics --tag <cell>
```
|