---
license: mit
tags:
  - sparse-autoencoder
  - dictionary-learning
  - interpretability
---

# ConSDL checkpoints

Model checkpoints for the vision and language experiments in *ConSDL:
Consistent Sparse Dictionary Learning*. This collection includes ConSDL
and six SAE baselines across four backbones, three dictionary sizes, and
three sparsity levels, with three training seeds per configuration.

## Layout

```
<modality>/<backbone>_layer<L>/<arch>/x<E>/<level>/seed<S>/
    config.json
    model.safetensors
    stats.<dataset>.split_<split>.seed_no/     per-latent firing statistics
<modality>/<backbone>_layer<L>/<arch>/x<E>/<level>/
    dict_metrics_step_final.json               cross-seed dictionary metrics (DCS, #dead, #dup)
    code_metrics_step_final.json               cross-seed code metrics (MCC, R²)
```

| field | values |
|---|---|
| modality / backbone | `vision/clip-vit-b-16_layer10`, `vision/dinov2-b_layer10`, `lm/pythia-160m_layer8`, `lm/gpt2-small_layer8` |
| arch | `consdl`, `relu`, `topk`, `batchtopk`, `matryoshka`, `mp`, `isae` |
| x\<E\> | dictionary size = E × 768: `x4`, `x8`, `x16` |
| level | target L0: `l0_32`, `l0_64`, `l0_128` |
| seed | `seed0`, `seed1`, `seed2` |

Vision SAEs are trained on ImageNet patch tokens (one epoch), language SAEs on
500M FineWeb tokens (residual stream after the given block). The exact
hyper-parameters of every checkpoint are in its `config.json`; the
per-cell values are also listed in the paper's appendix.

## Usage

```python
from consdl.io.hub import load_pretrained

sae = load_pretrained("vision/clip-vit-b-16_layer10/consdl/x16/l0_64/seed0")
codes = sae.encode(activations)          # (..., 768) -> (..., 12288)
recon = sae.decode(codes)
```

To recompute the cross-seed metrics of a cell, download it and run the
metric CLIs on it:

```python
from consdl.io.hub import download
cell = download("vision/clip-vit-b-16_layer10/consdl/x16/l0_64")
```
```bash
python -m consdl.cli.dict_metrics --tag <cell>
```