Hugging Face's logo

hematteo
/
parameter-trajectory-crosscoders

crosscoder
sparse-autoencoder
mech-interp
parameter-trajectory
pythia
olmo
Model card Files
xet
 Community
Matteo He commited on
Commit
a95a860
·
verified ·
1 Parent(s): 097677b

Add files using upload-large-folder tool

Browse files
Files changed (10) hide show
  1. .cache/._huggingface +0 -0
  2. README.md +120 -93
  3. index.json +1726 -0
  4. pythia-160m/W_E/cross-snapshot-32/d8192/seed0.metrics_recomputed.json +23 -0
  5. pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.config.json +2 -1
  6. pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.config.json +4 -2
  7. pythia-160m/W_U/cross-snapshot-16/d8192/seed0.metrics_recomputed.json +24 -0
  8. pythia-160m/W_U/cross-snapshot-32/d16384/seed0.metrics_recomputed.json +23 -0
  9. pythia-160m/W_U/cross-snapshot-32/d24576/seed0.metrics_recomputed.json +24 -0
  10. pythia-160m/W_U/cross-snapshot-32/d24576/seed1.metrics_recomputed.json +24 -0
.cache/._huggingface ADDED
Binary file (4.1 kB). View file
 
README.md CHANGED
@@ -1,5 +1,5 @@
1
  ---
2
- license: apache-2.0
3
  tags:
4
  - crosscoder
5
  - sparse-autoencoder
@@ -11,109 +11,136 @@ tags:
11
 
12
  # Parameter-trajectory crosscoders for vocabulary readout evolution
13
 
14
- Companion checkpoints to the paper *Parameter-trajectory crosscoders for
15
- vocabulary readout evolution* (NeurIPS 2026; arXiv:XXXX.XXXXX).
 
 
 
16
 
17
- We train **snapshot crosscoders** on parameter tensors (rather than
18
- activations) sampled across pretraining checkpoints. In Pythia-160M's output
19
- unembedding $W_U$, this reveals a sparse, reproducible **readout
20
- consolidation event** at training step ~1,000.
21
 
22
- ## Repository layout
23
-
24
- | What you want | Path |
25
- |---|---|
26
- | Headline 5-seed Pythia-160M $W_U$ crosscoder | `pythia-160m/W_U/cross-snapshot-32/d8192/seed{0..4}.safetensors` |
27
- | 32 single-snapshot SAEs (in-time view) | `pythia-160m/W_U/per-snapshot-saes/d8192/step*.safetensors` |
28
- | Higher-capacity Pythia-160M $W_U$ crosscoders | `pythia-160m/W_U/cross-snapshot-32/d{16384,24576}/` |
29
- | Architecture-invariance comparison (vs. JumpReLU baseline) | `pythia-160m/W_U/architecture-comparison/d8192/{batchtopk,gated,gated-retuned}/` |
30
- | Snapshot-density downsample | `pythia-160m/W_U/cross-snapshot-16/d8192/` |
31
- | Final-snapshot-only SAE Pareto | `pythia-160m/W_U/final-snapshot-saes/d{6144..65536}.safetensors` |
32
- | Read/write asymmetry: $W_E$ side | `pythia-160m/W_E/cross-snapshot-32/d{8192,24576}/` |
33
- | Cross-scale (Pythia-1B) | `pythia-1b/W_U/cross-snapshot-32/d{8192,16384,24576}/` |
34
- | Larger-scale (Pythia-6.9B) | `pythia-6.9b/W_U/cross-snapshot-32/d32768/` |
35
- | Cross-family (OLMo-2-7B) | `olmo-2-7b/W_U/cross-snapshot-32/d32768/` |
36
- | §5 attribution-patching artifacts | `attribution/pythia-160m/` |
37
-
38
- Each directory contains `<name>.safetensors` (weights),
39
- `<name>.config.json` (training hparams + quality metrics), and `<name>.md`
40
- (one-page model card). The metadata format is unified across all artifacts.
41
-
42
- ## Repository tree
43
 
44
- ```
45
- .
46
- ├── README.md
47
- │
48
- ├── attribution/pythia-160m/
49
- │ ├── induction-{results.pt, verdict.json}
50
- │ ├── ioi-{results.pt, verdict.json}
51
- │ └── sva-{results.pt, verdict.json}
52
- │
53
- ├── pythia-160m/
54
- │ ├── W_E/cross-snapshot-32/
55
- │ │ ├── d8192/seed{0..4}.{safetensors, config.json, md}
56
- │ │ └── d24576/seed0.{safetensors, config.json, md}
57
- │ │
58
- │ └── W_U/
59
- │ ├── cross-snapshot-32/
60
- │ │ ├── d8192/seed{0..4}.{safetensors, config.json, md}
61
- │ │ ├── d16384/seed0.{safetensors, config.json, md}
62
- │ │ └── d24576/seed{0..2}.{safetensors, config.json, md}
63
- │ │
64
- │ ├── cross-snapshot-16/d8192/seed0.{...}
65
- │ │
66
- │ ├── per-snapshot-saes/d8192/step{0..143000}.{safetensors, config.json, md} # 32 snapshots
67
- │ │
68
- │ ├── architecture-comparison/d8192/
69
- │ │ ├── batchtopk/seed0.{...}
70
- │ │ ├── gated/seed0.{...}
71
- │ │ └── gated-retuned/seed0.{...}
72
- │ │
73
- │ └── final-snapshot-saes/d{6144, 8192, 16384, 32768, 65536}.{safetensors, config.json, md}
74
- │
75
- └── pythia-1b/W_U/cross-snapshot-32/d{8192, 16384, 24576}/seed0.{safetensors, config.json, md}
76
  ```
77
 
78
- ## Quick start
 
 
79
 
80
- ```python
81
- from huggingface_hub import hf_hub_download
82
- from safetensors.torch import load_file
83
- import json
84
 
85
- # Download the headline Pythia-160M crosscoder, seed 0
86
- weights_path = hf_hub_download(
87
- "matteohe/parameter-trajectory-crosscoders",
88
- "pythia-160m/W_U/cross-snapshot-32/d8192/seed0.safetensors",
89
- )
90
- config_path = hf_hub_download(
91
- "matteohe/parameter-trajectory-crosscoders",
92
- "pythia-160m/W_U/cross-snapshot-32/d8192/seed0.config.json",
93
- )
94
- weights = load_file(weights_path) # dict[str, Tensor]
95
- config = json.load(open(config_path))
96
- print(config["public_label"]) # "Pythia-160M W_U crosscoder, ..."
97
- ```
98
 
99
- The crosscoder weight tensors are:
100
- - `W_E`: encoder, shape $(K, d_{\text{model}}, d_{\text{sae}})$
101
- - `W_D`: decoder, shape $(K, d_{\text{sae}}, d_{\text{model}})$
102
- - `b_E`: encoder bias, shape $(K, d_{\text{sae}})$
103
- - `b_D`: decoder bias, shape $(K, d_{\text{model}})$
104
- - `activation_function.log_jumprelu_threshold`: shape $(d_{\text{sae}},)$
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
105
 
106
- For per-snapshot SAEs `K=1`. The companion paper code (open-source, see
107
- `https://github.com/...`) provides `wu_adapter.build_crosscoder` which
108
- reconstructs the full module.
 
 
109
 
110
  ## Citation
111
 
112
  ```bibtex
113
- @inproceedings{{...,
114
- title={{Parameter-trajectory crosscoders for vocabulary readout evolution}},
115
- author={{...}},
116
- booktitle={{NeurIPS 2026}},
117
- year={{2026}},
118
- }}
 
119
  ```
 
 
 
 
 
1
  ---
2
+ license: mit
3
  tags:
4
  - crosscoder
5
  - sparse-autoencoder
 
11
 
12
  # Parameter-trajectory crosscoders for vocabulary readout evolution
13
 
14
+ Trained-dictionary release for **Learning to Read Out: Unembedding Dynamics in
15
+ Language Model Pretraining**. We train **snapshot crosscoders** on parameter
16
+ tensors (rather than activations) sampled across pretraining checkpoints. In
17
+ the output unembedding $W_U$ this reveals how a sparse vocabulary readout
18
+ forms, reorganizes, and becomes load-bearing during pretraining.
19
 
20
+ Code, figure-by-figure reproduction map, and retraining recipes:
21
+ **https://github.com/hematteo/learning-to-read-out** (see `docs/REPRODUCE.md`
22
+ and `docs/DATA.md`; per-run settings of record in `configs/runs/`).
 
23
 
24
+ ## Quick start
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
25
 
26
+ ```bash
27
+ # everything (~180 GB)
28
+ hf download matteohe/parameter-trajectory-crosscoders --local-dir $UM_SSD_ROOT/hf_release/parameter-trajectory-crosscoders
29
+ # one model only
30
+ hf download matteohe/parameter-trajectory-crosscoders --include "pythia-1b/**" --local-dir ...
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
  ```
32
 
33
+ Each artifact is `<name>.safetensors` + `<name>.config.json` (training
34
+ hyperparameters and recomputed quality metrics) + `<name>.md` (card).
35
+ `index.json` is the machine-readable inventory of everything below.
36
 
37
+ ## What you probably want
 
 
 
38
 
39
+ | What | Path |
40
+ |---|---|
41
+ | Headline 5-seed Pythia-160M $W_U$ crosscoder | `pythia-160m/W_U/cross-snapshot-32/d8192/seed{0..4}.safetensors` |
42
+ | High-resolution 160M instrument (atlas) | `pythia-160m/W_U/cross-snapshot-32/d24576/seed0.safetensors` |
43
+ | Cross-scale (Pythia-1B) | `pythia-1b/W_U/cross-snapshot-32/d24576/seed0.safetensors` |
44
+ | Large-scale, selected sparse run | `pythia-6.9b/W_U/cross-snapshot-32/d32768/seed0-sparse.safetensors` |
45
+ | Cross-family (OLMo-2-7B) | `olmo-2-7b/W_U/cross-snapshot-32/d32768/seed0.safetensors` |
46
+ | Read/write asymmetry ($W_E$ side) | `pythia-160m/W_E/cross-snapshot-32/...` |
47
+ | Activation-rate aggregates (lifecycle figures) | `derived/aggregates/`, `derived/rates/` |
48
+ | Attribution-patching artifacts | `attribution/pythia-160m/` |
49
+ | Held-out eval token corpus | `evaluation/eval-corpus/eval_tokens.pt` |
50
+
51
+ ## Full inventory
52
 
53
+ | Path | Model | Matrix | Kind | d_sae | Seed | Quality |
54
+ |---|---|---|---|---|---|---|
55
+ | `olmo-2-7b/W_U/cross-snapshot-32/d32768/seed0.safetensors` | allenai/OLMo-2-1124-7B | W_U | cross-snapshot-32 | 32768 | 0 | EV 0.853 / L0 557 |
56
+ | `pythia-160m/W_E/cross-snapshot-32/d24576/seed0.safetensors` | EleutherAI/pythia-160m | W_E | cross-snapshot-32 | 24576 | 0 | EV 0.831 / L0 118 |
57
+ | `pythia-160m/W_E/cross-snapshot-32/d8192/seed0.safetensors` | EleutherAI/pythia-160m | W_E | cross-snapshot-32 | 8192 | 0 | EV 0.581 / L0 82 |
58
+ | `pythia-160m/W_E/cross-snapshot-32/d8192/seed1.safetensors` | EleutherAI/pythia-160m | W_E | cross-snapshot-32 | 8192 | 1 | EV 0.580 / L0 82 |
59
+ | `pythia-160m/W_E/cross-snapshot-32/d8192/seed2.safetensors` | EleutherAI/pythia-160m | W_E | cross-snapshot-32 | 8192 | 2 | EV 0.582 / L0 82 |
60
+ | `pythia-160m/W_E/cross-snapshot-32/d8192/seed3.safetensors` | EleutherAI/pythia-160m | W_E | cross-snapshot-32 | 8192 | 3 | EV 0.581 / L0 82 |
61
+ | `pythia-160m/W_E/cross-snapshot-32/d8192/seed4.safetensors` | EleutherAI/pythia-160m | W_E | cross-snapshot-32 | 8192 | 4 | EV 0.583 / L0 83 |
62
+ | `pythia-160m/W_U/architecture-comparison/d8192/batchtopk/seed0.safetensors` | EleutherAI/pythia-160m | W_U | architecture-comparison/d8192 | 8192 | 0 | EV 0.725 / L0 203 |
63
+ | `pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.safetensors` | EleutherAI/pythia-160m | W_U | architecture-comparison/d8192 | 8192 | 0 | EV 0.214 / L0 12 |
64
+ | `pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.safetensors` | EleutherAI/pythia-160m | W_U | architecture-comparison/d8192 | 8192 | 0 | EV 0.827 / L0 654 |
65
+ | `pythia-160m/W_U/cross-snapshot-16/d8192/seed0.safetensors` | EleutherAI/pythia-160m | W_U | cross-snapshot-16 | 8192 | 0 | EV 0.773 / L0 216 |
66
+ | `pythia-160m/W_U/cross-snapshot-32/d16384/seed0.safetensors` | EleutherAI/pythia-160m | W_U | cross-snapshot-32 | 16384 | 0 | EV 0.780 / L0 103 |
67
+ | `pythia-160m/W_U/cross-snapshot-32/d24576/seed0.safetensors` | EleutherAI/pythia-160m | W_U | cross-snapshot-32 | 24576 | 0 | EV 0.920 / L0 286 |
68
+ | `pythia-160m/W_U/cross-snapshot-32/d24576/seed1.safetensors` | EleutherAI/pythia-160m | W_U | cross-snapshot-32 | 24576 | 1 | EV 0.920 / L0 286 |
69
+ | `pythia-160m/W_U/cross-snapshot-32/d24576/seed2.safetensors` | EleutherAI/pythia-160m | W_U | cross-snapshot-32 | 24576 | 2 | EV 0.920 / L0 286 |
70
+ | `pythia-160m/W_U/cross-snapshot-32/d8192/seed0.safetensors` | EleutherAI/pythia-160m | W_U | cross-snapshot-32 | 8192 | 0 | EV 0.776 / L0 203 |
71
+ | `pythia-160m/W_U/cross-snapshot-32/d8192/seed1.safetensors` | EleutherAI/pythia-160m | W_U | cross-snapshot-32 | 8192 | 1 | EV 0.776 / L0 203 |
72
+ | `pythia-160m/W_U/cross-snapshot-32/d8192/seed2.safetensors` | EleutherAI/pythia-160m | W_U | cross-snapshot-32 | 8192 | 2 | EV 0.776 / L0 203 |
73
+ | `pythia-160m/W_U/cross-snapshot-32/d8192/seed3.safetensors` | EleutherAI/pythia-160m | W_U | cross-snapshot-32 | 8192 | 3 | EV 0.776 / L0 203 |
74
+ | `pythia-160m/W_U/cross-snapshot-32/d8192/seed4.safetensors` | EleutherAI/pythia-160m | W_U | cross-snapshot-32 | 8192 | 4 | EV 0.777 / L0 203 |
75
+ | `pythia-160m/W_U/final-snapshot-saes/d16384.safetensors` | EleutherAI/pythia-160m | W_U | final-snapshot-saes | 16384 | 0 | EV 0.870 / L0 1913 |
76
+ | `pythia-160m/W_U/final-snapshot-saes/d32768.safetensors` | EleutherAI/pythia-160m | W_U | final-snapshot-saes | 32768 | 0 | EV 0.926 / L0 3410 |
77
+ | `pythia-160m/W_U/final-snapshot-saes/d6144.safetensors` | EleutherAI/pythia-160m | W_U | final-snapshot-saes | 6144 | 0 | EV 0.765 / L0 862 |
78
+ | `pythia-160m/W_U/final-snapshot-saes/d65536.safetensors` | EleutherAI/pythia-160m | W_U | final-snapshot-saes | 65536 | 0 | EV 0.964 / L0 5943 |
79
+ | `pythia-160m/W_U/final-snapshot-saes/d8192.safetensors` | EleutherAI/pythia-160m | W_U | final-snapshot-saes | 8192 | 0 | EV 0.799 / L0 1084 |
80
+ | `pythia-160m/W_U/lambda-sweep/d8192/lam0p40_seed0.safetensors` | EleutherAI/pythia-160m | W_U | lambda-sweep | 8192 | 0 | EV 0.748 / L0 160 |
81
+ | `pythia-160m/W_U/lambda-sweep/d8192/lam1p00_seed0.safetensors` | EleutherAI/pythia-160m | W_U | lambda-sweep | 8192 | 0 | EV 0.632 / L0 58 |
82
+ | `pythia-160m/W_U/lambda-sweep/d8192/lam1p20_seed0.safetensors` | EleutherAI/pythia-160m | W_U | lambda-sweep | 8192 | 0 | EV 0.603 / L0 45 |
83
+ | `pythia-160m/W_U/lambda-sweep/d8192/lam1p35_seed0.safetensors` | EleutherAI/pythia-160m | W_U | lambda-sweep | 8192 | 0 | EV 0.582 / L0 38 |
84
+ | `pythia-160m/W_U/lambda-sweep/d8192/lam1p35_seed1.safetensors` | EleutherAI/pythia-160m | W_U | lambda-sweep | 8192 | 1 | EV 0.582 / L0 38 |
85
+ | `pythia-160m/W_U/lambda-sweep/d8192/lam1p35_seed2.safetensors` | EleutherAI/pythia-160m | W_U | lambda-sweep | 8192 | 2 | EV 0.582 / L0 38 |
86
+ | `pythia-160m/W_U/lambda-sweep/d8192/lam1p80_seed0.safetensors` | EleutherAI/pythia-160m | W_U | lambda-sweep | 8192 | 0 | EV 0.528 / L0 23 |
87
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step0.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.729 / L0 1150 |
88
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step1.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.729 / L0 1150 |
89
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step1000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.786 / L0 997 |
90
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step102000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.800 / L0 983 |
91
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step116000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.812 / L0 958 |
92
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step128.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.729 / L0 1150 |
93
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step130000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.820 / L0 940 |
94
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step14000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.791 / L0 996 |
95
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step143000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.824 / L0 924 |
96
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step16.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.729 / L0 1150 |
97
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step2.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.729 / L0 1150 |
98
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step2000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.799 / L0 969 |
99
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step21000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.791 / L0 998 |
100
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step256.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.732 / L0 1142 |
101
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step27000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.791 / L0 999 |
102
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step3000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.799 / L0 972 |
103
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step32.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.729 / L0 1150 |
104
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step34000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.791 / L0 1000 |
105
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step4.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.729 / L0 1150 |
106
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step4000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.798 / L0 977 |
107
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step47000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.791 / L0 1000 |
108
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step5000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.796 / L0 982 |
109
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step512.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.754 / L0 1087 |
110
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step6000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.795 / L0 985 |
111
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step61000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.790 / L0 1002 |
112
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step64.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.729 / L0 1150 |
113
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step7000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.794 / L0 988 |
114
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step75000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.790 / L0 1004 |
115
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step8.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.729 / L0 1150 |
116
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step8000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.793 / L0 990 |
117
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step89000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.792 / L0 1002 |
118
+ | `pythia-160m/W_U/per-snapshot-saes/d8192/step9000.safetensors` | EleutherAI/pythia-160m | W_U | per-snapshot-saes | 8192 | 0 | EV 0.792 / L0 992 |
119
+ | `pythia-1b/W_U/cross-snapshot-32/d16384/seed0.safetensors` | EleutherAI/pythia-1b | W_U | cross-snapshot-32 | 16384 | 0 | EV 0.781 / L0 499 |
120
+ | `pythia-1b/W_U/cross-snapshot-32/d24576/seed0.safetensors` | EleutherAI/pythia-1b | W_U | cross-snapshot-32 | 24576 | 0 | EV 0.861 / L0 517 |
121
+ | `pythia-1b/W_U/cross-snapshot-32/d8192/seed0.safetensors` | EleutherAI/pythia-1b | W_U | cross-snapshot-32 | 8192 | 0 | EV 0.628 / L0 374 |
122
+ | `pythia-1b/W_U/cross-snapshot-32-matched-window/d24576/seed0.safetensors` | EleutherAI/pythia-1b | W_U | cross-snapshot-32-matched-window | 24576 | 0 | EV 0.884 / L0 264 |
123
+ | `pythia-6.9b/W_U/cross-snapshot-32/d32768/seed0-sparse.safetensors` | EleutherAI/pythia-6.9b | W_U | cross-snapshot-32 | 32768 | 0 | EV 0.808 / L0 742 |
124
+ | `pythia-6.9b/W_U/cross-snapshot-32/d32768/seed0.safetensors` | EleutherAI/pythia-6.9b | W_U | cross-snapshot-32 | 32768 | 0 | EV 0.833 / L0 1957 |
125
 
126
+ Quality metrics are recomputed from the released weights on the released
127
+ snapshot schedule (see the code repo's `scripts/eval/recompute_metrics.py`).
128
+ The `gated` architecture-comparison run intentionally documents
129
+ λ-transfer failure (default λ=0.3 moved across architectures); see
130
+ `gated-retuned` (λ=0.05) for the tuned comparison point.
131
 
132
  ## Citation
133
 
134
  ```bibtex
135
+ @misc{he2026learningtoreadout,
136
+ title = {Learning to Read Out: Unembedding Dynamics in Language Model Pretraining},
137
+ author = {He, Matteo and Shen, William F. and Iacob, Alex and Jovanovic, Andrej
138
+ and Qiu, Xinchi and Lane, Nicholas D.},
139
+ year = {2026},
140
+ note = {Under review. Code: https://github.com/hematteo/learning-to-read-out},
141
+ }
142
  ```
143
+
144
+ MIT. W_U/W_E source tensors derive from public Apache-2.0 checkpoints
145
+ (EleutherAI Pythia, AllenAI OLMo-2). The eval corpus derives from Wikipedia
146
+ (CC-BY-SA 4.0).
index.json ADDED
@@ -0,0 +1,1726 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "release": "parameter-trajectory-crosscoders",
3
+ "paper": "Learning to Read Out: Unembedding Dynamics in Language Model Pretraining",
4
+ "code": "https://github.com/hematteo/learning-to-read-out",
5
+ "loading_recipe": "from safetensors.torch import safe_open; f = safe_open(path, framework='pt'); W_D = f.get_tensor('W_D'); thr = f.get_tensor('activation_function.log_jumprelu_threshold').exp() # or use src.crosscoder.checkpoints.load_checkpoint from the code repo",
6
+ "checkpoints": [
7
+ {
8
+ "path": "olmo-2-7b/W_U/cross-snapshot-32/d32768/seed0.safetensors",
9
+ "model_name": "allenai/OLMo-2-1124-7B",
10
+ "model": "olmo-2-7b",
11
+ "matrix": "W_U",
12
+ "kind": "cross-snapshot-32",
13
+ "d_sae": 32768,
14
+ "seed": 0,
15
+ "n_snapshots": 32,
16
+ "arch": "jumprelu-crosscoder",
17
+ "public_label": "OLMo-2-7B W_U crosscoder, 32 snapshots, d_sae=32768, seed 0",
18
+ "quality": {
19
+ "explained_variance": 0.8526035831389717,
20
+ "mean_l0": 557.2795322963169,
21
+ "dead_rate": 0.0
22
+ },
23
+ "training": {
24
+ "lr": 1e-05,
25
+ "l1_coefficient": 0.3,
26
+ "batch_size": 1024,
27
+ "n_epochs": 300,
28
+ "input_preprocess": "center_scale"
29
+ }
30
+ },
31
+ {
32
+ "path": "pythia-160m/W_E/cross-snapshot-32/d24576/seed0.safetensors",
33
+ "model_name": "EleutherAI/pythia-160m",
34
+ "model": "pythia-160m",
35
+ "matrix": "W_E",
36
+ "kind": "cross-snapshot-32",
37
+ "d_sae": 24576,
38
+ "seed": 0,
39
+ "n_snapshots": 32,
40
+ "arch": "jumprelu-crosscoder",
41
+ "public_label": "Pythia-160M W_E crosscoder, 32 snapshots, d_sae=24576, seed 0",
42
+ "quality": {
43
+ "explained_variance": 0.8305442370487662,
44
+ "mean_l0": 117.50901977539063,
45
+ "dead_rate": 0.0
46
+ },
47
+ "training": {
48
+ "lr": 5e-05,
49
+ "l1_coefficient": 0.3,
50
+ "batch_size": 1024,
51
+ "n_epochs": 300,
52
+ "input_preprocess": "center_scale"
53
+ }
54
+ },
55
+ {
56
+ "path": "pythia-160m/W_E/cross-snapshot-32/d8192/seed0.safetensors",
57
+ "model_name": "EleutherAI/pythia-160m",
58
+ "model": "pythia-160m",
59
+ "matrix": "W_E",
60
+ "kind": "cross-snapshot-32",
61
+ "d_sae": 8192,
62
+ "seed": 0,
63
+ "n_snapshots": 32,
64
+ "arch": "jumprelu-crosscoder",
65
+ "public_label": "Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 0",
66
+ "quality": {
67
+ "explained_variance": 0.5807971126242004,
68
+ "mean_l0": 82.08435302734375
69
+ },
70
+ "training": {
71
+ "lr": 5e-05,
72
+ "l1_coefficient": 0.3,
73
+ "batch_size": 1024,
74
+ "n_epochs": 300,
75
+ "input_preprocess": "center_scale"
76
+ }
77
+ },
78
+ {
79
+ "path": "pythia-160m/W_E/cross-snapshot-32/d8192/seed1.safetensors",
80
+ "model_name": "EleutherAI/pythia-160m",
81
+ "model": "pythia-160m",
82
+ "matrix": "W_E",
83
+ "kind": "cross-snapshot-32",
84
+ "d_sae": 8192,
85
+ "seed": 1,
86
+ "n_snapshots": 32,
87
+ "arch": "jumprelu-crosscoder",
88
+ "public_label": "Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 1",
89
+ "quality": {
90
+ "explained_variance": 0.580104140268709,
91
+ "mean_l0": 82.01199279785156,
92
+ "dead_rate": 0.0
93
+ },
94
+ "training": {
95
+ "lr": 5e-05,
96
+ "l1_coefficient": 0.3,
97
+ "batch_size": 1024,
98
+ "n_epochs": 300,
99
+ "input_preprocess": "center_scale"
100
+ }
101
+ },
102
+ {
103
+ "path": "pythia-160m/W_E/cross-snapshot-32/d8192/seed2.safetensors",
104
+ "model_name": "EleutherAI/pythia-160m",
105
+ "model": "pythia-160m",
106
+ "matrix": "W_E",
107
+ "kind": "cross-snapshot-32",
108
+ "d_sae": 8192,
109
+ "seed": 2,
110
+ "n_snapshots": 32,
111
+ "arch": "jumprelu-crosscoder",
112
+ "public_label": "Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 2",
113
+ "quality": {
114
+ "explained_variance": 0.5815380601296635,
115
+ "mean_l0": 82.31076721191407,
116
+ "dead_rate": 0.0
117
+ },
118
+ "training": {
119
+ "lr": 5e-05,
120
+ "l1_coefficient": 0.3,
121
+ "batch_size": 1024,
122
+ "n_epochs": 300,
123
+ "input_preprocess": "center_scale"
124
+ }
125
+ },
126
+ {
127
+ "path": "pythia-160m/W_E/cross-snapshot-32/d8192/seed3.safetensors",
128
+ "model_name": "EleutherAI/pythia-160m",
129
+ "model": "pythia-160m",
130
+ "matrix": "W_E",
131
+ "kind": "cross-snapshot-32",
132
+ "d_sae": 8192,
133
+ "seed": 3,
134
+ "n_snapshots": 32,
135
+ "arch": "jumprelu-crosscoder",
136
+ "public_label": "Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 3",
137
+ "quality": {
138
+ "explained_variance": 0.5805984141666259,
139
+ "mean_l0": 82.13521545410157,
140
+ "dead_rate": 0.0
141
+ },
142
+ "training": {
143
+ "lr": 5e-05,
144
+ "l1_coefficient": 0.3,
145
+ "batch_size": 1024,
146
+ "n_epochs": 300,
147
+ "input_preprocess": "center_scale"
148
+ }
149
+ },
150
+ {
151
+ "path": "pythia-160m/W_E/cross-snapshot-32/d8192/seed4.safetensors",
152
+ "model_name": "EleutherAI/pythia-160m",
153
+ "model": "pythia-160m",
154
+ "matrix": "W_E",
155
+ "kind": "cross-snapshot-32",
156
+ "d_sae": 8192,
157
+ "seed": 4,
158
+ "n_snapshots": 32,
159
+ "arch": "jumprelu-crosscoder",
160
+ "public_label": "Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 4",
161
+ "quality": {
162
+ "explained_variance": 0.5825219599393371,
163
+ "mean_l0": 82.5084307861328,
164
+ "dead_rate": 0.0
165
+ },
166
+ "training": {
167
+ "lr": 5e-05,
168
+ "l1_coefficient": 0.3,
169
+ "batch_size": 1024,
170
+ "n_epochs": 300,
171
+ "input_preprocess": "center_scale"
172
+ }
173
+ },
174
+ {
175
+ "path": "pythia-160m/W_U/architecture-comparison/d8192/batchtopk/seed0.safetensors",
176
+ "model_name": "EleutherAI/pythia-160m",
177
+ "model": "pythia-160m",
178
+ "matrix": "W_U",
179
+ "kind": "architecture-comparison/d8192",
180
+ "d_sae": 8192,
181
+ "seed": 0,
182
+ "n_snapshots": 32,
183
+ "arch": "batchtopk",
184
+ "public_label": "Pythia-160M W_U crosscoder (batchtopk activation), 32 snapshots, d_sae=8192, seed 0",
185
+ "quality": {
186
+ "explained_variance": 0.7249162974116965,
187
+ "mean_l0": 203.0
188
+ },
189
+ "training": {
190
+ "lr": 5e-05,
191
+ "l1_coefficient": 0.3,
192
+ "batch_size": 1024,
193
+ "n_epochs": 100,
194
+ "input_preprocess": "center_scale"
195
+ }
196
+ },
197
+ {
198
+ "path": "pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.safetensors",
199
+ "model_name": "EleutherAI/pythia-160m",
200
+ "model": "pythia-160m",
201
+ "matrix": "W_U",
202
+ "kind": "architecture-comparison/d8192",
203
+ "d_sae": 8192,
204
+ "seed": 0,
205
+ "n_snapshots": 32,
206
+ "arch": "gated",
207
+ "public_label": "Pythia-160M W_U crosscoder (gated activation), 32 snapshots, d_sae=8192, seed 0",
208
+ "quality": {
209
+ "explained_variance": 0.21442979106557436,
210
+ "mean_l0": 11.51853515625
211
+ },
212
+ "training": {
213
+ "lr": 5e-05,
214
+ "l1_coefficient": 0.3,
215
+ "batch_size": 1024,
216
+ "n_epochs": 100,
217
+ "input_preprocess": "center_scale"
218
+ }
219
+ },
220
+ {
221
+ "path": "pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.safetensors",
222
+ "model_name": "EleutherAI/pythia-160m",
223
+ "model": "pythia-160m",
224
+ "matrix": "W_U",
225
+ "kind": "architecture-comparison/d8192",
226
+ "d_sae": 8192,
227
+ "seed": 0,
228
+ "n_snapshots": 32,
229
+ "arch": "gated",
230
+ "public_label": "Pythia-160M W_U crosscoder (Gated activation, L1=0.05), 32 snapshots, d_sae=8192, seed 0",
231
+ "quality": {
232
+ "explained_variance": 0.827462127951851,
233
+ "mean_l0": 653.64568359375
234
+ },
235
+ "training": {
236
+ "lr": 5e-05,
237
+ "l1_coefficient": 0.3,
238
+ "batch_size": 1024,
239
+ "n_epochs": 100,
240
+ "input_preprocess": "center_scale"
241
+ }
242
+ },
243
+ {
244
+ "path": "pythia-160m/W_U/cross-snapshot-16/d8192/seed0.safetensors",
245
+ "model_name": "EleutherAI/pythia-160m",
246
+ "model": "pythia-160m",
247
+ "matrix": "W_U",
248
+ "kind": "cross-snapshot-16",
249
+ "d_sae": 8192,
250
+ "seed": 0,
251
+ "n_snapshots": 16,
252
+ "arch": "jumprelu-crosscoder",
253
+ "public_label": "Pythia-160M W_U crosscoder, 16-snapshot downsample, d_sae=8192, seed 0",
254
+ "quality": {
255
+ "explained_variance": 0.7734329300485889,
256
+ "mean_l0": 215.73649780273436,
257
+ "dead_rate": 0.0
258
+ },
259
+ "training": {
260
+ "lr": 5e-05,
261
+ "l1_coefficient": 0.3,
262
+ "batch_size": 1024,
263
+ "n_epochs": 300,
264
+ "input_preprocess": "center_scale"
265
+ }
266
+ },
267
+ {
268
+ "path": "pythia-160m/W_U/cross-snapshot-32/d16384/seed0.safetensors",
269
+ "model_name": "EleutherAI/pythia-160m",
270
+ "model": "pythia-160m",
271
+ "matrix": "W_U",
272
+ "kind": "cross-snapshot-32",
273
+ "d_sae": 16384,
274
+ "seed": 0,
275
+ "n_snapshots": 32,
276
+ "arch": "jumprelu-crosscoder",
277
+ "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=16384, seed 0",
278
+ "quality": {
279
+ "explained_variance": 0.7802076016978367,
280
+ "mean_l0": 103.0708544921875
281
+ },
282
+ "training": {
283
+ "lr": 5e-05,
284
+ "l1_coefficient": 0.3,
285
+ "batch_size": 1024,
286
+ "n_epochs": 300,
287
+ "input_preprocess": "center_scale"
288
+ }
289
+ },
290
+ {
291
+ "path": "pythia-160m/W_U/cross-snapshot-32/d24576/seed0.safetensors",
292
+ "model_name": "EleutherAI/pythia-160m",
293
+ "model": "pythia-160m",
294
+ "matrix": "W_U",
295
+ "kind": "cross-snapshot-32",
296
+ "d_sae": 24576,
297
+ "seed": 0,
298
+ "n_snapshots": 32,
299
+ "arch": "jumprelu-crosscoder",
300
+ "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=24576, seed 0",
301
+ "quality": {
302
+ "explained_variance": 0.9196719747097695,
303
+ "mean_l0": 285.9946594238281,
304
+ "dead_rate": 0.0
305
+ },
306
+ "training": {
307
+ "lr": 5e-05,
308
+ "l1_coefficient": 0.3,
309
+ "batch_size": 1024,
310
+ "n_epochs": 300,
311
+ "input_preprocess": "center_scale"
312
+ }
313
+ },
314
+ {
315
+ "path": "pythia-160m/W_U/cross-snapshot-32/d24576/seed1.safetensors",
316
+ "model_name": "EleutherAI/pythia-160m",
317
+ "model": "pythia-160m",
318
+ "matrix": "W_U",
319
+ "kind": "cross-snapshot-32",
320
+ "d_sae": 24576,
321
+ "seed": 1,
322
+ "n_snapshots": 32,
323
+ "arch": "jumprelu-crosscoder",
324
+ "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=24576, seed 1",
325
+ "quality": {
326
+ "explained_variance": 0.9195844996740984,
327
+ "mean_l0": 286.09990661621094,
328
+ "dead_rate": 0.0
329
+ },
330
+ "training": {
331
+ "lr": 5e-05,
332
+ "l1_coefficient": 0.3,
333
+ "batch_size": 1024,
334
+ "n_epochs": 300,
335
+ "input_preprocess": "center_scale"
336
+ }
337
+ },
338
+ {
339
+ "path": "pythia-160m/W_U/cross-snapshot-32/d24576/seed2.safetensors",
340
+ "model_name": "EleutherAI/pythia-160m",
341
+ "model": "pythia-160m",
342
+ "matrix": "W_U",
343
+ "kind": "cross-snapshot-32",
344
+ "d_sae": 24576,
345
+ "seed": 2,
346
+ "n_snapshots": 32,
347
+ "arch": "jumprelu-crosscoder",
348
+ "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=24576, seed 2",
349
+ "quality": {
350
+ "explained_variance": 0.9196765967343927,
351
+ "mean_l0": 286.3446789550781,
352
+ "dead_rate": 0.0
353
+ },
354
+ "training": {
355
+ "lr": 5e-05,
356
+ "l1_coefficient": 0.3,
357
+ "batch_size": 1024,
358
+ "n_epochs": 300,
359
+ "input_preprocess": "center_scale"
360
+ }
361
+ },
362
+ {
363
+ "path": "pythia-160m/W_U/cross-snapshot-32/d8192/seed0.safetensors",
364
+ "model_name": "EleutherAI/pythia-160m",
365
+ "model": "pythia-160m",
366
+ "matrix": "W_U",
367
+ "kind": "cross-snapshot-32",
368
+ "d_sae": 8192,
369
+ "seed": 0,
370
+ "n_snapshots": 32,
371
+ "arch": "jumprelu-crosscoder",
372
+ "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 0",
373
+ "quality": {
374
+ "explained_variance": 0.7763740924044573,
375
+ "mean_l0": 203.14466369628906
376
+ },
377
+ "training": {
378
+ "lr": 5e-05,
379
+ "l1_coefficient": 0.3,
380
+ "batch_size": 1024,
381
+ "n_epochs": 300,
382
+ "input_preprocess": "center_scale"
383
+ }
384
+ },
385
+ {
386
+ "path": "pythia-160m/W_U/cross-snapshot-32/d8192/seed1.safetensors",
387
+ "model_name": "EleutherAI/pythia-160m",
388
+ "model": "pythia-160m",
389
+ "matrix": "W_U",
390
+ "kind": "cross-snapshot-32",
391
+ "d_sae": 8192,
392
+ "seed": 1,
393
+ "n_snapshots": 32,
394
+ "arch": "jumprelu-crosscoder",
395
+ "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 1",
396
+ "quality": {
397
+ "explained_variance": 0.7763600972452054,
398
+ "mean_l0": 203.26546752929687
399
+ },
400
+ "training": {
401
+ "lr": 5e-05,
402
+ "l1_coefficient": 0.3,
403
+ "batch_size": 1024,
404
+ "n_epochs": 300,
405
+ "input_preprocess": "center_scale"
406
+ }
407
+ },
408
+ {
409
+ "path": "pythia-160m/W_U/cross-snapshot-32/d8192/seed2.safetensors",
410
+ "model_name": "EleutherAI/pythia-160m",
411
+ "model": "pythia-160m",
412
+ "matrix": "W_U",
413
+ "kind": "cross-snapshot-32",
414
+ "d_sae": 8192,
415
+ "seed": 2,
416
+ "n_snapshots": 32,
417
+ "arch": "jumprelu-crosscoder",
418
+ "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 2",
419
+ "quality": {
420
+ "explained_variance": 0.7764523546944131,
421
+ "mean_l0": 203.14273071289062
422
+ },
423
+ "training": {
424
+ "lr": 5e-05,
425
+ "l1_coefficient": 0.3,
426
+ "batch_size": 1024,
427
+ "n_epochs": 300,
428
+ "input_preprocess": "center_scale"
429
+ }
430
+ },
431
+ {
432
+ "path": "pythia-160m/W_U/cross-snapshot-32/d8192/seed3.safetensors",
433
+ "model_name": "EleutherAI/pythia-160m",
434
+ "model": "pythia-160m",
435
+ "matrix": "W_U",
436
+ "kind": "cross-snapshot-32",
437
+ "d_sae": 8192,
438
+ "seed": 3,
439
+ "n_snapshots": 32,
440
+ "arch": "jumprelu-crosscoder",
441
+ "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 3",
442
+ "quality": {
443
+ "explained_variance": 0.7762290236129058,
444
+ "mean_l0": 203.09911560058595
445
+ },
446
+ "training": {
447
+ "lr": 5e-05,
448
+ "l1_coefficient": 0.3,
449
+ "batch_size": 1024,
450
+ "n_epochs": 300,
451
+ "input_preprocess": "center_scale"
452
+ }
453
+ },
454
+ {
455
+ "path": "pythia-160m/W_U/cross-snapshot-32/d8192/seed4.safetensors",
456
+ "model_name": "EleutherAI/pythia-160m",
457
+ "model": "pythia-160m",
458
+ "matrix": "W_U",
459
+ "kind": "cross-snapshot-32",
460
+ "d_sae": 8192,
461
+ "seed": 4,
462
+ "n_snapshots": 32,
463
+ "arch": "jumprelu-crosscoder",
464
+ "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 4",
465
+ "quality": {
466
+ "explained_variance": 0.7765168883427946,
467
+ "mean_l0": 203.4711474609375
468
+ },
469
+ "training": {
470
+ "lr": 5e-05,
471
+ "l1_coefficient": 0.3,
472
+ "batch_size": 1024,
473
+ "n_epochs": 300,
474
+ "input_preprocess": "center_scale"
475
+ }
476
+ },
477
+ {
478
+ "path": "pythia-160m/W_U/final-snapshot-saes/d16384.safetensors",
479
+ "model_name": "EleutherAI/pythia-160m",
480
+ "model": "pythia-160m",
481
+ "matrix": "W_U",
482
+ "kind": "final-snapshot-saes",
483
+ "d_sae": 16384,
484
+ "seed": 0,
485
+ "n_snapshots": 1,
486
+ "arch": "jumprelu-crosscoder",
487
+ "public_label": "Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=16384",
488
+ "quality": {
489
+ "explained_variance": 0.8703331309369664,
490
+ "mean_l0": 1913.29353515625
491
+ },
492
+ "training": {
493
+ "lr": 5e-05,
494
+ "l1_coefficient": 0.05,
495
+ "batch_size": 1024,
496
+ "n_epochs": 100,
497
+ "input_preprocess": "center_scale"
498
+ }
499
+ },
500
+ {
501
+ "path": "pythia-160m/W_U/final-snapshot-saes/d32768.safetensors",
502
+ "model_name": "EleutherAI/pythia-160m",
503
+ "model": "pythia-160m",
504
+ "matrix": "W_U",
505
+ "kind": "final-snapshot-saes",
506
+ "d_sae": 32768,
507
+ "seed": 0,
508
+ "n_snapshots": 1,
509
+ "arch": "jumprelu-crosscoder",
510
+ "public_label": "Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=32768",
511
+ "quality": {
512
+ "explained_variance": 0.926354609914279,
513
+ "mean_l0": 3410.36810546875
514
+ },
515
+ "training": {
516
+ "lr": 5e-05,
517
+ "l1_coefficient": 0.05,
518
+ "batch_size": 1024,
519
+ "n_epochs": 100,
520
+ "input_preprocess": "center_scale"
521
+ }
522
+ },
523
+ {
524
+ "path": "pythia-160m/W_U/final-snapshot-saes/d6144.safetensors",
525
+ "model_name": "EleutherAI/pythia-160m",
526
+ "model": "pythia-160m",
527
+ "matrix": "W_U",
528
+ "kind": "final-snapshot-saes",
529
+ "d_sae": 6144,
530
+ "seed": 0,
531
+ "n_snapshots": 1,
532
+ "arch": "jumprelu-crosscoder",
533
+ "public_label": "Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=6144",
534
+ "quality": {
535
+ "explained_variance": 0.7649826125102704,
536
+ "mean_l0": 861.8197265625
537
+ },
538
+ "training": {
539
+ "lr": 5e-05,
540
+ "l1_coefficient": 0.05,
541
+ "batch_size": 1024,
542
+ "n_epochs": 100,
543
+ "input_preprocess": "center_scale"
544
+ }
545
+ },
546
+ {
547
+ "path": "pythia-160m/W_U/final-snapshot-saes/d65536.safetensors",
548
+ "model_name": "EleutherAI/pythia-160m",
549
+ "model": "pythia-160m",
550
+ "matrix": "W_U",
551
+ "kind": "final-snapshot-saes",
552
+ "d_sae": 65536,
553
+ "seed": 0,
554
+ "n_snapshots": 1,
555
+ "arch": "jumprelu-crosscoder",
556
+ "public_label": "Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=65536",
557
+ "quality": {
558
+ "explained_variance": 0.9642323011052749,
559
+ "mean_l0": 5943.19078125
560
+ },
561
+ "training": {
562
+ "lr": 5e-05,
563
+ "l1_coefficient": 0.05,
564
+ "batch_size": 1024,
565
+ "n_epochs": 100,
566
+ "input_preprocess": "center_scale"
567
+ }
568
+ },
569
+ {
570
+ "path": "pythia-160m/W_U/final-snapshot-saes/d8192.safetensors",
571
+ "model_name": "EleutherAI/pythia-160m",
572
+ "model": "pythia-160m",
573
+ "matrix": "W_U",
574
+ "kind": "final-snapshot-saes",
575
+ "d_sae": 8192,
576
+ "seed": 0,
577
+ "n_snapshots": 1,
578
+ "arch": "jumprelu-crosscoder",
579
+ "public_label": "Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=8192",
580
+ "quality": {
581
+ "explained_variance": 0.7989936757858394,
582
+ "mean_l0": 1084.4333984375
583
+ },
584
+ "training": {
585
+ "lr": 5e-05,
586
+ "l1_coefficient": 0.05,
587
+ "batch_size": 1024,
588
+ "n_epochs": 100,
589
+ "input_preprocess": "center_scale"
590
+ }
591
+ },
592
+ {
593
+ "path": "pythia-160m/W_U/lambda-sweep/d8192/lam0p40_seed0.safetensors",
594
+ "model_name": "EleutherAI/pythia-160m",
595
+ "model": "pythia-160m",
596
+ "matrix": "W_U",
597
+ "kind": "lambda-sweep",
598
+ "d_sae": 8192,
599
+ "seed": 0,
600
+ "n_snapshots": 32,
601
+ "arch": "jumprelu-crosscoder",
602
+ "public_label": "Pythia-160M W_U crosscoder, \u03bb-sweep iso-L0 (\u03bb=0.4), 32 snapshots, d_sae=8192, seed 0",
603
+ "quality": {
604
+ "explained_variance": 0.7479858952216084,
605
+ "mean_l0": 160.0768243408203,
606
+ "dead_rate": 0.0
607
+ },
608
+ "training": {
609
+ "lr": 5e-05,
610
+ "l1_coefficient": 0.4,
611
+ "batch_size": 1024,
612
+ "n_epochs": 300,
613
+ "input_preprocess": "center_scale"
614
+ }
615
+ },
616
+ {
617
+ "path": "pythia-160m/W_U/lambda-sweep/d8192/lam1p00_seed0.safetensors",
618
+ "model_name": "EleutherAI/pythia-160m",
619
+ "model": "pythia-160m",
620
+ "matrix": "W_U",
621
+ "kind": "lambda-sweep",
622
+ "d_sae": 8192,
623
+ "seed": 0,
624
+ "n_snapshots": 32,
625
+ "arch": "jumprelu-crosscoder",
626
+ "public_label": "Pythia-160M W_U crosscoder, \u03bb-sweep iso-L0 (\u03bb=1.0), 32 snapshots, d_sae=8192, seed 0",
627
+ "quality": {
628
+ "explained_variance": 0.6323049070911215,
629
+ "mean_l0": 58.49865539550781,
630
+ "dead_rate": 0.0
631
+ },
632
+ "training": {
633
+ "lr": 5e-05,
634
+ "l1_coefficient": 1.0,
635
+ "batch_size": 1024,
636
+ "n_epochs": 300,
637
+ "input_preprocess": "center_scale"
638
+ }
639
+ },
640
+ {
641
+ "path": "pythia-160m/W_U/lambda-sweep/d8192/lam1p20_seed0.safetensors",
642
+ "model_name": "EleutherAI/pythia-160m",
643
+ "model": "pythia-160m",
644
+ "matrix": "W_U",
645
+ "kind": "lambda-sweep",
646
+ "d_sae": 8192,
647
+ "seed": 0,
648
+ "n_snapshots": 32,
649
+ "arch": "jumprelu-crosscoder",
650
+ "public_label": "Pythia-160M W_U crosscoder, \u03bb-sweep iso-L0 (\u03bb=1.2), 32 snapshots, d_sae=8192, seed 0",
651
+ "quality": {
652
+ "explained_variance": 0.6027639222075235,
653
+ "mean_l0": 45.41338317871094,
654
+ "dead_rate": 0.0
655
+ },
656
+ "training": {
657
+ "lr": 5e-05,
658
+ "l1_coefficient": 1.2,
659
+ "batch_size": 1024,
660
+ "n_epochs": 300,
661
+ "input_preprocess": "center_scale"
662
+ }
663
+ },
664
+ {
665
+ "path": "pythia-160m/W_U/lambda-sweep/d8192/lam1p35_seed0.safetensors",
666
+ "model_name": "EleutherAI/pythia-160m",
667
+ "model": "pythia-160m",
668
+ "matrix": "W_U",
669
+ "kind": "lambda-sweep",
670
+ "d_sae": 8192,
671
+ "seed": 0,
672
+ "n_snapshots": 32,
673
+ "arch": "jumprelu-crosscoder",
674
+ "public_label": "Pythia-160M W_U crosscoder, \u03bb-sweep iso-L0 (\u03bb=1.35), 32 snapshots, d_sae=8192, seed 0",
675
+ "quality": {
676
+ "explained_variance": 0.5819729868726617,
677
+ "mean_l0": 37.971342163085936,
678
+ "dead_rate": 0.0
679
+ },
680
+ "training": {
681
+ "lr": 5e-05,
682
+ "l1_coefficient": 1.35,
683
+ "batch_size": 1024,
684
+ "n_epochs": 300,
685
+ "input_preprocess": "center_scale"
686
+ }
687
+ },
688
+ {
689
+ "path": "pythia-160m/W_U/lambda-sweep/d8192/lam1p35_seed1.safetensors",
690
+ "model_name": "EleutherAI/pythia-160m",
691
+ "model": "pythia-160m",
692
+ "matrix": "W_U",
693
+ "kind": "lambda-sweep",
694
+ "d_sae": 8192,
695
+ "seed": 1,
696
+ "n_snapshots": 32,
697
+ "arch": "jumprelu-crosscoder",
698
+ "public_label": "Pythia-160M W_U crosscoder, \u03bb-sweep iso-L0 (\u03bb=1.35), 32 snapshots, d_sae=8192, seed 1",
699
+ "quality": {
700
+ "explained_variance": 0.581652004193779,
701
+ "mean_l0": 37.79685302734375,
702
+ "dead_rate": 0.0
703
+ },
704
+ "training": {
705
+ "lr": 5e-05,
706
+ "l1_coefficient": 1.35,
707
+ "batch_size": 1024,
708
+ "n_epochs": 300,
709
+ "input_preprocess": "center_scale"
710
+ }
711
+ },
712
+ {
713
+ "path": "pythia-160m/W_U/lambda-sweep/d8192/lam1p35_seed2.safetensors",
714
+ "model_name": "EleutherAI/pythia-160m",
715
+ "model": "pythia-160m",
716
+ "matrix": "W_U",
717
+ "kind": "lambda-sweep",
718
+ "d_sae": 8192,
719
+ "seed": 2,
720
+ "n_snapshots": 32,
721
+ "arch": "jumprelu-crosscoder",
722
+ "public_label": "Pythia-160M W_U crosscoder, \u03bb-sweep iso-L0 (\u03bb=1.35), 32 snapshots, d_sae=8192, seed 2",
723
+ "quality": {
724
+ "explained_variance": 0.582366028895355,
725
+ "mean_l0": 37.980972900390626,
726
+ "dead_rate": 0.0
727
+ },
728
+ "training": {
729
+ "lr": 5e-05,
730
+ "l1_coefficient": 1.35,
731
+ "batch_size": 1024,
732
+ "n_epochs": 300,
733
+ "input_preprocess": "center_scale"
734
+ }
735
+ },
736
+ {
737
+ "path": "pythia-160m/W_U/lambda-sweep/d8192/lam1p80_seed0.safetensors",
738
+ "model_name": "EleutherAI/pythia-160m",
739
+ "model": "pythia-160m",
740
+ "matrix": "W_U",
741
+ "kind": "lambda-sweep",
742
+ "d_sae": 8192,
743
+ "seed": 0,
744
+ "n_snapshots": 32,
745
+ "arch": "jumprelu-crosscoder",
746
+ "public_label": "Pythia-160M W_U crosscoder, \u03bb-sweep iso-L0 (\u03bb=1.8), 32 snapshots, d_sae=8192, seed 0",
747
+ "quality": {
748
+ "explained_variance": 0.527703732469858,
749
+ "mean_l0": 23.033436279296875,
750
+ "dead_rate": 0.0
751
+ },
752
+ "training": {
753
+ "lr": 5e-05,
754
+ "l1_coefficient": 1.8,
755
+ "batch_size": 1024,
756
+ "n_epochs": 300,
757
+ "input_preprocess": "center_scale"
758
+ }
759
+ },
760
+ {
761
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step0.safetensors",
762
+ "model_name": "EleutherAI/pythia-160m",
763
+ "model": "pythia-160m",
764
+ "matrix": "W_U",
765
+ "kind": "per-snapshot-saes",
766
+ "d_sae": 8192,
767
+ "seed": 0,
768
+ "n_snapshots": 1,
769
+ "arch": "jumprelu-crosscoder",
770
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 0, d_sae=8192",
771
+ "quality": {
772
+ "explained_variance": 0.7288661622516968,
773
+ "mean_l0": 1150.079375,
774
+ "dead_rate": 0.0
775
+ },
776
+ "training": {
777
+ "lr": 5e-05,
778
+ "l1_coefficient": 0.3,
779
+ "batch_size": 2048,
780
+ "n_epochs": 100,
781
+ "input_preprocess": "center_scale"
782
+ }
783
+ },
784
+ {
785
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step1.safetensors",
786
+ "model_name": "EleutherAI/pythia-160m",
787
+ "model": "pythia-160m",
788
+ "matrix": "W_U",
789
+ "kind": "per-snapshot-saes",
790
+ "d_sae": 8192,
791
+ "seed": 0,
792
+ "n_snapshots": 1,
793
+ "arch": "jumprelu-crosscoder",
794
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 1, d_sae=8192",
795
+ "quality": {
796
+ "explained_variance": 0.7288661622516968,
797
+ "mean_l0": 1150.079375,
798
+ "dead_rate": 0.0
799
+ },
800
+ "training": {
801
+ "lr": 5e-05,
802
+ "l1_coefficient": 0.3,
803
+ "batch_size": 2048,
804
+ "n_epochs": 100,
805
+ "input_preprocess": "center_scale"
806
+ }
807
+ },
808
+ {
809
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step1000.safetensors",
810
+ "model_name": "EleutherAI/pythia-160m",
811
+ "model": "pythia-160m",
812
+ "matrix": "W_U",
813
+ "kind": "per-snapshot-saes",
814
+ "d_sae": 8192,
815
+ "seed": 0,
816
+ "n_snapshots": 1,
817
+ "arch": "jumprelu-crosscoder",
818
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 1000, d_sae=8192",
819
+ "quality": {
820
+ "explained_variance": 0.7862491472291859,
821
+ "mean_l0": 996.89369140625,
822
+ "dead_rate": 0.0
823
+ },
824
+ "training": {
825
+ "lr": 5e-05,
826
+ "l1_coefficient": 0.3,
827
+ "batch_size": 2048,
828
+ "n_epochs": 100,
829
+ "input_preprocess": "center_scale"
830
+ }
831
+ },
832
+ {
833
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step102000.safetensors",
834
+ "model_name": "EleutherAI/pythia-160m",
835
+ "model": "pythia-160m",
836
+ "matrix": "W_U",
837
+ "kind": "per-snapshot-saes",
838
+ "d_sae": 8192,
839
+ "seed": 0,
840
+ "n_snapshots": 1,
841
+ "arch": "jumprelu-crosscoder",
842
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 102000, d_sae=8192",
843
+ "quality": {
844
+ "explained_variance": 0.7998127382951936,
845
+ "mean_l0": 983.041015625,
846
+ "dead_rate": 0.0
847
+ },
848
+ "training": {
849
+ "lr": 5e-05,
850
+ "l1_coefficient": 0.3,
851
+ "batch_size": 2048,
852
+ "n_epochs": 100,
853
+ "input_preprocess": "center_scale"
854
+ }
855
+ },
856
+ {
857
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step116000.safetensors",
858
+ "model_name": "EleutherAI/pythia-160m",
859
+ "model": "pythia-160m",
860
+ "matrix": "W_U",
861
+ "kind": "per-snapshot-saes",
862
+ "d_sae": 8192,
863
+ "seed": 0,
864
+ "n_snapshots": 1,
865
+ "arch": "jumprelu-crosscoder",
866
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 116000, d_sae=8192",
867
+ "quality": {
868
+ "explained_variance": 0.8120686966847851,
869
+ "mean_l0": 957.6841577148438,
870
+ "dead_rate": 0.0
871
+ },
872
+ "training": {
873
+ "lr": 5e-05,
874
+ "l1_coefficient": 0.3,
875
+ "batch_size": 2048,
876
+ "n_epochs": 100,
877
+ "input_preprocess": "center_scale"
878
+ }
879
+ },
880
+ {
881
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step128.safetensors",
882
+ "model_name": "EleutherAI/pythia-160m",
883
+ "model": "pythia-160m",
884
+ "matrix": "W_U",
885
+ "kind": "per-snapshot-saes",
886
+ "d_sae": 8192,
887
+ "seed": 0,
888
+ "n_snapshots": 1,
889
+ "arch": "jumprelu-crosscoder",
890
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 128, d_sae=8192",
891
+ "quality": {
892
+ "explained_variance": 0.7290331109474104,
893
+ "mean_l0": 1149.5578076171876,
894
+ "dead_rate": 0.0
895
+ },
896
+ "training": {
897
+ "lr": 5e-05,
898
+ "l1_coefficient": 0.3,
899
+ "batch_size": 2048,
900
+ "n_epochs": 100,
901
+ "input_preprocess": "center_scale"
902
+ }
903
+ },
904
+ {
905
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step130000.safetensors",
906
+ "model_name": "EleutherAI/pythia-160m",
907
+ "model": "pythia-160m",
908
+ "matrix": "W_U",
909
+ "kind": "per-snapshot-saes",
910
+ "d_sae": 8192,
911
+ "seed": 0,
912
+ "n_snapshots": 1,
913
+ "arch": "jumprelu-crosscoder",
914
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 130000, d_sae=8192",
915
+ "quality": {
916
+ "explained_variance": 0.8195671283611181,
917
+ "mean_l0": 940.0407983398437,
918
+ "dead_rate": 0.0
919
+ },
920
+ "training": {
921
+ "lr": 5e-05,
922
+ "l1_coefficient": 0.3,
923
+ "batch_size": 2048,
924
+ "n_epochs": 100,
925
+ "input_preprocess": "center_scale"
926
+ }
927
+ },
928
+ {
929
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step14000.safetensors",
930
+ "model_name": "EleutherAI/pythia-160m",
931
+ "model": "pythia-160m",
932
+ "matrix": "W_U",
933
+ "kind": "per-snapshot-saes",
934
+ "d_sae": 8192,
935
+ "seed": 0,
936
+ "n_snapshots": 1,
937
+ "arch": "jumprelu-crosscoder",
938
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 14000, d_sae=8192",
939
+ "quality": {
940
+ "explained_variance": 0.7912227686750525,
941
+ "mean_l0": 996.466455078125,
942
+ "dead_rate": 0.0
943
+ },
944
+ "training": {
945
+ "lr": 5e-05,
946
+ "l1_coefficient": 0.3,
947
+ "batch_size": 2048,
948
+ "n_epochs": 100,
949
+ "input_preprocess": "center_scale"
950
+ }
951
+ },
952
+ {
953
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step143000.safetensors",
954
+ "model_name": "EleutherAI/pythia-160m",
955
+ "model": "pythia-160m",
956
+ "matrix": "W_U",
957
+ "kind": "per-snapshot-saes",
958
+ "d_sae": 8192,
959
+ "seed": 0,
960
+ "n_snapshots": 1,
961
+ "arch": "jumprelu-crosscoder",
962
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 143000, d_sae=8192",
963
+ "quality": {
964
+ "explained_variance": 0.8244679775563315,
965
+ "mean_l0": 924.0657104492187,
966
+ "dead_rate": 0.0
967
+ },
968
+ "training": {
969
+ "lr": 5e-05,
970
+ "l1_coefficient": 0.3,
971
+ "batch_size": 2048,
972
+ "n_epochs": 100,
973
+ "input_preprocess": "center_scale"
974
+ }
975
+ },
976
+ {
977
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step16.safetensors",
978
+ "model_name": "EleutherAI/pythia-160m",
979
+ "model": "pythia-160m",
980
+ "matrix": "W_U",
981
+ "kind": "per-snapshot-saes",
982
+ "d_sae": 8192,
983
+ "seed": 0,
984
+ "n_snapshots": 1,
985
+ "arch": "jumprelu-crosscoder",
986
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 16, d_sae=8192",
987
+ "quality": {
988
+ "explained_variance": 0.7288671871864183,
989
+ "mean_l0": 1150.0855908203125,
990
+ "dead_rate": 0.0
991
+ },
992
+ "training": {
993
+ "lr": 5e-05,
994
+ "l1_coefficient": 0.3,
995
+ "batch_size": 2048,
996
+ "n_epochs": 100,
997
+ "input_preprocess": "center_scale"
998
+ }
999
+ },
1000
+ {
1001
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step2.safetensors",
1002
+ "model_name": "EleutherAI/pythia-160m",
1003
+ "model": "pythia-160m",
1004
+ "matrix": "W_U",
1005
+ "kind": "per-snapshot-saes",
1006
+ "d_sae": 8192,
1007
+ "seed": 0,
1008
+ "n_snapshots": 1,
1009
+ "arch": "jumprelu-crosscoder",
1010
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 2, d_sae=8192",
1011
+ "quality": {
1012
+ "explained_variance": 0.72886850366392,
1013
+ "mean_l0": 1150.0986572265624,
1014
+ "dead_rate": 0.0
1015
+ },
1016
+ "training": {
1017
+ "lr": 5e-05,
1018
+ "l1_coefficient": 0.3,
1019
+ "batch_size": 2048,
1020
+ "n_epochs": 100,
1021
+ "input_preprocess": "center_scale"
1022
+ }
1023
+ },
1024
+ {
1025
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step2000.safetensors",
1026
+ "model_name": "EleutherAI/pythia-160m",
1027
+ "model": "pythia-160m",
1028
+ "matrix": "W_U",
1029
+ "kind": "per-snapshot-saes",
1030
+ "d_sae": 8192,
1031
+ "seed": 0,
1032
+ "n_snapshots": 1,
1033
+ "arch": "jumprelu-crosscoder",
1034
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 2000, d_sae=8192",
1035
+ "quality": {
1036
+ "explained_variance": 0.7992791212506758,
1037
+ "mean_l0": 969.1734448242188,
1038
+ "dead_rate": 0.0
1039
+ },
1040
+ "training": {
1041
+ "lr": 5e-05,
1042
+ "l1_coefficient": 0.3,
1043
+ "batch_size": 2048,
1044
+ "n_epochs": 100,
1045
+ "input_preprocess": "center_scale"
1046
+ }
1047
+ },
1048
+ {
1049
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step21000.safetensors",
1050
+ "model_name": "EleutherAI/pythia-160m",
1051
+ "model": "pythia-160m",
1052
+ "matrix": "W_U",
1053
+ "kind": "per-snapshot-saes",
1054
+ "d_sae": 8192,
1055
+ "seed": 0,
1056
+ "n_snapshots": 1,
1057
+ "arch": "jumprelu-crosscoder",
1058
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 21000, d_sae=8192",
1059
+ "quality": {
1060
+ "explained_variance": 0.7907924441773572,
1061
+ "mean_l0": 998.4695678710938,
1062
+ "dead_rate": 0.0
1063
+ },
1064
+ "training": {
1065
+ "lr": 5e-05,
1066
+ "l1_coefficient": 0.3,
1067
+ "batch_size": 2048,
1068
+ "n_epochs": 100,
1069
+ "input_preprocess": "center_scale"
1070
+ }
1071
+ },
1072
+ {
1073
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step256.safetensors",
1074
+ "model_name": "EleutherAI/pythia-160m",
1075
+ "model": "pythia-160m",
1076
+ "matrix": "W_U",
1077
+ "kind": "per-snapshot-saes",
1078
+ "d_sae": 8192,
1079
+ "seed": 0,
1080
+ "n_snapshots": 1,
1081
+ "arch": "jumprelu-crosscoder",
1082
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 256, d_sae=8192",
1083
+ "quality": {
1084
+ "explained_variance": 0.7317386038736777,
1085
+ "mean_l0": 1142.36794921875,
1086
+ "dead_rate": 0.0
1087
+ },
1088
+ "training": {
1089
+ "lr": 5e-05,
1090
+ "l1_coefficient": 0.3,
1091
+ "batch_size": 2048,
1092
+ "n_epochs": 100,
1093
+ "input_preprocess": "center_scale"
1094
+ }
1095
+ },
1096
+ {
1097
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step27000.safetensors",
1098
+ "model_name": "EleutherAI/pythia-160m",
1099
+ "model": "pythia-160m",
1100
+ "matrix": "W_U",
1101
+ "kind": "per-snapshot-saes",
1102
+ "d_sae": 8192,
1103
+ "seed": 0,
1104
+ "n_snapshots": 1,
1105
+ "arch": "jumprelu-crosscoder",
1106
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 27000, d_sae=8192",
1107
+ "quality": {
1108
+ "explained_variance": 0.7907578681617007,
1109
+ "mean_l0": 999.3178100585938,
1110
+ "dead_rate": 0.0
1111
+ },
1112
+ "training": {
1113
+ "lr": 5e-05,
1114
+ "l1_coefficient": 0.3,
1115
+ "batch_size": 2048,
1116
+ "n_epochs": 100,
1117
+ "input_preprocess": "center_scale"
1118
+ }
1119
+ },
1120
+ {
1121
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step3000.safetensors",
1122
+ "model_name": "EleutherAI/pythia-160m",
1123
+ "model": "pythia-160m",
1124
+ "matrix": "W_U",
1125
+ "kind": "per-snapshot-saes",
1126
+ "d_sae": 8192,
1127
+ "seed": 0,
1128
+ "n_snapshots": 1,
1129
+ "arch": "jumprelu-crosscoder",
1130
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 3000, d_sae=8192",
1131
+ "quality": {
1132
+ "explained_variance": 0.7993412040992963,
1133
+ "mean_l0": 972.4029614257812,
1134
+ "dead_rate": 0.0
1135
+ },
1136
+ "training": {
1137
+ "lr": 5e-05,
1138
+ "l1_coefficient": 0.3,
1139
+ "batch_size": 2048,
1140
+ "n_epochs": 100,
1141
+ "input_preprocess": "center_scale"
1142
+ }
1143
+ },
1144
+ {
1145
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step32.safetensors",
1146
+ "model_name": "EleutherAI/pythia-160m",
1147
+ "model": "pythia-160m",
1148
+ "matrix": "W_U",
1149
+ "kind": "per-snapshot-saes",
1150
+ "d_sae": 8192,
1151
+ "seed": 0,
1152
+ "n_snapshots": 1,
1153
+ "arch": "jumprelu-crosscoder",
1154
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 32, d_sae=8192",
1155
+ "quality": {
1156
+ "explained_variance": 0.7288681571269175,
1157
+ "mean_l0": 1150.0745947265625,
1158
+ "dead_rate": 0.0
1159
+ },
1160
+ "training": {
1161
+ "lr": 5e-05,
1162
+ "l1_coefficient": 0.3,
1163
+ "batch_size": 2048,
1164
+ "n_epochs": 100,
1165
+ "input_preprocess": "center_scale"
1166
+ }
1167
+ },
1168
+ {
1169
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step34000.safetensors",
1170
+ "model_name": "EleutherAI/pythia-160m",
1171
+ "model": "pythia-160m",
1172
+ "matrix": "W_U",
1173
+ "kind": "per-snapshot-saes",
1174
+ "d_sae": 8192,
1175
+ "seed": 0,
1176
+ "n_snapshots": 1,
1177
+ "arch": "jumprelu-crosscoder",
1178
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 34000, d_sae=8192",
1179
+ "quality": {
1180
+ "explained_variance": 0.7906181564574442,
1181
+ "mean_l0": 999.5078881835938,
1182
+ "dead_rate": 0.0
1183
+ },
1184
+ "training": {
1185
+ "lr": 5e-05,
1186
+ "l1_coefficient": 0.3,
1187
+ "batch_size": 2048,
1188
+ "n_epochs": 100,
1189
+ "input_preprocess": "center_scale"
1190
+ }
1191
+ },
1192
+ {
1193
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step4.safetensors",
1194
+ "model_name": "EleutherAI/pythia-160m",
1195
+ "model": "pythia-160m",
1196
+ "matrix": "W_U",
1197
+ "kind": "per-snapshot-saes",
1198
+ "d_sae": 8192,
1199
+ "seed": 0,
1200
+ "n_snapshots": 1,
1201
+ "arch": "jumprelu-crosscoder",
1202
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 4, d_sae=8192",
1203
+ "quality": {
1204
+ "explained_variance": 0.7288676993559831,
1205
+ "mean_l0": 1150.087177734375,
1206
+ "dead_rate": 0.0
1207
+ },
1208
+ "training": {
1209
+ "lr": 5e-05,
1210
+ "l1_coefficient": 0.3,
1211
+ "batch_size": 2048,
1212
+ "n_epochs": 100,
1213
+ "input_preprocess": "center_scale"
1214
+ }
1215
+ },
1216
+ {
1217
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step4000.safetensors",
1218
+ "model_name": "EleutherAI/pythia-160m",
1219
+ "model": "pythia-160m",
1220
+ "matrix": "W_U",
1221
+ "kind": "per-snapshot-saes",
1222
+ "d_sae": 8192,
1223
+ "seed": 0,
1224
+ "n_snapshots": 1,
1225
+ "arch": "jumprelu-crosscoder",
1226
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 4000, d_sae=8192",
1227
+ "quality": {
1228
+ "explained_variance": 0.7977749784323522,
1229
+ "mean_l0": 977.3629370117187,
1230
+ "dead_rate": 0.0
1231
+ },
1232
+ "training": {
1233
+ "lr": 5e-05,
1234
+ "l1_coefficient": 0.3,
1235
+ "batch_size": 2048,
1236
+ "n_epochs": 100,
1237
+ "input_preprocess": "center_scale"
1238
+ }
1239
+ },
1240
+ {
1241
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step47000.safetensors",
1242
+ "model_name": "EleutherAI/pythia-160m",
1243
+ "model": "pythia-160m",
1244
+ "matrix": "W_U",
1245
+ "kind": "per-snapshot-saes",
1246
+ "d_sae": 8192,
1247
+ "seed": 0,
1248
+ "n_snapshots": 1,
1249
+ "arch": "jumprelu-crosscoder",
1250
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 47000, d_sae=8192",
1251
+ "quality": {
1252
+ "explained_variance": 0.790620026518176,
1253
+ "mean_l0": 1000.2795385742188,
1254
+ "dead_rate": 0.0
1255
+ },
1256
+ "training": {
1257
+ "lr": 5e-05,
1258
+ "l1_coefficient": 0.3,
1259
+ "batch_size": 2048,
1260
+ "n_epochs": 100,
1261
+ "input_preprocess": "center_scale"
1262
+ }
1263
+ },
1264
+ {
1265
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step5000.safetensors",
1266
+ "model_name": "EleutherAI/pythia-160m",
1267
+ "model": "pythia-160m",
1268
+ "matrix": "W_U",
1269
+ "kind": "per-snapshot-saes",
1270
+ "d_sae": 8192,
1271
+ "seed": 0,
1272
+ "n_snapshots": 1,
1273
+ "arch": "jumprelu-crosscoder",
1274
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 5000, d_sae=8192",
1275
+ "quality": {
1276
+ "explained_variance": 0.7960998419245482,
1277
+ "mean_l0": 981.8336376953125,
1278
+ "dead_rate": 0.0
1279
+ },
1280
+ "training": {
1281
+ "lr": 5e-05,
1282
+ "l1_coefficient": 0.3,
1283
+ "batch_size": 2048,
1284
+ "n_epochs": 100,
1285
+ "input_preprocess": "center_scale"
1286
+ }
1287
+ },
1288
+ {
1289
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step512.safetensors",
1290
+ "model_name": "EleutherAI/pythia-160m",
1291
+ "model": "pythia-160m",
1292
+ "matrix": "W_U",
1293
+ "kind": "per-snapshot-saes",
1294
+ "d_sae": 8192,
1295
+ "seed": 0,
1296
+ "n_snapshots": 1,
1297
+ "arch": "jumprelu-crosscoder",
1298
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 512, d_sae=8192",
1299
+ "quality": {
1300
+ "explained_variance": 0.7542853862655656,
1301
+ "mean_l0": 1086.609990234375,
1302
+ "dead_rate": 0.0
1303
+ },
1304
+ "training": {
1305
+ "lr": 5e-05,
1306
+ "l1_coefficient": 0.3,
1307
+ "batch_size": 2048,
1308
+ "n_epochs": 100,
1309
+ "input_preprocess": "center_scale"
1310
+ }
1311
+ },
1312
+ {
1313
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step6000.safetensors",
1314
+ "model_name": "EleutherAI/pythia-160m",
1315
+ "model": "pythia-160m",
1316
+ "matrix": "W_U",
1317
+ "kind": "per-snapshot-saes",
1318
+ "d_sae": 8192,
1319
+ "seed": 0,
1320
+ "n_snapshots": 1,
1321
+ "arch": "jumprelu-crosscoder",
1322
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 6000, d_sae=8192",
1323
+ "quality": {
1324
+ "explained_variance": 0.7947783648659816,
1325
+ "mean_l0": 985.470498046875,
1326
+ "dead_rate": 0.0
1327
+ },
1328
+ "training": {
1329
+ "lr": 5e-05,
1330
+ "l1_coefficient": 0.3,
1331
+ "batch_size": 2048,
1332
+ "n_epochs": 100,
1333
+ "input_preprocess": "center_scale"
1334
+ }
1335
+ },
1336
+ {
1337
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step61000.safetensors",
1338
+ "model_name": "EleutherAI/pythia-160m",
1339
+ "model": "pythia-160m",
1340
+ "matrix": "W_U",
1341
+ "kind": "per-snapshot-saes",
1342
+ "d_sae": 8192,
1343
+ "seed": 0,
1344
+ "n_snapshots": 1,
1345
+ "arch": "jumprelu-crosscoder",
1346
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 61000, d_sae=8192",
1347
+ "quality": {
1348
+ "explained_variance": 0.7904375219119095,
1349
+ "mean_l0": 1001.8048901367188,
1350
+ "dead_rate": 0.0
1351
+ },
1352
+ "training": {
1353
+ "lr": 5e-05,
1354
+ "l1_coefficient": 0.3,
1355
+ "batch_size": 2048,
1356
+ "n_epochs": 100,
1357
+ "input_preprocess": "center_scale"
1358
+ }
1359
+ },
1360
+ {
1361
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step64.safetensors",
1362
+ "model_name": "EleutherAI/pythia-160m",
1363
+ "model": "pythia-160m",
1364
+ "matrix": "W_U",
1365
+ "kind": "per-snapshot-saes",
1366
+ "d_sae": 8192,
1367
+ "seed": 0,
1368
+ "n_snapshots": 1,
1369
+ "arch": "jumprelu-crosscoder",
1370
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 64, d_sae=8192",
1371
+ "quality": {
1372
+ "explained_variance": 0.728867046997959,
1373
+ "mean_l0": 1150.0748681640625,
1374
+ "dead_rate": 0.0
1375
+ },
1376
+ "training": {
1377
+ "lr": 5e-05,
1378
+ "l1_coefficient": 0.3,
1379
+ "batch_size": 2048,
1380
+ "n_epochs": 100,
1381
+ "input_preprocess": "center_scale"
1382
+ }
1383
+ },
1384
+ {
1385
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step7000.safetensors",
1386
+ "model_name": "EleutherAI/pythia-160m",
1387
+ "model": "pythia-160m",
1388
+ "matrix": "W_U",
1389
+ "kind": "per-snapshot-saes",
1390
+ "d_sae": 8192,
1391
+ "seed": 0,
1392
+ "n_snapshots": 1,
1393
+ "arch": "jumprelu-crosscoder",
1394
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 7000, d_sae=8192",
1395
+ "quality": {
1396
+ "explained_variance": 0.7938394341974695,
1397
+ "mean_l0": 988.107509765625,
1398
+ "dead_rate": 0.0
1399
+ },
1400
+ "training": {
1401
+ "lr": 5e-05,
1402
+ "l1_coefficient": 0.3,
1403
+ "batch_size": 2048,
1404
+ "n_epochs": 100,
1405
+ "input_preprocess": "center_scale"
1406
+ }
1407
+ },
1408
+ {
1409
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step75000.safetensors",
1410
+ "model_name": "EleutherAI/pythia-160m",
1411
+ "model": "pythia-160m",
1412
+ "matrix": "W_U",
1413
+ "kind": "per-snapshot-saes",
1414
+ "d_sae": 8192,
1415
+ "seed": 0,
1416
+ "n_snapshots": 1,
1417
+ "arch": "jumprelu-crosscoder",
1418
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 75000, d_sae=8192",
1419
+ "quality": {
1420
+ "explained_variance": 0.790395516481567,
1421
+ "mean_l0": 1004.148408203125,
1422
+ "dead_rate": 0.0
1423
+ },
1424
+ "training": {
1425
+ "lr": 5e-05,
1426
+ "l1_coefficient": 0.3,
1427
+ "batch_size": 2048,
1428
+ "n_epochs": 100,
1429
+ "input_preprocess": "center_scale"
1430
+ }
1431
+ },
1432
+ {
1433
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step8.safetensors",
1434
+ "model_name": "EleutherAI/pythia-160m",
1435
+ "model": "pythia-160m",
1436
+ "matrix": "W_U",
1437
+ "kind": "per-snapshot-saes",
1438
+ "d_sae": 8192,
1439
+ "seed": 0,
1440
+ "n_snapshots": 1,
1441
+ "arch": "jumprelu-crosscoder",
1442
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 8, d_sae=8192",
1443
+ "quality": {
1444
+ "explained_variance": 0.7288677026395104,
1445
+ "mean_l0": 1150.0938623046875,
1446
+ "dead_rate": 0.0
1447
+ },
1448
+ "training": {
1449
+ "lr": 5e-05,
1450
+ "l1_coefficient": 0.3,
1451
+ "batch_size": 2048,
1452
+ "n_epochs": 100,
1453
+ "input_preprocess": "center_scale"
1454
+ }
1455
+ },
1456
+ {
1457
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step8000.safetensors",
1458
+ "model_name": "EleutherAI/pythia-160m",
1459
+ "model": "pythia-160m",
1460
+ "matrix": "W_U",
1461
+ "kind": "per-snapshot-saes",
1462
+ "d_sae": 8192,
1463
+ "seed": 0,
1464
+ "n_snapshots": 1,
1465
+ "arch": "jumprelu-crosscoder",
1466
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 8000, d_sae=8192",
1467
+ "quality": {
1468
+ "explained_variance": 0.7931325520036884,
1469
+ "mean_l0": 990.1047631835937,
1470
+ "dead_rate": 0.0
1471
+ },
1472
+ "training": {
1473
+ "lr": 5e-05,
1474
+ "l1_coefficient": 0.3,
1475
+ "batch_size": 2048,
1476
+ "n_epochs": 100,
1477
+ "input_preprocess": "center_scale"
1478
+ }
1479
+ },
1480
+ {
1481
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step89000.safetensors",
1482
+ "model_name": "EleutherAI/pythia-160m",
1483
+ "model": "pythia-160m",
1484
+ "matrix": "W_U",
1485
+ "kind": "per-snapshot-saes",
1486
+ "d_sae": 8192,
1487
+ "seed": 0,
1488
+ "n_snapshots": 1,
1489
+ "arch": "jumprelu-crosscoder",
1490
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 89000, d_sae=8192",
1491
+ "quality": {
1492
+ "explained_variance": 0.7924073402250866,
1493
+ "mean_l0": 1001.9916259765625,
1494
+ "dead_rate": 0.0
1495
+ },
1496
+ "training": {
1497
+ "lr": 5e-05,
1498
+ "l1_coefficient": 0.3,
1499
+ "batch_size": 2048,
1500
+ "n_epochs": 100,
1501
+ "input_preprocess": "center_scale"
1502
+ }
1503
+ },
1504
+ {
1505
+ "path": "pythia-160m/W_U/per-snapshot-saes/d8192/step9000.safetensors",
1506
+ "model_name": "EleutherAI/pythia-160m",
1507
+ "model": "pythia-160m",
1508
+ "matrix": "W_U",
1509
+ "kind": "per-snapshot-saes",
1510
+ "d_sae": 8192,
1511
+ "seed": 0,
1512
+ "n_snapshots": 1,
1513
+ "arch": "jumprelu-crosscoder",
1514
+ "public_label": "Pythia-160M W_U per-snapshot SAE at training step 9000, d_sae=8192",
1515
+ "quality": {
1516
+ "explained_variance": 0.7924251809723794,
1517
+ "mean_l0": 992.0130297851563,
1518
+ "dead_rate": 0.0
1519
+ },
1520
+ "training": {
1521
+ "lr": 5e-05,
1522
+ "l1_coefficient": 0.3,
1523
+ "batch_size": 2048,
1524
+ "n_epochs": 100,
1525
+ "input_preprocess": "center_scale"
1526
+ }
1527
+ },
1528
+ {
1529
+ "path": "pythia-1b/W_U/cross-snapshot-32/d16384/seed0.safetensors",
1530
+ "model_name": "EleutherAI/pythia-1b",
1531
+ "model": "pythia-1b",
1532
+ "matrix": "W_U",
1533
+ "kind": "cross-snapshot-32",
1534
+ "d_sae": 16384,
1535
+ "seed": 0,
1536
+ "n_snapshots": 32,
1537
+ "arch": "jumprelu-crosscoder",
1538
+ "public_label": "Pythia-1B W_U crosscoder, 32 snapshots, d_sae=16384, seed 0",
1539
+ "quality": {
1540
+ "explained_variance": 0.7812400637275599,
1541
+ "mean_l0": 499.1909545898437,
1542
+ "dead_rate": 0.0
1543
+ },
1544
+ "training": {
1545
+ "lr": 2e-05,
1546
+ "l1_coefficient": 0.3,
1547
+ "batch_size": 1024,
1548
+ "n_epochs": 300,
1549
+ "input_preprocess": "center_scale"
1550
+ }
1551
+ },
1552
+ {
1553
+ "path": "pythia-1b/W_U/cross-snapshot-32/d24576/seed0.safetensors",
1554
+ "model_name": "EleutherAI/pythia-1b",
1555
+ "model": "pythia-1b",
1556
+ "matrix": "W_U",
1557
+ "kind": "cross-snapshot-32",
1558
+ "d_sae": 24576,
1559
+ "seed": 0,
1560
+ "n_snapshots": 32,
1561
+ "arch": "jumprelu-crosscoder",
1562
+ "public_label": "Pythia-1B W_U crosscoder, 32 snapshots, d_sae=24576, seed 0",
1563
+ "quality": {
1564
+ "explained_variance": 0.861217602606241,
1565
+ "mean_l0": 517.0205554199218,
1566
+ "dead_rate": 0.0
1567
+ },
1568
+ "training": {
1569
+ "lr": 2e-05,
1570
+ "l1_coefficient": 0.3,
1571
+ "batch_size": 1024,
1572
+ "n_epochs": 300,
1573
+ "input_preprocess": "center_scale"
1574
+ }
1575
+ },
1576
+ {
1577
+ "path": "pythia-1b/W_U/cross-snapshot-32/d8192/seed0.safetensors",
1578
+ "model_name": "EleutherAI/pythia-1b",
1579
+ "model": "pythia-1b",
1580
+ "matrix": "W_U",
1581
+ "kind": "cross-snapshot-32",
1582
+ "d_sae": 8192,
1583
+ "seed": 0,
1584
+ "n_snapshots": 32,
1585
+ "arch": "jumprelu-crosscoder",
1586
+ "public_label": "Pythia-1B W_U crosscoder, 32 snapshots, d_sae=8192, seed 0",
1587
+ "quality": {
1588
+ "explained_variance": 0.628030172619609,
1589
+ "mean_l0": 374.1408770751953,
1590
+ "dead_rate": 0.0
1591
+ },
1592
+ "training": {
1593
+ "lr": 2e-05,
1594
+ "l1_coefficient": 0.3,
1595
+ "batch_size": 1024,
1596
+ "n_epochs": 300,
1597
+ "input_preprocess": "center_scale"
1598
+ }
1599
+ },
1600
+ {
1601
+ "path": "pythia-1b/W_U/cross-snapshot-32-matched-window/d24576/seed0.safetensors",
1602
+ "model_name": "EleutherAI/pythia-1b",
1603
+ "model": "pythia-1b",
1604
+ "matrix": "W_U",
1605
+ "kind": "cross-snapshot-32-matched-window",
1606
+ "d_sae": 24576,
1607
+ "seed": 0,
1608
+ "n_snapshots": 32,
1609
+ "arch": "jumprelu-crosscoder",
1610
+ "public_label": "Pythia-1B W_U crosscoder, OLMo-matched late-start 32-snapshot window, d_sae=24576, seed 0",
1611
+ "quality": {
1612
+ "explained_variance": 0.8844067825487556,
1613
+ "mean_l0": 263.98009787989025,
1614
+ "dead_rate": 0.0
1615
+ },
1616
+ "training": {
1617
+ "lr": 5e-05,
1618
+ "l1_coefficient": 0.3,
1619
+ "batch_size": 1024,
1620
+ "n_epochs": 300,
1621
+ "input_preprocess": "center_scale"
1622
+ }
1623
+ },
1624
+ {
1625
+ "path": "pythia-6.9b/W_U/cross-snapshot-32/d32768/seed0-sparse.safetensors",
1626
+ "model_name": "EleutherAI/pythia-6.9b",
1627
+ "model": "pythia-6.9b",
1628
+ "matrix": "W_U",
1629
+ "kind": "cross-snapshot-32",
1630
+ "d_sae": 32768,
1631
+ "seed": 0,
1632
+ "n_snapshots": 32,
1633
+ "arch": "jumprelu-crosscoder",
1634
+ "public_label": "Pythia-6.9B W_U crosscoder, 32 snapshots, d_sae=32768, seed 0, selected sparse run (lambda=0.6)",
1635
+ "quality": {
1636
+ "explained_variance": 0.8080785315044828,
1637
+ "mean_l0": 741.8141662287833,
1638
+ "dead_rate": 0.0
1639
+ },
1640
+ "training": {
1641
+ "lr": 1e-05,
1642
+ "l1_coefficient": 0.6,
1643
+ "batch_size": 1024,
1644
+ "n_epochs": 300,
1645
+ "input_preprocess": "center_scale"
1646
+ }
1647
+ },
1648
+ {
1649
+ "path": "pythia-6.9b/W_U/cross-snapshot-32/d32768/seed0.safetensors",
1650
+ "model_name": "EleutherAI/pythia-6.9b",
1651
+ "model": "pythia-6.9b",
1652
+ "matrix": "W_U",
1653
+ "kind": "cross-snapshot-32",
1654
+ "d_sae": 32768,
1655
+ "seed": 0,
1656
+ "n_snapshots": 32,
1657
+ "arch": "jumprelu-crosscoder",
1658
+ "public_label": "Pythia-6.9B W_U crosscoder, 32 snapshots, d_sae=32768, seed 0",
1659
+ "quality": {
1660
+ "explained_variance": 0.8331843988901371,
1661
+ "mean_l0": 1956.59887943171,
1662
+ "dead_rate": 0.0
1663
+ },
1664
+ "training": {
1665
+ "lr": 1e-05,
1666
+ "l1_coefficient": 0.3,
1667
+ "batch_size": 1024,
1668
+ "n_epochs": 300,
1669
+ "input_preprocess": "center_scale"
1670
+ }
1671
+ }
1672
+ ],
1673
+ "auxiliary": {
1674
+ "derived/aggregates": [
1675
+ "aggregates_dsae16384_seed0.pt",
1676
+ "aggregates_dsae24576_seed0.pt",
1677
+ "aggregates_dsae8192_seed0.pt",
1678
+ "aggregates_pythia-160m_d24576_seed0.pt",
1679
+ "aggregates_pythia-1b_d16384_seed0.pt",
1680
+ "aggregates_pythia-1b_d24576_seed0.pt",
1681
+ "aggregates_pythia-1b_d8192_seed0.pt",
1682
+ "aggregates_pythia-6.9b_d32768_seed0.pt"
1683
+ ],
1684
+ "derived/rates": [
1685
+ "we-d24576/we_cc_dsae24576_seed0_norms.npy",
1686
+ "we-d24576/we_rates_dsae24576_seed0.pt",
1687
+ "we-d8192-multiseed/we_cc_dsae8192_seed0_norms.npy",
1688
+ "we-d8192-multiseed/we_cc_dsae8192_seed1_norms.npy",
1689
+ "we-d8192-multiseed/we_cc_dsae8192_seed2_norms.npy",
1690
+ "we-d8192-multiseed/we_cc_dsae8192_seed3_norms.npy",
1691
+ "we-d8192-multiseed/we_cc_dsae8192_seed4_norms.npy",
1692
+ "we-d8192-multiseed/we_rates_dsae8192_seed0.pt",
1693
+ "we-d8192-multiseed/we_rates_dsae8192_seed1.pt",
1694
+ "we-d8192-multiseed/we_rates_dsae8192_seed2.pt",
1695
+ "we-d8192-multiseed/we_rates_dsae8192_seed3.pt",
1696
+ "we-d8192-multiseed/we_rates_dsae8192_seed4.pt",
1697
+ "wu-1b-d24576/decoder_norms_dsae24576_seed0.npy",
1698
+ "wu-d24576-multiseed/decoder_norms_dsae24576_seed0.npy",
1699
+ "wu-d24576-multiseed/decoder_norms_dsae24576_seed1.npy",
1700
+ "wu-d24576-multiseed/decoder_norms_dsae24576_seed2.npy",
1701
+ "wu-d24576-multiseed/wu_cc_dsae24576_seed0_norms.npy",
1702
+ "wu-d24576-multiseed/wu_cc_dsae24576_seed1_norms.npy",
1703
+ "wu-d24576-multiseed/wu_cc_dsae24576_seed2_norms.npy",
1704
+ "wu-d24576-multiseed/wu_rates_dsae24576_seed0.pt",
1705
+ "wu-d24576-multiseed/wu_rates_dsae24576_seed1.pt",
1706
+ "wu-d24576-multiseed/wu_rates_dsae24576_seed2.pt",
1707
+ "wu-d8192-cs16/wu_cc_dsae8192_seed0_norms.npy",
1708
+ "wu-d8192-cs16/wu_rates_dsae8192_seed0.pt",
1709
+ "wu-d8192-multiseed/decoder_norms_all_seeds.npy",
1710
+ "wu-d8192-multiseed/firing_rates_all_seeds.npy",
1711
+ "wu-d8192-persnap/train.log"
1712
+ ],
1713
+ "evaluation": [
1714
+ "eval-corpus/README.md",
1715
+ "eval-corpus/eval_tokens.pt"
1716
+ ],
1717
+ "attribution": [
1718
+ "pythia-160m/induction-results.pt",
1719
+ "pythia-160m/induction-verdict.json",
1720
+ "pythia-160m/ioi-results.pt",
1721
+ "pythia-160m/ioi-verdict.json",
1722
+ "pythia-160m/sva-results.pt",
1723
+ "pythia-160m/sva-verdict.json"
1724
+ ]
1725
+ }
1726
+ }
pythia-160m/W_E/cross-snapshot-32/d8192/seed0.metrics_recomputed.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "recomputed": {
3
+ "explained_variance": 0.5807971329492893,
4
+ "reconstruction_mse": 0.4182338160276413,
5
+ "mean_l0": 82.08435302734375,
6
+ "dead_rate": 0.0,
7
+ "d_sae": 8192,
8
+ "n_snapshots": 32,
9
+ "n_rows": 50304,
10
+ "n_batches": 50,
11
+ "preprocess_mode": "center_scale",
12
+ "model_name": "EleutherAI/pythia-160m",
13
+ "runtime_s": 53.8
14
+ },
15
+ "stored": {
16
+ "explained_variance": 0.5807971126242004,
17
+ "reconstruction_mse": 0.4182338178157806,
18
+ "mean_l0": 82.08435302734375,
19
+ "d_sae": 8192,
20
+ "n_snapshots": 32,
21
+ "n_rows": 50304
22
+ }
23
+ }
pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.config.json CHANGED
@@ -5,7 +5,8 @@
5
  "d_sae": 8192,
6
  "n_snapshots": 32,
7
  "d_model": 768,
8
- "batchtopk_k": null
 
9
  },
10
  "training": {
11
  "lr": 5e-05,
 
5
  "d_sae": 8192,
6
  "n_snapshots": 32,
7
  "d_model": 768,
8
+ "batchtopk_k": null,
9
+ "gated_l1_coefficient": 0.05
10
  },
11
  "training": {
12
  "lr": 5e-05,
pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.config.json CHANGED
@@ -5,7 +5,8 @@
5
  "d_sae": 8192,
6
  "n_snapshots": 32,
7
  "d_model": 768,
8
- "batchtopk_k": null
 
9
  },
10
  "training": {
11
  "lr": 5e-05,
@@ -70,5 +71,6 @@
70
  "model_name": "EleutherAI/pythia-160m",
71
  "seed": 0,
72
  "kind": "cross-snapshot",
73
- "preprocess_mode": "center_scale"
 
74
  }
 
5
  "d_sae": 8192,
6
  "n_snapshots": 32,
7
  "d_model": 768,
8
+ "batchtopk_k": null,
9
+ "gated_l1_coefficient": 0.3
10
  },
11
  "training": {
12
  "lr": 5e-05,
 
71
  "model_name": "EleutherAI/pythia-160m",
72
  "seed": 0,
73
  "kind": "cross-snapshot",
74
+ "preprocess_mode": "center_scale",
75
+ "note": "Default-lambda gated run (lambda=0.3 transferred from the JumpReLU recipe): low EV/L0 is the documented lambda-transfer failure, not a broken artifact. See gated-retuned (lambda=0.05) for the tuned comparison point."
76
  }
pythia-160m/W_U/cross-snapshot-16/d8192/seed0.metrics_recomputed.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "recomputed": {
3
+ "explained_variance": 0.7734329322856591,
4
+ "reconstruction_mse": 0.22638386040925979,
5
+ "mean_l0": 215.73648681640626,
6
+ "dead_rate": 0.0,
7
+ "d_sae": 8192,
8
+ "n_snapshots": 16,
9
+ "n_rows": 50304,
10
+ "n_batches": 50,
11
+ "preprocess_mode": "center_scale",
12
+ "model_name": "EleutherAI/pythia-160m",
13
+ "runtime_s": 12.7
14
+ },
15
+ "stored": {
16
+ "explained_variance": 0.7734329300485889,
17
+ "reconstruction_mse": 0.22638385832309724,
18
+ "mean_l0": 215.73649780273436,
19
+ "dead_rate": 0.0,
20
+ "d_sae": 8192,
21
+ "n_snapshots": 16,
22
+ "n_rows": 50304
23
+ }
24
+ }
pythia-160m/W_U/cross-snapshot-32/d16384/seed0.metrics_recomputed.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "recomputed": {
3
+ "explained_variance": 0.7802076050852021,
4
+ "reconstruction_mse": 0.21972236514091492,
5
+ "mean_l0": 103.07085510253906,
6
+ "dead_rate": 0.0,
7
+ "d_sae": 16384,
8
+ "n_snapshots": 32,
9
+ "n_rows": 50304,
10
+ "n_batches": 50,
11
+ "preprocess_mode": "center_scale",
12
+ "model_name": "EleutherAI/pythia-160m",
13
+ "runtime_s": 169.3
14
+ },
15
+ "stored": {
16
+ "explained_variance": 0.7802076016978367,
17
+ "reconstruction_mse": 0.21972236931324005,
18
+ "mean_l0": 103.0708544921875,
19
+ "d_sae": 16384,
20
+ "n_snapshots": 32,
21
+ "n_rows": 50304
22
+ }
23
+ }
pythia-160m/W_U/cross-snapshot-32/d24576/seed0.metrics_recomputed.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "recomputed": {
3
+ "explained_variance": 0.9196719720912131,
4
+ "reconstruction_mse": 0.08030243396759033,
5
+ "mean_l0": 285.9946276855469,
6
+ "dead_rate": 0.0,
7
+ "d_sae": 24576,
8
+ "n_snapshots": 32,
9
+ "n_rows": 50304,
10
+ "n_batches": 50,
11
+ "preprocess_mode": "center_scale",
12
+ "model_name": "EleutherAI/pythia-160m",
13
+ "runtime_s": 416.6
14
+ },
15
+ "stored": {
16
+ "explained_variance": 0.9196719747097695,
17
+ "reconstruction_mse": 0.08030243307352065,
18
+ "mean_l0": 285.9946594238281,
19
+ "dead_rate": 0.0,
20
+ "d_sae": 24576,
21
+ "n_snapshots": 32,
22
+ "n_rows": 50304
23
+ }
24
+ }
pythia-160m/W_U/cross-snapshot-32/d24576/seed1.metrics_recomputed.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "recomputed": {
3
+ "explained_variance": 0.9195844969046052,
4
+ "reconstruction_mse": 0.08038988128304482,
5
+ "mean_l0": 286.0998828125,
6
+ "dead_rate": 0.0,
7
+ "d_sae": 24576,
8
+ "n_snapshots": 32,
9
+ "n_rows": 50304,
10
+ "n_batches": 50,
11
+ "preprocess_mode": "center_scale",
12
+ "model_name": "EleutherAI/pythia-160m",
13
+ "runtime_s": 508.2
14
+ },
15
+ "stored": {
16
+ "explained_variance": 0.9195844996740984,
17
+ "reconstruction_mse": 0.08038988023996353,
18
+ "mean_l0": 286.09990661621094,
19
+ "dead_rate": 0.0,
20
+ "d_sae": 24576,
21
+ "n_snapshots": 32,
22
+ "n_rows": 50304
23
+ }
24
+ }