jln.shk_64x4: JSAE (resid_mid -> resid_out) 7.5k steps
j.mlp_layer.shk_64x4: JSAE (mlp_in -> mlp_out) 7.5k steps
David Quarel
davidquarel
·
AI & ML interests
None yet
Recent Activity
updated
a model
about 2 months ago
davidquarel/jaxgmg_smoke
updated
a model
about 2 months ago
davidquarel/jaxgmg_test
updated
a model
about 2 months ago
davidquarel/jaxgmg_test
Organizations
Pairs of Jacobian SAEs, 8 in total.
Each pair is placed across the MLP blocks,
one on resid_mid, one on resid_post. Uses DyT
A sweep of JSAE training with jacobian penalty from 1e-3 to 1e-2, trained for 20k steps. (added zero sparsity for baseline)
Depricated. Just a bunch of bad SAEs. Probably don't use. Trained between transformer blocks
Jacobian SAEs with all combo of jacobian_coeff - Layer0 in (2e-4, 4e-4, 6e-4, 8e-4)
Layer1 = 3e-3
Layer2 = 3e-3
Layer3 in (2e-2, 4e-2, 6e-2, 8e-2)
-
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_8.0e-02
216k • Updated -
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_6.0e-02
216k • Updated -
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_4.0e-02
216k • Updated -
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_2.0e-02
216k • Updated
A collection of SAEs, trained on the input and output of each MLP layer of Peter's shakespeare_64x4 GPT model
jln.shk_64x4: JSAE (resid_mid -> resid_out) 7.5k steps
j.mlp_layer.shk_64x4: JSAE (mlp_in -> mlp_out) 7.5k steps
Pairs of Jacobian SAEs, 8 in total.
Each pair is placed across the MLP blocks,
one on resid_mid, one on resid_post. Uses DyT
Jacobian SAEs with all combo of jacobian_coeff - Layer0 in (2e-4, 4e-4, 6e-4, 8e-4)
Layer1 = 3e-3
Layer2 = 3e-3
Layer3 in (2e-2, 4e-2, 6e-2, 8e-2)
-
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_8.0e-02
216k • Updated -
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_6.0e-02
216k • Updated -
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_4.0e-02
216k • Updated -
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_2.0e-02
216k • Updated
A sweep of JSAE training with jacobian penalty from 1e-3 to 1e-2, trained for 20k steps. (added zero sparsity for baseline)
A collection of SAEs, trained on the input and output of each MLP layer of Peter's shakespeare_64x4 GPT model
Depricated. Just a bunch of bad SAEs. Probably don't use. Trained between transformer blocks