Merge branch 'master'

.gitattributes

@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+2.8b_l2 filter=lfs diff=lfs merge=lfs -text
+2.8b_l0 filter=lfs diff=lfs merge=lfs -text
+2.8b_l1 filter=lfs diff=lfs merge=lfs -text
+2.8b_l15 filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+checkpoints/*.pt filter=lfs diff=lfs merge=lfs -text

2.8b_l0/checkpoints/ae_390000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c34f8f87a979745fcb6cfa259fe49a17f51c87023e6e1805a69bb31b27b2cd9d
+size 839037024

2.8b_l1/checkpoints/ae_390000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1226f41519f04c462311f16c93476bd5f0c72b32a8eea3555e0df10a84155f64
+size 839037024

2.8b_l1/checkpoints/ae_740000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:52fcd56fdd9587095373f3bd7a5d91eb1358aea921244d39cceb7e810195628c
+size 839037024

2.8b_l1/checkpoints/ae_750000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2bd5ae298ffdfd942c7961b98ec6cc21087bd06714158d3ca2acb8ae09307b57
+size 839037024

2.8b_l15/checkpoints/ae_490000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b4c7ba419fee5dbbb5d446df44da5a824575e7e051b8b4203ef1869c4dc7b1c
+size 839037024

2.8b_l2/checkpoints/ae_390000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02638ada96593ac34ad77861ca51aea5dfdecdf88e707e444f47fa5c43d99b40
+size 839037024

README.md

@@ -1,3 +1,32 @@
+<<<<<<< HEAD
 ---
 license: apache-2.0
 ---
+=======
+# Trained Sparse Autoencoders on Pythia 2.8B
+I trained SAEs on the MLP_out activations of the Pythia 2.8B dataset. I trained using https://github.com/magikarp01/facts-sae.git, a fork of https://github.com/saprmarks/dictionary_learning.
+
+## SAE Setup
+- **Training Dataset**: Uncopyrighted Pile, at monology/pile-uncopyrighted
+- **Model**: 32-layer Pythia 2.8B
+- **Activation**: MLP_out
+- **Layers Trained**: 0, 1, 2, 15
+- **Batch Size**: 2048 for layer 15, 2560 for layers 0, 1, 2
+- **Training Tokens**: 1e9 for layers 15, 0, 2, slightly less than 2e9 for layer 1
+- **Training Steps**: 4e5 for layers 0, 2, 5e5 for layer 15, 7.5e5 for layer 1
+
+## Training Hyperparamaters
+- **Learning Rate**: 3e-4
+- **Sparsity Penalty**: 1e-3
+- **Warmup Steps**: 5000
+- **Resample Steps**: 50000
+- **Optimizer**: Constrained Adam
+- **Scheduler**: LambdaLR, linear warmup lr between 0 and warmup_steps
+
+## SAE Metrics
+![2.8b L0 390k Steps](results/2.8b_l0_390k.png)
+![2.8b L1 390k Steps](results/2.8b_l1_390k.png)
+![2.8b L1 740k Steps](results/2.8b_l1_740k.png)
+![2.8b L2 390k Steps](results/2.8b_l2_390k.png)
+![2.8b L15 490k Steps](results/2.8b_l15_490k.png)
+>>>>>>> master