End of training

README.md

@@ -15,7 +15,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 2.3180
+- Loss: 2.3188
 
 ## Model description
 

model-00001-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bd055300c3324c20c166c5f7e4afd48032101896e5a7fdf5652cbfae69196c1f
+oid sha256:2b8e3f0f4ee18f293ef8a0c738a2f07294f981f71de3dbf285804fe95830179f
 size 4943163992

model-00002-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:99d6e4309f63cdb9fe277e26c0b35b9ee752b28608a478df6e94f7310cad7f05
+oid sha256:ee2167da978685b018574d420a96d32ccd07c57bb4ccd69c268fc219f8dbcd90
 size 4999821144

model-00003-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7c60d789b7bdb792280d7f4baa1514003ad70385c2e000ba328707627e3d9744
+oid sha256:07599734b7f002e9cf552f9fccc5140536b656f50385aa0bca03f88173093d49
 size 4540517840

sparsification_sftt.py

@@ -384,7 +384,6 @@ class MistralSparseSiluMLP(MistralMLP):
             pre_act = self.gate_proj(x)
             post_act = self.act_fn(pre_act)
             if self.kill_sparse_swish_outputs:
-                print("POST ACT:", post_act.abs().mean())
                 dead_neurons = post_act.abs() <= self.dead_threshold
                 # print("pre act sparsity: ", (pre_act==0).float().mean())
 
@@ -400,7 +399,6 @@ class MistralSparseSiluMLP(MistralMLP):
                     ) / (self.visit_counts + 1)
                     self.visit_counts += 1
 
-                    print("DEAD PERCENTAGE: ", self.dead_percentage)
                     self.a = dead_percentage
 
                     # print(self.agg_sparsity)