Update README.md

README.md

@@ -1,7 +1,29 @@
----
-license: mit
-datasets:
-- tiiuae/falcon-refinedweb
-language:
-- en
----
+## Description
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+Models trained on 300B tokens, including dense FFN ones and low-rank FFN ones.
+
+
+
+## Citation 
+
+If you find it useful, please consider citing the paper:
+
+```
+@article{wei2024building,
+  title={Building on efficient foundations: Effective training of LLMs with structured feedforward layers},
+  author={Wei, Xiuying and Moalla, Skander and Pascanu, Razvan and Gulcehre, Caglar},
+  journal={Advances in Neural Information Processing Systems},
+  volume={37},
+  pages={4689--4717},
+  year={2024}
+}
+
+@article{wei2024investigating,
+  title={Investigating low-rank training in transformer language models: Efficiency and scaling analysis},
+  author={Wei, Xiuying and Moalla, Skander and Pascanu, Razvan and Gulcehre, Caglar},
+  journal={arXiv preprint arXiv:2407.09835},
+  year={2024}
+}
+```