Create README.md

README.md

@@ -0,0 +1,9 @@
+---
+language:
+- en
+---
+A tiny 55 million parameter trained on 1.3 billion tokens on a custom dataset mixture.
+    HuggingFaceFW/fineweb-edu 50%
+    epfml/FineWeb-HQ 30%
+    HuggingFaceTB/cosmopedia (stories split) 20%
+The tokenizer is a basic bpe tokenizer that was trained on a smaller subset of 80_000 samples of this same data mixture with a vocab size of 8000.