Post
102
Hello, people of Hugging Face!
I recently released FlameF0X/TinyMoE-100m-2x8-retrained, a small Mixture of Experts language model trained on the Smollm-Corpus. Built on top of the Mixtral architecture, it’s fully compatible with 🤗 Transformers right out of the box!
The model can produce somewhat coherent text on its own, and for some reason, it generates even more coherent responses when given a ChatLM template.
I’m excited to see what you all come up with, and feel free to fine-tune it if you’d like. In the meantime, I’ll be working on developing the chat-trained version.
Demo: FlameF0X/TinyMoE-Playground
Collection: https://huggingface.co/collections/FlameF0X/tinymoe
I recently released FlameF0X/TinyMoE-100m-2x8-retrained, a small Mixture of Experts language model trained on the Smollm-Corpus. Built on top of the Mixtral architecture, it’s fully compatible with 🤗 Transformers right out of the box!
The model can produce somewhat coherent text on its own, and for some reason, it generates even more coherent responses when given a ChatLM template.
I’m excited to see what you all come up with, and feel free to fine-tune it if you’d like. In the meantime, I’ll be working on developing the chat-trained version.
Demo: FlameF0X/TinyMoE-Playground
Collection: https://huggingface.co/collections/FlameF0X/tinymoe