Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
CodeSoft 
posted an update 1 day ago
Post
806
Sorbet Mini Experimental is out!

It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.

Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.
Banaxi-Tech

Hi @CodeSoft We want to invite you to join https://huggingface.co/bananamind-discuss due to a decision on our BananaMind 2.1 models. I've invited you to the org, huggingface should have send you our invitation via email.

We would really like you to join it.

·
CodeSoft

Thanks for reaching out! Since I'm not part of the BananaMind project, could you clarify what kind of feedback you're looking for?
Hoglet-33

@CodeSoft If you want some tips for training decent models in this size range what i did/do is at least have fineweb-edu and cosmopedia-v2, and it would be recommended to have DCLM, math data like FineMath-4+, and a tiny bit of code data like python but for 5M you would probably be fine skipping code, use the Muon opimizer for 2D weights and AdamW for everything else for 5M and i would aim for at least 500M tokens, and i would settle for a 2048 vocab tokenizer just to leave the most room possible for non embedding params. I wish you the best of luck and i hope this helps
In this post