Spaces:
Running
Running
| title: README | |
| emoji: 💤 | |
| colorFrom: indigo | |
| colorTo: blue | |
| sdk: static | |
| pinned: true | |
| We build **stuff** and **share it** with the community. | |
| That's pretty much it. And yes, "stuff" was the right word to use. | |
| ### Why "**FromZero**"? | |
| **FromZero** is inspired by **Re:Zero - Starting Life in Another World**. Just like the main character, all of our models begin from zero. | |
| Thankfully, they don't spend their time dying repeatedly or having mental breakdowns. | |
| I mean, training is already painful enough. | |
| ### Our goal | |
| Well, there isn't actually a definitive goal we're chasing. I mean, yeah, we want to be the **greatest**. But so does everyone else. | |
| We're just here to build ***weird*** models, share them with the community, and grow side-by-side with everyone. | |
| ### Current Models | |
| - **Syn-2.6M**: A 2.6M-parameter Tiny Language Model (TLM) trained only on synthetic data (tiny-textbooks, tiny-stories-v2, longpage, and finephrase). | |
| - **Er-Medium**: A 12.4M-parameter model trained on 34B tokens. It achieves state-of-the-art performance on many benchmarks. | |
| ### Future Releases | |
| - **Orez** - Tiny (1.3M), Small (5M), Medium (12.4M), Large (30M): A family of Small Language Models ranging from 1.2M to 30M parameters. Each of them will be massively overtrained on a minium of 1000+ tokens per parameter. | |
| ### Discord | |
| Join our Discord community: [FromZero](https://discord.gg/mGPMfme9s). | |
| ### Website | |
| Our website: **[AskJASP](https://askjasp.com)** |