| datasets: | |
| - HuggingFaceTB/smollm-corpus | |
| language: | |
| - en | |
| pipeline_tag: text-generation | |
| tags: | |
| - SLM | |
| - ROSA | |
| license: mit | |
| # SRLM-1M | |
| A **S**mall **R**OSA based **L**anguage **M**odel, of 900k parameters. | |
| ## Training inforation | |
| Trained on smollm-corpus fineweb-edu-dedup subset, over 16M tokens. | |
| ## Evaluations | |
| - Wikitext v2 byte_perplexity: 4.4553 | |
| - Arc_easy acc_norm: 28.66% | |
| - Arc_easy acc: 26.39% | |
| - Blimp acc: 53.1% | |
| - Hellaswag acc: 26.65% | |
| - Hellaswag acc_norm: 27.07% | |
| ## How to run it | |
| To run this model, you have to: | |
| 1. Download the files | |
| 2. Compile rosa.pyx using `python setup.py build_ext --inplace` | |
| 3. Load the model with the correct configuration | |
| 4. Call the model |