README.md

metadata

datasets:
  - HuggingFaceTB/smollm-corpus
language:
  - en
pipeline_tag: text-generation
tags:
  - SLM
  - ROSA
license: mit

SRLM-1M

A Small ROSA based Language Model, of 900k parameters.

Training inforation

Trained on smollm-corpus fineweb-edu-dedup subset, over 16M tokens.

Evaluations

• Wikitext v2 byte_perplexity: 4.4553
• Arc_easy acc_norm: 28.66%
• Arc_easy acc: 26.39%
• Blimp acc: 53.1%
• Hellaswag acc: 26.65%
• Hellaswag acc_norm: 27.07%

How to run it

To run this model, you have to:

1. Download the files
2. Compile rosa.pyx using python setup.py build_ext --inplace
3. Load the model with the correct configuration
4. Call the model