| --- |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| - HuggingFaceTB/cosmopedia |
| - epfml/FineWeb2-HQ |
| language: |
| - en |
| --- |
| An ultra-tiny 28 million parameter model trained on 2.6 billion tokens using a custom dataset mixture. Context length of 1024 tokens. |
|
|
| | Dataset | Weight | |
| |---|---| |
| | `HuggingFaceFW/fineweb-edu` | 50% | |
| | `epfml/FineWeb-HQ` | 30% | |
| | `HuggingFaceTB/cosmopedia` (stories split) | 20% | |
|
|
| The tokenizer is a basic bpe tokenizer that was trained on a smaller subset of 80_000 samples of this same data mixture with a vocab size of 8000. |
| |
| This model has not undergone any post-training. |
| |
| This base model is best suited for fine-tuning on specific tasks. On its own, it is very limited, but it is a pretty flexible foundation for applications such as toxic comment detection or sentiment analysis. |
| |
| | Benchmark | Metric | Value | |
| |---|---|---| |
| | ARC-Challenge | Acc | 0.1980 | |
| | ARC-Challenge | Acc Norm | 0.2312 | |
| | ARC-Easy | Acc | 0.4091 | |
| | ARC-Easy | Acc Norm | 0.3859 | |
| | BoolQ | Acc | 0.5850 | |
| | HellaSwag | Acc | 0.2749 | |
| | HellaSwag | Acc Norm | 0.2823 | |
| | PIQA | Acc | 0.5963 | |
| | PIQA | Acc Norm | 0.5762 | |
| | WinoGrande | Acc | 0.4917 | |
| | WikiText | Word PPL | 86.6418 | |
| | WikiText | Byte PPL | 2.3034 | |
| | WikiText | Bits/Byte | 1.2037 | |
| | BLIMP (Avg) | Acc | 0.6975 | |
