metadata
license: apache-2.0
September Update - this is the official model used in dot, as of 09/2024, it consistently outperforms every sota model below 700m parameters.
benchmarks
zero-shot evaluations performed on current sota ~0.5b models against the best language model below 2b parameters.
| Parameters | Model | MMLU | ARC-C | HellaSwag | PIQA | Winogrande | Average |
|---|---|---|---|---|---|---|---|
| 0.5b | qwen2 | 0.4413 | 0.2892 | 0.4905 | 0.6931 | 0.5699 | 0.4968 |
| 0.6b | mobilellm | - | 0.3580 | 0.5590 | 0.7230 | 0.5860 | - |
| 0.5b | danube3 | 0.2481 | 0.3618 | 0.6046 | 0.7378 | 0.6101 | 0.5125 |
| 0.5b | palmer | 0.2617 | 0.3729 | 0.6288 | 0.7437 | 0.6227 | 0.5260 |
| 1.7b | smollm | 0.2765 | 0.4626 | 0.6574 | 0.7606 | 0.6093 | 0.5533 |
supporters
