metadata
license: apache-2.0
September Update - this is the official model used in dot, as of 09/2024, it consistently outperforms every sota model below 700m parameters.
benchmarks
zero-shot evaluations performed on current sota ~0.5b models against the best language model below 2b parameters.
| Parameters | Model | MMLU | ARC-C | HellaSwag | PIQA | Winogrande | Average |
|---|---|---|---|---|---|---|---|
| 0.5b | qwen2 | 0.4413 | 0.2892 | 0.4905 | 0.6931 | 0.5699 | 0.4968 |
| 0.6b | mobilellm | - | 0.3580 | 0.5590 | 0.7230 | 0.5860 | - |
| 0.5b | danube3 | 0.2554 | 0.3626 | 0.6072 | 0.7432 | 0.6140 | 0.5164 |
| 1.1b | palmer | 0.2661 | 0.3490 | 0.6173 | 0.7481 | 0.6417 | 0.5244 |
| 0.5b | arco | 0.2617 | 0.3729 | 0.6288 | 0.7437 | 0.6227 | 0.5260 |
supporters
