metadata
license: apache-2.0
language:
- en
MetaNova 1
README.md currently being written... ☕
| Benchmark | MetaNova-Test | MetaNova-0.1 | MetaNova-1 | Supra-50M-Base | OpenAI/GPT-2 |
|---|---|---|---|---|---|
| Params | 70.55M | 70.55M | 62.70M | 51.79M | 124M |
| HellaSwag | 25.37% | 27.40% | 27.08% | 31.65% | 31.26% |
| ARC-Easy | 26.89% | 31.73% | 30.56% | 45.58% | 39.35% |
| ARC-Challenge | 25.09% | 25.00% | 24.15% | 24.66% | 22.35% |
| PIQA | 52.29% | 58.54% | 56.04% | 61.53% | 62.08% |
| ArithMark-2 | 24.12% | 27.12% | 34.12% | 26.08% | 26.48% |
| ARC Avg | 25.99% | 28.37% | 27.35% | 35.12% | 30.85% |
| Final Avg | 31.94% | 35.36% | 36.15% | 38.60% | 37.67% |
DeepSeek and Qwen Format
| Thinking Mode | Non-Thinking Mode |
|---|---|
<|im_start|>user {query} /think<|im_end|> <|im_start|>assistant <think> {thinking_content} </think> |
<|im_start|>user {query} /no_think<|im_end|> <|im_start|>assistant <think> </think> {response}<|im_end|> |