--- license: apache-2.0 --- Test Pipeline (run2)
Benchmark WhirlwindAI/MetaNova-Test WhirlwindAI/MetaNova-0.1-BF16 SupraLabs/Supra-50M-Base
Params 70.55M 70.55M 51.79M
HellaSwag 25.37% 27.40% 31.65%
ARC-Easy 26.89% 31.73% 45.58%
ARC-Challenge 25.09% 25.00% 24.66%
PIQA 52.29% 58.54% 61.53%
ArithMark-2 24.12% 27.12% 26.08%
ARC Avg 25.99% 28.37% 35.12%
Final Avg 31.94% 35.36% 38.60%

DeepSeek and Qwen Format
Thinking Mode Non-Thinking Mode 
<|im_start|>user
{query} /think<|im_end|>
<|im_start|>assistant
<think>
{thinking_content}
</think>

{response}<|im_end|>
 
<|im_start|>user
{query} /no_think<|im_end|>
<|im_start|>assistant
<think>

</think>

{response}<|im_end|>