Not-really-but-originally-Olmo-3 architecture 10M model with 2:1 SWA:GQA and HoPE embeddings. Trained on like 17B tokens of Fineweb-Edu (filtered to the splits before ChatGPT's release), DCLM-Edu and UltraData-Math.
(arch stability experiment more than anything)
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 0 | acc | ↑ | 0.1766 | ± | 0.0111 |
| none | 0 | acc_norm | ↑ | 0.2176 | ± | 0.0121 | ||
| arc_easy | 1 | none | 0 | acc | ↑ | 0.3721 | ± | 0.0099 |
| none | 0 | acc_norm | ↑ | 0.3460 | ± | 0.0098 | ||
| hellaswag | 1 | none | 0 | acc | ↑ | 0.2665 | ± | 0.0044 |
| none | 0 | acc_norm | ↑ | 0.2691 | ± | 0.0044 | ||
| piqa | 1 | none | 0 | acc | ↑ | 0.5631 | ± | 0.0116 |
| none | 0 | acc_norm | ↑ | 0.5539 | ± | 0.0116 |
Loading weights: 100% 156/156 [00:00<00:00, 10204.64it/s]
10,762,998 parameters (torch.bfloat16)
arithmark-3: 100% 32/32 [00:00<00:00, 34.61it/s]
arithmark-3: raw 31.70% (317/1000) normalized 31.90% (319/1000)
speed: 1081.5 examples/s (tokenize 0.17s, evaluate 0.92s)
Category N Raw Normalized
----------------------------------------------------------------------------------------------
elementary_school_math_continuation::addition::grades_1_2::easy 128 19.53% 19.53%
elementary_school_math_continuation::comparison::grades_2_3::medium 44 25.00% 25.00%
elementary_school_math_continuation::comparison_difference::grades_2_3::medium 48 20.83% 20.83%
elementary_school_math_continuation::data::grades_2_3::easy 43 27.91% 25.58%
elementary_school_math_continuation::division::grades_3_4::medium 54 31.48% 27.78%
elementary_school_math_continuation::fractions_counting::grades_3_4::medium 50 20.00% 24.00%
elementary_school_math_continuation::geometry_area::grades_4_5::medium 52 51.92% 51.92%
elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium 45 28.89% 28.89%
elementary_school_math_continuation::measurement::grades_2_3::easy 76 39.47% 40.79%
elementary_school_math_continuation::money::grades_3_4::medium 64 25.00% 25.00%
elementary_school_math_continuation::multiplication::grades_3_4::medium 74 41.89% 41.89%
elementary_school_math_continuation::patterns::grades_3_4::medium 53 26.42% 26.42%
elementary_school_math_continuation::subtraction::grades_1_2::easy 117 19.66% 21.37%
elementary_school_math_continuation::time::grades_2_3::easy 55 100.00% 100.00%
elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium 46 23.91% 23.91%
elementary_school_math_continuation::two_step_addition::grades_2_3::medium 19 15.79% 15.79%
elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium 32 28.12% 28.12%
====================================================================
../exported_hf (10,762,998 params) RESULTS
====================================================================
Raw continuation accuracy 31.70%
Length-normalized accuracy 31.90%
Primary (acc_norm) 31.90%
====================================================================
- Downloads last month
- 26
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support