~64M (active) model trained on ~12.9B tokens of DCLM-Edu, UltraData-Math, and other assorted datasets.
Evals
Zero-shot
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 0 | acc | β | 0.2065 | Β± | 0.0118 |
| none | 0 | acc_norm | β | 0.2423 | Β± | 0.0125 | ||
| arc_easy | 1 | none | 0 | acc | β | 0.4651 | Β± | 0.0102 |
| none | 0 | acc_norm | β | 0.4213 | Β± | 0.0101 | ||
| hellaswag | 1 | none | 0 | acc | β | 0.2896 | Β± | 0.0045 |
| none | 0 | acc_norm | β | 0.3149 | Β± | 0.0046 | ||
| piqa | 1 | none | 0 | acc | β | 0.6251 | Β± | 0.0113 |
| none | 0 | acc_norm | β | 0.6110 | Β± | 0.0114 |
Five-shot
Gains from n-shot tasks aren't equal:
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | Ξ zero-shot | ||
|---|---|---|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 5 | acc | β | 0.2108 | Β± | 0.0119 | |
| none | 5 | acc_norm | β | 0.2398 | Β± | 0.0125 | stderr | ||
| arc_easy | 1 | none | 5 | acc | β | 0.4861 | Β± | 0.0103 | |
| none | 5 | acc_norm | β | 0.4714 | Β± | 0.0102 | +0.0501 | ||
| hellaswag | 1 | none | 5 | acc | β | 0.2876 | Β± | 0.0045 | |
| none | 5 | acc_norm | β | 0.3157 | Β± | 0.0046 | stderr | ||
| piqa | 1 | none | 5 | acc | β | 0.6192 | Β± | 0.0113 | |
| none | 5 | acc_norm | β | 0.6050 | Β± | 0.0114 | stderr |
Architecture Details
- 2.5x intermediate SwiGLU MLPs
- Block attention residuals (every 3 layers)
- Headwise gated attention (headwise chosen instead of elementwise to save params for the smaller model families)
- Learned attention sinks on every attention head
- SWA:SWA:GQA attention pattern, with the final layer forced to GQA
- 512 sliding window size
- 2048 native context length
- HoPE positional embeddings (on both SWA and GQA layers)
- One MTP head for auxiliary CE loss at 0.3x multiplier (currently unused at inference)
- Optimizers:
- Muon at 0.02 LR for matrices with Hyperball weight norm constraints
- AdamW at 0.002 LR for embeddings and vectors, norms, etc. with no weight decay or Hyperball
total parameter counts
- 67,397,848 total params
- 61,106,392 non-embedding params
- 67,397,848 trainable params
- 3,494,728 MTP params
- 63,903,120 params without MTP
Training Data
Full mix available here
Code
Pretraining codebase: https://code.allura.moe/fizz/OLMo-core
Other Benchmarks
====================================================================
Loading ../exported_hf...
====================================================================
Loading weights: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 326/326 [00:00<00:00, 2850.58it/s]
67,397,848 parameters (torch.bfloat16)
arithmark-3: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 32/32 [00:01<00:00, 21.67it/s]
arithmark-3: raw 35.40% (354/1000) normalized 35.50% (355/1000)
speed: 677.0 examples/s (tokenize 0.24s, evaluate 1.48s)
Category N Raw Normalized
----------------------------------------------------------------------------------------------
elementary_school_math_continuation::addition::grades_1_2::easy 128 20.31% 20.31%
elementary_school_math_continuation::comparison::grades_2_3::medium 44 18.18% 18.18%
elementary_school_math_continuation::comparison_difference::grades_2_3::medium 48 25.00% 25.00%
elementary_school_math_continuation::data::grades_2_3::easy 43 18.60% 18.60%
elementary_school_math_continuation::division::grades_3_4::medium 54 31.48% 31.48%
elementary_school_math_continuation::fractions_counting::grades_3_4::medium 50 24.00% 24.00%
elementary_school_math_continuation::geometry_area::grades_4_5::medium 52 59.62% 61.54%
elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium 45 57.78% 57.78%
elementary_school_math_continuation::measurement::grades_2_3::easy 76 38.16% 38.16%
elementary_school_math_continuation::money::grades_3_4::medium 64 28.12% 28.12%
elementary_school_math_continuation::multiplication::grades_3_4::medium 74 50.00% 50.00%
elementary_school_math_continuation::patterns::grades_3_4::medium 53 30.19% 30.19%
elementary_school_math_continuation::subtraction::grades_1_2::easy 117 28.21% 28.21%
elementary_school_math_continuation::time::grades_2_3::easy 55 100.00% 100.00%
elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium 46 21.74% 21.74%
elementary_school_math_continuation::two_step_addition::grades_2_3::medium 19 31.58% 31.58%
elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium 32 31.25% 31.25%
====================================================================
../exported_hf (67,397,848 params) RESULTS
====================================================================
Raw continuation accuracy 35.40%
Length-normalized accuracy 35.50%
Primary (acc_norm) 35.50%
====================================================================
- Downloads last month
- 5
Inference Providers NEW
This model isn't deployed by any Inference Provider. π Ask for provider support