Hugging Face's logo

stepfun-ai
/
Step-3.5-Flash-Base-Midtrain

Text Generation
Transformers
Safetensors
step3p5
conversational
custom_code
Model card Files
xet
Community
4
Randomize commited on
Commit
acb3a16
·
verified ·
1 Parent(s): ef79978

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +13 -13
README.md CHANGED
@@ -55,23 +55,23 @@ Performance of Step 3.5 Flash measured across **Reasoning**, **Coding**, and **A
55
  | # Activated Params | - | 11B | 11B | 15B | 32B | 37B | 37B | 32B |
56
  | # Total Params | - | 196B | 196B | 309B | 355B | 671B | 671B | 1043B |
57
  | General | | | | | | | | |
58
- | BBH | 3-shot | 87.6 | 88.2 | 88.5 | 86.2 | 88.2† | 88.7† | 88.7 |
59
- | MMLU | 5-shot | 83.5 | 85.8 | 86.7 | 86.1 | 87.4† | 87.8† | 87.8 |
60
  | MMLU‑Redux | 5-shot | 87.2 | 89.2 | 90.6 | - | 90.0† | 90.4† | 90.2 |
61
- | MMLU‑Pro | 5-shot | 60.4 | 62.3 | 73.2 | - | 58.8† | 62.1† | 69.2 |
62
- | HellaSwag | 10-shot | 91.2 | 90.2 | 88.5 | 87.1 | 89.2† | 89.4† | 94.6 |
63
- | WinoGrande | 5-shot | 77.0 | 79.1 | 83.8 | - | 85.9† | 85.6† | 85.3 |
64
- | GPQA | 5-shot | 42.2 | 41.7 | 43.5* | 33.5* | 43.1* | 37.3* | 43.1* |
65
  | SuperGPQA | 5-shot | 41.2 | 41.0 | 41.1 | - | 42.3† | 43.6† | 44.7 |
66
- | SimpleQA | 5-shot | 28.3 | 31.6 | 20.6 | 30.0 | 26.3† | 27.0† | 35.3 |
67
  | Mathematics | | | | | | | | |
68
- | GSM8K | 8-shot | 88.4 | 88.2 | 92.3 | 87.6 | 91.4† | 91.1† | 92.1 |
69
- | MATH | 4-shot | 65.4 | 66.8 | 71.0 | 62.6 | 62.6† | 62.5† | 70.2 |
70
  | Code | | | | | | | | |
71
- | HumanEval | 3-shot | 65.2 | 81.1 | 77.4* | 79.8* | 72.5* | 67.7* | 84.8* |
72
- | MBPP | 3-shot | 79.2 | 79.4 | 81.0* | 81.6* | 74.6* | 75.6* | 89.0* |
73
- | HumanEval+ | 0-shot | 73.8 | 72.0 | 70.7 | - | 64.6† | 67.7† | - |
74
- | MBPP+ | 0-shot | 63.8 | 70.6 | 71.4 | - | 72.2† | 69.8† | - |
75
  | MultiPL‑E HumanEval | 0-shot | 63.0 | 67.7 | 59.5 | - | 45.9† | 45.7† | 60.5 |
76
  | MultiPL‑E MBPP | 0-shot | 47.9 | 58.0 | 56.7 | - | 52.5† | 50.6† | 58.8 |
77
  | Chinese | | | | | | | | |
 
55
  | # Activated Params | - | 11B | 11B | 15B | 32B | 37B | 37B | 32B |
56
  | # Total Params | - | 196B | 196B | 309B | 355B | 671B | 671B | 1043B |
57
  | General | | | | | | | | |
58
+ | BBH | 3-shot | 87.3 | 88.2 | 88.5 | 86.2 | 88.2† | 88.7† | 88.7 |
59
+ | MMLU | 5-shot | 83.4 | 85.8 | 86.7 | 86.1 | 87.4† | 87.8† | 87.8 |
60
  | MMLU‑Redux | 5-shot | 87.2 | 89.2 | 90.6 | - | 90.0† | 90.4† | 90.2 |
61
+ | MMLU‑Pro | 5-shot | 63.6 | 62.3 | 73.2 | - | 58.8† | 62.1† | 69.2 |
62
+ | HellaSwag | 10-shot | 91.0 | 90.2 | 88.5 | 87.1 | 89.2† | 89.4† | 94.6 |
63
+ | WinoGrande | 5-shot | 75.8 | 79.1 | 83.8 | - | 85.9† | 85.6† | 85.3 |
64
+ | GPQA | 5-shot | 43.8 | 41.7 | 43.5* | 33.5* | 43.1* | 37.3* | 43.1* |
65
  | SuperGPQA | 5-shot | 41.2 | 41.0 | 41.1 | - | 42.3† | 43.6† | 44.7 |
66
+ | SimpleQA | 5-shot | 28.4 | 31.6 | 20.6 | 30.0 | 26.3† | 27.0† | 35.3 |
67
  | Mathematics | | | | | | | | |
68
+ | GSM8K | 8-shot | 88.9 | 88.2 | 92.3 | 87.6 | 91.4† | 91.1† | 92.1 |
69
+ | MATH | 4-shot | 65.7 | 66.8 | 71.0 | 62.6 | 62.6† | 62.5† | 70.2 |
70
  | Code | | | | | | | | |
71
+ | HumanEval | 3-shot | 67.0 | 81.1 | 77.4* | 79.8* | 72.5* | 67.7* | 84.8* |
72
+ | MBPP | 3-shot | 79.0 | 79.4 | 81.0* | 81.6* | 74.6* | 75.6* | 89.0* |
73
+ | HumanEval+ | 0-shot | 75.0 | 72.0 | 70.7 | - | 64.6† | 67.7† | - |
74
+ | MBPP+ | 0-shot | 62.4 | 70.6 | 71.4 | - | 72.2† | 69.8† | - |
75
  | MultiPL‑E HumanEval | 0-shot | 63.0 | 67.7 | 59.5 | - | 45.9† | 45.7† | 60.5 |
76
  | MultiPL‑E MBPP | 0-shot | 47.9 | 58.0 | 56.7 | - | 52.5† | 50.6† | 58.8 |
77
  | Chinese | | | | | | | | |