deepanshupillm commited on
Commit
f0d79e8
·
verified ·
1 Parent(s): 7eed28a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +7 -15
README.md CHANGED
@@ -106,7 +106,7 @@ Training used eight NVIDIA H100 GPUs, with 3,034 GRPO steps across five phases.
106
  >
107
  > Alpie Seed scores are from the FP16 model and do not apply to the 2-bit, 3-bit, or 4-bit variants. **Bold** indicates the best score in each row. Comparison-model scores are taken from published model cards and reports (including the [Qwen3.5-9B model card](https://huggingface.co/Qwen/Qwen3.5-9B)).
108
 
109
- ### Language
110
 
111
  <table style="width: 100%; border-collapse: collapse; text-align: center; font-size: 14px;">
112
  <thead>
@@ -127,7 +127,7 @@ Training used eight NVIDIA H100 GPUs, with 3,034 GRPO steps across five phases.
127
  </tr>
128
  <tr style="border-bottom: 1px solid #F1F5F9;">
129
  <td style="text-align: left; padding: 8px 12px;">MMLU</td>
130
- <td>--</td><td>--</td><td>--</td><td>85.3</td><td>85.1</td><td>--</td>
131
  <td><b>87.30</b></td>
132
  </tr>
133
  <tr style="border-bottom: 1px solid #F1F5F9;">
@@ -143,16 +143,6 @@ Training used eight NVIDIA H100 GPUs, with 3,034 GRPO steps across five phases.
143
  <tr style="background-color: #F5F3FF; color: #6D28D9; font-weight: bold; text-align: left;">
144
  <td colspan="8" style="padding: 8px 12px;">Mathematics</td>
145
  </tr>
146
- <tr style="border-bottom: 1px solid #F1F5F9;">
147
- <td style="text-align: left; padding: 8px 12px;">GSM8K</td>
148
- <td>--</td><td>--</td><td>--</td><td>--</td><td>--</td><td>--</td>
149
- <td><b>98.00</b></td>
150
- </tr>
151
- <tr style="border-bottom: 1px solid #F1F5F9;">
152
- <td style="text-align: left; padding: 8px 12px;">Math-500</td>
153
- <td>--</td><td>--</td><td>--</td><td>94.2</td><td><b>97.0</b></td><td>--</td>
154
- <td>91.00</td>
155
- </tr>
156
  <tr style="border-bottom: 1px solid #F1F5F9;">
157
  <td style="text-align: left; padding: 8px 12px;">AIME 2026</td>
158
  <td>69.4</td><td>77.5</td><td>--</td><td>--</td><td>--</td><td>--</td>
@@ -174,7 +164,7 @@ Training used eight NVIDIA H100 GPUs, with 3,034 GRPO steps across five phases.
174
  </tbody>
175
  </table>
176
 
177
- ### Vision Language
178
 
179
  <table style="width: 100%; border-collapse: collapse; text-align: center; font-size: 14px;">
180
  <thead>
@@ -234,13 +224,15 @@ Training used eight NVIDIA H100 GPUs, with 3,034 GRPO steps across five phases.
234
  - SWE-Bench Verified: resolved rate using Mini-SWE-agent, with retries disabled.
235
  - Comparison-model scores were not re-run by 169Pi, and harnesses, prompts, and benchmark versions differ, so cross-model comparisons are indicative rather than strictly like-for-like. Scores vary with dataset versions, prompts, evaluation harnesses, and inference settings.
236
  - Empty cells (--) indicate scores not yet available or not applicable.
237
- - MMLU for GPT-OSS-20B is from OpenAI's model card (high reasoning); MMLU for Sarvam-30B and Math-500 for Sarvam-30B and GPT-OSS-20B are from Sarvam's release comparison. Qwen3.5-9B AIME 2026 is from a third-party evaluation.
 
 
238
 
239
  ## Quickstart
240
 
241
  ### Install dependencies
242
 
243
- Alpie Seed support requires a recent Transformers release (Transformers 5 or later).
244
 
245
  ```bash
246
  pip install -U "transformers>=5.2.0" torch torchvision pillow accelerate
 
106
  >
107
  > Alpie Seed scores are from the FP16 model and do not apply to the 2-bit, 3-bit, or 4-bit variants. **Bold** indicates the best score in each row. Comparison-model scores are taken from published model cards and reports (including the [Qwen3.5-9B model card](https://huggingface.co/Qwen/Qwen3.5-9B)).
108
 
109
+ ### 🧮 Language
110
 
111
  <table style="width: 100%; border-collapse: collapse; text-align: center; font-size: 14px;">
112
  <thead>
 
127
  </tr>
128
  <tr style="border-bottom: 1px solid #F1F5F9;">
129
  <td style="text-align: left; padding: 8px 12px;">MMLU</td>
130
+ <td>--</td><td>--</td><td>76.1*</td><td>85.3</td><td>85.1</td><td>78.63*</td>
131
  <td><b>87.30</b></td>
132
  </tr>
133
  <tr style="border-bottom: 1px solid #F1F5F9;">
 
143
  <tr style="background-color: #F5F3FF; color: #6D28D9; font-weight: bold; text-align: left;">
144
  <td colspan="8" style="padding: 8px 12px;">Mathematics</td>
145
  </tr>
 
 
 
 
 
 
 
 
 
 
146
  <tr style="border-bottom: 1px solid #F1F5F9;">
147
  <td style="text-align: left; padding: 8px 12px;">AIME 2026</td>
148
  <td>69.4</td><td>77.5</td><td>--</td><td>--</td><td>--</td><td>--</td>
 
164
  </tbody>
165
  </table>
166
 
167
+ ### 👁️ Vision Language
168
 
169
  <table style="width: 100%; border-collapse: collapse; text-align: center; font-size: 14px;">
170
  <thead>
 
224
  - SWE-Bench Verified: resolved rate using Mini-SWE-agent, with retries disabled.
225
  - Comparison-model scores were not re-run by 169Pi, and harnesses, prompts, and benchmark versions differ, so cross-model comparisons are indicative rather than strictly like-for-like. Scores vary with dataset versions, prompts, evaluation harnesses, and inference settings.
226
  - Empty cells (--) indicate scores not yet available or not applicable.
227
+ - MMLU for GPT-OSS-20B is from OpenAI's model card (high reasoning); MMLU for Sarvam-30B is from Sarvam's release comparison. Ministral 3 8B (76.1) is the published 5-shot MMLU score for its base checkpoint, and Nemotron 3.5 Lightning (78.63) is the published base-model exact-match score. These settings/checkpoints are not strictly like-for-like with every other entry. Qwen3.5-9B and Gemma 4 12B are shown as -- because their cited model cards report MMLU-Pro / MMLU-Redux rather than the same standard MMLU result; those metrics should not be substituted into the MMLU row.
228
+ - AIME 2026 entries are shown only where a score for that exact benchmark is available in the cited material. AIME 2025 results are not substituted for AIME 2026. Qwen3.5-9B AIME 2026 is from a third-party evaluation.
229
+ - Sources: [Ministral 3 8B model card](https://www.modelscope.cn/models/mistralai/Ministral-3-8B-Instruct-2512-BF16), [NVIDIA Nemotron 3.5 Lightning evaluation](https://docs.nvidia.com/nemotron/nightly/nemotron/lightning35/evaluate.html), [Qwen3.5-9B model card](https://huggingface.co/Qwen/Qwen3.5-9B), [Gemma 4 model card](https://huggingface.co/google/gemma-4-12B), [GPT-OSS evaluation](https://deploymentsafety.openai.com/gpt-oss), and [Sarvam-30B model card](https://huggingface.co/sarvamai/sarvam-30b).
230
 
231
  ## Quickstart
232
 
233
  ### Install dependencies
234
 
235
+ Qwen3.5 support requires a recent Transformers release (Transformers 5 or later).
236
 
237
  ```bash
238
  pip install -U "transformers>=5.2.0" torch torchvision pillow accelerate