[doc] Convert accuracy table to Markdown pipe format, add EvalScope version, fix AA-LCR code block

#1
Files changed (1) hide show
  1. README.md +18 -61
README.md CHANGED
@@ -91,71 +91,25 @@ python quantize_quark.py \
91
  ```
92
 
93
  # Evaluation
94
- The model was evaluated on GSM8K using vLLM and EvalScope.
95
 
96
  ### Accuracy
97
 
98
- <table>
99
- <tr>
100
- <td><strong>Benchmark</strong></td>
101
- <td><strong>MiniMax/MiniMax-M3.1-Flash</strong></td>
102
- <td><strong>amd/MiniMax-M3.1-Flash-Quark-MXFP4 (this model)</strong></td>
103
- <td><strong>Recovery</strong></td>
104
- </tr>
105
- <tr>
106
- <td>GSM8K (4-shot)</td>
107
- <td>97.20</td>
108
- <td>96.90</td>
109
- <td>99.69%</td>
110
- </tr>
111
- <tr>
112
- <td>GPQA Diamond (mean@8)</td>
113
- <td>86.36</td>
114
- <td>87.37</td>
115
- <td>101.17%</td>
116
- </tr>
117
- <tr>
118
- <td>AIME26 (mean@16)</td>
119
- <td>93.54</td>
120
- <td>93.75</td>
121
- <td>100.22%</td>
122
- </tr>
123
- <tr>
124
- <td>MMMU Pro, standard (mean@1)</td>
125
- <td>79.19</td>
126
- <td>79.13</td>
127
- <td>99.92%</td>
128
- </tr>
129
- <tr>
130
- <td>MMMU Pro, vision (mean@1)</td>
131
- <td>76.99</td>
132
- <td>77.57</td>
133
- <td>100.75%</td>
134
- </tr>
135
- <tr>
136
- <td>SciCode, subproblem (mean@8)</td>
137
- <td>48.11</td>
138
- <td>49.27</td>
139
- <td>102.41%</td>
140
- </tr>
141
- <tr>
142
- <td>AA-LCR (mean@8)</td>
143
- <td>73.13</td>
144
- <td>74.00</td>
145
- <td>101.19%</td>
146
- </tr>
147
- <tr>
148
- <td>IFBench (mean@2)</td>
149
- <td>50.33</td>
150
- <td>50.83</td>
151
- <td>100.99%</td>
152
- </tr>
153
- </table>
154
 
155
 
156
  ### Reproduction
157
 
158
- The following results were obtained using EvalScope and vLLM.
159
 
160
  #### Launching server
161
  We launch the vLLM server using the [vLLM Recipes](https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3.1-Flash?hardware=mi355x&variant=mxfp4) with docker image `vllm/vllm-openai-rocm:minimax-m31-flash`.
@@ -257,8 +211,9 @@ evalscope eval \
257
 
258
  **AA-LCR (v1.0.0)**
259
 
260
- Judge configuration (aa-lcr-judge.json; replace <AMD_SUBSCRIPTION_KEY>):
261
- ```bash
 
262
  {
263
  "strategy": "llm",
264
  "models": [
@@ -275,13 +230,15 @@ Judge configuration (aa-lcr-judge.json; replace <AMD_SUBSCRIPTION_KEY>):
275
  "api-version": "2025-04-01-preview"
276
  },
277
  "extra_headers": {
278
- "Ocp-Apim-Subscription-Key": "<AMD_SUBSCRIPTION_KEY>"
279
  }
280
  }
281
  }
282
  ]
283
  }
 
284
 
 
285
  evalscope eval \
286
  --model amd/MiniMax-M3.1-Flash-Quark-MXFP4 \
287
  --api-url http://127.0.0.1:8000/v1/chat/completions \
 
91
  ```
92
 
93
  # Evaluation
94
+ The model was evaluated using vLLM and EvalScope.
95
 
96
  ### Accuracy
97
 
98
+ | **Benchmark** | **MiniMax/MiniMax-M3.1-Flash** | **amd/MiniMax-M3.1-Flash-Quark-MXFP4 (this model)** | **Recovery** |
99
+ | --- | --- | --- | --- |
100
+ | GSM8K (4-shot) | 97.20 | 96.90 | 99.69% |
101
+ | GPQA Diamond (mean@8) | 86.36 | 87.37 | 101.17% |
102
+ | AIME26 (mean@16) | 93.54 | 93.75 | 100.22% |
103
+ | MMMU Pro, standard (mean@1) | 79.19 | 79.13 | 99.92% |
104
+ | MMMU Pro, vision (mean@1) | 76.99 | 77.57 | 100.75% |
105
+ | SciCode, subproblem (mean@8) | 48.11 | 49.27 | 102.41% |
106
+ | AA-LCR (mean@8) | 73.13 | 74.00 | 101.19% |
107
+ | IFBench (mean@2) | 50.33 | 50.83 | 100.99% |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
108
 
109
 
110
  ### Reproduction
111
 
112
+ The following results were obtained using vLLM and EvalScope `1.14.1`.
113
 
114
  #### Launching server
115
  We launch the vLLM server using the [vLLM Recipes](https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3.1-Flash?hardware=mi355x&variant=mxfp4) with docker image `vllm/vllm-openai-rocm:minimax-m31-flash`.
 
211
 
212
  **AA-LCR (v1.0.0)**
213
 
214
+ Judge configuration (aa-lcr-judge.json; replace `<YOUR_SUBSCRIPTION_KEY>`):
215
+
216
+ ```json
217
  {
218
  "strategy": "llm",
219
  "models": [
 
230
  "api-version": "2025-04-01-preview"
231
  },
232
  "extra_headers": {
233
+ "Ocp-Apim-Subscription-Key": "<YOUR_SUBSCRIPTION_KEY>"
234
  }
235
  }
236
  }
237
  ]
238
  }
239
+ ```
240
 
241
+ ```bash
242
  evalscope eval \
243
  --model amd/MiniMax-M3.1-Flash-Quark-MXFP4 \
244
  --api-url http://127.0.0.1:8000/v1/chat/completions \