[doc] Convert accuracy table to Markdown pipe format, add EvalScope version, fix AA-LCR code block
#1
by linzhao-amd - opened
README.md
CHANGED
|
@@ -91,71 +91,25 @@ python quantize_quark.py \
|
|
| 91 |
```
|
| 92 |
|
| 93 |
# Evaluation
|
| 94 |
-
The model was evaluated
|
| 95 |
|
| 96 |
### Accuracy
|
| 97 |
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
<td>96.90</td>
|
| 109 |
-
<td>99.69%</td>
|
| 110 |
-
</tr>
|
| 111 |
-
<tr>
|
| 112 |
-
<td>GPQA Diamond (mean@8)</td>
|
| 113 |
-
<td>86.36</td>
|
| 114 |
-
<td>87.37</td>
|
| 115 |
-
<td>101.17%</td>
|
| 116 |
-
</tr>
|
| 117 |
-
<tr>
|
| 118 |
-
<td>AIME26 (mean@16)</td>
|
| 119 |
-
<td>93.54</td>
|
| 120 |
-
<td>93.75</td>
|
| 121 |
-
<td>100.22%</td>
|
| 122 |
-
</tr>
|
| 123 |
-
<tr>
|
| 124 |
-
<td>MMMU Pro, standard (mean@1)</td>
|
| 125 |
-
<td>79.19</td>
|
| 126 |
-
<td>79.13</td>
|
| 127 |
-
<td>99.92%</td>
|
| 128 |
-
</tr>
|
| 129 |
-
<tr>
|
| 130 |
-
<td>MMMU Pro, vision (mean@1)</td>
|
| 131 |
-
<td>76.99</td>
|
| 132 |
-
<td>77.57</td>
|
| 133 |
-
<td>100.75%</td>
|
| 134 |
-
</tr>
|
| 135 |
-
<tr>
|
| 136 |
-
<td>SciCode, subproblem (mean@8)</td>
|
| 137 |
-
<td>48.11</td>
|
| 138 |
-
<td>49.27</td>
|
| 139 |
-
<td>102.41%</td>
|
| 140 |
-
</tr>
|
| 141 |
-
<tr>
|
| 142 |
-
<td>AA-LCR (mean@8)</td>
|
| 143 |
-
<td>73.13</td>
|
| 144 |
-
<td>74.00</td>
|
| 145 |
-
<td>101.19%</td>
|
| 146 |
-
</tr>
|
| 147 |
-
<tr>
|
| 148 |
-
<td>IFBench (mean@2)</td>
|
| 149 |
-
<td>50.33</td>
|
| 150 |
-
<td>50.83</td>
|
| 151 |
-
<td>100.99%</td>
|
| 152 |
-
</tr>
|
| 153 |
-
</table>
|
| 154 |
|
| 155 |
|
| 156 |
### Reproduction
|
| 157 |
|
| 158 |
-
The following results were obtained using
|
| 159 |
|
| 160 |
#### Launching server
|
| 161 |
We launch the vLLM server using the [vLLM Recipes](https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3.1-Flash?hardware=mi355x&variant=mxfp4) with docker image `vllm/vllm-openai-rocm:minimax-m31-flash`.
|
|
@@ -257,8 +211,9 @@ evalscope eval \
|
|
| 257 |
|
| 258 |
**AA-LCR (v1.0.0)**
|
| 259 |
|
| 260 |
-
Judge configuration (aa-lcr-judge.json; replace <
|
| 261 |
-
|
|
|
|
| 262 |
{
|
| 263 |
"strategy": "llm",
|
| 264 |
"models": [
|
|
@@ -275,13 +230,15 @@ Judge configuration (aa-lcr-judge.json; replace <AMD_SUBSCRIPTION_KEY>):
|
|
| 275 |
"api-version": "2025-04-01-preview"
|
| 276 |
},
|
| 277 |
"extra_headers": {
|
| 278 |
-
"Ocp-Apim-Subscription-Key": "<
|
| 279 |
}
|
| 280 |
}
|
| 281 |
}
|
| 282 |
]
|
| 283 |
}
|
|
|
|
| 284 |
|
|
|
|
| 285 |
evalscope eval \
|
| 286 |
--model amd/MiniMax-M3.1-Flash-Quark-MXFP4 \
|
| 287 |
--api-url http://127.0.0.1:8000/v1/chat/completions \
|
|
|
|
| 91 |
```
|
| 92 |
|
| 93 |
# Evaluation
|
| 94 |
+
The model was evaluated using vLLM and EvalScope.
|
| 95 |
|
| 96 |
### Accuracy
|
| 97 |
|
| 98 |
+
| **Benchmark** | **MiniMax/MiniMax-M3.1-Flash** | **amd/MiniMax-M3.1-Flash-Quark-MXFP4 (this model)** | **Recovery** |
|
| 99 |
+
| --- | --- | --- | --- |
|
| 100 |
+
| GSM8K (4-shot) | 97.20 | 96.90 | 99.69% |
|
| 101 |
+
| GPQA Diamond (mean@8) | 86.36 | 87.37 | 101.17% |
|
| 102 |
+
| AIME26 (mean@16) | 93.54 | 93.75 | 100.22% |
|
| 103 |
+
| MMMU Pro, standard (mean@1) | 79.19 | 79.13 | 99.92% |
|
| 104 |
+
| MMMU Pro, vision (mean@1) | 76.99 | 77.57 | 100.75% |
|
| 105 |
+
| SciCode, subproblem (mean@8) | 48.11 | 49.27 | 102.41% |
|
| 106 |
+
| AA-LCR (mean@8) | 73.13 | 74.00 | 101.19% |
|
| 107 |
+
| IFBench (mean@2) | 50.33 | 50.83 | 100.99% |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 108 |
|
| 109 |
|
| 110 |
### Reproduction
|
| 111 |
|
| 112 |
+
The following results were obtained using vLLM and EvalScope `1.14.1`.
|
| 113 |
|
| 114 |
#### Launching server
|
| 115 |
We launch the vLLM server using the [vLLM Recipes](https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3.1-Flash?hardware=mi355x&variant=mxfp4) with docker image `vllm/vllm-openai-rocm:minimax-m31-flash`.
|
|
|
|
| 211 |
|
| 212 |
**AA-LCR (v1.0.0)**
|
| 213 |
|
| 214 |
+
Judge configuration (aa-lcr-judge.json; replace `<YOUR_SUBSCRIPTION_KEY>`):
|
| 215 |
+
|
| 216 |
+
```json
|
| 217 |
{
|
| 218 |
"strategy": "llm",
|
| 219 |
"models": [
|
|
|
|
| 230 |
"api-version": "2025-04-01-preview"
|
| 231 |
},
|
| 232 |
"extra_headers": {
|
| 233 |
+
"Ocp-Apim-Subscription-Key": "<YOUR_SUBSCRIPTION_KEY>"
|
| 234 |
}
|
| 235 |
}
|
| 236 |
}
|
| 237 |
]
|
| 238 |
}
|
| 239 |
+
```
|
| 240 |
|
| 241 |
+
```bash
|
| 242 |
evalscope eval \
|
| 243 |
--model amd/MiniMax-M3.1-Flash-Quark-MXFP4 \
|
| 244 |
--api-url http://127.0.0.1:8000/v1/chat/completions \
|