| # Paired significance: `brainmed-8b-v1__soup-last-a0.3` vs `base-HuatuoGPT-o1-8B` |
| |
| McNemar's exact test on the items both runs answered. `only B` counts items the |
| candidate gets right and the baseline gets wrong; `only A` the reverse. Items both |
| get right or both get wrong are uninformative about which model is better. |
| |
| | Benchmark | n | base-HuatuoGPT-o1-8B | brainmed-8b-v1__soup-last-a0.3 | Delta | only A | only B | p | verdict (a=0.05) | |
| |---|---|---|---|---|---|---|---|---| |
| | MB-op4 | 298 | 59.73 | 62.42 | +2.68 | 30 | 38 | 0.396 | no difference | |
| | MB-op5 | 298 | 55.37 | 53.36 | -2.01 | 37 | 31 | 0.545 | no difference | |
| | MedMCQA | 4183 | 63.28 | 64.04 | +0.77 | 358 | 390 | 0.257 | no difference | |
| | MedQA | 1273 | 77.93 | 76.59 | -1.34 | 103 | 86 | 0.244 | no difference | |
| | MedXpert | 1449 | 16.98 | 18.63 | +1.66 | 93 | 117 | 0.112 | no difference | |
| | PubMedQA | 1000 | 80.2 | 79.1 | -1.10 | 30 | 19 | 0.152 | no difference | |
|
|
| ## Pooled across benchmarks |
|
|
| - items only `base-HuatuoGPT-o1-8B` gets right: **651** |
| - items only `brainmed-8b-v1__soup-last-a0.3` gets right: **681** |
| - McNemar exact p = **0.4269** |
|
|
| Pooled, the two runs are **statistically indistinguishable** (p = 0.427). Any average difference between them is within sampling noise. |
|
|
| Per benchmark: 0 significantly better, 0 significantly worse, 6 indistinguishable. |
|
|
| Pooling treats benchmarks as one sample and so weights them by size; read it |
| alongside the per-benchmark rows, not instead of them. |
|
|
