# Comparison against the published 7B-8B models

Our rows are **measured by this pipeline**. Every other row is **transcribed from**
MedReason (arXiv:2504.00993) and was not re-run.

> Excluded from this table: MedReason-8B.

> Backbone shown with its published values (Huatuo-o1-RL-8B), not the score measured on this harness.

> Harness offset measured on the untouched backbone: +2.89 pt vs the published protocol. Subtract it before reading our-vs-published gaps.

## Table A — Ranking on the paper's six-benchmark suite

| # | Model | MB-op4 | MB-op5 | MedXpert | MedQA | MedMCQA | PubMedQA | Avg |
|---|---|---|---|---|---|---|---|---|
| **1** | **brainmed-8b-final (ours)** | **62.01** | **54.87** | **18.63** | **76.67** | **64.07** | **79.10** | **59.23** |
| **2** | **brainmed-8b-v1__soup-last-a0.3 (ours)** | **62.01** | **52.92** | **18.63** | **76.67** | **64.07** | **79.10** | **58.90** |
| **3** | **brainmed-8b-v1__soup-last-a0.7 (ours)** | **60.71** | **56.49** | **19.32** | **76.12** | **62.87** | **77.90** | **58.90** |
| **4** | **brainmed-8b-v1__soup-last-a0.5 (ours)** | **58.77** | **54.55** | **18.91** | **76.90** | **64.36** | **78.90** | **58.73** |
| **5** | **brainmed-8b-v1__soup-a0.3 (ours)** | **59.42** | **54.87** | **16.43** | **76.28** | **62.54** | **80.00** | **58.26** |
| **6** | **brainmed-8b-v1__soup-a0.15 (ours)** | **60.06** | **50.97** | **16.08** | **77.85** | **63.21** | **81.20** | **58.23** |
| **7** | **brainmed-8b-v1__epoch-3 (ours)** | **58.77** | **56.82** | **18.08** | **72.11** | **61.20** | **77.10** | **57.35** |
| **8** | **brainmed-8b-v1 (ours)** | **56.49** | **50.65** | **17.81** | **72.58** | **60.22** | **77.70** | **55.91** |
| **9** | **brainmed-8b-v1__best (ours)** | **56.49** | **50.65** | **17.81** | **72.58** | **60.22** | **77.70** | **55.91** |
| 10 | Huatuo-o1-RL-8B | 55.20 | 51.30 | 16.70 | 72.60 | 60.40 | 79.20 | 55.90 |
| **11** | **brainmed-8b-v1__epoch-1 (ours)** | **55.52** | **50.00** | **15.87** | **70.62** | **59.77** | **77.50** | **54.88** |
| 12 | Huatuo-o1-SFT-8B | 53.30 | 49.70 | 17.30 | 70.20 | 58.20 | 76.10 | 54.10 |
| 13 | Qwen2.5-Instruct-7B | 50.00 | 41.60 | 12.60 | 57.00 | 55.60 | 72.70 | 48.20 |
| 14 | Llama3.1-Instruct-8B | 43.20 | 40.90 | 14.30 | 58.70 | 56.00 | 75.20 | 48.00 |
| 15 | OpenBioLLM-8B | 39.20 | 35.70 | 10.70 | 57.70 | 54.10 | 74.10 | 45.30 |
| 16 | DeepSeek-Distill-8B | 41.90 | 35.10 | 13.50 | 55.40 | 49.00 | 73.90 | 44.80 |
| 17 | Medical-CoT-8B | 39.30 | 34.10 | 12.60 | 49.00 | 42.60 | 68.00 | 40.90 |
| 18 | BioMistral-7B | 46.40 | 33.10 | 12.40 | 45.00 | 40.20 | 66.90 | 40.70 |
| 19 | Mistral-Instruct-7B | 43.50 | 33.40 | 11.40 | 48.20 | 44.90 | 50.10 | 38.60 |
| 20 | Medical-Llama3-8B | 33.40 | 25.30 | 9.00 | 40.30 | 46.80 | 48.00 | 33.80 |

## Table B — `brainmed-8b-final` vs published **Huatuo-o1-RL-8B**

| Benchmark | Huatuo-o1-RL-8B
(published) | brainmed-8b-final
(measured) | Delta |  |
|---|---|---|---|---|
| MB-op4 | 55.20 | 62.01 | +6.81 | win |
| MB-op5 | 51.30 | 54.87 | +3.57 | win |
| MedXpert | 16.70 | 18.63 | +1.93 | win |
| MedQA | 72.60 | 76.67 | +4.07 | win |
| MedMCQA | 60.40 | 64.07 | +3.67 | win |
| PubMedQA | 79.20 | 79.10 | -0.10 |  |
| **Average** | **55.90** | **59.23** | **+3.33** | **win** |

## Table B — `brainmed-8b-v1` vs published **Huatuo-o1-RL-8B**

| Benchmark | Huatuo-o1-RL-8B
(published) | brainmed-8b-v1
(measured) | Delta |  |
|---|---|---|---|---|
| MB-op4 | 55.20 | 56.49 | +1.29 | win |
| MB-op5 | 51.30 | 50.65 | -0.65 |  |
| MedXpert | 16.70 | 17.81 | +1.11 | win |
| MedQA | 72.60 | 72.58 | -0.02 |  |
| MedMCQA | 60.40 | 60.22 | -0.18 |  |
| PubMedQA | 79.20 | 77.70 | -1.50 |  |
| **Average** | **55.90** | **55.91** | **+0.01** | **win** |

## Table B — `brainmed-8b-v1__best` vs published **Huatuo-o1-RL-8B**

| Benchmark | Huatuo-o1-RL-8B
(published) | brainmed-8b-v1__best
(measured) | Delta |  |
|---|---|---|---|---|
| MB-op4 | 55.20 | 56.49 | +1.29 | win |
| MB-op5 | 51.30 | 50.65 | -0.65 |  |
| MedXpert | 16.70 | 17.81 | +1.11 | win |
| MedQA | 72.60 | 72.58 | -0.02 |  |
| MedMCQA | 60.40 | 60.22 | -0.18 |  |
| PubMedQA | 79.20 | 77.70 | -1.50 |  |
| **Average** | **55.90** | **55.91** | **+0.01** | **win** |

## Table B — `brainmed-8b-v1__epoch-1` vs published **Huatuo-o1-RL-8B**

| Benchmark | Huatuo-o1-RL-8B
(published) | brainmed-8b-v1__epoch-1
(measured) | Delta |  |
|---|---|---|---|---|
| MB-op4 | 55.20 | 55.52 | +0.32 | win |
| MB-op5 | 51.30 | 50.00 | -1.30 |  |
| MedXpert | 16.70 | 15.87 | -0.83 |  |
| MedQA | 72.60 | 70.62 | -1.98 |  |
| MedMCQA | 60.40 | 59.77 | -0.63 |  |
| PubMedQA | 79.20 | 77.50 | -1.70 |  |
| **Average** | **55.90** | **54.88** | **-1.02** |  |

## Table B — `brainmed-8b-v1__epoch-3` vs published **Huatuo-o1-RL-8B**

| Benchmark | Huatuo-o1-RL-8B
(published) | brainmed-8b-v1__epoch-3
(measured) | Delta |  |
|---|---|---|---|---|
| MB-op4 | 55.20 | 58.77 | +3.57 | win |
| MB-op5 | 51.30 | 56.82 | +5.52 | win |
| MedXpert | 16.70 | 18.08 | +1.38 | win |
| MedQA | 72.60 | 72.11 | -0.49 |  |
| MedMCQA | 60.40 | 61.20 | +0.80 | win |
| PubMedQA | 79.20 | 77.10 | -2.10 |  |
| **Average** | **55.90** | **57.35** | **+1.45** | **win** |

## Table B — `brainmed-8b-v1__soup-a0.15` vs published **Huatuo-o1-RL-8B**

| Benchmark | Huatuo-o1-RL-8B
(published) | brainmed-8b-v1__soup-a0.15
(measured) | Delta |  |
|---|---|---|---|---|
| MB-op4 | 55.20 | 60.06 | +4.86 | win |
| MB-op5 | 51.30 | 50.97 | -0.33 |  |
| MedXpert | 16.70 | 16.08 | -0.62 |  |
| MedQA | 72.60 | 77.85 | +5.25 | win |
| MedMCQA | 60.40 | 63.21 | +2.81 | win |
| PubMedQA | 79.20 | 81.20 | +2.00 | win |
| **Average** | **55.90** | **58.23** | **+2.33** | **win** |

## Table B — `brainmed-8b-v1__soup-a0.3` vs published **Huatuo-o1-RL-8B**

| Benchmark | Huatuo-o1-RL-8B
(published) | brainmed-8b-v1__soup-a0.3
(measured) | Delta |  |
|---|---|---|---|---|
| MB-op4 | 55.20 | 59.42 | +4.22 | win |
| MB-op5 | 51.30 | 54.87 | +3.57 | win |
| MedXpert | 16.70 | 16.43 | -0.27 |  |
| MedQA | 72.60 | 76.28 | +3.68 | win |
| MedMCQA | 60.40 | 62.54 | +2.14 | win |
| PubMedQA | 79.20 | 80.00 | +0.80 | win |
| **Average** | **55.90** | **58.26** | **+2.36** | **win** |

## Table B — `brainmed-8b-v1__soup-last-a0.3` vs published **Huatuo-o1-RL-8B**

| Benchmark | Huatuo-o1-RL-8B
(published) | brainmed-8b-v1__soup-last-a0.3
(measured) | Delta |  |
|---|---|---|---|---|
| MB-op4 | 55.20 | 62.01 | +6.81 | win |
| MB-op5 | 51.30 | 52.92 | +1.62 | win |
| MedXpert | 16.70 | 18.63 | +1.93 | win |
| MedQA | 72.60 | 76.67 | +4.07 | win |
| MedMCQA | 60.40 | 64.07 | +3.67 | win |
| PubMedQA | 79.20 | 79.10 | -0.10 |  |
| **Average** | **55.90** | **58.90** | **+3.00** | **win** |

## Table B — `brainmed-8b-v1__soup-last-a0.5` vs published **Huatuo-o1-RL-8B**

| Benchmark | Huatuo-o1-RL-8B
(published) | brainmed-8b-v1__soup-last-a0.5
(measured) | Delta |  |
|---|---|---|---|---|
| MB-op4 | 55.20 | 58.77 | +3.57 | win |
| MB-op5 | 51.30 | 54.55 | +3.25 | win |
| MedXpert | 16.70 | 18.91 | +2.21 | win |
| MedQA | 72.60 | 76.90 | +4.30 | win |
| MedMCQA | 60.40 | 64.36 | +3.96 | win |
| PubMedQA | 79.20 | 78.90 | -0.30 |  |
| **Average** | **55.90** | **58.73** | **+2.83** | **win** |

## Table B — `brainmed-8b-v1__soup-last-a0.7` vs published **Huatuo-o1-RL-8B**

| Benchmark | Huatuo-o1-RL-8B
(published) | brainmed-8b-v1__soup-last-a0.7
(measured) | Delta |  |
|---|---|---|---|---|
| MB-op4 | 55.20 | 60.71 | +5.51 | win |
| MB-op5 | 51.30 | 56.49 | +5.19 | win |
| MedXpert | 16.70 | 19.32 | +2.62 | win |
| MedQA | 72.60 | 76.12 | +3.52 | win |
| MedMCQA | 60.40 | 62.87 | +2.47 | win |
| PubMedQA | 79.20 | 77.90 | -1.30 |  |
| **Average** | **55.90** | **58.90** | **+3.00** | **win** |

## Table D — Eight-benchmark suite (MedReason Table 2 columns)

| Model / data | MedQA | MedMCQA | PubMedQA | MMLU-Pro | MB-op4 | MB-op5 | MedXpert | HLE(med) | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Llama3.1-Instruct-8B (base) | 58.70 | 56.00 | 75.20 | 58.20 | 48.70 | 42.50 | 13.20 | 13.60 | 45.80 |
| Llama3.1-Instruct-8B + Huatuo CoT | 70.20 | 58.20 | 76.10 | 59.90 | 53.30 | 49.70 | 17.30 | 14.60 | 49.90 |
| Llama3.1-Instruct-8B + MedReason | 68.40 | 57.50 | 77.60 | 63.10 | 57.50 | 52.30 | 16.40 | 16.50 | 51.20 |
| Mistral-Instruct-7B (base) | 48.20 | 44.90 | 50.10 | 42.70 | 43.50 | 33.40 | 11.40 | 14.60 | 36.10 |
| Mistral-Instruct-7B + Huatuo CoT | 59.90 | 46.90 | 57.50 | 47.60 | 50.00 | 46.10 | 14.40 | 14.60 | 42.10 |
| Mistral-Instruct-7B + MedReason | 58.70 | 48.90 | 59.20 | 50.80 | 52.30 | 47.10 | 16.60 | 24.30 | 44.70 |
| **brainmed-8b-final (ours)** | **76.67** | **64.07** | **79.10** | **65.73** | **62.01** | **54.87** | **18.63** | **10.68** | **53.97** |
| **brainmed-8b-v1 (ours)** | **72.58** | **60.22** | **77.70** | **61.82** | **56.49** | **50.65** | **17.81** | **20.39** | **52.21** |