evaluation/_card/README.md

---
license: apache-2.0
base_model: FreedomIntelligence/HuatuoGPT-o1-8B
library_name: transformers
pipeline_tag: text-generation
tags:
- medical
- clinical-reasoning
- chain-of-thought
- sft
language:
- en
---

# BrainMed-8B

Full-parameter fine-tune of `FreedomIntelligence/HuatuoGPT-o1-8B` on
[`Williamsanderson/MedReason-MedO1-Reasoning-46K`](https://huggingface.co/datasets/Williamsanderson/MedReason-MedO1-Reasoning-46K) — a union of KG-grounded
MedReason reasoning traces and verifier-checked medical-o1 traces.

> **Research model. Not medical advice, not a medical device, not clinically validated.**
> Do not use it for decisions about any real person.

## Results

Accuracy (%) on the six benchmarks of MedReason Table 4. The **BrainMed-8B** row was
measured here; every other row is transcribed from the MedReason paper (arXiv:2504.00993) and
was not re-run.

| Model | MedBullets op4 | MedBullets op5 | MedXpertQA | MedQA | MedMCQA | PubMedQA | Avg | Source |
|---|---|---|---|---|---|---|---|---|
| Medical-Llama3-8B | 33.4 | 25.3 | 9.0 | 40.3 | 46.8 | 48.0 | 33.8 | paper |
| Mistral-Instruct-7B | 43.5 | 33.4 | 11.4 | 48.2 | 44.9 | 50.1 | 38.6 | paper |
| BioMistral-7B | 46.4 | 33.1 | 12.4 | 45.0 | 40.2 | 66.9 | 40.7 | paper |
| Medical-CoT-8B | 39.3 | 34.1 | 12.6 | 49.0 | 42.6 | 68.0 | 40.9 | paper |
| DeepSeek-Distill-8B | 41.9 | 35.1 | 13.5 | 55.4 | 49.0 | 73.9 | 44.8 | paper |
| OpenBioLLM-8B | 39.2 | 35.7 | 10.7 | 57.7 | 54.1 | 74.1 | 45.3 | paper |
| Llama3.1-Instruct-8B | 43.2 | 40.9 | 14.3 | 58.7 | 56.0 | 75.2 | 48.0 | paper |
| Qwen2.5-Instruct-7B | 50.0 | 41.6 | 12.6 | 57.0 | 55.6 | 72.7 | 48.2 | paper |
| Huatuo-o1-SFT-8B | 53.3 | 49.7 | 17.3 | 70.2 | 58.2 | 76.1 | 54.1 | paper |
| Huatuo-o1-RL-8B | 55.2 | 51.3 | 16.7 | 72.6 | 60.4 | 79.2 | 55.9 | paper |
| MedReason-8B | 57.5 | 55.5 | 19.0 | 71.8 | 60.7 | 79.4 | 57.3 | paper |
| **BrainMed-8B** | **62.01** | **54.87** | **18.63** | **76.67** | **64.07** | **79.10** | **59.23** | measured |

### Additional benchmarks

| Benchmark | n | BrainMed-8B |
|---|---|---|
| MMLU-Pro (Med) | 1535 | 65.73 |
| HLE (med) | 103 | 10.68 |
| GPQA (Med) | 390 | 59.74 |
| MedQA (5-opt) | 1273 | 73.68 |

**Read the comparison with this caveat.** Our numbers come from our own evaluation harness
(greedy decoding, the training system prompt applied at inference). Scoring the *untouched*
backbone on that harness gives **+2.89 points** over its published row, so part of the gap to
the paper rows is protocol, not model. The comparison that is free of this effect is the
before/after against the backbone measured on the same harness, published in
[`evaluation/REPORT.md`](https://huggingface.co/BrainHealthAI/BrainMed-8B/blob/main/evaluation/REPORT.md).

![fig8_paper_ranking](https://huggingface.co/BrainHealthAI/BrainMed-8B/resolve/main/assets/fig8_paper_ranking.png)

![fig4_training_curves](https://huggingface.co/BrainHealthAI/BrainMed-8B/resolve/main/assets/fig4_training_curves.png)

![table_A_ranking](https://huggingface.co/BrainHealthAI/BrainMed-8B/resolve/main/assets/table_A_ranking.png)

## Sample answers

Produced by this checkpoint, greedy decoding, with the system prompt below.

**Multiple choice, strict format**

![multiple choice, strict format](https://huggingface.co/BrainHealthAI/BrainMed-8B/resolve/main/assets/sample_2_multiple_choice.png)

**Differential diagnosis**

![differential diagnosis](https://huggingface.co/BrainHealthAI/BrainMed-8B/resolve/main/assets/sample_3_differential_diagnosis.png)

Note what these show and what they do not: the model is trained and evaluated on medical
**question answering**. It handles multiple choice and differential diagnosis well. It is
weaker on open-ended acute management — its corpus contains QA pairs, not treatment
protocols — and it should not be relied on there.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BrainHealthAI/BrainMed-8B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

SYSTEM = ("You are a medical reasoning assistant. Work through the clinical problem step by "
          "step inside <think>...</think>, grounding every step in established medical "
          "knowledge, then give the final, complete answer inside <answer>...</answer>.")

messages = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": "How is eclampsia-related seizure managed?"}]
inputs = tok(tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True),
             return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**inputs, max_new_tokens=1024)[0], skip_special_tokens=True))
```

The system prompt is part of the contract: the model was trained under it and answers as
`<think>…</think><answer>…</answer>`. Dropping it measurably lowers accuracy.

## Training

MedReason recipe (arXiv:2504.00993): lr 5e-6, effective batch 128, 3 epochs, cosine schedule
with 5% warmup, weight decay 0.1, DeepSpeed ZeRO-3, bf16, full parameters — no adapter.
4×H100, ~1.5 h. Full pipeline, evaluation logs and figures are under
[`evaluation/`](https://huggingface.co/BrainHealthAI/BrainMed-8B/tree/main/evaluation).

## Citation

```bibtex
@misc{wu2025medreason,
  title={MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs},
  author={Wu, Juncheng and others}, year={2025}, eprint={2504.00993}, archivePrefix={arXiv}
}
```