Instructions to use BrainHealthAI/BrainMed-8B-SFT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use BrainHealthAI/BrainMed-8B-SFT with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="BrainHealthAI/BrainMed-8B-SFT")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("BrainHealthAI/BrainMed-8B-SFT")
model = AutoModelForCausalLM.from_pretrained("BrainHealthAI/BrainMed-8B-SFT", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use BrainHealthAI/BrainMed-8B-SFT with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "BrainHealthAI/BrainMed-8B-SFT"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BrainHealthAI/BrainMed-8B-SFT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/BrainHealthAI/BrainMed-8B-SFT

SGLang

How to use BrainHealthAI/BrainMed-8B-SFT with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "BrainHealthAI/BrainMed-8B-SFT" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BrainHealthAI/BrainMed-8B-SFT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "BrainHealthAI/BrainMed-8B-SFT" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BrainHealthAI/BrainMed-8B-SFT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use BrainHealthAI/BrainMed-8B-SFT with Docker Model Runner:
```
docker model run hf.co/BrainHealthAI/BrainMed-8B-SFT
```

BrainMed-8B-SFT / evaluation /REPORT.md

Williamsanderson

evaluation results and figures

8a33135 verified 16 days ago

preview code

Raw

History Blame Contribute Delete

36.7 kB

BrainMed reasoning SFT - evaluation report

Rows marked (ours) were measured by this pipeline (greedy decoding, strict prompt, MedReason scorer with <answer> support, training system prompt applied at inference). All other rows are transcribed from MedReason (arXiv:2504.00993, Table 4) and were not re-run.

Comparison with published 7B-8B models (MedReason Table 4)

Model	MB-op4	MB-op5	MedXpert	MedQA	MedMCQA	PubMedQA	Avg
Llama3.1-Instruct-8B	43.2	40.9	14.3	58.7	56.0	75.2	48.0
Qwen2.5-Instruct-7B	50.0	41.6	12.6	57.0	55.6	72.7	48.2
Mistral-Instruct-7B	43.5	33.4	11.4	48.2	44.9	50.1	38.6
Medical-Llama3-8B	33.4	25.3	9.0	40.3	46.8	48.0	33.8
OpenBioLLM-8B	39.2	35.7	10.7	57.7	54.1	74.1	45.3
BioMistral-7B	46.4	33.1	12.4	45.0	40.2	66.9	40.7
Medical-CoT-8B	39.3	34.1	12.6	49.0	42.6	68.0	40.9
DeepSeek-Distill-8B	41.9	35.1	13.5	55.4	49.0	73.9	44.8
Huatuo-o1-SFT-8B	53.3	49.7	17.3	70.2	58.2	76.1	54.1
Huatuo-o1-RL-8B	55.2	51.3	16.7	72.6	60.4	79.2	55.9
MedReason-8B	57.5	55.5	19.0	71.8	60.7	79.4	57.3
base-HuatuoGPT-o1-8B (ours)	59.09	55.19	17.05	77.93	63.3	80.2	58.79
brainmed-8b-final (ours)	62.01	54.87	18.63	76.67	64.07	79.1	59.23
brainmed-8b-v1 (ours)	56.49	50.65	17.81	72.58	60.22	77.7	55.91
brainmed-8b-v1__best (ours)	56.49	50.65	17.81	72.58	60.22	77.7	55.91
brainmed-8b-v1__epoch-1 (ours)	55.52	50.0	15.87	70.62	59.77	77.5	54.88
brainmed-8b-v1__epoch-2 (ours)	56.49	50.65	17.81	72.58	60.22	77.7	55.91
brainmed-8b-v1__epoch-3 (ours)	58.77	56.82	18.08	72.11	61.2	77.1	57.35
brainmed-8b-v1__last (ours)	58.77	56.82	18.08	72.11	61.2	77.1	57.35
brainmed-8b-v1__soup-a0.15 (ours)	60.06	50.97	16.08	77.85	63.21	81.2	58.23
brainmed-8b-v1__soup-a0.3 (ours)	59.42	54.87	16.43	76.28	62.54	80.0	58.26
brainmed-8b-v1__soup-last-a0.3 (ours)	62.01	52.92	18.63	76.67	64.07	79.1	58.9
brainmed-8b-v1__soup-last-a0.5 (ours)	58.77	54.55	18.91	76.9	64.36	78.9	58.73
brainmed-8b-v1__soup-last-a0.7 (ours)	60.71	56.49	19.32	76.12	62.87	77.9	58.9

Table 2 style - effect of the training data (MedReason Table 2)

Published rows compare the same backbone trained on Huatuo CoT vs MedReason. Our row adds the union corpus. Only rows sharing a backbone are comparable: read the Llama3.1 block against a Llama3.1 run, not against a Huatuo-o1 run.

Model / data	MedQA	MedMCQA	PubMedQA	MMLU-Pro	MB-op4	MB-op5	MedXpert	HLE(med)	Avg
Llama3.1-Instruct-8B (base)	58.7	56.0	75.2	58.2	48.7	42.5	13.2	13.6	45.8
Llama3.1-Instruct-8B + Huatuo CoT	70.2	58.2	76.1	59.9	53.3	49.7	17.3	14.6	49.9
Llama3.1-Instruct-8B + MedReason	68.4	57.5	77.6	63.1	57.5	52.3	16.4	16.5	51.2
Mistral-Instruct-7B (base)	48.2	44.9	50.1	42.7	43.5	33.4	11.4	14.6	36.1
Mistral-Instruct-7B + Huatuo CoT	59.9	46.9	57.5	47.6	50.0	46.1	14.4	14.6	42.1
Mistral-Instruct-7B + MedReason	58.7	48.9	59.2	50.8	52.3	47.1	16.6	24.3	44.7
base-HuatuoGPT-o1-8B + ours (union) (ours)	77.93	63.3	80.2	64.56	59.09	55.19	17.05	11.65	53.62
brainmed-8b-final + ours (union) (ours)	76.67	64.07	79.1	65.73	62.01	54.87	18.63	10.68	53.97
brainmed-8b-v1 + ours (union) (ours)	72.58	60.22	77.7	61.82	56.49	50.65	17.81	20.39	52.21

Table 3 style - common vs challenging averages (MedReason Table 3)

Model / data	Avg common (4)	Avg challenging (4)	Avg overall (8)
Medical-CoT-8B (base)	52.1	25.4	38.75
Medical-CoT-8B + MedReason	57.4	30.6	44.0
DeepSeek-Distill-8B (base)	58.0	25.5	41.75
DeepSeek-Distill-8B + MedReason	61.5	33.3	47.4
base-HuatuoGPT-o1-8B (ours)	71.5	35.75	53.62
brainmed-8b-final (ours)	71.39	36.55	53.97
brainmed-8b-v1 (ours)	68.08	36.34	52.21

HuatuoGPT-o1 Table 1/2 style - core medical benchmarks

MMLU-Pro (Med) and GPQA (Med) are the merged medical tracks shipped in eval_data.json; Huatuo's Table 1 splits them per track, which the released files do not allow us to reproduce, so this uses their Table 2 granularity.

Model	MedQA	MedMCQA	PubMedQA	MMLU-Pro	GPQA(med)	Avg
LLaMA-3.1-8B-Instruct (base)	58.7	56.0	75.2	58.2	44.1	58.44
SFT w/ original exam data	60.0	55.5	74.1	54.3	46.9	58.16
SFT w/o CoT	65.2	58.1	75.4	58.5	48.7	61.18
SFT w/ Simple CoT	66.6	59.2	75.4	57.0	46.7	60.98
SFT w/ Complex CoT	69.0	57.9	77.7	59.4	51.0	63.0
SFT w/o CoT + RL (PPO)	66.4	58.6	76.3	60.1	49.8	62.24
SFT w/ Simple CoT + RL (PPO)	68.7	58.4	77.5	60.2	53.1	63.58
SFT w/ Complex CoT + RL (PPO) = o1-8B	72.6	60.4	79.2	63.1	57.5	66.56
SFT w/ Complex CoT + RL (DPO)	72.2	58.4	77.3	60.4	52.5	64.16
SFT w/ Complex CoT + RL (RLOO)	71.1	60.1	78.1	60.9	58.2	65.68
base-HuatuoGPT-o1-8B (ours)	77.93	63.3	80.2	64.56	49.74	67.15
brainmed-8b-final (ours)	76.67	64.07	79.1	65.73	59.74	69.06
brainmed-8b-v1 (ours)	72.58	60.22	77.7	61.82	49.23	64.31

Table 5 style - pipeline ablations

MedReason's Table 5 ablates quality filtering. The equivalent knobs here are decontamination (--no_decontaminate) and MCQ answer-format alignment (--no_answer_alignment). Each additional arm costs one more training run - the table below fills in from whatever runs exist.

Run	MedQA	MedMCQA	PubMedQA	MMLU-Pro	MB-op4	MB-op5	MedXpert	HLE(med)	Avg (8)	Avg (Table 4, 6)
brainmed-8b-final	76.67	64.07	79.1	65.73	62.01	54.87	18.63	10.68	53.97	59.23
brainmed-8b-v1	72.58	60.22	77.7	61.82	56.49	50.65	17.81	20.39	52.21	55.91
brainmed-8b-v1__best	72.58	60.22	77.7		56.49	50.65	17.81		-	55.91
brainmed-8b-v1__epoch-1	70.62	59.77	77.5		55.52	50.0	15.87		-	54.88
brainmed-8b-v1__epoch-2	72.58	60.22	77.7		56.49	50.65	17.81		-	55.91
brainmed-8b-v1__epoch-3	72.11	61.2	77.1		58.77	56.82	18.08		-	57.35
brainmed-8b-v1__last	72.11	61.2	77.1		58.77	56.82	18.08		-	57.35
brainmed-8b-v1__soup-a0.15	77.85	63.21	81.2		60.06	50.97	16.08		-	58.23
brainmed-8b-v1__soup-a0.3	76.28	62.54	80.0		59.42	54.87	16.43		-	58.26
brainmed-8b-v1__soup-last-a0.3	76.67	64.07	79.1		62.01	52.92	18.63		-	58.9
brainmed-8b-v1__soup-last-a0.5	76.9	64.36	78.9		58.77	54.55	18.91		-	58.73
brainmed-8b-v1__soup-last-a0.7	76.12	62.87	77.9		60.71	56.49	19.32		-	58.9

The two averages cover different column sets and must never be compared to each other; Avg (Table 4, 6) is the one the published tables use.

All evaluated benchmarks

Every benchmark scored in this run (10 of them). The paper's averages cover only the eight it reports; the extras below are measured and kept for the record but excluded from those averages.

Benchmark	n	In paper avg	base-HuatuoGPT-o1-8B	brainmed-8b-final	brainmed-8b-v1	brainmed-8b-v1__best	brainmed-8b-v1__epoch-1	brainmed-8b-v1__epoch-2	brainmed-8b-v1__epoch-3	brainmed-8b-v1__last	brainmed-8b-v1__soup-a0.15	brainmed-8b-v1__soup-a0.3	brainmed-8b-v1__soup-last-a0.3	brainmed-8b-v1__soup-last-a0.5	brainmed-8b-v1__soup-last-a0.7
MedQA	1273	yes	77.93	76.67	72.58	72.58	70.62	72.58	72.11	72.11	77.85	76.28	76.67	76.9	76.12
MedMCQA	4183	yes	63.3	64.07	60.22	60.22	59.77	60.22	61.2	61.2	63.21	62.54	64.07	64.36	62.87
PubMedQA	1000	yes	80.2	79.1	77.7	77.7	77.5	77.7	77.1	77.1	81.2	80.0	79.1	78.9	77.9
MMLU-Pro	1535	yes	64.56	65.73	61.82	-	-	-	-	-	-	-	-	-	-
MB-op4	308	yes	59.09	62.01	56.49	56.49	55.52	56.49	58.77	58.77	60.06	59.42	62.01	58.77	60.71
MB-op5	308	yes	55.19	54.87	50.65	50.65	50.0	50.65	56.82	56.82	50.97	54.87	52.92	54.55	56.49
MedXpert	1449	yes	17.05	18.63	17.81	17.81	15.87	17.81	18.08	18.08	16.08	16.43	18.63	18.91	19.32
HLE(med)	103	yes	11.65	10.68	20.39	-	-	-	-	-	-	-	-	-	-
GPQA(med)	390	extra	49.74	59.74	49.23	-	-	-	-	-	-	-	-	-	-
MedQA-5opt	1273	extra	72.82	73.68	68.81	-	-	-	-	-	-	-	-	-	-

Paper-comparable suite (8 benchmarks)

Run	MedQA	MedMCQA	PubMedQA	MMLU-Pro	Avg common	MB-op4	MB-op5	MedXpert	HLE(med)	Avg challenging	Avg overall
base-HuatuoGPT-o1-8B	77.93	63.3	80.2	64.56	71.5	59.09	55.19	17.05	11.65	35.75	53.62
brainmed-8b-final	76.67	64.07	79.1	65.73	71.39	62.01	54.87	18.63	10.68	36.55	53.97
brainmed-8b-v1	72.58	60.22	77.7	61.82	68.08	56.49	50.65	17.81	20.39	36.34	52.21
brainmed-8b-v1__best	72.58	60.22	77.7		-	56.49	50.65	17.81		-	-
brainmed-8b-v1__epoch-1	70.62	59.77	77.5		-	55.52	50.0	15.87		-	-
brainmed-8b-v1__epoch-2	72.58	60.22	77.7		-	56.49	50.65	17.81		-	-
brainmed-8b-v1__epoch-3	72.11	61.2	77.1		-	58.77	56.82	18.08		-	-
brainmed-8b-v1__last	72.11	61.2	77.1		-	58.77	56.82	18.08		-	-
brainmed-8b-v1__soup-a0.15	77.85	63.21	81.2		-	60.06	50.97	16.08		-	-
brainmed-8b-v1__soup-a0.3	76.28	62.54	80.0		-	59.42	54.87	16.43		-	-
brainmed-8b-v1__soup-last-a0.3	76.67	64.07	79.1		-	62.01	52.92	18.63		-	-
brainmed-8b-v1__soup-last-a0.5	76.9	64.36	78.9		-	58.77	54.55	18.91		-	-
brainmed-8b-v1__soup-last-a0.7	76.12	62.87	77.9		-	60.71	56.49	19.32		-	-

- marks an average that cannot be computed: brainmed-8b-v1__best, brainmed-8b-v1__epoch-1, brainmed-8b-v1__epoch-2, brainmed-8b-v1__epoch-3, brainmed-8b-v1__last, brainmed-8b-v1__soup-a0.15, brainmed-8b-v1__soup-a0.3, brainmed-8b-v1__soup-last-a0.3, brainmed-8b-v1__soup-last-a0.5, brainmed-8b-v1__soup-last-a0.7 were scored on a subset of the suite (QUICK sweep). Re-run the full bundle before quoting an 8-benchmark average for them.

Win conditions

Two things must both hold: the model beats the backbone it started from, and it beats every published 7B-8B model in MedReason Table 4. The backbone is compared on numbers measured here; the others on their published rows.

Run	Condition	Reference	Ours	Delta	Verdict
brainmed-8b-final	beats backbone (base-HuatuoGPT-o1-8B, measured)	58.79	59.23	+0.44	PASS
brainmed-8b-final	beats published best (MedReason-8B)	57.3	59.23	+1.93	PASS
brainmed-8b-final	beats all 11 published models	-	59.23	11/11	PASS
brainmed-8b-v1	beats backbone (base-HuatuoGPT-o1-8B, measured)	58.79	55.91	-2.88	FAIL
brainmed-8b-v1	beats published best (MedReason-8B)	57.3	55.91	-1.39	FAIL
brainmed-8b-v1	beats all 11 published models	-	55.91	10/11	FAIL (MedReason-8B)
brainmed-8b-v1__best	beats backbone (base-HuatuoGPT-o1-8B, measured)	58.79	55.91	-2.88	FAIL
brainmed-8b-v1__best	beats published best (MedReason-8B)	57.3	55.91	-1.39	FAIL
brainmed-8b-v1__best	beats all 11 published models	-	55.91	10/11	FAIL (MedReason-8B)
brainmed-8b-v1__epoch-1	beats backbone (base-HuatuoGPT-o1-8B, measured)	58.79	54.88	-3.91	FAIL
brainmed-8b-v1__epoch-1	beats published best (MedReason-8B)	57.3	54.88	-2.42	FAIL
brainmed-8b-v1__epoch-1	beats all 11 published models	-	54.88	9/11	FAIL (Huatuo-o1-RL-8B, MedReason-8B)
brainmed-8b-v1__epoch-2	beats backbone (base-HuatuoGPT-o1-8B, measured)	58.79	55.91	-2.88	FAIL
brainmed-8b-v1__epoch-2	beats published best (MedReason-8B)	57.3	55.91	-1.39	FAIL
brainmed-8b-v1__epoch-2	beats all 11 published models	-	55.91	10/11	FAIL (MedReason-8B)
brainmed-8b-v1__epoch-3	beats backbone (base-HuatuoGPT-o1-8B, measured)	58.79	57.35	-1.44	FAIL
brainmed-8b-v1__epoch-3	beats published best (MedReason-8B)	57.3	57.35	+0.05	PASS
brainmed-8b-v1__epoch-3	beats all 11 published models	-	57.35	11/11	PASS
brainmed-8b-v1__last	beats backbone (base-HuatuoGPT-o1-8B, measured)	58.79	57.35	-1.44	FAIL
brainmed-8b-v1__last	beats published best (MedReason-8B)	57.3	57.35	+0.05	PASS
brainmed-8b-v1__last	beats all 11 published models	-	57.35	11/11	PASS
brainmed-8b-v1__soup-a0.15	beats backbone (base-HuatuoGPT-o1-8B, measured)	58.79	58.23	-0.56	FAIL
brainmed-8b-v1__soup-a0.15	beats published best (MedReason-8B)	57.3	58.23	+0.93	PASS
brainmed-8b-v1__soup-a0.15	beats all 11 published models	-	58.23	11/11	PASS
brainmed-8b-v1__soup-a0.3	beats backbone (base-HuatuoGPT-o1-8B, measured)	58.79	58.26	-0.53	FAIL
brainmed-8b-v1__soup-a0.3	beats published best (MedReason-8B)	57.3	58.26	+0.96	PASS
brainmed-8b-v1__soup-a0.3	beats all 11 published models	-	58.26	11/11	PASS
brainmed-8b-v1__soup-last-a0.3	beats backbone (base-HuatuoGPT-o1-8B, measured)	58.79	58.9	+0.11	PASS
brainmed-8b-v1__soup-last-a0.3	beats published best (MedReason-8B)	57.3	58.9	+1.60	PASS
brainmed-8b-v1__soup-last-a0.3	beats all 11 published models	-	58.9	11/11	PASS
brainmed-8b-v1__soup-last-a0.5	beats backbone (base-HuatuoGPT-o1-8B, measured)	58.79	58.73	-0.06	FAIL
brainmed-8b-v1__soup-last-a0.5	beats published best (MedReason-8B)	57.3	58.73	+1.43	PASS
brainmed-8b-v1__soup-last-a0.5	beats all 11 published models	-	58.73	11/11	PASS
brainmed-8b-v1__soup-last-a0.7	beats backbone (base-HuatuoGPT-o1-8B, measured)	58.79	58.9	+0.11	PASS
brainmed-8b-v1__soup-last-a0.7	beats published best (MedReason-8B)	57.3	58.9	+1.60	PASS
brainmed-8b-v1__soup-last-a0.7	beats all 11 published models	-	58.9	11/11	PASS

Controlled before/after - same harness (base-HuatuoGPT-o1-8B -> brainmed-8b-final)

Benchmark	Before	After	Delta	Gate
MedQA	77.93	76.67	-1.26	REGRESSION
MedMCQA	63.3	64.07	+0.77	OK
PubMedQA	80.2	79.1	-1.10	REGRESSION
MMLU-Pro	64.56	65.73	+1.17	OK
MB-op4	59.09	62.01	+2.92	OK
MB-op5	55.19	54.87	-0.32	OK
MedXpert	17.05	18.63	+1.58	OK
HLE(med)	11.65	10.68	-0.97	REGRESSION
Average	53.62	53.97	+0.35

GATE FAILED - 3 benchmark(s) below the -0.5 pt tolerance: MedQA -1.26, PubMedQA -1.10, HLE(med) -0.97.

The fine-tune traded existing strengths for its gains. Before shipping, either lower the learning rate / shorten the schedule, or interpolate towards the backbone with scripts/weight_soup.py and re-evaluate.

Controlled before/after - same harness (base-HuatuoGPT-o1-8B -> brainmed-8b-v1)

Benchmark	Before	After	Delta	Gate
MedQA	77.93	72.58	-5.35	REGRESSION
MedMCQA	63.3	60.22	-3.08	REGRESSION
PubMedQA	80.2	77.7	-2.50	REGRESSION
MMLU-Pro	64.56	61.82	-2.74	REGRESSION
MB-op4	59.09	56.49	-2.60	REGRESSION
MB-op5	55.19	50.65	-4.54	REGRESSION
MedXpert	17.05	17.81	+0.76	OK
HLE(med)	11.65	20.39	+8.74	OK
Average	53.62	52.21	-1.41

GATE FAILED - 6 benchmark(s) below the -0.5 pt tolerance: MedQA -5.35, MedMCQA -3.08, PubMedQA -2.50, MMLU-Pro -2.74, MB-op4 -2.60, MB-op5 -4.54.

Controlled before/after - same harness (base-HuatuoGPT-o1-8B -> brainmed-8b-v1__best)

Benchmark	Before	After	Delta	Gate
MedQA	77.93	72.58	-5.35	REGRESSION
MedMCQA	63.3	60.22	-3.08	REGRESSION
PubMedQA	80.2	77.7	-2.50	REGRESSION
MB-op4	59.09	56.49	-2.60	REGRESSION
MB-op5	55.19	50.65	-4.54	REGRESSION
MedXpert	17.05	17.81	+0.76	OK
Average	58.79	55.91	-2.88

GATE FAILED - 5 benchmark(s) below the -0.5 pt tolerance: MedQA -5.35, MedMCQA -3.08, PubMedQA -2.50, MB-op4 -2.60, MB-op5 -4.54.

Controlled before/after - same harness (base-HuatuoGPT-o1-8B -> brainmed-8b-v1__epoch-1)

Benchmark	Before	After	Delta	Gate
MedQA	77.93	70.62	-7.31	REGRESSION
MedMCQA	63.3	59.77	-3.53	REGRESSION
PubMedQA	80.2	77.5	-2.70	REGRESSION
MB-op4	59.09	55.52	-3.57	REGRESSION
MB-op5	55.19	50.0	-5.19	REGRESSION
MedXpert	17.05	15.87	-1.18	REGRESSION
Average	58.79	54.88	-3.91

GATE FAILED - 6 benchmark(s) below the -0.5 pt tolerance: MedQA -7.31, MedMCQA -3.53, PubMedQA -2.70, MB-op4 -3.57, MB-op5 -5.19, MedXpert -1.18.

Controlled before/after - same harness (base-HuatuoGPT-o1-8B -> brainmed-8b-v1__epoch-2)

Benchmark	Before	After	Delta	Gate
MedQA	77.93	72.58	-5.35	REGRESSION
MedMCQA	63.3	60.22	-3.08	REGRESSION
PubMedQA	80.2	77.7	-2.50	REGRESSION
MB-op4	59.09	56.49	-2.60	REGRESSION
MB-op5	55.19	50.65	-4.54	REGRESSION
MedXpert	17.05	17.81	+0.76	OK
Average	58.79	55.91	-2.88

GATE FAILED - 5 benchmark(s) below the -0.5 pt tolerance: MedQA -5.35, MedMCQA -3.08, PubMedQA -2.50, MB-op4 -2.60, MB-op5 -4.54.

Controlled before/after - same harness (base-HuatuoGPT-o1-8B -> brainmed-8b-v1__epoch-3)

Benchmark	Before	After	Delta	Gate
MedQA	77.93	72.11	-5.82	REGRESSION
MedMCQA	63.3	61.2	-2.10	REGRESSION
PubMedQA	80.2	77.1	-3.10	REGRESSION
MB-op4	59.09	58.77	-0.32	OK
MB-op5	55.19	56.82	+1.63	OK
MedXpert	17.05	18.08	+1.03	OK
Average	58.79	57.35	-1.45

GATE FAILED - 3 benchmark(s) below the -0.5 pt tolerance: MedQA -5.82, MedMCQA -2.10, PubMedQA -3.10.

Controlled before/after - same harness (base-HuatuoGPT-o1-8B -> brainmed-8b-v1__last)

Benchmark	Before	After	Delta	Gate
MedQA	77.93	72.11	-5.82	REGRESSION
MedMCQA	63.3	61.2	-2.10	REGRESSION
PubMedQA	80.2	77.1	-3.10	REGRESSION
MB-op4	59.09	58.77	-0.32	OK
MB-op5	55.19	56.82	+1.63	OK
MedXpert	17.05	18.08	+1.03	OK
Average	58.79	57.35	-1.45

GATE FAILED - 3 benchmark(s) below the -0.5 pt tolerance: MedQA -5.82, MedMCQA -2.10, PubMedQA -3.10.

Controlled before/after - same harness (base-HuatuoGPT-o1-8B -> brainmed-8b-v1__soup-a0.15)

Benchmark	Before	After	Delta	Gate
MedQA	77.93	77.85	-0.08	OK
MedMCQA	63.3	63.21	-0.09	OK
PubMedQA	80.2	81.2	+1.00	OK
MB-op4	59.09	60.06	+0.97	OK
MB-op5	55.19	50.97	-4.22	REGRESSION
MedXpert	17.05	16.08	-0.97	REGRESSION
Average	58.79	58.23	-0.56

GATE FAILED - 2 benchmark(s) below the -0.5 pt tolerance: MB-op5 -4.22, MedXpert -0.97.

Controlled before/after - same harness (base-HuatuoGPT-o1-8B -> brainmed-8b-v1__soup-a0.3)

Benchmark	Before	After	Delta	Gate
MedQA	77.93	76.28	-1.65	REGRESSION
MedMCQA	63.3	62.54	-0.76	REGRESSION
PubMedQA	80.2	80.0	-0.20	OK
MB-op4	59.09	59.42	+0.33	OK
MB-op5	55.19	54.87	-0.32	OK
MedXpert	17.05	16.43	-0.62	REGRESSION
Average	58.79	58.26	-0.54

GATE FAILED - 3 benchmark(s) below the -0.5 pt tolerance: MedQA -1.65, MedMCQA -0.76, MedXpert -0.62.

Controlled before/after - same harness (base-HuatuoGPT-o1-8B -> brainmed-8b-v1__soup-last-a0.3)

Benchmark	Before	After	Delta	Gate
MedQA	77.93	76.67	-1.26	REGRESSION
MedMCQA	63.3	64.07	+0.77	OK
PubMedQA	80.2	79.1	-1.10	REGRESSION
MB-op4	59.09	62.01	+2.92	OK
MB-op5	55.19	52.92	-2.27	REGRESSION
MedXpert	17.05	18.63	+1.58	OK
Average	58.79	58.9	+0.11

GATE FAILED - 3 benchmark(s) below the -0.5 pt tolerance: MedQA -1.26, PubMedQA -1.10, MB-op5 -2.27.

Controlled before/after - same harness (base-HuatuoGPT-o1-8B -> brainmed-8b-v1__soup-last-a0.5)

Benchmark	Before	After	Delta	Gate
MedQA	77.93	76.9	-1.03	REGRESSION
MedMCQA	63.3	64.36	+1.06	OK
PubMedQA	80.2	78.9	-1.30	REGRESSION
MB-op4	59.09	58.77	-0.32	OK
MB-op5	55.19	54.55	-0.64	REGRESSION
MedXpert	17.05	18.91	+1.86	OK
Average	58.79	58.73	-0.06

GATE FAILED - 3 benchmark(s) below the -0.5 pt tolerance: MedQA -1.03, PubMedQA -1.30, MB-op5 -0.64.

Controlled before/after - same harness (base-HuatuoGPT-o1-8B -> brainmed-8b-v1__soup-last-a0.7)

Benchmark	Before	After	Delta	Gate
MedQA	77.93	76.12	-1.81	REGRESSION
MedMCQA	63.3	62.87	-0.43	OK
PubMedQA	80.2	77.9	-2.30	REGRESSION
MB-op4	59.09	60.71	+1.62	OK
MB-op5	55.19	56.49	+1.30	OK
MedXpert	17.05	19.32	+2.27	OK
Average	58.79	58.9	+0.11

GATE FAILED - 2 benchmark(s) below the -0.5 pt tolerance: MedQA -1.81, PubMedQA -2.30.

Harness calibration - backbone measured here vs its published row

Backbone base-HuatuoGPT-o1-8B against the published Huatuo-o1-RL-8B. Small gaps are expected; a large one means the harness, not the training, is moving the numbers, and every delta downstream has to be read with that in mind.

Benchmark	Published	Measured here	Gap
MB-op4	55.2	59.09	+3.89
MB-op5	51.3	55.19	+3.89
MedXpert	16.7	17.05	+0.35
MedQA	72.6	77.93	+5.33
MedMCQA	60.4	63.3	+2.90
PubMedQA	79.2	80.2	+1.00
Mean gap			+2.89

Largest single-benchmark gap: 5.33 pt. That is large enough to confound a data-driven gain - prefer the measured before/after over comparisons with published rows.

Raw score vs clean subset

The training set kept the rows overlapping these benchmarks (the upstream setting, which is what the published tables were produced under). Raw is therefore the number comparable to the paper; Clean excludes the overlapping items and is the number to quote as an unbiased estimate. A large gap between the two means the score leans on memorised evaluation items.

Run	Benchmark	n	Contaminated	Raw	Clean	Bias
base-HuatuoGPT-o1-8B	MedQA	1273	1	77.93	77.99	-0.06
base-HuatuoGPT-o1-8B	MMLU-Pro	1535	340	64.56	64.52	+0.04
base-HuatuoGPT-o1-8B	MB-op4	308	35	59.09	58.97	+0.12
base-HuatuoGPT-o1-8B	MB-op5	308	35	55.19	53.48	+1.71
base-HuatuoGPT-o1-8B	MedXpert	1449	15	17.05	16.88	+0.17
base-HuatuoGPT-o1-8B	HLE(med)	103	30	11.65	8.22	+3.43
brainmed-8b-final	MedQA	1273	1	76.67	76.73	-0.06
brainmed-8b-final	MMLU-Pro	1535	340	65.73	65.44	+0.29
brainmed-8b-final	MB-op4	308	35	62.01	62.64	-0.63
brainmed-8b-final	MB-op5	308	35	54.87	54.21	+0.66
brainmed-8b-final	MedXpert	1449	15	18.63	18.34	+0.29
brainmed-8b-final	HLE(med)	103	30	10.68	6.85	+3.83
brainmed-8b-v1	MedQA	1273	1	72.58	72.56	+0.02
brainmed-8b-v1	MMLU-Pro	1535	340	61.82	60.67	+1.15
brainmed-8b-v1	MB-op4	308	35	56.49	56.41	+0.08
brainmed-8b-v1	MB-op5	308	35	50.65	49.08	+1.57
brainmed-8b-v1	MedXpert	1449	15	17.81	17.5	+0.31
brainmed-8b-v1	HLE(med)	103	30	20.39	17.81	+2.58
brainmed-8b-v1__best	MedQA	1273	1	72.58	72.56	+0.02
brainmed-8b-v1__best	MB-op4	308	35	56.49	56.41	+0.08
brainmed-8b-v1__best	MB-op5	308	35	50.65	49.08	+1.57
brainmed-8b-v1__best	MedXpert	1449	15	17.81	17.5	+0.31
brainmed-8b-v1__epoch-1	MedQA	1273	1	70.62	70.6	+0.02
brainmed-8b-v1__epoch-1	MB-op4	308	35	55.52	55.31	+0.21
brainmed-8b-v1__epoch-1	MB-op5	308	35	50.0	48.72	+1.28
brainmed-8b-v1__epoch-1	MedXpert	1449	15	15.87	15.83	+0.04
brainmed-8b-v1__epoch-2	MedQA	1273	1	72.58	72.56	+0.02
brainmed-8b-v1__epoch-2	MB-op4	308	35	56.49	56.41	+0.08
brainmed-8b-v1__epoch-2	MB-op5	308	35	50.65	49.08	+1.57
brainmed-8b-v1__epoch-2	MedXpert	1449	15	17.81	17.5	+0.31
brainmed-8b-v1__epoch-3	MedQA	1273	1	72.11	72.17	-0.06
brainmed-8b-v1__epoch-3	MB-op4	308	35	58.77	57.14	+1.63
brainmed-8b-v1__epoch-3	MB-op5	308	35	56.82	57.51	-0.69
brainmed-8b-v1__epoch-3	MedXpert	1449	15	18.08	17.78	+0.30
brainmed-8b-v1__last	MedQA	1273	1	72.11	72.17	-0.06
brainmed-8b-v1__last	MB-op4	308	35	58.77	57.14	+1.63
brainmed-8b-v1__last	MB-op5	308	35	56.82	57.51	-0.69
brainmed-8b-v1__last	MedXpert	1449	15	18.08	17.78	+0.30
brainmed-8b-v1__soup-a0.15	MedQA	1273	1	77.85	77.91	-0.06
brainmed-8b-v1__soup-a0.15	MB-op4	308	35	60.06	60.44	-0.38
brainmed-8b-v1__soup-a0.15	MB-op5	308	35	50.97	50.92	+0.05
brainmed-8b-v1__soup-a0.15	MedXpert	1449	15	16.08	15.76	+0.32
brainmed-8b-v1__soup-a0.3	MedQA	1273	1	76.28	76.26	+0.02
brainmed-8b-v1__soup-a0.3	MB-op4	308	35	59.42	60.07	-0.65
brainmed-8b-v1__soup-a0.3	MB-op5	308	35	54.87	54.95	-0.08
brainmed-8b-v1__soup-a0.3	MedXpert	1449	15	16.43	16.11	+0.32
brainmed-8b-v1__soup-last-a0.3	MedQA	1273	1	76.67	76.73	-0.06
brainmed-8b-v1__soup-last-a0.3	MB-op4	308	35	62.01	62.64	-0.63
brainmed-8b-v1__soup-last-a0.3	MB-op5	308	35	52.92	52.75	+0.17
brainmed-8b-v1__soup-last-a0.3	MedXpert	1449	15	18.63	18.34	+0.29
brainmed-8b-v1__soup-last-a0.5	MedQA	1273	1	76.9	76.97	-0.07
brainmed-8b-v1__soup-last-a0.5	MB-op4	308	35	58.77	57.88	+0.89
brainmed-8b-v1__soup-last-a0.5	MB-op5	308	35	54.55	53.11	+1.44
brainmed-8b-v1__soup-last-a0.5	MedXpert	1449	15	18.91	18.55	+0.36
brainmed-8b-v1__soup-last-a0.7	MedQA	1273	1	76.12	76.18	-0.06
brainmed-8b-v1__soup-last-a0.7	MB-op4	308	35	60.71	59.71	+1.00
brainmed-8b-v1__soup-last-a0.7	MB-op5	308	35	56.49	54.58	+1.91
brainmed-8b-v1__soup-last-a0.7	MedXpert	1449	15	19.32	19.11	+0.21
base-HuatuoGPT-o1-8B	Average (8)			53.62	52.95	+0.67
brainmed-8b-final	Average (8)			53.97	53.42	+0.55
brainmed-8b-v1	Average (8)			52.21	51.49	+0.72

Data integrity

training rows after decontamination: 44351 (removed 0, 0.0%)

Contamination found in the source corpus and removed before training:

Benchmark	n	Leaked items	%	Train rows dropped
gpqa_medical.jsonl	390	0	0.0%	0
hle_med.jsonl	103	30	29.13%	30
medbullets_op4.jsonl	308	35	11.36%	35
medbullets_op5.jsonl	308	35	11.36%	35
medmcqa_val.jsonl	4183	0	0.0%	0
medqa_4opt.jsonl	1273	1	0.08%	1
medqa_5opt.jsonl	1273	1	0.08%	1
medxpertqa.jsonl	1449	15	1.04%	17
mmlu_pro_medical.jsonl	1535	340	22.15%	261
pubmedqa_test.jsonl	1000	0	0.0%	0

Answer-format compliance

Share of outputs where the answer was read from an explicit The answer is X. rather than recovered by the scorer's fuzzy fallbacks. A low value means the accuracy above leans on string matching and should be treated with suspicion.

Run	Benchmark	Strict-format %
base-HuatuoGPT-o1-8B	MedQA	99.84%
base-HuatuoGPT-o1-8B	MedMCQA	99.81%
base-HuatuoGPT-o1-8B	PubMedQA	100.0%
base-HuatuoGPT-o1-8B	MMLU-Pro	99.15%
base-HuatuoGPT-o1-8B	MB-op4	100.0%
base-HuatuoGPT-o1-8B	MB-op5	99.68%
base-HuatuoGPT-o1-8B	MedXpert	96.27%
base-HuatuoGPT-o1-8B	HLE(med)	96.12%
base-HuatuoGPT-o1-8B	GPQA(med)	98.21%
base-HuatuoGPT-o1-8B	MedQA-5opt	99.76%
brainmed-8b-final	MedQA	99.14%
brainmed-8b-final	MedMCQA	99.83%
brainmed-8b-final	PubMedQA	99.9%
brainmed-8b-final	MMLU-Pro	99.48%
brainmed-8b-final	MB-op4	99.68%
brainmed-8b-final	MB-op5	99.35%
brainmed-8b-final	MedXpert	99.24%
brainmed-8b-final	HLE(med)	97.09%
brainmed-8b-final	GPQA(med)	98.46%
brainmed-8b-final	MedQA-5opt	98.9%
brainmed-8b-v1	MedQA	100.0%
brainmed-8b-v1	MedMCQA	99.88%
brainmed-8b-v1	PubMedQA	100.0%
brainmed-8b-v1	MMLU-Pro	99.61%
brainmed-8b-v1	MB-op4	100.0%
brainmed-8b-v1	MB-op5	100.0%
brainmed-8b-v1	MedXpert	100.0%
brainmed-8b-v1	HLE(med)	95.15%
brainmed-8b-v1	GPQA(med)	99.74%
brainmed-8b-v1	MedQA-5opt	99.92%
brainmed-8b-v1__best	MedQA	100.0%
brainmed-8b-v1__best	MedMCQA	99.88%
brainmed-8b-v1__best	PubMedQA	100.0%
brainmed-8b-v1__best	MB-op4	100.0%
brainmed-8b-v1__best	MB-op5	100.0%
brainmed-8b-v1__best	MedXpert	100.0%
brainmed-8b-v1__epoch-1	MedQA	99.92%
brainmed-8b-v1__epoch-1	MedMCQA	99.62%
brainmed-8b-v1__epoch-1	PubMedQA	99.9%
brainmed-8b-v1__epoch-1	MB-op4	100.0%
brainmed-8b-v1__epoch-1	MB-op5	100.0%
brainmed-8b-v1__epoch-1	MedXpert	99.72%
brainmed-8b-v1__epoch-2	MedQA	100.0%
brainmed-8b-v1__epoch-2	MedMCQA	99.88%
brainmed-8b-v1__epoch-2	PubMedQA	100.0%
brainmed-8b-v1__epoch-2	MB-op4	100.0%
brainmed-8b-v1__epoch-2	MB-op5	100.0%
brainmed-8b-v1__epoch-2	MedXpert	100.0%
brainmed-8b-v1__epoch-3	MedQA	100.0%
brainmed-8b-v1__epoch-3	MedMCQA	99.9%
brainmed-8b-v1__epoch-3	PubMedQA	100.0%
brainmed-8b-v1__epoch-3	MB-op4	100.0%
brainmed-8b-v1__epoch-3	MB-op5	100.0%
brainmed-8b-v1__epoch-3	MedXpert	100.0%
brainmed-8b-v1__last	MedQA	100.0%
brainmed-8b-v1__last	MedMCQA	99.9%
brainmed-8b-v1__last	PubMedQA	100.0%
brainmed-8b-v1__last	MB-op4	100.0%
brainmed-8b-v1__last	MB-op5	100.0%
brainmed-8b-v1__last	MedXpert	100.0%
brainmed-8b-v1__soup-a0.15	MedQA	89.32%
brainmed-8b-v1__soup-a0.15	MedMCQA	99.19%
brainmed-8b-v1__soup-a0.15	PubMedQA	99.6%
brainmed-8b-v1__soup-a0.15	MB-op4	86.69%
brainmed-8b-v1__soup-a0.15	MB-op5	87.66%
brainmed-8b-v1__soup-a0.15	MedXpert	92.41%
brainmed-8b-v1__soup-a0.3	MedQA	99.69%
brainmed-8b-v1__soup-a0.3	MedMCQA	99.64%
brainmed-8b-v1__soup-a0.3	PubMedQA	100.0%
brainmed-8b-v1__soup-a0.3	MB-op4	99.03%
brainmed-8b-v1__soup-a0.3	MB-op5	99.68%
brainmed-8b-v1__soup-a0.3	MedXpert	99.59%
brainmed-8b-v1__soup-last-a0.3	MedQA	99.14%
brainmed-8b-v1__soup-last-a0.3	MedMCQA	99.83%
brainmed-8b-v1__soup-last-a0.3	PubMedQA	99.9%
brainmed-8b-v1__soup-last-a0.3	MB-op4	99.68%
brainmed-8b-v1__soup-last-a0.3	MB-op5	99.35%
brainmed-8b-v1__soup-last-a0.3	MedXpert	99.24%
brainmed-8b-v1__soup-last-a0.5	MedQA	99.84%
brainmed-8b-v1__soup-last-a0.5	MedMCQA	99.86%
brainmed-8b-v1__soup-last-a0.5	PubMedQA	100.0%
brainmed-8b-v1__soup-last-a0.5	MB-op4	100.0%
brainmed-8b-v1__soup-last-a0.5	MB-op5	100.0%
brainmed-8b-v1__soup-last-a0.5	MedXpert	100.0%
brainmed-8b-v1__soup-last-a0.7	MedQA	100.0%
brainmed-8b-v1__soup-last-a0.7	MedMCQA	99.93%
brainmed-8b-v1__soup-last-a0.7	PubMedQA	100.0%
brainmed-8b-v1__soup-last-a0.7	MB-op4	100.0%
brainmed-8b-v1__soup-last-a0.7	MB-op5	100.0%
brainmed-8b-v1__soup-last-a0.7	MedXpert	100.0%