Instructions to use BrainHealthAI/BrainMed-8B-SFT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use BrainHealthAI/BrainMed-8B-SFT with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="BrainHealthAI/BrainMed-8B-SFT")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("BrainHealthAI/BrainMed-8B-SFT")
model = AutoModelForCausalLM.from_pretrained("BrainHealthAI/BrainMed-8B-SFT", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use BrainHealthAI/BrainMed-8B-SFT with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "BrainHealthAI/BrainMed-8B-SFT"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BrainHealthAI/BrainMed-8B-SFT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/BrainHealthAI/BrainMed-8B-SFT

SGLang

How to use BrainHealthAI/BrainMed-8B-SFT with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "BrainHealthAI/BrainMed-8B-SFT" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BrainHealthAI/BrainMed-8B-SFT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "BrainHealthAI/BrainMed-8B-SFT" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BrainHealthAI/BrainMed-8B-SFT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use BrainHealthAI/BrainMed-8B-SFT with Docker Model Runner:
```
docker model run hf.co/BrainHealthAI/BrainMed-8B-SFT
```

BrainMed-8B-SFT / evaluation /COMPARISON.md

Williamsanderson

evaluation results and figures

8a33135 verified 16 days ago

preview code

Raw

History Blame Contribute Delete

8.88 kB

Comparison against the published 7B-8B models

Our rows are measured by this pipeline. Every other row is transcribed from MedReason (arXiv:2504.00993) and was not re-run.

Excluded from this table: MedReason-8B.

Backbone shown with its published values (Huatuo-o1-RL-8B), not the score measured on this harness.

Harness offset measured on the untouched backbone: +2.89 pt vs the published protocol. Subtract it before reading our-vs-published gaps.

Table A — Ranking on the paper's six-benchmark suite

#	Model	MB-op4	MB-op5	MedXpert	MedQA	MedMCQA	PubMedQA	Avg
1	brainmed-8b-final (ours)	62.01	54.87	18.63	76.67	64.07	79.10	59.23
2	brainmed-8b-v1__soup-last-a0.3 (ours)	62.01	52.92	18.63	76.67	64.07	79.10	58.90
3	brainmed-8b-v1__soup-last-a0.7 (ours)	60.71	56.49	19.32	76.12	62.87	77.90	58.90
4	brainmed-8b-v1__soup-last-a0.5 (ours)	58.77	54.55	18.91	76.90	64.36	78.90	58.73
5	brainmed-8b-v1__soup-a0.3 (ours)	59.42	54.87	16.43	76.28	62.54	80.00	58.26
6	brainmed-8b-v1__soup-a0.15 (ours)	60.06	50.97	16.08	77.85	63.21	81.20	58.23
7	brainmed-8b-v1__epoch-3 (ours)	58.77	56.82	18.08	72.11	61.20	77.10	57.35
8	brainmed-8b-v1 (ours)	56.49	50.65	17.81	72.58	60.22	77.70	55.91
9	brainmed-8b-v1__best (ours)	56.49	50.65	17.81	72.58	60.22	77.70	55.91
10	Huatuo-o1-RL-8B	55.20	51.30	16.70	72.60	60.40	79.20	55.90
11	brainmed-8b-v1__epoch-1 (ours)	55.52	50.00	15.87	70.62	59.77	77.50	54.88
12	Huatuo-o1-SFT-8B	53.30	49.70	17.30	70.20	58.20	76.10	54.10
13	Qwen2.5-Instruct-7B	50.00	41.60	12.60	57.00	55.60	72.70	48.20
14	Llama3.1-Instruct-8B	43.20	40.90	14.30	58.70	56.00	75.20	48.00
15	OpenBioLLM-8B	39.20	35.70	10.70	57.70	54.10	74.10	45.30
16	DeepSeek-Distill-8B	41.90	35.10	13.50	55.40	49.00	73.90	44.80
17	Medical-CoT-8B	39.30	34.10	12.60	49.00	42.60	68.00	40.90
18	BioMistral-7B	46.40	33.10	12.40	45.00	40.20	66.90	40.70
19	Mistral-Instruct-7B	43.50	33.40	11.40	48.20	44.90	50.10	38.60
20	Medical-Llama3-8B	33.40	25.30	9.00	40.30	46.80	48.00	33.80

Table B — `brainmed-8b-final` vs published Huatuo-o1-RL-8B

| Benchmark | Huatuo-o1-RL-8B (published) | brainmed-8b-final

(measured) | Delta | | |---|---|---|---|---| | MB-op4 | 55.20 | 62.01 | +6.81 | win | | MB-op5 | 51.30 | 54.87 | +3.57 | win | | MedXpert | 16.70 | 18.63 | +1.93 | win | | MedQA | 72.60 | 76.67 | +4.07 | win | | MedMCQA | 60.40 | 64.07 | +3.67 | win | | PubMedQA | 79.20 | 79.10 | -0.10 | | | Average | 55.90 | 59.23 | +3.33 | win |

Table B — `brainmed-8b-v1` vs published Huatuo-o1-RL-8B

| Benchmark | Huatuo-o1-RL-8B (published) | brainmed-8b-v1

(measured) | Delta | | |---|---|---|---|---| | MB-op4 | 55.20 | 56.49 | +1.29 | win | | MB-op5 | 51.30 | 50.65 | -0.65 | | | MedXpert | 16.70 | 17.81 | +1.11 | win | | MedQA | 72.60 | 72.58 | -0.02 | | | MedMCQA | 60.40 | 60.22 | -0.18 | | | PubMedQA | 79.20 | 77.70 | -1.50 | | | Average | 55.90 | 55.91 | +0.01 | win |

Table B — `brainmed-8b-v1__best` vs published Huatuo-o1-RL-8B

| Benchmark | Huatuo-o1-RL-8B (published) | brainmed-8b-v1__best

Table B — `brainmed-8b-v1__epoch-1` vs published Huatuo-o1-RL-8B

| Benchmark | Huatuo-o1-RL-8B (published) | brainmed-8b-v1__epoch-1

(measured) | Delta | | |---|---|---|---|---| | MB-op4 | 55.20 | 55.52 | +0.32 | win | | MB-op5 | 51.30 | 50.00 | -1.30 | | | MedXpert | 16.70 | 15.87 | -0.83 | | | MedQA | 72.60 | 70.62 | -1.98 | | | MedMCQA | 60.40 | 59.77 | -0.63 | | | PubMedQA | 79.20 | 77.50 | -1.70 | | | Average | 55.90 | 54.88 | -1.02 | |

Table B — `brainmed-8b-v1__epoch-3` vs published Huatuo-o1-RL-8B

| Benchmark | Huatuo-o1-RL-8B (published) | brainmed-8b-v1__epoch-3

(measured) | Delta | | |---|---|---|---|---| | MB-op4 | 55.20 | 58.77 | +3.57 | win | | MB-op5 | 51.30 | 56.82 | +5.52 | win | | MedXpert | 16.70 | 18.08 | +1.38 | win | | MedQA | 72.60 | 72.11 | -0.49 | | | MedMCQA | 60.40 | 61.20 | +0.80 | win | | PubMedQA | 79.20 | 77.10 | -2.10 | | | Average | 55.90 | 57.35 | +1.45 | win |

Table B — `brainmed-8b-v1__soup-a0.15` vs published Huatuo-o1-RL-8B

| Benchmark | Huatuo-o1-RL-8B (published) | brainmed-8b-v1__soup-a0.15

(measured) | Delta | | |---|---|---|---|---| | MB-op4 | 55.20 | 60.06 | +4.86 | win | | MB-op5 | 51.30 | 50.97 | -0.33 | | | MedXpert | 16.70 | 16.08 | -0.62 | | | MedQA | 72.60 | 77.85 | +5.25 | win | | MedMCQA | 60.40 | 63.21 | +2.81 | win | | PubMedQA | 79.20 | 81.20 | +2.00 | win | | Average | 55.90 | 58.23 | +2.33 | win |

Table B — `brainmed-8b-v1__soup-a0.3` vs published Huatuo-o1-RL-8B

| Benchmark | Huatuo-o1-RL-8B (published) | brainmed-8b-v1__soup-a0.3

(measured) | Delta | | |---|---|---|---|---| | MB-op4 | 55.20 | 59.42 | +4.22 | win | | MB-op5 | 51.30 | 54.87 | +3.57 | win | | MedXpert | 16.70 | 16.43 | -0.27 | | | MedQA | 72.60 | 76.28 | +3.68 | win | | MedMCQA | 60.40 | 62.54 | +2.14 | win | | PubMedQA | 79.20 | 80.00 | +0.80 | win | | Average | 55.90 | 58.26 | +2.36 | win |

Table B — `brainmed-8b-v1__soup-last-a0.3` vs published Huatuo-o1-RL-8B

| Benchmark | Huatuo-o1-RL-8B (published) | brainmed-8b-v1__soup-last-a0.3

(measured) | Delta | | |---|---|---|---|---| | MB-op4 | 55.20 | 62.01 | +6.81 | win | | MB-op5 | 51.30 | 52.92 | +1.62 | win | | MedXpert | 16.70 | 18.63 | +1.93 | win | | MedQA | 72.60 | 76.67 | +4.07 | win | | MedMCQA | 60.40 | 64.07 | +3.67 | win | | PubMedQA | 79.20 | 79.10 | -0.10 | | | Average | 55.90 | 58.90 | +3.00 | win |

Table B — `brainmed-8b-v1__soup-last-a0.5` vs published Huatuo-o1-RL-8B

| Benchmark | Huatuo-o1-RL-8B (published) | brainmed-8b-v1__soup-last-a0.5

(measured) | Delta | | |---|---|---|---|---| | MB-op4 | 55.20 | 58.77 | +3.57 | win | | MB-op5 | 51.30 | 54.55 | +3.25 | win | | MedXpert | 16.70 | 18.91 | +2.21 | win | | MedQA | 72.60 | 76.90 | +4.30 | win | | MedMCQA | 60.40 | 64.36 | +3.96 | win | | PubMedQA | 79.20 | 78.90 | -0.30 | | | Average | 55.90 | 58.73 | +2.83 | win |

Table B — `brainmed-8b-v1__soup-last-a0.7` vs published Huatuo-o1-RL-8B

| Benchmark | Huatuo-o1-RL-8B (published) | brainmed-8b-v1__soup-last-a0.7

(measured) | Delta | | |---|---|---|---|---| | MB-op4 | 55.20 | 60.71 | +5.51 | win | | MB-op5 | 51.30 | 56.49 | +5.19 | win | | MedXpert | 16.70 | 19.32 | +2.62 | win | | MedQA | 72.60 | 76.12 | +3.52 | win | | MedMCQA | 60.40 | 62.87 | +2.47 | win | | PubMedQA | 79.20 | 77.90 | -1.30 | | | Average | 55.90 | 58.90 | +3.00 | win |

Table D — Eight-benchmark suite (MedReason Table 2 columns)

Model / data	MedQA	MedMCQA	PubMedQA	MMLU-Pro	MB-op4	MB-op5	MedXpert	HLE(med)	Avg
Llama3.1-Instruct-8B (base)	58.70	56.00	75.20	58.20	48.70	42.50	13.20	13.60	45.80
Llama3.1-Instruct-8B + Huatuo CoT	70.20	58.20	76.10	59.90	53.30	49.70	17.30	14.60	49.90
Llama3.1-Instruct-8B + MedReason	68.40	57.50	77.60	63.10	57.50	52.30	16.40	16.50	51.20
Mistral-Instruct-7B (base)	48.20	44.90	50.10	42.70	43.50	33.40	11.40	14.60	36.10
Mistral-Instruct-7B + Huatuo CoT	59.90	46.90	57.50	47.60	50.00	46.10	14.40	14.60	42.10
Mistral-Instruct-7B + MedReason	58.70	48.90	59.20	50.80	52.30	47.10	16.60	24.30	44.70
brainmed-8b-final (ours)	76.67	64.07	79.10	65.73	62.01	54.87	18.63	10.68	53.97
brainmed-8b-v1 (ours)	72.58	60.22	77.70	61.82	56.49	50.65	17.81	20.39	52.21

Comparison against the published 7B-8B models

Table A — Ranking on the paper's six-benchmark suite

Table B — brainmed-8b-final vs published Huatuo-o1-RL-8B

Table B — brainmed-8b-v1 vs published Huatuo-o1-RL-8B

Table B — brainmed-8b-v1__best vs published Huatuo-o1-RL-8B

Table B — brainmed-8b-v1__epoch-1 vs published Huatuo-o1-RL-8B

Table B — brainmed-8b-v1__epoch-3 vs published Huatuo-o1-RL-8B

Table B — brainmed-8b-v1__soup-a0.15 vs published Huatuo-o1-RL-8B

Table B — brainmed-8b-v1__soup-a0.3 vs published Huatuo-o1-RL-8B

Table B — brainmed-8b-v1__soup-last-a0.3 vs published Huatuo-o1-RL-8B

Table B — brainmed-8b-v1__soup-last-a0.5 vs published Huatuo-o1-RL-8B

Table B — brainmed-8b-v1__soup-last-a0.7 vs published Huatuo-o1-RL-8B