bechir09 commited on
Commit
da53f7f
·
verified ·
1 Parent(s): 0a51fc3

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +222 -203
README.md CHANGED
@@ -2,285 +2,304 @@
2
  license: apache-2.0
3
  language:
4
  - en
 
 
5
  library_name: transformers
6
  pipeline_tag: text-generation
7
  tags:
8
  - medical
9
- - healthcare
10
- - reasoning
11
- - llm
12
- - deepseek
13
- - qwen
14
  - tanit
15
- base_model: deepseek-ai/DeepSeek-R1-0528-Qwen3-8B
 
 
 
 
 
 
 
16
  model-index:
17
- - name: Tanit-MedReason-8B
18
  results:
19
- - task:
20
- type: question-answering
21
- name: Medical QA
22
- metrics:
23
- - type: accuracy
24
- value: 60.3
25
- name: Test Accuracy
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  ---
27
 
28
- <div align="center">
29
 
30
- # 🏥 Tanit-MedReason-8B
31
 
32
- ### Advanced Medical Reasoning Model by TANIT Healthcare Technologies
33
 
34
- [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
35
- [![Model](https://img.shields.io/badge/Model-8B%20Parameters-green.svg)]()
36
- [![Phase](https://img.shields.io/badge/Phase-Final%20(v1.0)-orange.svg)]()
37
 
38
- </div>
39
 
40
  ---
41
 
42
- ## 🌟 Model Overview
43
 
44
- **Tanit-MedReason-8B** is a specialized medical reasoning model developed by [TANIT Healthcare Technologies](https://tanitai.com).
 
 
 
 
 
 
 
 
 
45
 
46
- Production-ready medical reasoning model trained through a 4-phase curriculum: Broad SFT → Reasoning SFT → DPO Alignment → MedReason Chain-of-Thought fine-tuning. Achieves strong performance across medical benchmarks with transparent reasoning via <think> tags.
47
 
48
- ### Key Features
49
 
50
- - 🧠 **Advanced Medical Reasoning**: Deep chain-of-thought reasoning for complex medical scenarios
51
- - 🔬 **Evidence-Based Responses**: Trained on validated medical knowledge sources
52
- - 💭 **Transparent Thinking**: Exposes reasoning process via `<think>` tags
53
- - ⚡ **Efficient**: 8B parameters optimized for deployment
54
- - 🛡️ **Safety-Aligned**: DPO-trained for safer, more helpful responses
55
 
 
56
 
57
- ## 🔬 Training Pipeline
 
 
 
 
 
 
 
 
 
 
 
 
58
 
59
- Tanit-MedReason-8B was developed through a carefully designed 4-phase training pipeline:
60
 
61
- ```
62
- ┌─────────────────────────────────────────────────────────────────┐
63
- │ Phase 1: Broad Medical SFT (24.5 hours) │
64
- │ ├── 800K+ diverse medical samples │
65
- │ ├── Foundation knowledge acquisition │
66
- │ └── Loss: 1.129 → Comprehensive medical understanding │
67
- ├─────────────────────────────────────────────────────────────────┤
68
- │ Phase 2: Reasoning-Focused SFT (6.8 hours) │
69
- │ ├── 155K high-quality reasoning chains │
70
- │ ├── Focus: Medical-R1-Distill + Huatuo-o1 │
71
- │ └── Loss: 1.014 → Enhanced reasoning capabilities │
72
- ├─────────────────────────────────────────────────────────────────┤
73
- │ Phase 3A: DPO Preference Alignment (5.9 hours) │
74
- │ ├── 33K preference pairs (FineMed-DPO) │
75
- │ ├── Direct Preference Optimization │
76
- │ └── Loss: 0.263 → Human-aligned responses │
77
- ├─────────────────────────────────────────────────────────────────┤
78
- │ Phase 4: MedReason CoT Fine-Tuning (3 hours) │
79
- │ ├── 35K samples from UCSC-VLAA/MedReason │
80
- │ ├── Structured medical reasoning with <think> tags │
81
- │ └── Loss: 1.662 → Production-ready model │
82
- └─────────────────────────────────────────────────────────────────┘
83
- ```
84
 
85
- **Total Training Time**: ~42 hours on NVIDIA H200 144GB
86
 
 
87
 
88
- ## 🚀 Quick Start
 
 
 
 
 
 
 
 
 
 
 
 
89
 
90
- ### Installation
91
 
92
- ```bash
93
- pip install transformers torch accelerate
94
- ```
95
 
96
- ### Using with Transformers
97
 
98
- ```python
99
- from transformers import AutoModelForCausalLM, AutoTokenizer
100
- import torch
101
 
102
- model_name = "TanitAI/Tanit-MedReason-8B"
103
 
104
- tokenizer = AutoTokenizer.from_pretrained(model_name)
105
- model = AutoModelForCausalLM.from_pretrained(
106
- model_name,
107
- torch_dtype=torch.bfloat16,
108
- device_map="auto",
109
- trust_remote_code=True
110
- )
111
 
112
- # Medical question
113
- question = """A 45-year-old male presents with sudden onset chest pain radiating to
114
- the left arm, diaphoresis, and shortness of breath. ECG shows ST elevation in
115
- leads V1-V4. What is the most likely diagnosis and immediate management?"""
116
 
117
- messages = [
118
- {"role": "user", "content": question}
119
- ]
 
 
 
 
 
120
 
121
- # Apply chat template
122
- input_text = tokenizer.apply_chat_template(
123
- messages,
124
- tokenize=False,
125
- add_generation_prompt=True
126
- )
127
 
128
- inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
129
 
130
- # Generate response
131
- with torch.no_grad():
132
- outputs = model.generate(
133
- **inputs,
134
- max_new_tokens=2048,
135
- temperature=0.6,
136
- top_p=0.95,
137
- do_sample=True
138
- )
139
 
140
- response = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
141
- print(response)
142
- ```
143
 
144
- ### Using with vLLM (Recommended for Production)
145
 
146
- ```python
147
- from vllm import LLM, SamplingParams
 
 
 
 
148
 
149
- model_name = "TanitAI/Tanit-MedReason-8B"
150
 
151
- llm = LLM(
152
- model=model_name,
153
- dtype="bfloat16",
154
- tensor_parallel_size=1, # Adjust based on your GPU setup
155
- trust_remote_code=True,
156
- max_model_len=8192
157
- )
158
 
159
- sampling_params = SamplingParams(
160
- temperature=0.6,
161
- top_p=0.95,
162
- max_tokens=2048
163
- )
164
 
165
- question = "What are the diagnostic criteria for Type 2 Diabetes Mellitus?"
166
 
167
- # Format with chat template
168
- prompt = f"<|im_start|>user\n{question}<|im_end|>\n<|im_start|>assistant\n"
169
 
170
- outputs = llm.generate([prompt], sampling_params)
171
- print(outputs[0].outputs[0].text)
172
- ```
 
 
 
173
 
174
- ### Using via API (Without Loading Model)
175
 
176
- For team members who need access without loading the model locally, use the HuggingFace Inference API:
 
 
 
 
 
 
 
 
 
 
 
177
 
178
- ```python
179
- import requests
180
-
181
- API_URL = "https://api-inference.huggingface.co/models/TanitAI/Tanit-MedReason-8B"
182
- headers = {"Authorization": "Bearer YOUR_HF_TOKEN"}
183
-
184
- def query(payload):
185
- response = requests.post(API_URL, headers=headers, json=payload)
186
- return response.json()
187
-
188
- # Example query
189
- output = query({
190
- "inputs": "What is the first-line treatment for hypertension?",
191
- "parameters": {
192
- "max_new_tokens": 1024,
193
- "temperature": 0.6
194
- }
195
- })
196
- print(output)
197
- ```
198
 
199
- ### Accessing via HuggingFace Hub
 
 
 
 
 
 
 
 
200
 
201
  ```python
202
- from huggingface_hub import InferenceClient
203
 
204
- client = InferenceClient(
205
- model="TanitAI/Tanit-MedReason-8B",
206
- token="YOUR_HF_TOKEN"
207
- )
208
 
209
- response = client.text_generation(
210
- "Explain the pathophysiology of heart failure.",
211
- max_new_tokens=1024,
212
- temperature=0.6
 
 
 
 
 
 
 
213
  )
214
- print(response)
 
 
215
  ```
216
 
217
- ## 📊 Evaluation Results
218
 
219
- | Benchmark | Accuracy |
220
- |-----------|----------|
221
- | MedQA | 60.3% |
222
- | MMLU Medical | 66.4% |
223
- | MedExQA | 62.1% |
224
- | PubMedQA | 74.6% |
225
- | MedMCQA | 52.9% |
226
 
227
- *Evaluated using zero-shot prompting with chain-of-thought reasoning.*
228
 
 
229
 
230
- ## 📋 Training Details
 
 
 
 
 
 
 
 
 
 
231
 
232
- | Parameter | Value |
233
- |-----------|-------|
234
- | **Base Model** | `deepseek-ai/DeepSeek-R1-0528-Qwen3-8B` |
235
- | **Training Phase** | Final (v1.0) |
236
- | **Training Data** | 35K samples from UCSC-VLAA/MedReason dataset with structured medical reasoning |
237
- | **Training Steps** | 4,442 |
238
- | **Training Time** | 3.0 hours |
239
- | **Final Loss** | 1.662 |
240
- | **Precision** | bfloat16 |
241
- | **Context Length** | 8,192 tokens |
242
 
243
- ## ⚠️ Limitations & Intended Use
244
 
245
- ### Intended Use
246
- - Medical education and research
247
- - Clinical decision support (with physician oversight)
248
- - Medical question answering
249
- - Healthcare documentation assistance
 
250
 
251
- ### Limitations
252
- - **Not a replacement for professional medical advice**
253
- - May generate plausible-sounding but incorrect information
254
- - Performance varies across medical specialties
255
- - Should always be used with human oversight in clinical settings
256
 
257
- ### Ethical Considerations
258
- - This model is intended to assist, not replace, healthcare professionals
259
- - Always verify medical information with authoritative sources
260
- - Do not use for diagnosis or treatment without professional consultation
261
 
262
- ## 📜 License
263
 
264
- This model is released under the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0).
265
 
266
- ## 🙏 Acknowledgments
267
 
268
- - Built on [DeepSeek-R1-0528-Qwen3-8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B)
269
- - Training data from FreedomIntelligence, UCSC-VLAA, Intelligent-Internet, and other open-source medical datasets
270
- - Developed by [TANIT Healthcare Technologies](https://tanitai.com)
271
 
272
- ## 📧 Contact
273
 
274
- For questions, collaborations, or access requests:
275
- - **Organization**: [TANIT Healthcare Technologies](https://huggingface.co/TanitAI)
276
- - **Email**: contact@tanitai.com
277
 
278
  ---
279
 
280
- <div align="center">
281
 
282
- **Made with ❤️ by TANIT Healthcare Technologies**
 
 
 
 
 
 
 
283
 
284
- *Advancing healthcare through AI*
285
 
286
- </div>
 
2
  license: apache-2.0
3
  language:
4
  - en
5
+ base_model:
6
+ - Qwen/Qwen3-8B
7
  library_name: transformers
8
  pipeline_tag: text-generation
9
  tags:
10
  - medical
11
+ - clinical-reasoning
12
+ - chain-of-thought
13
+ - qwen3
14
+ - sft
15
+ - dpo
16
  - tanit
17
+ datasets:
18
+ - FreedomIntelligence/medical-o1-reasoning-SFT
19
+ - FreedomIntelligence/Medical-R1-Distill-Data
20
+ - UCSC-VLAA/MedReason
21
+ - UCSC-VLAA/m23k-tokenized
22
+ - Intelligent-Internet/II-Medical-RL
23
+ - hongzhouyu/FineMed-DPO
24
+ - super-dainiu/medagents-benchmark
25
  model-index:
26
+ - name: Tanit-Med-8B
27
  results:
28
+ - task: {type: question-answering, name: Medical QA}
29
+ dataset: {name: 'MedAgentsBench: MedQA', type: super-dainiu/medagents-benchmark, config: MedQA, split: test}
30
+ metrics: [{type: accuracy, value: 65.0, name: Accuracy}]
31
+ - task: {type: question-answering, name: Medical QA}
32
+ dataset: {name: 'MedAgentsBench: PubMedQA', type: super-dainiu/medagents-benchmark, config: PubMedQA, split: test}
33
+ metrics: [{type: accuracy, value: 68.0, name: Accuracy}]
34
+ - task: {type: question-answering, name: Medical QA}
35
+ dataset: {name: 'MedAgentsBench: MedMCQA', type: super-dainiu/medagents-benchmark, config: MedMCQA, split: test}
36
+ metrics: [{type: accuracy, value: 57.8, name: Accuracy}]
37
+ - task: {type: question-answering, name: Medical QA}
38
+ dataset: {name: 'MedAgentsBench: MedBullets', type: super-dainiu/medagents-benchmark, config: MedBullets, split: test}
39
+ metrics: [{type: accuracy, value: 42.9, name: Accuracy}]
40
+ - task: {type: question-answering, name: Medical QA}
41
+ dataset: {name: 'MedAgentsBench: MMLU (medical)', type: super-dainiu/medagents-benchmark, config: MMLU, split: test}
42
+ metrics: [{type: accuracy, value: 77.5, name: Accuracy}]
43
+ - task: {type: question-answering, name: Medical QA}
44
+ dataset: {name: 'MedAgentsBench: MMLU-Pro (medical)', type: super-dainiu/medagents-benchmark, config: MMLU-Pro, split: test}
45
+ metrics: [{type: accuracy, value: 47.3, name: Accuracy}]
46
+ - task: {type: question-answering, name: Medical QA}
47
+ dataset: {name: 'MedAgentsBench: MedExQA', type: super-dainiu/medagents-benchmark, config: MedExQA, split: test}
48
+ metrics: [{type: accuracy, value: 73.3, name: Accuracy}]
49
+ - task: {type: question-answering, name: Medical QA}
50
+ dataset: {name: 'MedAgentsBench: MedXpertQA-R', type: super-dainiu/medagents-benchmark, config: MedXpertQA-R, split: test}
51
+ metrics: [{type: accuracy, value: 12.1, name: Accuracy}]
52
+ - task: {type: question-answering, name: Medical QA}
53
+ dataset: {name: 'MedAgentsBench: MedXpertQA-U', type: super-dainiu/medagents-benchmark, config: MedXpertQA-U, split: test}
54
+ metrics: [{type: accuracy, value: 14.6, name: Accuracy}]
55
+ - task: {type: question-answering, name: Medical QA}
56
+ dataset: {name: 'MedAgentsBench: AfriMedQA', type: super-dainiu/medagents-benchmark, config: AfrimedQA, split: test}
57
+ metrics: [{type: accuracy, value: 48.9, name: Accuracy}]
58
  ---
59
 
60
+ # Tanit-Med-8B
61
 
62
+ *An 8B medical reasoning model that thinks before it answers — and a model card that tells you where it doesn't.*
63
 
64
+ Tanit-Med-8B is a full fine-tune of [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) for clinical multiple-choice reasoning and medical question answering. It was trained in four stages — broad medical SFT, reasoning SFT, DPO, and a short chain-of-thought polish — and evaluated across the ten benchmarks in [MedAgentsBench](https://huggingface.co/datasets/super-dainiu/medagents-benchmark), on both the standard and the hard splits.
65
 
66
+ It is named for Tanit, the Carthaginian goddess who watched over the western Mediterranean. The name is a commitment as much as a nod: this model is built by a North African team, and AfriMedQA is a first-class benchmark here, not a footnote.
 
 
67
 
68
+ **The short version:** on the standard splits, Tanit-Med-8B is, to our knowledge, the strongest open 8B medical model on MedAgentsBench — a 50.7% macro average, against 34.6% for the next-best 8B peer. On the *hard* splits it is not meaningfully better than any other 8B model, and neither is anyone else. We think both halves of that sentence matter, so both are in this card.
69
 
70
  ---
71
 
72
+ ## At a glance
73
 
74
+ | | |
75
+ |---|---|
76
+ | **Base model** | Qwen/Qwen3-8B (dense, 8.2B params) |
77
+ | **Precision** | BF16 |
78
+ | **Context** | 32K native (trained at 8,192-token packed sequences) |
79
+ | **Reasoning** | `<think>` blocks, always on by default (≈99% of responses) |
80
+ | **Language** | English (only language evaluated) |
81
+ | **License** | Apache-2.0 |
82
+ | **Best at** | Multiple-choice clinical QA, USMLE-style vignettes, medical exam reasoning |
83
+ | **Not for** | Diagnosis, treatment, dosing, triage, or anything touching a real patient |
84
 
85
+ ---
86
 
87
+ ## Results
88
 
89
+ All numbers below are **zero-shot, greedy decoding**, using the MedAgentsBench prompt and answer-index extraction. `FULL` is the standard test split; `HARD` is the MedAgentsBench hard subset. Baseline numbers for the other 8B models were produced by us under the same harness.
 
 
 
 
90
 
91
+ ### Against open 8B medical models
92
 
93
+ | Benchmark | **Tanit-Med-8B** | DeepSeek-R1-0528-Qwen3-8B | Falcon-H1R-7B | HuatuoGPT-o1-8B | MedReason-8B | Ministral-3-8B-Reasoning |
94
+ |---|---|---|---|---|---|---|
95
+ | MedQA | **65.0** | 44.0 | 28.9 | 29.5 | 28.0 | 27.8 |
96
+ | PubMedQA | **68.0** | 60.2 | 60.0 | 55.2 | 55.2 | 57.2 |
97
+ | MedMCQA | **57.8** | 42.9 | 33.6 | 35.8 | 35.0 | 34.3 |
98
+ | MedBullets | **42.9** | 26.0 | 20.1 | 20.5 | 19.2 | 17.2 |
99
+ | MMLU (med) | **77.5** | 50.5 | 30.3 | 26.6 | 22.8 | 22.2 |
100
+ | MMLU-Pro (med) | **47.3** | 20.7 | 13.6 | 13.6 | 14.4 | 9.4 |
101
+ | MedExQA | **73.3** | 49.3 | 35.1 | 28.7 | 21.7 | 21.7 |
102
+ | MedXpertQA-R | 12.1 | 10.5 | 9.6 | 11.4 | **18.1** | 11.0 |
103
+ | MedXpertQA-U | 14.6 | 9.7 | 10.4 | 11.5 | **16.5** | 9.7 |
104
+ | AfriMedQA | **48.9** | 32.2 | 25.3 | 12.6 | 12.6 | 10.9 |
105
+ | **Macro avg** | **50.7** | 34.6 | 26.7 | 24.5 | 24.4 | 22.1 |
106
 
107
+ Tanit-Med-8B wins eight of ten. The two it loses are the two where nobody is really winning (see below).
108
 
109
+ The AfriMedQA gap is the one we're proudest of: **48.9 vs 32.2** for the strongest baseline, and 4× the score of HuatuoGPT-o1 and MedReason. Medical models trained on US-exam corpora tend to fall over on questions grounded in African clinical practice. This one falls over less.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
110
 
111
+ ### Against frontier models, for scale
112
 
113
+ These are the published MedAgentsBench reference numbers ([arXiv:2503.07459](https://arxiv.org/abs/2503.07459)), **not re-run by us**. Cross-harness comparison, so read them as a ruler rather than a leaderboard. Averaged over the nine benchmarks the paper reports.
114
 
115
+ | Model | FULL (9-bench avg) | HARD (9-bench avg) |
116
+ |---|---|---|
117
+ | DeepSeek-R1 | 73.9 | 32.5 |
118
+ | o3-mini | 71.8 | 28.0 |
119
+ | GPT-4o | 68.2 | 18.0 |
120
+ | o1-mini | 67.8 | 25.3 |
121
+ | DeepSeek-V3 | 63.1 | 12.2 |
122
+ | Claude-3.5-Sonnet | 61.8 | 12.3 |
123
+ | Llama-3.3-70B | 61.8 | 12.4 |
124
+ | QwQ-32B | 61.0 | 17.1 |
125
+ | GPT-4o-mini | 57.8 | 10.8 |
126
+ | Claude-3.5-Haiku | 55.0 | 12.0 |
127
+ | **Tanit-Med-8B** | **50.9** | **20.4** |
128
 
129
+ An 8B model running on a single consumer GPU lands about 5 points behind GPT-4o-mini and 4 behind Claude-3.5-Haiku on the standard splits. That is the honest position: competitive with last-generation *small* frontier models, not with frontier models.
130
 
131
+ Please do **not** read the HARD column as "an 8B model beats GPT-4o." See the next section for why that number is a mirage.
 
 
132
 
133
+ ---
134
 
135
+ ## How to read the hard splits (please read this)
 
 
136
 
137
+ The MedAgentsBench hard subsets were built by keeping only questions that **fewer than half of a panel of baseline LLMs answered correctly** — a panel that included GPT-4o. Those models are therefore *adversarially selected against* on this split. GPT-4o's 18.0 is not a measurement of GPT-4o being worse at medicine than an 8B model; it is a measurement of the filter having done its job.
138
 
139
+ Three things follow, and we'd rather say them ourselves than have someone say them in the community tab:
 
 
 
 
 
 
140
 
141
+ **1. At 8B, the hard splits do not separate models.** Here is every 8B-class model we tested, macro-averaged over the ten hard subsets:
 
 
 
142
 
143
+ | Model | HARD macro avg |
144
+ |---|---|
145
+ | Falcon-H1R-7B | 21.5 |
146
+ | MedReason-8B | 21.2 |
147
+ | HuatuoGPT-o1-8B | 20.7 |
148
+ | **Tanit-Med-8B** | **20.3** |
149
+ | Ministral-3-8B-Reasoning | 19.6 |
150
+ | DeepSeek-R1-0528-Qwen3-8B | 17.4 |
151
 
152
+ A 4-point spread across six models with wildly different training. Our +16-point advantage on the standard splits evaporates entirely. We do not claim a hard-split win, because there isn't one.
 
 
 
 
 
153
 
154
+ **2. The samples are tiny.** AfriMedQA-hard is *32 questions* — one item is worth 3.1 points. Seven of the ten hard subsets are n=100. Treat any difference under ~8 points on these splits as noise, ours included.
155
 
156
+ **3. Bigger doesn't fix it, and self-consistency can make it worse.** In our own baseline sweep, Qwen3-30B-A3B with sampling + self-consistency (k=3) reached **60.2%** micro-average on the standard splits and **14.9%** on the hard ones — *below* Qwen3-0.6B's 20.2%. On adversarially-filtered items, majority voting appears to converge confidently on the attractive distractor. This was the most useful negative result of the project and it is why we ship a greedy, single-sample model.
 
 
 
 
 
 
 
 
157
 
158
+ ---
 
 
159
 
160
+ ## Where this model fails
161
 
162
+ - **MedXpertQA is unsolved at this scale.** Its items carry up to ten answer options, which puts chance around 10%. We score 12.1 (R) and 14.6 (U) — barely off the floor. So does every other 8B model here. DeepSeek-R1 gets 37.3. This is not a benchmark we are competitive on; it's a benchmark that shows where 8B runs out of road.
163
+ - **MedReason-8B beats us on both MedXpertQA subsets** (18.1 / 16.5). Knowledge-graph-grounded training seems to buy something on deep-reasoning items that our curriculum doesn't.
164
+ - **Hard splits: no better than the field.** See above.
165
+ - **Long-form clinical advice is untested.** Every number in this card comes from multiple-choice benchmarks. We have not evaluated free-text safety, hedging, refusal behaviour, hallucinated citations, or drug dosing.
166
+ - **English only.** AfriMedQA is English-language. We have run no non-English evaluation.
167
+ - **Answer-format brittleness.** The phase-4 checkpoint is tuned to emit a strict final-answer line. Prompt it off-format and extraction gets flaky (see *Evaluation protocol*).
168
 
169
+ ### A note we owe you
170
 
171
+ We have not yet published a like-for-like **Qwen3-8B baseline** under this exact frozen harness. Our internal sweep of Qwen3-8B in thinking mode lands in a similar range to Tanit-Med-8B on several standard splits, and we are not going to make a claim of the form *"medical fine-tuning beats the base model"* until that number is measured under the same conditions as everything else in this card. Until it is up, read the peer table as a comparison against **other medical fine-tunes**, not as proof that medical SFT was worth it. That number is coming, and it will go here whatever it says.
 
 
 
 
 
 
172
 
173
+ ---
 
 
 
 
174
 
175
+ ## Training
176
 
177
+ Four stages, full fine-tune (no LoRA), ZeRO-3 / FSDP, bf16.
 
178
 
179
+ | Phase | Method | Data | Purpose |
180
+ |---|---|---|---|
181
+ | 1 | SFT | medical-o1-reasoning (Huatuo-o1) + Medical-R1-Distill + MedReason | Broad medical foundation, clinical language |
182
+ | 2 | SFT | m23k + II-Medical-RL + ChatDoctor-RL + MedReason | MCQ robustness, multi-step reasoning structure |
183
+ | 3 | DPO | FineMed-DPO (32.9K pairs) | Preference alignment; fewer confidently-wrong answers |
184
+ | 4 | SFT | MedReason + Huatuo-o1, outputs rewritten to ≤6 reasoning bullets + final answer line | Concise, extractable chain-of-thought |
185
 
186
+ ### Hyperparameters
187
 
188
+ | | |
189
+ |---|---|
190
+ | Optimizer | AdamW, β = (0.9, 0.95), ε = 1e-8, weight decay 0.1 |
191
+ | Grad clip | 1.0 |
192
+ | Max sequence length | 8,192, packing on |
193
+ | Attention | FlashAttention |
194
+ | Global batch | ~1,048,576 tokens/step (≈128 × 8,192 seqs), grad-accum to match |
195
+ | Schedule | 2% warmup, cosine decay to 10% of peak |
196
+ | Peak LR — phase 1 | 2.0e-5 |
197
+ | Peak LR — phase 2 | 1.0e-5 |
198
+ | Peak LR — phase 3 (DPO) | 5.0e-7 |
199
+ | Peak LR — phase 4 | 5.0e-6 |
200
 
201
+ ### What each phase actually bought
202
+
203
+ Measured on our internal eval harness, macro-averaged across all ten standard splits:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
204
 
205
+ - **Phase 1 2** was the single biggest jump. Reasoning SFT took MedBullets from 30.5 to 47.1 and MMLU-Pro from 25.7 to 43.8 — the compositional benchmarks, exactly where you'd hope forced multi-step reasoning would pay.
206
+ - **Phase 3 (DPO)** *cost* raw accuracy. The DPO checkpoint sits several points below phase 4 on the standard splits and only reasons on 40–80% of prompts. What it bought was calibration: markedly fewer confident wrong answers. We ship it separately as [Tanit-Med-8B-DPO](https://huggingface.co/TanitAI/Tanit-Med-8B-DPO) because for some downstream uses that trade is the right one.
207
+ - **Phase 4 (CoT polish)** recovered the accuracy *and* pushed the reasoning rate to ~99% of responses while shortening the traces. It also cut extraction failures by an order of magnitude, which — see below — turned out to matter more than we expected.
208
+
209
+ We capped the reasoning budget at 4,096 tokens throughout. This follows [m1](https://arxiv.org/abs/2504.00869), which identifies an optimal medical-reasoning threshold around 4K tokens, past which accuracy *declines* — unlike in maths, forcing more reasoning on a medical question mostly gives a model with a shaky knowledge prior more rope to talk itself out of a correct answer. We saw the same thing and stopped fighting it.
210
+
211
+ ---
212
+
213
+ ## Usage
214
 
215
  ```python
216
+ from transformers import AutoModelForCausalLM, AutoTokenizer
217
 
218
+ model_id = "TanitAI/Tanit-Med-8B"
219
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
220
+ model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
 
221
 
222
+ messages = [
223
+ {"role": "system", "content": "You are Tanit, an expert medical AI assistant."},
224
+ {"role": "user", "content": (
225
+ "A 45-year-old presents with acute monoarthritis of the first MTP joint.\n"
226
+ "What is the most likely diagnosis?\n\n"
227
+ "A. Gout\nB. Pseudogout\nC. Septic arthritis\nD. Rheumatoid arthritis"
228
+ )},
229
+ ]
230
+
231
+ prompt = tokenizer.apply_chat_template(
232
+ messages, tokenize=False, add_generation_prompt=True, enable_thinking=True
233
  )
234
+ inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
235
+ out = model.generate(**inputs, max_new_tokens=4096, temperature=0.6, top_p=0.95, top_k=20)
236
+ print(tokenizer.decode(out[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))
237
  ```
238
 
239
+ **Serving with vLLM:**
240
 
241
+ ```bash
242
+ vllm serve TanitAI/Tanit-Med-8B --max-model-len 32768 --reasoning-parser qwen3
243
+ ```
 
 
 
 
244
 
245
+ **Generation settings.** For open-ended use, follow Qwen3's thinking-mode defaults: `temperature=0.6`, `top_p=0.95`, `top_k=20`. For benchmark reproduction, use **greedy** (`temperature=0`) — every number in this card was produced that way. Do not use greedy decoding with `enable_thinking=True` for long open-ended generations; it degenerates into repetition, as it does for the base model.
246
 
247
+ **Quantized:** [Tanit-Med-8B-NVFP4](https://huggingface.co/TanitAI/Tanit-Med-8B-NVFP4) runs on a single consumer GPU.
248
 
249
+ ---
250
+
251
+ ## Evaluation protocol
252
+
253
+ Reproducing our numbers requires matching three things, in descending order of how much they'll bite you:
254
+
255
+ 1. **Answer extraction.** This is a bigger confound than most training decisions. Between two revisions of our own harness, the strict-format extraction-failure rate on this checkpoint moved from **31.3% → 4.8%** on MedQA (and 63% → 5% on MedQA-hard), while accuracy moved by about a point. If you benchmark against us and get a different number, check your extraction rate *first*. Ours is under 7% on every split.
256
+ 2. **Prompt.** We use the MedAgentsBench zero-shot prompt verbatim — knowledgeable-medical-assistant system message, question, options, "reply with the answer index only."
257
+ 3. **Decoding.** Greedy. Single sample. Reasoning budget 4,096 tokens, generation cap 16,384.
258
+
259
+ Self-consistency (k=5) is worth roughly **+3–10% relative** on the standard splits if you can afford 5× the compute — but re-read the hard-split warning above before you assume it helps everywhere. It doesn't.
260
 
261
+ ---
 
 
 
 
 
 
 
 
 
262
 
263
+ ## The collection
264
 
265
+ | Model | Stage | Use it if |
266
+ |---|---|---|
267
+ | [**Tanit-Med-8B**](https://huggingface.co/TanitAI/Tanit-Med-8B) | Phase 4 (CoT polish) | You want the best accuracy. **Start here.** |
268
+ | [Tanit-Med-8B-DPO](https://huggingface.co/TanitAI/Tanit-Med-8B-DPO) | Phase 3 (DPO) | You'd trade a few points of accuracy for better-calibrated confidence |
269
+ | [Tanit-MedReason-8B](https://huggingface.co/TanitAI/Tanit-MedReason-8B) | Phase 2 (reasoning SFT) | You want an un-DPO'd reasoning checkpoint to build on |
270
+ | [Tanit-Med-8B-NVFP4](https://huggingface.co/TanitAI/Tanit-Med-8B-NVFP4) | Phase 4, FP4 | You're deploying on a consumer GPU |
271
 
272
+ ---
 
 
 
 
273
 
274
+ ## Intended use and limitations
 
 
 
275
 
276
+ **Intended for:** medical NLP research, benchmark development, medical education tooling, retrieval-augmented clinical QA prototypes, and as a base for further fine-tuning.
277
 
278
+ **Not intended for, and not safe for:** clinical decision support, diagnosis, treatment or dosing recommendations, triage, or any patient-facing deployment. Tanit-Med-8B is **not a medical device**. It has not been reviewed or cleared by any regulator, has not been evaluated for clinical safety, and has been measured only on multiple-choice exam questions — which correlate with medical knowledge but are not a proxy for clinical judgment. It answers with a `<think>` trace that *looks* like reasoning; a confident, fluent, well-structured trace can and does precede a wrong answer.
279
 
280
+ **Known risks:** inherits the biases of its training corpora, which are overwhelmingly Western and exam-derived. Will hallucinate drug names, doses, and citations. Will not reliably refuse out-of-scope or unsafe requests — DPO here optimized for *correctness* under ambiguity, not for safety refusals.
281
 
282
+ If you deploy anything downstream of this model in a setting where a person could be harmed, that safety work is yours to do, and it has not been done here.
 
 
283
 
284
+ ---
285
 
286
+ ## Licensing
287
+
288
+ Model weights are released under **Apache-2.0**, inherited from Qwen3-8B. Note that the training corpora carry their own terms — several are research-oriented and some derive from sources with non-commercial restrictions. If you intend to use this model commercially, verify the licence of each dataset listed in the metadata; we are releasing weights, not indemnity.
289
 
290
  ---
291
 
292
+ ## Citation
293
 
294
+ ```bibtex
295
+ @misc{tanit-med-8b-2026,
296
+ title = {Tanit-Med-8B: A Four-Stage Curriculum for Open Medical Reasoning at 8B},
297
+ author = {Tanit Healthcare Technologies},
298
+ year = {2026},
299
+ url = {https://huggingface.co/TanitAI/Tanit-Med-8B}
300
+ }
301
+ ```
302
 
303
+ Built on [Qwen3](https://huggingface.co/Qwen/Qwen3-8B) (Qwen Team). Evaluated with [MedAgentsBench](https://arxiv.org/abs/2503.07459) (Zhang et al., 2025). Trained on data from [FreedomIntelligence](https://huggingface.co/FreedomIntelligence), [UCSC-VLAA](https://huggingface.co/UCSC-VLAA), [Intelligent-Internet](https://huggingface.co/Intelligent-Internet), and [FineMed](https://huggingface.co/hongzhouyu). Our thanks to all of them — the open medical-AI stack is a shared one.
304
 
305
+ **Found a problem with these numbers?** Open a discussion. We'd rather be corrected than cited wrongly.