MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications
Paper • 2409.07314 • Published • 56
🩺 Diagnostic-Medicine-RL1
A Bayesian Clinical Reasoning AI Model
How to Use
You can load this model using the transformers library:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "Clinical-Reasoning-Hub/Diagnostic-Reasoning-RL1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
# Example inference
input_text = "Patient presents with..."
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
[](https://huggingface.co/Clinical-Reasoning-Hub/Diagnostic-Medicine-R1)
[](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B)
[](https://creativecommons.org/licenses/by-nc-nd/4.0/)
*Developed by the Clinical Reasoning Lab*
</div>
---
## 🎯 Overview
**Diagnostic-Medicine-R1** represents a novel approach to medical AI training that mirrors how physicians actually learn and reason. Unlike conventional medical QA models that rely on pattern matching, this system is trained using **evidence-based Bayesian clinical reasoning methodology**, incorporating likelihood ratios from peer-reviewed sources and structured diagnostic frameworks.
The model demonstrates that small language models (8B parameters) can achieve competitive medical reasoning performance (if trained like medical curriculum of students) through **methodological innovation** rather than scale.
---
## ✨ Key Innovations
| Innovation | Description |
|------------|-------------|
| **Evidence-Based Reasoning** | > 500 likelihood ratios extracted clinical scenarios and textbooks |
| **Diagnostic Frameworks** | Structured approaches from textbooks to identify diagnosis based on presentation |
| **Clinical Behavior Training** | 125+ real anonymized clinical cases using clinical reasoning methodology |
| **Weighted Curriculum Learning** | Prevents catastrophic forgetting through strategic data mixing |
| **Transparent Reasoning** | Uses `<think>` tags for explicit step-by-step clinical analysis |
---
## 📊 Training Methodology: The Three Pillars
Our training methodology is built on three core components that mirror how expert clinicians develop their diagnostic skills:
### 1️⃣ Numbers: Evidence-Based Likelihood Ratios
The model learns to quantify diagnostic certainty using likelihood ratios (LRs) from peer-reviewed sources:
- **LR+ > 10**: Strong rule-IN (+45% probability shift)
- **LR+ 5-10**: Moderate rule-in (+30% probability shift)
- **LR- < 0.1**: Strong rule-OUT (-45% probability shift)
### 2️⃣ Logic: Diagnostic Frameworks
Structured approaches to common clinical presentations including:
- Pivot points (findings that dramatically change probability)
- Must-not-miss diagnoses
- Systematic differential generation
### 3️⃣ Behavior: Clinical Decision Patterns
Tacit knowledge of experienced clinicians including:
- Golden Rules (e.g., "Always test for pregnancy before imaging")
- Cannot-Miss Diagnoses
- Structured clinical reasoning worksheets
---
## 📈 Performance Metrics
| Metric | Before Training | After Training |
|--------|-----------------|----------------|
| **MedMCQA Accuracy** | 22% | 65-75% |
| **Training Loss** | 4.86 | 1.0-1.5 |
| **Token Accuracy** | 27% | 66-68% |
### Training Data Composition
Trademark Secrete
| **Total** | | | **~420,000** |
---
## 🔧 Model Details
| Attribute | Value |
|-----------|-------|
| **Base Model** | [DeepSeek-R1-Distill-Llama-8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B) |
| **Architecture** | Transformers with Chain-of-Thought (CoT) Reasoning |
| **Parameters** | 8 Billion |
| **Context Length** | 4,096 tokens |
| **Training Method** | QLoRA (4-bit quantization) |
| **LoRA Rank** | 64 |
| **Developer** | Clinical Reasoning Lab |
| **License** | CC-BY-NC-ND-4.0 |
### Attribution
This model is a fine-tune of **DeepSeek-R1-Distill-Llama-8B**, which is distilled from **Llama-3.1-8B**.
- Built with Llama
- Original DeepSeek-R1 Copyright © 2023 DeepSeek
-
- Disclaimer:
- This model is for research and educational purposes only.
- It is not intended to provide medical advice or replace professional medical judgment.
- It may hallucinate or produce inaccurate differential diagnoses.
---
## 💡 Example Output
User: A 65-year-old male presents with crushing chest pain radiating to the left arm, diaphoresis, and ST elevation in leads V1-V4. What is the most likely diagnosis?
Model: Let me analyze this case using Bayesian clinical reasoning.
Key Findings:
- 65-year-old male (cardiac risk age)
- Crushing chest pain radiating to left arm (classic ACS presentation)
- Diaphoresis (autonomic response to MI)
- ST elevation in V1-V4 (anterior leads)
Pre-test Probability: Male >55 with chest pain in ED setting: ~30% baseline for ACS
Applying Likelihood Ratios:
- Radiation to left arm: LR+ 2.3 → probability increases to ~50%
- Diaphoresis: LR+ 2.0 → probability increases to ~65%
- ST elevation in V1-V4: LR+ >10 → probability increases to >95%
Localization: V1-V4 = anterior wall = LAD territory
Diagnosis: Acute Anterior STEMI
Post-test probability: >95%
This patient requires immediate cardiac catheterization. The ST elevation in leads V1-V4 indicates LAD occlusion with anterior wall infarction.
---
## 🔒 Access Policy
Due to the sensitive nature of medical AI, access to this model is **gated and manually reviewed**.
### Requirements for Access:
1. ✅ Verifiable real name
2. ✅ Institutional affiliation (university, hospital, research organization)
3. ✅ Clear research purpose with specific use case
4. ✅ Agreement to non-commercial, research-only use
### Requests that will be declined:
- ❌ Anonymous accounts
- ❌ Vague or missing research justification
- ❌ Commercial deployment intentions
- ❌ Clinical decision-making applications
---
## ⚠️ Intended Use & Limitations
### ✅ Intended Use
| Use Case | Description |
|----------|-------------|
| **Clinical Research** | Analyzing patterns in de-identified medical data |
| **Medical Education** | Simulating diagnostic scenarios for training |
| **AI Benchmarking** | Evaluating against other medical reasoning systems |
| **Methodology Research** | Studying Bayesian reasoning in AI systems |
### ❌ Limitations & Warnings
> **⚠️ NOT A MEDICAL DEVICE**
>
> This model is **NOT** a licensed medical professional and must **NOT** be used to provide medical advice, diagnosis, or treatment to real patients.
| Limitation | Description |
|------------|-------------|
| **Hallucination Risk** | Like all LLMs, outputs may contain fabricated information |
| **No Real-Time Knowledge** | Training data has a cutoff date |
| **Bias** | May reflect biases present in medical literature |
| **Validation Required** | All outputs must be verified by qualified professionals |
| **Not FDA Approved** | Not cleared for clinical use |
---
## 📄 Documentation
Complete AI Development Methodology (DOCX) available on request only
**Document Contents:**
1. Executive Summary
2. Project Objectives & Target Benchmarks
3. Theoretical Framework (Numbers, Logic, Behavior)
4. Training Data Architecture (6-tier system)
5. Training Methodology & Catastrophic Forgetting Solution
6. Training Phases & Results
7. Technical Implementation Details
8. Key Innovations
9. Conclusion & Future Work
10. References
---
## 📚 References
1. Medical texbooks (finely distilled knowledge as per medical curriculum of undergraduate medical students)
2. DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs. arXiv:2401.02954, 2024.
3. M42-Health. MEDIC: A Comprehensive Framework for Evaluating LLMs in Clinical Applications. arXiv:2409.07314, 2024.
4. AI4LIFE-GROUP. MedSafetyBench: Evaluating and Developing Safety of Medical AI. NeurIPS 2024.
---
## 💼 Commercial Licensing
For commercial licensing inquiries, enterprise deployment, or partnership
opportunities, contact: [adnanagha11@gmail.com]
The publicly available model is for research only. Commercial licenses
with support, customization, and deployment rights are available.
📖 Citation
If you use this model in your research, please cite:
@misc{diagnostic-medicine-rl1-2026,
author = {Clinical Reasoning Lab},
title = {Diagnostic-Medicine-RL1: A Bayesian Clinical Reasoning Model},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/Clinical-Reasoning-Hub/Diagnostic-Medicine-RL1}},
note = {Fine-tuned from DeepSeek-R1-Distill-Llama-8B using evidence-based
Bayesian clinical reasoning methodology}
}
Developed with ❤️ by the Clinical Reasoning Lab
For research and educational purposes only
- Downloads last month
- -
Model tree for Clinical-Reasoning-Hub/Diagnostic-Reasoning-RL1
Base model
deepseek-ai/DeepSeek-R1-Distill-Llama-8B