README.md

---
base_model: unsloth/deepseek-r1-distill-llama-8b-unsloth-bnb-4bit
library_name: peft
license: mit
datasets:
- FreedomIntelligence/medical-o1-reasoning-SFT
language:
- en
tags:
- medical
---

# Model Card for DeepSeek-R1-Medical-COT

## Model Details

### Model Description

DeepSeek-R1-Medical-COT is a fine-tuned version of the DeepSeek-R1 model, optimized for medical chain-of-thought (COT) reasoning. It is designed to assist in medical-related tasks such as question-answering, reasoning, and decision support. This model is particularly useful for applications requiring structured reasoning in the medical domain.

- **Developed by:** Mohamed Mahmoud
- **Funded by [optional]:** Independent project
- **Shared by:** Mohamed Mahmoud
- **Model type:** Transformer-based Large Language Model (LLM)
- **Language(s) (NLP):** English (en)
- **License:** MIT
- **Finetuned from model:** unsloth/deepseek-r1-distill-llama-8b-unsloth-bnb-4bit

### Model Sources

- **Repository:** [Hugging Face Model Repo](https://huggingface.co/thesnak/DeepSeek-R1-Medical-COT)
  
- **LinkedIn:** [Mohamed Mahmoud](https://www.linkedin.com/in/mohamed-thesnak)

## Uses

### Direct Use

The model can be used directly for medical reasoning tasks, including:

- Answering medical questions
- Assisting in medical decision-making
- Supporting clinical research and literature review

### Downstream Use

- Fine-tuning for specialized medical applications
- Integration into chatbots and virtual assistants for medical advice
- Educational tools for medical students

### Out-of-Scope Use

- This model is not a replacement for professional medical advice.
- Should not be used for clinical decision-making without expert validation.
- May not perform well in languages other than English.

## Bias, Risks, and Limitations

While fine-tuned for medical reasoning, the model may still have biases due to the limitations of its training data. Users should exercise caution and validate critical outputs with medical professionals.

### Recommendations

Users should verify outputs, particularly in sensitive medical contexts. The model is best used as an assistive tool rather than a primary decision-making system.

## How to Get Started with the Model

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "thesnak/DeepSeek-R1-Medical-COT"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")

input_text = "What are the symptoms of pneumonia?"
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Training Details

### Training Data

The model was fine-tuned using the **FreedomIntelligence/medical-o1-reasoning-SFT** dataset, which contains medical question-answer pairs designed to improve reasoning capabilities.

### Training Procedure

#### Preprocessing

- Tokenization using LLaMA tokenizer
- Text cleaning and normalization

#### Training Hyperparameters

- **Precision:** bf16 mixed precision
- **Optimizer:** AdamW
- **Batch size:** 16
- **Learning rate:** 2e-5
- **Epochs:** 3

#### Speeds, Sizes, Times

- **Training time:** Approximately 12 hours on a P100 GPU (Kaggle)
- **Model size:** 8B parameters (bnb 4-bit quantized)

#### Training Loss

| Step | Training Loss |
| ---- | ------------- |
| 10   | 1.919000      |
| 20   | 1.461800      |
| 30   | 1.402500      |
| 40   | 1.309000      |
| 50   | 1.344400      |
| 60   | 1.314100      |

## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

- The model was evaluated on held-out samples from **FreedomIntelligence/medical-o1-reasoning-SFT**.

#### Factors

- Performance was assessed on medical reasoning tasks.

#### Metrics

- **Perplexity:** Measured for general coherence.
- **Accuracy:** Evaluated based on expert-verified responses.
- **BLEU Score:** Used to assess response relevance.

### Results

- **Perplexity:**  
- **Accuracy:**  
- **BLEU Score:**
  
## Model Examination

Further interpretability analyses can be conducted using tools like Captum and SHAP to analyze how the model derives its medical reasoning responses.

## Environmental Impact

- **Hardware Type:** P100 GPU (Kaggle)
- **Hours used:** 2 hours
- **Cloud Provider:** Kaggle
- **Compute Region:** N/A
- **Carbon Emitted:** Estimated at 9.5 kg CO2eq
- **[Kaggle Notebook](https://www.kaggle.com/code/thesnak/fine-tune-deepseek)**
## Technical Specifications

### Compute Infrastructure

#### Hardware

- P100 GPU (16GB VRAM) on Kaggle


## Citation

**BibTeX:**

```bibtex
@misc{mahmoud2025deepseekmedcot,
  title={DeepSeek-R1-Medical-COT},
  author={Mohamed Mahmoud},
  year={2025},
  url={https://huggingface.co/thesnak/DeepSeek-R1-Medical-COT}
}
```

## Model Card Authors

- Mohamed Mahmoud

## Model Card Contact

- [LinkedIn](https://www.linkedin.com/in/mohamed-thesnak)