Instructions to use Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical") model = AutoModelForCausalLM.from_pretrained("Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical
- SGLang
How to use Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Unsloth Desktop
- Docker Model Runner
How to use Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical with Docker Model Runner:
docker model run hf.co/Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical
DeepSeek-R1-Distill-Llama-8B-Medical
DeepSeek-R1-Distill-Llama-8B-Medical is an 8B-parameter reasoning model for medical question answering. It is DeepSeek-R1-Distill-Llama-8B fine-tuned on medical reasoning traces, with the LoRA adapter merged into the original BF16 weights. Like its base model, it thinks step by step inside a reasoning block and then gives a separate final answer.
Training
| Base model | deepseek-ai/DeepSeek-R1-Distill-Llama-8B |
| Dataset | FreedomIntelligence/Medical-R1-Distill-Data, medical questions with reasoning and answers distilled from DeepSeek-R1 |
| Samples | First 5,000 samples; 151 longer than 6,144 tokens were excluded (no sample was truncated), leaving 4,849 |
| Method | QLoRA with Unsloth: r=16, alpha=16, all linear layers, learning rate 1e-4 (cosine), 1 epoch, effective batch size 16 |
| Format | <think> reasoning </think> followed by the answer; loss computed on the reasoning and answer only |
| Hardware | Single NVIDIA T4 (Kaggle) |
Usage
For best results use temperature 0.6, top-p 0.95 and no system prompt. The chat template starts the response with <think> automatically, so the output begins with the reasoning, followed by a single </think> and the final answer.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "A 30-year-old woman has an itchy annular rash under her right breast. What is the most likely causative organism?"}]
inputs = tok.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
output = model.generate(**inputs, max_new_tokens=2048, do_sample=True, temperature=0.6, top_p=0.95)
print(tok.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
To see the reasoning and the answer separated by the </think> marker, decode with skip_special_tokens=False.
Limitations
- Research model. It has not been clinically validated or benchmarked yet and can state incorrect medical facts with confidence. Do not use it for diagnosis or treatment decisions.
- Training data was generated by another language model (DeepSeek-R1), so errors in those traces can be learned and repeated.
- Trained on a 4,849-sample subset in English only; coverage of specialties, rare conditions and other languages is limited.
- Long reasoning chains can contain mistakes or be repetitive. Always verify outputs against authoritative clinical sources.
- Downloads last month
- 122
Model tree for Rumiii/DeepSeek-R1-Distill-Llama-8B-Medical
Base model
deepseek-ai/DeepSeek-R1-Distill-Llama-8B