Instructions to use EPFLiGHT/Meditron3-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use EPFLiGHT/Meditron3-8B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="EPFLiGHT/Meditron3-8B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("EPFLiGHT/Meditron3-8B")
model = AutoModelForCausalLM.from_pretrained("EPFLiGHT/Meditron3-8B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use EPFLiGHT/Meditron3-8B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "EPFLiGHT/Meditron3-8B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EPFLiGHT/Meditron3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/EPFLiGHT/Meditron3-8B

SGLang

How to use EPFLiGHT/Meditron3-8B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "EPFLiGHT/Meditron3-8B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EPFLiGHT/Meditron3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "EPFLiGHT/Meditron3-8B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EPFLiGHT/Meditron3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use EPFLiGHT/Meditron3-8B with Docker Model Runner:
```
docker model run hf.co/EPFLiGHT/Meditron3-8B
```

[Feature Request] Evaluate Qwen 3.6 as an alternative base for future Meditron variants

by Milor123 - opened Jul 2

Discussion

Milor123

Jul 2

Thank you guys for the continued work on Meditron — it remains one of the most important open medical LLM initiatives.

I wanted to ask whether the team has explored Qwen 3.6 as a potential base model for future Meditron releases. While Llama 3.1 has served the project well, Qwen 3.6 introduces several architectural and licensing characteristics that seem particularly well-suited to medical domain fine-tuning, and I would love to understand if this is on the roadmap.

What makes Qwen 3.6 interesting for Meditron

1. Licensing flexibility

Qwen 3.6 is released under Apache 2.0, which differs from Llama 3.1's Community License in a few practical ways:

No user-count thresholds (Llama requires Meta approval above 700M MAU).
No mandatory "Built with Llama" branding or naming conventions.
No Acceptable Use Policy that can be updated unilaterally by the licensor.

For downstream deployers in hospitals, startups, or low-resource settings, this can simplify compliance and reduce legal uncertainty.

2. Architecture aligned with medical workflows

Long context: 262K tokens natively (up to ~1M with YaRN), compared to Llama 3.1's 128K. This could allow single-prompt ingestion of full patient histories or lengthy clinical guidelines without retrieval augmentation.
Hybrid attention: 75% linear-attention layers (Gated DeltaNet) reduce KV-cache pressure significantly. For long-document medical inference, this may translate to lower serving costs and higher throughput.
Thinking preservation: Qwen 3.6 retains <think> blocks across multi-turn conversations, which could be useful for diagnostic reasoning that spans multiple follow-up questions.

3. Performance baseline

Qwen 3.6-27B matches Llama 3.1-70B on coding benchmarks while running ~1.7–2× faster, and offers strong multilingual coverage (29+ languages). It would be interesting to see how these characteristics translate to medical benchmarks.

A related signal from the community

The recent "Fully Open Meditron" paper (May 2026) explicitly experiments with fully open base models and references Qwen3 in its evaluation setup. This suggests the broader community is curious about this direction.

Questions for the team

Has Qwen 3.6 been evaluated internally as a Meditron base?
Would you be open to publishing a small experimental comparison (e.g., Meditron-Llama vs. Meditron-Qwen on a medical benchmark suite)?
Are there technical or licensing blockers we might not be seeing from the outside?

I would be glad to help run benchmarks or test a Qwen-based variant if that would be useful.

Qwen 3.6 (Apache 2.0): https://github.com/QwenLM/Qwen3.6
Architecture overview: https://tosea.ai/blog/qwen-3-6-27b-complete-guide
Llama 3.1 License analysis: https://shujisado.org/2025/01/27/why-is-the-llama-license-not-open-source/
Fully Open Meditron (May 2026): https://arxiv.org/html/2605.16215v2

Thanks again for the great work, and looking forward to your thoughts.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment