Instructions to use mkurman/Llama-3.2-MedIT-3B-R1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mkurman/Llama-3.2-MedIT-3B-R1 with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="mkurman/Llama-3.2-MedIT-3B-R1",
	filename="Llama-3.2-MedIT-3B-R1-Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use mkurman/Llama-3.2-MedIT-3B-R1 with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M

Use Docker

docker model run hf.co/mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M

LM Studio
Jan
Ollama
How to use mkurman/Llama-3.2-MedIT-3B-R1 with Ollama:
```
ollama run hf.co/mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M
```

Unsloth Studio new

How to use mkurman/Llama-3.2-MedIT-3B-R1 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mkurman/Llama-3.2-MedIT-3B-R1 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mkurman/Llama-3.2-MedIT-3B-R1 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for mkurman/Llama-3.2-MedIT-3B-R1 to start chatting

Pi new

How to use mkurman/Llama-3.2-MedIT-3B-R1 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use mkurman/Llama-3.2-MedIT-3B-R1 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use mkurman/Llama-3.2-MedIT-3B-R1 with Docker Model Runner:
```
docker model run hf.co/mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M
```

Lemonade

How to use mkurman/Llama-3.2-MedIT-3B-R1 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mkurman/Llama-3.2-MedIT-3B-R1:Q4_K_M

Run and chat with the model

lemonade run user.Llama-3.2-MedIT-3B-R1-Q4_K_M

List all available models

lemonade list

mkurman/Llama-3.2-MedIT-3B-R1

Important Notice:
This model is provided strictly for research purposes and is not intended for production use. It should not be considered a validated source of medical or professional advice. Use only in controlled experimental settings.

Model Overview

mkurman/Llama-3.2-MedIT-3B-R1 is a fine-tuned variant of meta-llama/Llama-3.2-3B-Instruct, adapted specifically for exploring natural language understanding and reasoning. This model leverages a multi-stage training approach, combining Blurred Thoughts Supervised Fine-Tuning (BT-SFT) and Group Relative Policy Optimization (GRPO) with an LLM evaluator to enhance its performance on specialized tasks.

Training Procedure

The model was developed through the following sequential steps:

Initial Blurred Thoughts Supervised Fine-Tuning (BT-SFT):
- Base Model: meta-llama/Llama-3.2-3B-Instruct
- Parameters: 2000 steps, batch size 2, accumulation iterations 16, learning rate 1e-6
- Dataset: open-thoughts/OpenThoughts-114k
- Details: For further information on BT-SFT, see the detailed post and the GitHub repository.
Group Relative Policy Optimization (GRPO) Stage 1:
- Dataset: FreedomIntelligence/medical-o1-verifiable-problem
- Training: 200 steps
- LLM Evaluator mkurman/Qwen2.5-14B-DeepSeek-R1-1M
- Details: For further information on GRPO with LLM evaluators, see the GitHub repository.
Group Relative Policy Optimization (GRPO) Stage 2:
- Dataset: open-r1/OpenR1-Math-220k
- Training: 200 steps
- LLM Evaluator deepseek/deepseek-r1-distill-qwen-14b (OpenRouterAI)

Datasets Utilized

open-thoughts/OpenThoughts-114k:
A dataset consisting of open-ended thoughts that supports diverse conversational contexts during the initial supervised fine-tuning.
FreedomIntelligence/medical-o1-verifiable-problem:
A dataset curated for enhancing the model's capabilities in addressing verifiable medical problems.
open-r1/OpenR1-Math-220k: A dataset designed to improve the model's reasoning and problem-solving skills in mathematical contexts.

Intended Use

Research and Experimental Applications:
This model is optimized for academic research and exploratory projects. It is ideal for investigating advanced fine-tuning methods and evaluating performance on task-oriented conversational scenarios.
Controlled Environments:
Users should deploy this model only within controlled experimental frameworks where rigorous evaluation and proper safety guardrails are in place.

Limitations and Ethical Considerations

Not for Clinical or Production Use:
The model’s outputs have not been validated for clinical accuracy or professional decision-making. It must not be used as a primary source for medical, legal, or safety-critical information.
Safety and Guardrails:
All users must implement appropriate safety measures and validation protocols. The model may produce biased or inaccurate results and should be used with caution.
Experimental Nature:
Given its research-oriented design, the model’s performance can vary widely based on input and context. It is essential to perform thorough testing and validation before drawing any conclusions from its outputs.

License

This model is released under the Llama 3.2 license. Users must adhere to the terms specified in the license when utilizing this model.

Final Notice

All outputs from mkurman/Llama-3.2-MedIT-3B-R1 are intended solely for research purposes. This model is not a comprehensive knowledge source and should not be used as a substitute for professional advice or decision-making. Ensure that all necessary guardrails and safety protocols are in place when conducting any experiments with this model.

Downloads last month: 13

Safetensors

Model size

3B params

Tensor type

BF16

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for mkurman/Llama-3.2-MedIT-3B-R1

Base model

meta-llama/Llama-3.2-3B-Instruct

Quantized

(453)

this model

Quantizations

1 model

mkurman
/

Llama-3.2-MedIT-3B-R1