Instructions to use unsloth/DeepSeek-R1-GGUF-UD with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/DeepSeek-R1-GGUF-UD with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="unsloth/DeepSeek-R1-GGUF-UD", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("unsloth/DeepSeek-R1-GGUF-UD", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("unsloth/DeepSeek-R1-GGUF-UD", trust_remote_code=True)

llama-cpp-python

How to use unsloth/DeepSeek-R1-GGUF-UD with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="unsloth/DeepSeek-R1-GGUF-UD",
	filename="BF16/DeepSeek-R1-BF16-00001-of-00030.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use unsloth/DeepSeek-R1-GGUF-UD with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL
# Run inference directly in the terminal:
llama-cli -hf unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL
# Run inference directly in the terminal:
llama-cli -hf unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL

LM Studio
Jan

vLLM

How to use unsloth/DeepSeek-R1-GGUF-UD with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/DeepSeek-R1-GGUF-UD"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/DeepSeek-R1-GGUF-UD",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL

SGLang

How to use unsloth/DeepSeek-R1-GGUF-UD with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "unsloth/DeepSeek-R1-GGUF-UD" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/DeepSeek-R1-GGUF-UD",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "unsloth/DeepSeek-R1-GGUF-UD" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/DeepSeek-R1-GGUF-UD",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use unsloth/DeepSeek-R1-GGUF-UD with Ollama:
```
ollama run hf.co/unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL
```

Unsloth Studio

How to use unsloth/DeepSeek-R1-GGUF-UD with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/DeepSeek-R1-GGUF-UD to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/DeepSeek-R1-GGUF-UD to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/DeepSeek-R1-GGUF-UD to start chatting

Docker Model Runner
How to use unsloth/DeepSeek-R1-GGUF-UD with Docker Model Runner:
```
docker model run hf.co/unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL
```

Lemonade

How to use unsloth/DeepSeek-R1-GGUF-UD with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/DeepSeek-R1-GGUF-UD:UD-Q4_K_XL

Run and chat with the model

lemonade run user.DeepSeek-R1-GGUF-UD-UD-Q4_K_XL

List all available models

lemonade list

danielhanchen commited on Apr 26, 2025

Commit

253ed86

verified ·

1 Parent(s): 62527e6

Add files using upload-large-folder tool

Browse files

Files changed (23) hide show

UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00004.gguf +2 -2
UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00002-of-00004.gguf +2 -2
UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00003-of-00004.gguf +2 -2
UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00004-of-00004.gguf +2 -2
UD-IQ2_M/DeepSeek-R1-UD-IQ2_M-00001-of-00005.gguf +2 -2
UD-IQ2_M/DeepSeek-R1-UD-IQ2_M-00002-of-00005.gguf +2 -2
UD-IQ2_M/DeepSeek-R1-UD-IQ2_M-00003-of-00005.gguf +2 -2
UD-IQ2_M/DeepSeek-R1-UD-IQ2_M-00004-of-00005.gguf +2 -2
UD-IQ2_M/DeepSeek-R1-UD-IQ2_M-00005-of-00005.gguf +2 -2
UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00001-of-00006.gguf +2 -2
UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00002-of-00006.gguf +2 -2
UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00003-of-00006.gguf +2 -2
UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00004-of-00006.gguf +2 -2
UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00005-of-00006.gguf +2 -2
UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00006-of-00006.gguf +2 -2
UD-Q4_K_XL/DeepSeek-R1-UD-Q4_K_XL-00001-of-00008.gguf +2 -2
UD-Q4_K_XL/DeepSeek-R1-UD-Q4_K_XL-00002-of-00008.gguf +2 -2
UD-Q4_K_XL/DeepSeek-R1-UD-Q4_K_XL-00003-of-00008.gguf +2 -2
UD-Q4_K_XL/DeepSeek-R1-UD-Q4_K_XL-00004-of-00008.gguf +2 -2
UD-Q4_K_XL/DeepSeek-R1-UD-Q4_K_XL-00005-of-00008.gguf +2 -2
UD-Q4_K_XL/DeepSeek-R1-UD-Q4_K_XL-00006-of-00008.gguf +2 -2
UD-Q4_K_XL/DeepSeek-R1-UD-Q4_K_XL-00007-of-00008.gguf +2 -2
UD-Q4_K_XL/DeepSeek-R1-UD-Q4_K_XL-00008-of-00008.gguf +2 -2

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dc59b0954f01e9da75891d70f23ec11da033ef698e846d903e442bf5a06ad283
-size 49194505088

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5d040660f2f588a9b9fc85973a4a012d425a220e288a719221427810a05359ea
-size 49222053888

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7bbdf02cc2f77545813324b55c6bec3dba92b8a9bf9adfc886c59095cb9c559b
-size 49566458016

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7e1cf1104e59b02020c2fd66e8b8e957dab3db150549eed6262eda437cd57b09
-size 44269775232

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3258cacd0f32b3e1c1ce2205ac95c4c065baefe609e689f32538fcf408c57b93
-size 49993592928

 version https://git-lfs.github.com/spec/v1
+oid sha256:14c75e219fae276b8b47dd163d1a6cbbadf571fce3de97149c6f0eef4216ee3f
+size 49987028640

 version https://git-lfs.github.com/spec/v1
+oid sha256:ac5742cdbaea7cc12bca39f4805ac8af1b4c24d439b4161056004564b529efa7
+size 49425995648

 version https://git-lfs.github.com/spec/v1
+oid sha256:dd5eae6a1f988c8da3370fd163a468596846fa262a3c5ae0f0f05cea2f14e6e5
+size 49781744160

 version https://git-lfs.github.com/spec/v1
+oid sha256:81ffcb392319ec2d72f6aaf088e8ee574fe889b78eceba990a1994af140beb8d
+size 35984078656

 version https://git-lfs.github.com/spec/v1
+oid sha256:35a0a1b471676761b24960e43e7af95cc916b1ce07a262086534c540efc3e4a4
+size 49141873504

 version https://git-lfs.github.com/spec/v1
+oid sha256:15a097171a7c514c648cd7ef1fe34b23dfe8814465e4881ef6c5a04046a71d76
+size 49349330912

 version https://git-lfs.github.com/spec/v1
+oid sha256:2f54126d1070645dc689a48d22117177f9a0af6047d0f94c5b3cd0ea3a6631a4
+size 49313289952

 version https://git-lfs.github.com/spec/v1
+oid sha256:f2016f93251912b2e114b5d038abfce8bf12ea5603fdd10d270fa8cc061fd4f6
+size 49895274720

 version https://git-lfs.github.com/spec/v1
+oid sha256:0322d552a6624b165443d9054a82c283f743629a63d859192034bd9a791d2799
+size 30203786496

 version https://git-lfs.github.com/spec/v1
+oid sha256:5ed724f55b3f482378fd9cf1138791113f31dff266227628bafb882b90be7c5b
+size 48884849376

 version https://git-lfs.github.com/spec/v1
+oid sha256:8c21e5a15c463e5f28d62fc75df8db73b1d78c7fe9bab712332f4b5a07b6857b
+size 49475565120

 version https://git-lfs.github.com/spec/v1
+oid sha256:7a8edd8bd9e5c6559d7bd6e9e92afe4ccdd8fd588e9e4f6ff3fcc01ef5b35a40
+size 49105036992

 version https://git-lfs.github.com/spec/v1
+oid sha256:f8c29265662789c59bd97a577d5ff12c9e4f459e96e1be2664b169d98ebeb5e4
+size 49382563936

 version https://git-lfs.github.com/spec/v1
+oid sha256:8bc4ecffbd35e1527c933ff6d0855c71947f326d6e09ee5880a2578edd937064
+size 48916035008

 version https://git-lfs.github.com/spec/v1
+oid sha256:3e724cbd52487e211b749a1770a4933feaa39e26325bac76214ece5ff9933ff6
+size 4616106656

 version https://git-lfs.github.com/spec/v1
+oid sha256:bae4d933e9c9915427b8098498087ed3cf2b0c6865298f16ba47f8df449600c5
+size 48837204896

 version https://git-lfs.github.com/spec/v1
+oid sha256:edd7f42743e12d90ef6efe4b97d248e5b498b1bc1b378ae3065ab511551768b9
+size 48507439872

 version https://git-lfs.github.com/spec/v1
+oid sha256:89f63e15b022e415c51939436b781e1e463e85987f1895a1cbb7ac6533e75e9b
+size 48021659008

 version https://git-lfs.github.com/spec/v1
+oid sha256:8b4c7f35e9acfe7eafb3b8d698584f0d3f99c4fa12a3f0af35deea7db0163a8a
+size 49068004224

 version https://git-lfs.github.com/spec/v1
+oid sha256:6ed05d8a4506712637505f2d1b147b1f4b8b11c2bb1f8dd3a37522986960408d
+size 48043297536

 version https://git-lfs.github.com/spec/v1
+oid sha256:bece41b1ccb99edb47dbb1cd9cf7d4d518e39619058f8ad0d3a171fa3da6e8c4
+size 49790338176

 version https://git-lfs.github.com/spec/v1
+oid sha256:69b63b72e8b4f422a72b83863fe4837b76602d26d4ac3ec02da9db121ed226ea
+size 49691287040

 version https://git-lfs.github.com/spec/v1
+oid sha256:1a53e39e050456fc22782ad3492d13db5b3949d48328d5a638bffec35d4b7d58
+size 42042478720