Instructions to use unsloth/MAI-DS-R1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/MAI-DS-R1-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="unsloth/MAI-DS-R1-GGUF", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("unsloth/MAI-DS-R1-GGUF", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("unsloth/MAI-DS-R1-GGUF", trust_remote_code=True)

llama-cpp-python

How to use unsloth/MAI-DS-R1-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="unsloth/MAI-DS-R1-GGUF",
	filename="BF16/MAI-DS-R1-BF16-00001-of-00030.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use unsloth/MAI-DS-R1-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama-cli -hf unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama-cli -hf unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL

LM Studio
Jan

vLLM

How to use unsloth/MAI-DS-R1-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/MAI-DS-R1-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/MAI-DS-R1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL

SGLang

How to use unsloth/MAI-DS-R1-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "unsloth/MAI-DS-R1-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/MAI-DS-R1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "unsloth/MAI-DS-R1-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/MAI-DS-R1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use unsloth/MAI-DS-R1-GGUF with Ollama:
```
ollama run hf.co/unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL
```

Unsloth Studio new

How to use unsloth/MAI-DS-R1-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/MAI-DS-R1-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/MAI-DS-R1-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/MAI-DS-R1-GGUF to start chatting

Docker Model Runner
How to use unsloth/MAI-DS-R1-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL
```

Lemonade

How to use unsloth/MAI-DS-R1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/MAI-DS-R1-GGUF:UD-Q4_K_XL

Run and chat with the model

lemonade run user.MAI-DS-R1-GGUF-UD-Q4_K_XL

List all available models

lemonade list

danielhanchen commited on Apr 27, 2025

Commit

669d675

verified ·

1 Parent(s): 74afc4e

Add files using upload-large-folder tool

Browse files

Files changed (23) hide show

.gitattributes +22 -0
UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00001-of-00005.gguf +3 -0
UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00002-of-00005.gguf +3 -0
UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00003-of-00005.gguf +3 -0
UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00004-of-00005.gguf +3 -0
UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00005-of-00005.gguf +3 -0
UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00001-of-00005.gguf +3 -0
UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00002-of-00005.gguf +3 -0
UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00003-of-00005.gguf +3 -0
UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00004-of-00005.gguf +3 -0
UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00005-of-00005.gguf +3 -0
UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00001-of-00006.gguf +3 -0
UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00002-of-00006.gguf +3 -0
UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00003-of-00006.gguf +3 -0
UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00004-of-00006.gguf +3 -0
UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00005-of-00006.gguf +3 -0
UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00006-of-00006.gguf +3 -0
UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00001-of-00006.gguf +3 -0
UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00002-of-00006.gguf +3 -0
UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00003-of-00006.gguf +3 -0
UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00004-of-00006.gguf +3 -0
UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00005-of-00006.gguf +3 -0
UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00006-of-00006.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -71,3 +71,25 @@ UD-IQ1_S/MAI-DS-R1-UD-IQ1_S-00003-of-00004.gguf filter=lfs diff=lfs merge=lfs -t
 UD-IQ1_S/MAI-DS-R1-UD-IQ1_S-00001-of-00004.gguf filter=lfs diff=lfs merge=lfs -text
 UD-Q4_K_XL/MAI-DS-R1-UD-Q4_K_XL-00001-of-00008.gguf filter=lfs diff=lfs merge=lfs -text
 UD-IQ2_M/MAI-DS-R1-UD-IQ2_M-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text

 UD-IQ1_S/MAI-DS-R1-UD-IQ1_S-00001-of-00004.gguf filter=lfs diff=lfs merge=lfs -text
 UD-Q4_K_XL/MAI-DS-R1-UD-Q4_K_XL-00001-of-00008.gguf filter=lfs diff=lfs merge=lfs -text
 UD-IQ2_M/MAI-DS-R1-UD-IQ2_M-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00006-of-00006.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00006-of-00006.gguf filter=lfs diff=lfs merge=lfs -text
+UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00004-of-00006.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00005-of-00006.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00005-of-00006.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00002-of-00006.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00003-of-00006.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00004-of-00006.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00003-of-00006.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00001-of-00006.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00002-of-00006.gguf filter=lfs diff=lfs merge=lfs -text
+UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00001-of-00006.gguf filter=lfs diff=lfs merge=lfs -text

UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00001-of-00005.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:523682f3eca44ecd1e30d36aa8602a9c2ead16f1fce5b95e3f7bf820ad247a66
+size 48578503328

UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00002-of-00005.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:743c6b185ca8d958f6f5fcf89f5cfa84c3582bd2c8cffdab5691ceacc217f84f
+size 48023079040

UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00003-of-00005.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8b10572f0c9de8177ac1e579ce1d87170e5bb267e6ab124ddda725748f02b981
+size 49472649728

UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00004-of-00005.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:799b86acca23974251c030e79325d9100f703bd4a69b679246b2272516ab1ba2
+size 49571488000

UD-IQ1_M/MAI-DS-R1-UD-IQ1_M-00005-of-00005.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cb838fd288b00bf227ac1f4f3a726865716f8ed942320d8d7e5babffe28217e4
+size 5082290592

UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00001-of-00005.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:94e9c2387a5100824223b0cb1276ed6ff07ce60d726552c9230cb1fd1431af42
+size 48675752480

UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00002-of-00005.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0cc2281053b386486126ee1f15fecbb8107765695b58e3cef201c5e0869578b9
+size 49756273024

UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00003-of-00005.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:89dfaa331323f12aa8b1b01a810b40c132544d4bcc6c74888eb0c01bb4247533
+size 48830682848

UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00004-of-00005.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:da92830f91f75fbf0e2bab7237f0da8bbe0ef61a91b94997913a7216ce38500d
+size 49256204416

UD-IQ2_XXS/MAI-DS-R1-UD-IQ2_XXS-00005-of-00005.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:647acbcaa65984120d44192392ddf1622c37572352260c171aaded36fa76f2f1
+size 19999794400

UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00001-of-00006.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5ee20d95266fb6f4e4524144cf3c0dc7a3aa8b6980eae2ddc56c1a9d5a0454e0
+size 49228577280

UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00002-of-00006.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:850989283eded9870c4546a4d919c8b434f5d592b6b52fc3fd3a76bd1ab3db9b
+size 49235766720

UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00003-of-00006.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b72b3793590045d0a19cb58d8fd5ccaef888bf2efa85873b522501ccdf0a7309
+size 49023616544

UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00004-of-00006.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:07a37db9b1650479d1b946b679629d3d31b88754b630a1a1e02246ca9fe77618
+size 49200580288

UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00005-of-00006.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8d24a7277fe3affc644bab5412e0f2276e91dafb804271e7d9a5d1d75e84836f
+size 49039099584

UD-IQ3_XXS/MAI-DS-R1-UD-IQ3_XXS-00006-of-00006.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:751543fe455184d7d6410ba8a9271c4b717738ce3cafaaa82c8b579758eb0dbf
+size 27591222496

UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00001-of-00006.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:519e457c41a9414794b189b55908726b7df2fb4e8f0973fd179cf12df67c2420
+size 48232198240

UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00002-of-00006.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:16c3307c077fcc23735236ec28e5ae0a2b2089655ac1e3ecb64cc6f749bb85c6
+size 49192734528

UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00003-of-00006.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c4e6453ef893e452e6145471caa35db0d8937ad4b3721cdb1e691bd3b8dd100e
+size 49776641312

UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00004-of-00006.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:359e9bbfdc7b6b9ec8541877e9711ff670e1e06ee15cf45f6af2e019339e62c8
+size 49913607712

UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00005-of-00006.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8b6d916e6f0ab303db186c53ad605176640e0cd587cce393ab713462acfac6d0
+size 49925076512

UD-Q3_K_XL/MAI-DS-R1-UD-Q3_K_XL-00006-of-00006.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b9974677a7d4ba7da1f3e76bb40bfd86e1c35fe39889b2c8077ea619eb06fcd7
+size 49223429920