Instructions to use tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF", dtype="auto")

MLX

How to use tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF with MLX:

# Download the model from the Hub
pip install huggingface_hub[hf_xet]

huggingface-cli download --local-dir DeepSeek-R1-Distill-Qwen-14B-GGUF tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF

llama-cpp-python

How to use tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF",
	filename="DeepSeek-R1-Distill-Qwen-14B-Q2_K.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q2_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q2_K
# Run inference directly in the terminal:
./llama-cli -hf tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q2_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q2_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q2_K

Use Docker

docker model run hf.co/tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q2_K

LM Studio
Jan
Ollama
How to use tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF with Ollama:
```
ollama run hf.co/tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q2_K
```

Unsloth Studio

How to use tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF with Docker Model Runner:
```
docker model run hf.co/tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q2_K
```

Lemonade

How to use tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull tensorblock/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q2_K

Run and chat with the model

lemonade run user.DeepSeek-R1-Distill-Qwen-14B-GGUF-Q2_K

List all available models

lemonade list

morriszms commited on Mar 6, 2025

Commit

93baa93

verified ·

1 Parent(s): 4e5c41f

Upload folder using huggingface_hub

Browse files

Files changed (13) hide show

DeepSeek-R1-Distill-Qwen-14B-Q2_K.gguf +2 -2
DeepSeek-R1-Distill-Qwen-14B-Q3_K_L.gguf +2 -2
DeepSeek-R1-Distill-Qwen-14B-Q3_K_M.gguf +2 -2
DeepSeek-R1-Distill-Qwen-14B-Q3_K_S.gguf +2 -2
DeepSeek-R1-Distill-Qwen-14B-Q4_0.gguf +2 -2
DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf +2 -2
DeepSeek-R1-Distill-Qwen-14B-Q4_K_S.gguf +2 -2
DeepSeek-R1-Distill-Qwen-14B-Q5_0.gguf +2 -2
DeepSeek-R1-Distill-Qwen-14B-Q5_K_M.gguf +2 -2
DeepSeek-R1-Distill-Qwen-14B-Q5_K_S.gguf +2 -2
DeepSeek-R1-Distill-Qwen-14B-Q6_K.gguf +2 -2
DeepSeek-R1-Distill-Qwen-14B-Q8_0.gguf +2 -2
README.md +7 -5

DeepSeek-R1-Distill-Qwen-14B-Q2_K.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8c68d60536951243a0c77e462dfe8504fe31af9c7c4ba8b6668217c7e67a8db0
-size 5770497184

 version https://git-lfs.github.com/spec/v1
+oid sha256:9c455ec6df638cb2ab50fa5aecf0865ad1ad2846058b9294e362d044a3efbae7
+size 5770497536

DeepSeek-R1-Distill-Qwen-14B-Q3_K_L.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b223a03bcfee852fd74fff46c49fdbbdf24f124ac7981bd593b781bb496e2313
-size 7924767904

 version https://git-lfs.github.com/spec/v1
+oid sha256:b27aaa81a1d47b169fd453837c2dd8934b9ecf8e3534b1ff6a10ebf5461e3ace
+size 7924768256

DeepSeek-R1-Distill-Qwen-14B-Q3_K_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:280f0c73e26199e77f057f6dad6e23a3649e76dae30793c7c7466f1e51734e85
-size 7339203744

 version https://git-lfs.github.com/spec/v1
+oid sha256:0581832bab6d3d56cc13776c9d609e1bbaddcdd8bbae782a5781302355a2a3c3
+size 7339204096

DeepSeek-R1-Distill-Qwen-14B-Q3_K_S.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:72c228886f47aabe75a6e5200cc9233ff57042d25c5d8979af1cbe1111071f79
-size 6659595424

 version https://git-lfs.github.com/spec/v1
+oid sha256:5564a917f5fde18fea695233d79f8dec908c980fbac3c58cb9679f699405f12b
+size 6659595776

DeepSeek-R1-Distill-Qwen-14B-Q4_0.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:af3a10dcd7a3477d10da57885cdacc01a2f716e792cc9f3aed341137a1345ee0
-size 8517725344

 version https://git-lfs.github.com/spec/v1
+oid sha256:a363ccf8b23575f38dc4c87162e18141dc4ee2eefd25ea19bdf711650d28e93a
+size 8517725696

DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:37592056644085b197085d27f4cc2079df47a9edd3abaec2cd9e20065347dfbc
-size 8988109984

 version https://git-lfs.github.com/spec/v1
+oid sha256:34384a268ba02c8d62d0147a47ff63629d0d74dbe821ce3c76ec07d1637fc6da
+size 8988110336

DeepSeek-R1-Distill-Qwen-14B-Q4_K_S.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5fbaff3cadca2f82263c6af3c7f425b23ba634f9f3fe5671db8de9f31ca3acca
-size 8573430944

 version https://git-lfs.github.com/spec/v1
+oid sha256:f32015fc149588d3a0fda93e108ff63d369d84c0985a7acbdc9860eab85c5cac
+size 8573431296

DeepSeek-R1-Distill-Qwen-14B-Q5_0.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2a3c80221d7f5c13c25de5791aa58107a1c2569042f90781610cfd001b8bfdb5
-size 10266553504

 version https://git-lfs.github.com/spec/v1
+oid sha256:a1237424dff5b958459750ee6b3f585356fe902ad901e67fb0a6a90ffd7f6d5d
+size 10266553856

DeepSeek-R1-Distill-Qwen-14B-Q5_K_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e5b4e806325762bf65c7292f7ac304517f75e7a0aeae9a188417c01579d4c140
-size 10508872864

 version https://git-lfs.github.com/spec/v1
+oid sha256:1137ff7db186f43f73eb13b11c9dfe3b020c28e4248ee2601abde0d850e68e21
+size 10508873216

DeepSeek-R1-Distill-Qwen-14B-Q5_K_S.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9bfd33fb9c6577994de849c31580a064f0bdc971405509595763d8b4c6fa82c6
-size 10266553504

 version https://git-lfs.github.com/spec/v1
+oid sha256:10d8fc0e5a99a8ede0d3b9a50679e5c29a3a7c6151ec0e86caef61cdbc9f158a
+size 10266553856

DeepSeek-R1-Distill-Qwen-14B-Q6_K.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f5e743edc4c813e16bfa7dc4a73f08cccdd8fd5d6dd780627074b7370ecf9064
-size 12124683424

 version https://git-lfs.github.com/spec/v1
+oid sha256:45292e81c764e2e7fc22f28d30f381cedda806249d16565df1e994a2079f2c6c
+size 12124683776

DeepSeek-R1-Distill-Qwen-14B-Q8_0.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e209e1547e22d399db53be6c9903bce6d3aa485789e0b278c78f38ac54790255
-size 15701597344

 version https://git-lfs.github.com/spec/v1
+oid sha256:3e2bab18bd44f253461ddf3c2e53089ffe56d049bb9a7fcc0e65d9b246b208fc
+size 15701597696

README.md CHANGED Viewed

@@ -1,8 +1,10 @@
 ---
-base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
 tags:
 - TensorBlock
 - GGUF
 ---
 <div style="width: auto; margin-left: auto; margin-right: auto">
@@ -16,11 +18,11 @@ tags:
     </div>
 </div>
-## deepseek-ai/DeepSeek-R1-Distill-Qwen-14B - GGUF
-This repo contains GGUF format model files for [deepseek-ai/DeepSeek-R1-Distill-Qwen-14B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B).
-The files were quantized using machines provided by [TensorBlock](https://tensorblock.co/), and they are compatible with llama.cpp as of [commit ec7f3ac](https://github.com/ggerganov/llama.cpp/commit/ec7f3ac9ab33e46b136eb5ab6a76c4d81f57c7f1).
 <div style="text-align: left; margin: 20px 0;">
     <a href="https://tensorblock.co/waitlist/client" style="display: inline-block; padding: 10px 20px; background-color: #007bff; color: white; text-decoration: none; border-radius: 5px; font-weight: bold;">
@@ -31,7 +33,7 @@ The files were quantized using machines provided by [TensorBlock](https://tensor
 ## Prompt template
 ```
-<｜begin▁of▁sentence｜>{system_prompt}<｜User｜>{prompt}<｜Assistant｜>
 ```
 ## Model file specification

 ---
+library_name: transformers
 tags:
+- mlx
 - TensorBlock
 - GGUF
+base_model: mlx-community/DeepSeek-R1-Distill-Qwen-14B
 ---
 <div style="width: auto; margin-left: auto; margin-right: auto">
     </div>
 </div>
+## mlx-community/DeepSeek-R1-Distill-Qwen-14B - GGUF
+This repo contains GGUF format model files for [mlx-community/DeepSeek-R1-Distill-Qwen-14B](https://huggingface.co/mlx-community/DeepSeek-R1-Distill-Qwen-14B).
+The files were quantized using machines provided by [TensorBlock](https://tensorblock.co/), and they are compatible with llama.cpp as of [commit b4823](https://github.com/ggml-org/llama.cpp/commit/5bbe6a9fe9a8796a9389c85accec89dbc4d91e39).
 <div style="text-align: left; margin: 20px 0;">
     <a href="https://tensorblock.co/waitlist/client" style="display: inline-block; padding: 10px 20px; background-color: #007bff; color: white; text-decoration: none; border-radius: 5px; font-weight: bold;">
 ## Prompt template
 ```
+<｜begin▁of▁sentence｜>{system_prompt}<｜User｜>{prompt}<｜Assistant｜><think>
 ```
 ## Model file specification