Instructions to use tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q2_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q2_K
# Run inference directly in the terminal:
./llama-cli -hf tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q2_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q2_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q2_K

Use Docker

docker model run hf.co/tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q2_K

LM Studio
Jan
Ollama
How to use tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF with Ollama:
```
ollama run hf.co/tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q2_K
```
Unsloth Desktop
Docker Model Runner
How to use tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF with Docker Model Runner:
```
docker model run hf.co/tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q2_K
```

Lemonade

How to use tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull tensorblock/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q2_K

Run and chat with the model

lemonade run user.DeepSeek-R1-Distill-Qwen-1.5B-GGUF-Q2_K

List all available models

lemonade list

Atomic Chat

morriszms commited on Mar 6, 2025

Commit

ebe200b

verified ·

1 Parent(s): 78bf11d

Upload folder using huggingface_hub

Browse files

Files changed (13) hide show

DeepSeek-R1-Distill-Qwen-1.5B-Q2_K.gguf +2 -2
DeepSeek-R1-Distill-Qwen-1.5B-Q3_K_L.gguf +2 -2
DeepSeek-R1-Distill-Qwen-1.5B-Q3_K_M.gguf +2 -2
DeepSeek-R1-Distill-Qwen-1.5B-Q3_K_S.gguf +2 -2
DeepSeek-R1-Distill-Qwen-1.5B-Q4_0.gguf +2 -2
DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf +2 -2
DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_S.gguf +2 -2
DeepSeek-R1-Distill-Qwen-1.5B-Q5_0.gguf +2 -2
DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_M.gguf +2 -2
DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_S.gguf +2 -2
DeepSeek-R1-Distill-Qwen-1.5B-Q6_K.gguf +2 -2
DeepSeek-R1-Distill-Qwen-1.5B-Q8_0.gguf +2 -2
README.md +14 -5

DeepSeek-R1-Distill-Qwen-1.5B-Q2_K.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dff3ea1a5df98bdf1c2bab5cfed078458fbe83785d3bb371438f43df67ccf382
-size 752879968

 version https://git-lfs.github.com/spec/v1
+oid sha256:2271c260fbaeddc9aed2be4759666d18e6685784072e64dae323b73989579c93
+size 752880480

DeepSeek-R1-Distill-Qwen-1.5B-Q3_K_L.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7ae67a65c81b13dbf17d5c269fb568cf9c3227eed0d625bbe43f7c2402bc145a
-size 980439904

 version https://git-lfs.github.com/spec/v1
+oid sha256:9337d8c9716a7f2060b4fb57ddff9ee2d7386cc7e6df5187643629986368cd42
+size 980440416

DeepSeek-R1-Distill-Qwen-1.5B-Q3_K_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bb1701725a950fc33805648b5631d0962c632237c8916aa0dce2361ce7b49a21
-size 924455776

 version https://git-lfs.github.com/spec/v1
+oid sha256:1bc36f5fb4f3985286f910871a748e345382584c66e2abb4b170ec50e8fa5d00
+size 924456288

DeepSeek-R1-Distill-Qwen-1.5B-Q3_K_S.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:23a74068964e55c83db1d8040bfb8dcdc7abc3590766fbb1099469b0b877a6c2
-size 861221728

 version https://git-lfs.github.com/spec/v1
+oid sha256:017a60015ec397aa885706651a59cf06d0954b9ce40c2e43cfe423937b1c756e
+size 861222240

DeepSeek-R1-Distill-Qwen-1.5B-Q4_0.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:58b11be136431e30ed25f29f19ca965f2ed3a7c90b11d5113d3bdc89da28efaa
-size 1066227040

 version https://git-lfs.github.com/spec/v1
+oid sha256:3210fb37aa87ed004aeaf238a92b7776823189399b5f699b823005f310f0aa17
+size 1066227552

DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7b3c5a5f32d5b775d73c569b034caf3cac9e682a3a799465084569defbe29bf1
-size 1117320544

 version https://git-lfs.github.com/spec/v1
+oid sha256:c42eb89982af54f77212e90afa8c8f6040a34d3801dee18f12dc502348c2df1c
+size 1117321056

DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_S.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f89e2eff6bdcd310b72da1910999cf7e71639c94a57beb874433830b575205db
-size 1071584608

 version https://git-lfs.github.com/spec/v1
+oid sha256:f51779bb1149e9bd3564458bf28a892bc23d9fa37861b571bf98ed883fae8f95
+size 1071585120

DeepSeek-R1-Distill-Qwen-1.5B-Q5_0.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aee082401e742d64db14189390dc1c39d510b283637df811cf83bce7b6a637fe
-size 1259173216

 version https://git-lfs.github.com/spec/v1
+oid sha256:72befdd2d324ccf4dab50a6f76628c7db50f862364a26ce1190ff296c751b372
+size 1259173728

DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a03fe25c8118439b01e8d7a7724186cde17e895114bb0419b8a1232bf73c1dce
-size 1285494112

 version https://git-lfs.github.com/spec/v1
+oid sha256:9adb7beadf181c485a1376eb84484781f6ff63289169c9bbe65f73e61f693ba3
+size 1285494624

DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_S.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:488601285cc063e20cff650884f534374fb53061b587d2eb6426614d0b790602
-size 1259173216

 version https://git-lfs.github.com/spec/v1
+oid sha256:9889e62f83466c743d6fddaa17c65bff9daf8bd0dc5b20010636ee06b6a6d60f
+size 1259173728

DeepSeek-R1-Distill-Qwen-1.5B-Q6_K.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ebc6d3690e16bb0af250083249a7fe0788fa2ed8d1e366352b0dc1d100097f04
-size 1464178528

 version https://git-lfs.github.com/spec/v1
+oid sha256:a78cb2c2f5f0f78ddd3cf8bfec6ed05a7292b14f6947da9a6be84eaceb8b4ad1
+size 1464179040

DeepSeek-R1-Distill-Qwen-1.5B-Q8_0.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:de5c0b2972296dddc21e54a1ad61fc0e5ee314d05d3288c42984e1980bd26aaa
-size 1894531936

 version https://git-lfs.github.com/spec/v1
+oid sha256:a04f5028b4e3d65f73758dd5e04ca9c3f5a9b81f6d9b6b1dd3b7c642b0196423
+size 1894532448

README.md CHANGED Viewed

@@ -1,6 +1,15 @@
 ---
-base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
 tags:
 - TensorBlock
 - GGUF
 ---
@@ -16,11 +25,11 @@ tags:
     </div>
 </div>
-## deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B - GGUF
-This repo contains GGUF format model files for [deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B).
-The files were quantized using machines provided by [TensorBlock](https://tensorblock.co/), and they are compatible with llama.cpp as of [commit ec7f3ac](https://github.com/ggerganov/llama.cpp/commit/ec7f3ac9ab33e46b136eb5ab6a76c4d81f57c7f1).
 <div style="text-align: left; margin: 20px 0;">
     <a href="https://tensorblock.co/waitlist/client" style="display: inline-block; padding: 10px 20px; background-color: #007bff; color: white; text-decoration: none; border-radius: 5px; font-weight: bold;">
@@ -31,7 +40,7 @@ The files were quantized using machines provided by [TensorBlock](https://tensor
 ## Prompt template
 ```
-<｜begin▁of▁sentence｜>{system_prompt}<｜User｜>{prompt}<｜Assistant｜>
 ```
 ## Model file specification

 ---
+base_model: unsloth/DeepSeek-R1-Distill-Qwen-1.5B
+language:
+- en
+license: apache-2.0
+library_name: transformers
 tags:
+- deepseek
+- qwen
+- qwen2
+- unsloth
+- transformers
 - TensorBlock
 - GGUF
 ---
     </div>
 </div>
+## unsloth/DeepSeek-R1-Distill-Qwen-1.5B - GGUF
+This repo contains GGUF format model files for [unsloth/DeepSeek-R1-Distill-Qwen-1.5B](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-1.5B).
+The files were quantized using machines provided by [TensorBlock](https://tensorblock.co/), and they are compatible with llama.cpp as of [commit b4823](https://github.com/ggml-org/llama.cpp/commit/5bbe6a9fe9a8796a9389c85accec89dbc4d91e39).
 <div style="text-align: left; margin: 20px 0;">
     <a href="https://tensorblock.co/waitlist/client" style="display: inline-block; padding: 10px 20px; background-color: #007bff; color: white; text-decoration: none; border-radius: 5px; font-weight: bold;">
 ## Prompt template
 ```
+<｜begin▁of▁sentence｜>{system_prompt}<｜User｜>{prompt}<｜Assistant｜><think>
 ```
 ## Model file specification