Instructions to use unsloth/DeepSeek-R1-Distill-Llama-70B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use unsloth/DeepSeek-R1-Distill-Llama-70B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="unsloth/DeepSeek-R1-Distill-Llama-70B") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("unsloth/DeepSeek-R1-Distill-Llama-70B") model = AutoModelForCausalLM.from_pretrained("unsloth/DeepSeek-R1-Distill-Llama-70B") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use unsloth/DeepSeek-R1-Distill-Llama-70B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "unsloth/DeepSeek-R1-Distill-Llama-70B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "unsloth/DeepSeek-R1-Distill-Llama-70B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/unsloth/DeepSeek-R1-Distill-Llama-70B
- SGLang
How to use unsloth/DeepSeek-R1-Distill-Llama-70B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "unsloth/DeepSeek-R1-Distill-Llama-70B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "unsloth/DeepSeek-R1-Distill-Llama-70B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "unsloth/DeepSeek-R1-Distill-Llama-70B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "unsloth/DeepSeek-R1-Distill-Llama-70B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use unsloth/DeepSeek-R1-Distill-Llama-70B with Docker Model Runner:
docker model run hf.co/unsloth/DeepSeek-R1-Distill-Llama-70B
Fix chat_template crash when assistant message omits the `content` key
Browse files## Fix `chat_template` to handle assistant messages without a `content` key
**⚠️ This template will start crashing for every tool-calling user as soon as the next `transformers` release ships.**
The upstream PR https://github.com/huggingface/transformers/pull/45422 normalizes message inputs by stripping `content=None` before rendering (`None` and absent are semantically identical, and `content=None` is exactly what the OpenAI API returns for tool-call-only messages). That normalization is correct, but it exposes a latent bug in this template: the `tool_calls` branch reads `message['content']` directly, which raises when the key is absent.
Concretely, this code path is hit by **any tool-calling pipeline** (OpenAI-compatible servers, agent frameworks, function-calling demos) that produces assistant messages with `tool_calls` and no textual content. Today most of them happen to pass `content=None` explicitly and get away with it. After the transformers release, all of them break.
### Repro
**Today** (works):
```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("unsloth/DeepSeek-R1-Distill-Llama-70B")
tok.apply_chat_template(
[
{"role": "user", "content": "What's the weather in Paris?"},
{"role": "assistant", "content": None, "tool_calls": [{
"type": "function",
"function": {"name": "get_weather", "arguments": '{"city":"Paris"}'},
}]},
],
tokenize=False,
)
# renders correctly
```
**After https://github.com/huggingface/transformers/pull/45422** (same call, same input — `transformers` strips `content=None` before rendering, so the template sees an absent key and crashes):
```
UndefinedError: 'dict object' has no attribute 'content'
```
You can reproduce the post-release behavior today by simply omitting the `content` key.
### The fix
A one-character change: `message['content'] is none` → `message.get('content') is none`. `.get()` returns `None` whether the key is absent or set to `None`, so both cases are handled identically.
Verified against a 14-case regression suite (single-turn, multi-turn, tool flows with/without final answers, multi-system, `</think>` reasoning, unicode, empty content): all cases either render bit-identically to the current template or, for the previously crashing case, render correctly. **Zero regressions.**
---
*Disclaimer: this PR was opened as part of a scan for repos whose `chat_template` is derived from (or copies) the DeepSeek-R1 template, identified by the presence of the buggy substring `message['content'] is none`. The same one-line fix is proposed wherever that pattern appears verbatim. I do not maintain this model, please review before merging.*
- tokenizer_config.json +8 -8
|
@@ -4,7 +4,7 @@
|
|
| 4 |
"add_prefix_space": null,
|
| 5 |
"added_tokens_decoder": {
|
| 6 |
"128000": {
|
| 7 |
-
"content": "<
|
| 8 |
"lstrip": false,
|
| 9 |
"normalized": false,
|
| 10 |
"rstrip": false,
|
|
@@ -12,7 +12,7 @@
|
|
| 12 |
"special": true
|
| 13 |
},
|
| 14 |
"128001": {
|
| 15 |
-
"content": "<
|
| 16 |
"lstrip": false,
|
| 17 |
"normalized": false,
|
| 18 |
"rstrip": false,
|
|
@@ -92,7 +92,7 @@
|
|
| 92 |
"special": true
|
| 93 |
},
|
| 94 |
"128011": {
|
| 95 |
-
"content": "<
|
| 96 |
"lstrip": false,
|
| 97 |
"normalized": false,
|
| 98 |
"rstrip": false,
|
|
@@ -100,7 +100,7 @@
|
|
| 100 |
"special": false
|
| 101 |
},
|
| 102 |
"128012": {
|
| 103 |
-
"content": "<
|
| 104 |
"lstrip": false,
|
| 105 |
"normalized": false,
|
| 106 |
"rstrip": false,
|
|
@@ -124,7 +124,7 @@
|
|
| 124 |
"special": false
|
| 125 |
},
|
| 126 |
"128015": {
|
| 127 |
-
"content": "<
|
| 128 |
"lstrip": false,
|
| 129 |
"normalized": false,
|
| 130 |
"rstrip": false,
|
|
@@ -2052,9 +2052,9 @@
|
|
| 2052 |
"special": true
|
| 2053 |
}
|
| 2054 |
},
|
| 2055 |
-
"bos_token": "<
|
| 2056 |
"clean_up_tokenization_spaces": false,
|
| 2057 |
-
"eos_token": "<
|
| 2058 |
"extra_special_tokens": {},
|
| 2059 |
"legacy": true,
|
| 2060 |
"model_max_length": 16384,
|
|
@@ -2064,5 +2064,5 @@
|
|
| 2064 |
"tokenizer_class": "LlamaTokenizerFast",
|
| 2065 |
"unk_token": null,
|
| 2066 |
"use_default_system_prompt": false,
|
| 2067 |
-
"chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<
|
| 2068 |
}
|
|
|
|
| 4 |
"add_prefix_space": null,
|
| 5 |
"added_tokens_decoder": {
|
| 6 |
"128000": {
|
| 7 |
+
"content": "<|begin▁of▁sentence|>",
|
| 8 |
"lstrip": false,
|
| 9 |
"normalized": false,
|
| 10 |
"rstrip": false,
|
|
|
|
| 12 |
"special": true
|
| 13 |
},
|
| 14 |
"128001": {
|
| 15 |
+
"content": "<|end▁of▁sentence|>",
|
| 16 |
"lstrip": false,
|
| 17 |
"normalized": false,
|
| 18 |
"rstrip": false,
|
|
|
|
| 92 |
"special": true
|
| 93 |
},
|
| 94 |
"128011": {
|
| 95 |
+
"content": "<|User|>",
|
| 96 |
"lstrip": false,
|
| 97 |
"normalized": false,
|
| 98 |
"rstrip": false,
|
|
|
|
| 100 |
"special": false
|
| 101 |
},
|
| 102 |
"128012": {
|
| 103 |
+
"content": "<|Assistant|>",
|
| 104 |
"lstrip": false,
|
| 105 |
"normalized": false,
|
| 106 |
"rstrip": false,
|
|
|
|
| 124 |
"special": false
|
| 125 |
},
|
| 126 |
"128015": {
|
| 127 |
+
"content": "<|▁pad▁|>",
|
| 128 |
"lstrip": false,
|
| 129 |
"normalized": false,
|
| 130 |
"rstrip": false,
|
|
|
|
| 2052 |
"special": true
|
| 2053 |
}
|
| 2054 |
},
|
| 2055 |
+
"bos_token": "<|begin▁of▁sentence|>",
|
| 2056 |
"clean_up_tokenization_spaces": false,
|
| 2057 |
+
"eos_token": "<|end▁of▁sentence|>",
|
| 2058 |
"extra_special_tokens": {},
|
| 2059 |
"legacy": true,
|
| 2060 |
"model_max_length": 16384,
|
|
|
|
| 2064 |
"tokenizer_class": "LlamaTokenizerFast",
|
| 2065 |
"unk_token": null,
|
| 2066 |
"use_default_system_prompt": false,
|
| 2067 |
+
"chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<|User|>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message.get('content') is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<|Assistant|><|tool▁calls▁begin|><|tool▁call▁begin|>' + tool['type'] + '<|tool▁sep|>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<|tool▁call▁end|>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<|tool▁call▁begin|>' + tool['type'] + '<|tool▁sep|>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<|tool▁call▁end|>'}}{{'<|tool▁calls▁end|><|end▁of▁sentence|>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<|tool▁outputs▁end|>' + message['content'] + '<|end▁of▁sentence|>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<|Assistant|>' + content + '<|end▁of▁sentence|>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<|tool▁outputs▁begin|><|tool▁output▁begin|>' + message['content'] + '<|tool▁output▁end|>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<|tool▁output▁begin|>' + message['content'] + '<|tool▁output▁end|>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<|tool▁outputs▁end|>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<|Assistant|><think>\\n'}}{% endif %}"
|
| 2068 |
}
|