Instructions to use FoolDev/Thanatos-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use FoolDev/Thanatos-27B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="FoolDev/Thanatos-27B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("FoolDev/Thanatos-27B", dtype="auto") - llama-cpp-python
How to use FoolDev/Thanatos-27B with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="FoolDev/Thanatos-27B", filename="Thanatos-27B.Q4_K_M.gguf", )
llm.create_chat_completion( messages = [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] ) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- llama.cpp
How to use FoolDev/Thanatos-27B with llama.cpp:
Install from brew
brew install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf FoolDev/Thanatos-27B:Q4_K_M # Run inference directly in the terminal: llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf FoolDev/Thanatos-27B:Q4_K_M # Run inference directly in the terminal: llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf FoolDev/Thanatos-27B:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf FoolDev/Thanatos-27B:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M
Use Docker
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
- LM Studio
- Jan
- vLLM
How to use FoolDev/Thanatos-27B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "FoolDev/Thanatos-27B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FoolDev/Thanatos-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
- SGLang
How to use FoolDev/Thanatos-27B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "FoolDev/Thanatos-27B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FoolDev/Thanatos-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "FoolDev/Thanatos-27B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FoolDev/Thanatos-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Ollama
How to use FoolDev/Thanatos-27B with Ollama:
ollama run hf.co/FoolDev/Thanatos-27B:Q4_K_M
- Unsloth Studio new
How to use FoolDev/Thanatos-27B with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for FoolDev/Thanatos-27B to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for FoolDev/Thanatos-27B to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for FoolDev/Thanatos-27B to start chatting
- Pi new
How to use FoolDev/Thanatos-27B with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "FoolDev/Thanatos-27B:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use FoolDev/Thanatos-27B with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default FoolDev/Thanatos-27B:Q4_K_M
Run Hermes
hermes
- Docker Model Runner
How to use FoolDev/Thanatos-27B with Docker Model Runner:
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
- Lemonade
How to use FoolDev/Thanatos-27B with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull FoolDev/Thanatos-27B:Q4_K_M
Run and chat with the model
lemonade run user.Thanatos-27B-Q4_K_M
List all available models
lemonade list
Document Ollama tool-calling gap honestly
Browse filesOllama refuses tool-call requests with "<model> does not support tools"
on both /api/chat and /v1/chat/completions because our Modelfile has no
TEMPLATE directive exposing the Qwen 3.6 tool-jinja blocks — Ollama
falls back to the trivial {{ .Prompt }} template and its capability
detector reports completion only.
README "Tool / function calling" and Known limitations now reflect
reality. examples/ollama_chat.py:tool_round_trip keeps the function as
a reference for the request shape but adds a docstring warning, and the
demo wraps the call in try/except so it prints [skip] instead of
crashing with HTTPError 400.
Plain chat, streaming, system overrides, and the OpenAI-compat endpoint
all still work — verified end-to-end against the running daemon.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- CHANGELOG.md +10 -0
- README.md +18 -4
- examples/ollama_chat.py +17 -2
|
@@ -7,6 +7,16 @@ and documentation**, not the underlying base model.
|
|
| 7 |
|
| 8 |
## [Unreleased]
|
| 9 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
### Fixed
|
| 11 |
- `Modelfile`: added explicit `PARAMETER stop` directives for `<|im_end|>`,
|
| 12 |
`<|endoftext|>`, and `<|im_start|>`. Ollama was only picking up
|
|
|
|
| 7 |
|
| 8 |
## [Unreleased]
|
| 9 |
|
| 10 |
+
### Documented
|
| 11 |
+
- Tool calling via Ollama is currently rejected with `does not support
|
| 12 |
+
tools` because the Modelfile has no `TEMPLATE` directive exposing the
|
| 13 |
+
Qwen 3.6 tool-jinja blocks. Verified against both `/api/chat` and
|
| 14 |
+
`/v1/chat/completions`. README "Tool / function calling" + Known
|
| 15 |
+
limitations updated to reflect reality. `examples/ollama_chat.py:
|
| 16 |
+
tool_round_trip` keeps the helper as a reference shape but adds a
|
| 17 |
+
docstring warning, and the demo wraps the call in a try/except so it
|
| 18 |
+
prints `[skip]` instead of crashing.
|
| 19 |
+
|
| 20 |
### Fixed
|
| 21 |
- `Modelfile`: added explicit `PARAMETER stop` directives for `<|im_end|>`,
|
| 22 |
`<|endoftext|>`, and `<|im_start|>`. Ollama was only picking up
|
|
@@ -295,7 +295,7 @@ client doesn't.
|
|
| 295 |
|
| 296 |
#### Tool / function calling
|
| 297 |
|
| 298 |
-
|
| 299 |
|
| 300 |
```text
|
| 301 |
<tool_call>
|
|
@@ -306,15 +306,29 @@ The embedded template uses Qwen's XML format:
|
|
| 306 |
</tool_call>
|
| 307 |
```
|
| 308 |
|
| 309 |
-
|
| 310 |
-
|
| 311 |
-
`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 312 |
|
| 313 |
## Known limitations
|
| 314 |
|
| 315 |
- **Slower per token than the 35B-A3B sibling.** Dense 27B beats sparse 35B/3B-active on steps-per-second benchmarks because every parameter contributes; if you optimize for tokens-per-second, the MoE wins.
|
| 316 |
- **No mmproj in this release**, and **vision via Ollama is broken upstream** (qwen35/qwen35moe arch entries missing from Ollama's vendored llama.cpp fork — see the [Vision](#vision) section). For image input use llama.cpp directly until that's fixed.
|
| 317 |
- **Q4_K_M quality loss** is real. Use Q5_K_M or Q6_K if you have the VRAM (~20-22 GB).
|
|
|
|
| 318 |
- **No formal evaluation in this card.** Numbers above are estimates.
|
| 319 |
|
| 320 |
## Related models
|
|
|
|
| 295 |
|
| 296 |
#### Tool / function calling
|
| 297 |
|
| 298 |
+
Qwen 3.6's chat template uses Qwen's XML format:
|
| 299 |
|
| 300 |
```text
|
| 301 |
<tool_call>
|
|
|
|
| 306 |
</tool_call>
|
| 307 |
```
|
| 308 |
|
| 309 |
+
> **Tool calling via Ollama is currently disabled for this Modelfile.**
|
| 310 |
+
> Both `/api/chat` and `/v1/chat/completions` reject requests with
|
| 311 |
+
> `"<model> does not support tools"` because Ollama's tool-capability
|
| 312 |
+
> detection requires an explicit Modelfile `TEMPLATE` directive
|
| 313 |
+
> containing tool-jinja blocks, and we currently fall back to the
|
| 314 |
+
> trivial `{{ .Prompt }}` template (the GGUF's embedded jinja isn't
|
| 315 |
+
> picked up by Ollama's detector). Plain chat, streaming, and
|
| 316 |
+
> system-prompt overrides all work — only the `tools` array is
|
| 317 |
+
> rejected.
|
| 318 |
+
>
|
| 319 |
+
> If you need tool calling, use **llama.cpp** / **llama-cpp-python**
|
| 320 |
+
> directly (they read the GGUF's embedded chat template), or write a
|
| 321 |
+
> Modelfile `TEMPLATE` mirroring the official Qwen 3.6 chat template.
|
| 322 |
+
> The reference Python helper `examples/ollama_chat.py:tool_round_trip`
|
| 323 |
+
> is shipped for documentation but raises `HTTPError 400` against
|
| 324 |
+
> Ollama until the above is fixed.
|
| 325 |
|
| 326 |
## Known limitations
|
| 327 |
|
| 328 |
- **Slower per token than the 35B-A3B sibling.** Dense 27B beats sparse 35B/3B-active on steps-per-second benchmarks because every parameter contributes; if you optimize for tokens-per-second, the MoE wins.
|
| 329 |
- **No mmproj in this release**, and **vision via Ollama is broken upstream** (qwen35/qwen35moe arch entries missing from Ollama's vendored llama.cpp fork — see the [Vision](#vision) section). For image input use llama.cpp directly until that's fixed.
|
| 330 |
- **Q4_K_M quality loss** is real. Use Q5_K_M or Q6_K if you have the VRAM (~20-22 GB).
|
| 331 |
+
- **Tool calling via Ollama is currently disabled** because the Modelfile has no `TEMPLATE` directive exposing the Qwen 3.6 tool-jinja blocks; Ollama returns `does not support tools` for any request with a `tools` array. Use llama.cpp directly for tool calling, or contribute a Modelfile `TEMPLATE`. See [Tool / function calling](#tool--function-calling).
|
| 332 |
- **No formal evaluation in this card.** Numbers above are estimates.
|
| 333 |
|
| 334 |
## Related models
|
|
@@ -109,7 +109,15 @@ def fake_weather(city: str, unit: str) -> str:
|
|
| 109 |
|
| 110 |
|
| 111 |
def tool_round_trip(prompt: str) -> str:
|
| 112 |
-
"""Single-shot tool call: model -> tool -> model -> final answer.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 113 |
history: list[dict[str, Any]] = [{"role": "user", "content": prompt}]
|
| 114 |
r = requests.post(
|
| 115 |
f"{HOST}/api/chat",
|
|
@@ -183,7 +191,14 @@ def _demo() -> None:
|
|
| 183 |
print()
|
| 184 |
|
| 185 |
print("\n=== 3. tool round-trip ===")
|
| 186 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 187 |
|
| 188 |
print("\n=== 4. OpenAI-compat ===")
|
| 189 |
print(openai_chat("Say 'OpenAI endpoint OK' and nothing else."))
|
|
|
|
| 109 |
|
| 110 |
|
| 111 |
def tool_round_trip(prompt: str) -> str:
|
| 112 |
+
"""Single-shot tool call: model -> tool -> model -> final answer.
|
| 113 |
+
|
| 114 |
+
NOTE: Currently fails against Ollama with HTTPError 400
|
| 115 |
+
"<model> does not support tools" because the project Modelfile has
|
| 116 |
+
no TEMPLATE directive exposing the Qwen 3.6 tool-jinja blocks. The
|
| 117 |
+
function is shipped as a reference for the request shape — wire it
|
| 118 |
+
against llama-cpp-python or a custom-templated Modelfile to actually
|
| 119 |
+
run it. See README "Tool / function calling".
|
| 120 |
+
"""
|
| 121 |
history: list[dict[str, Any]] = [{"role": "user", "content": prompt}]
|
| 122 |
r = requests.post(
|
| 123 |
f"{HOST}/api/chat",
|
|
|
|
| 191 |
print()
|
| 192 |
|
| 193 |
print("\n=== 3. tool round-trip ===")
|
| 194 |
+
try:
|
| 195 |
+
print(tool_round_trip("What is the weather in Paris in celsius?"))
|
| 196 |
+
except requests.HTTPError as e:
|
| 197 |
+
if e.response is not None and "does not support tools" in e.response.text:
|
| 198 |
+
print("[skip] Ollama refuses tools for this Modelfile (no TEMPLATE).")
|
| 199 |
+
print(" See README 'Tool / function calling' for context.")
|
| 200 |
+
else:
|
| 201 |
+
raise
|
| 202 |
|
| 203 |
print("\n=== 4. OpenAI-compat ===")
|
| 204 |
print(openai_chat("Say 'OpenAI endpoint OK' and nothing else."))
|