Instructions to use FoolDev/Thanatos-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FoolDev/Thanatos-27B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="FoolDev/Thanatos-27B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("FoolDev/Thanatos-27B", dtype="auto")

llama-cpp-python

How to use FoolDev/Thanatos-27B with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="FoolDev/Thanatos-27B",
	filename="Thanatos-27B.Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": [
				{
					"type": "text",
					"text": "Describe this image in one sentence."
				},
				{
					"type": "image_url",
					"image_url": {
						"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
					}
				}
			]
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use FoolDev/Thanatos-27B with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Use Docker

docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M

LM Studio
Jan

vLLM

How to use FoolDev/Thanatos-27B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FoolDev/Thanatos-27B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FoolDev/Thanatos-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M

SGLang

How to use FoolDev/Thanatos-27B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FoolDev/Thanatos-27B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FoolDev/Thanatos-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FoolDev/Thanatos-27B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FoolDev/Thanatos-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Ollama
How to use FoolDev/Thanatos-27B with Ollama:
```
ollama run hf.co/FoolDev/Thanatos-27B:Q4_K_M
```

Unsloth Studio new

How to use FoolDev/Thanatos-27B with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FoolDev/Thanatos-27B to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FoolDev/Thanatos-27B to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for FoolDev/Thanatos-27B to start chatting

Pi new

How to use FoolDev/Thanatos-27B with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "FoolDev/Thanatos-27B:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use FoolDev/Thanatos-27B with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default FoolDev/Thanatos-27B:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use FoolDev/Thanatos-27B with Docker Model Runner:
```
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
```

Lemonade

How to use FoolDev/Thanatos-27B with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull FoolDev/Thanatos-27B:Q4_K_M

Run and chat with the model

lemonade run user.Thanatos-27B-Q4_K_M

List all available models

lemonade list

FoolDev Claude Opus 4.7 commited on 10 days ago

Commit

2bf2adf

1 Parent(s): c336f44

release: v0.6.0

Browse files

First tagged release under the Thanatos name. Cuts the
two-week-burst of changes since [0.5.0] (2026-05-02) into a
versioned section. Headline:

- Project renamed `Janus-27B` -> `Thanatos-27B` (HF repo, GGUFs,
default Ollama tag, system-prompt identity, banner). MoE sibling
`FoolDev/Janus-35B` keeps the Janus name.
- Bundle GGUF re-stamped `qwen35` -> `qwen36` (the
architecturally-honest label; breaks stock loaders until upstream
adds the arch entry; as of 2026-05-19 there's no PR or tracking
issue for it in either ggml-org/llama.cpp or ollama/ollama).
- `make load-bundle`: one-shot bundle -> local Ollama tag.
- `make heal-hf`: rebadge an already-pulled hf.co/... tag in store
+ rewrite its manifest's model-layer digest, so the same tag
loads in place.
- `scripts/rename_arch.py`: generic GGUF arch renamer (metadata
only, tensors byte-identical).
- Tool-calling end-to-end via Ollama (Modelfile TEMPLATE) and HF
bridge (root-level template / system / params; kept in sync via
`scripts/check_bridge_sync.py`).
- `make smoke` / `make smoke-tools` / `make bench` for regression
and tok/s measurement. Three measured Strix Halo data points
logged in the Modelfile and CHANGELOG.
- Vision section restructured around `llama-server --mmproj` (the
only reliably working vision path; Ollama's C++ fallback for
mmproj still lacks the qwen35/qwen35moe arch entries — tracked
in ollama/ollama#15898).
- Bundled GGUF reduced to Q4_K_M only (17 GB); Q3_K_S no longer
redistributed, built locally via `make build QUANT=Q3_K_S`.

CITATION.cff has no `version` field, so no bump needed there. The
git tag v0.6.0 (added in this commit) is the canonical reference;
the CHANGELOG header uses the conventional last-pre-release commit
sha (`c336f44`).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Files changed (1) hide show

CHANGELOG.md +69 -0

CHANGELOG.md CHANGED Viewed

@@ -7,6 +7,75 @@ and documentation**, not the underlying base model.
 ## [Unreleased]
 ### Added
 - `scripts/heal_hf_pull.sh` + `make heal-hf`: heal an already-pulled
   `hf.co/FoolDev/Thanatos-27B:...` tag in-store by rebadging its

 ## [Unreleased]
+## [0.6.0] - 2026-05-19 — `c336f44`
+Headline changes since 0.5.0 (over a two-week burst on 2026-05-19):
+- **Project renamed `Janus-27B` → `Thanatos-27B`** (HF repo, GGUF
+  filenames, default Ollama tag, system-prompt identity, banner
+  wordmark). MoE 35B sibling `FoolDev/Janus-35B` keeps the Janus
+  name; the dense 27B splits off as its own identity.
+- **Bundle GGUF re-stamped `general.architecture: 'qwen35'` →
+  `'qwen36'`** — the architecturally-honest label. Breaks
+  `ollama run hf.co/FoolDev/Thanatos-27B` and direct
+  `llama-server -m Thanatos-27B.Q4_K_M.gguf` on every stock loader
+  until upstream adds the arch entry (which, as of 2026-05-19,
+  has no PR or tracking issue in either ggml-org/llama.cpp or
+  ollama/ollama).
+- **`make load-bundle`** — one-shot bundle → loadable local Ollama
+  tag (LFS smudge + qwen36 → qwen35 rebadge + `ollama create`).
+- **`make heal-hf`** — heal an already-pulled HF-bridge tag in
+  store by rebadging its model blob qwen36 → qwen35 and rewriting
+  the manifest's model-layer digest, so the same `hf.co/...` tag
+  becomes loadable in place without switching tag names.
+- **`scripts/rename_arch.py`** — generic GGUF arch renamer
+  (metadata only, tensor data byte-identical). Used by load-bundle
+  and heal-hf; reusable for future arch renames.
+- **Tool-calling** end-to-end via Ollama (Modelfile TEMPLATE wires
+  `.Tools` / `.ToolCalls` so `ollama show` reports the `tools`
+  capability and `/api/chat` / `/v1/chat/completions` accept tools
+  arrays). HF Ollama bridge gets parallel coverage via root-level
+  `template` / `system` / `params` files (HF's bridge does **not**
+  read `Modelfile`); kept in sync via `scripts/check_bridge_sync.py`,
+  wired into `make check`.
+- **`scripts/bench.sh` + `make bench`** — 3-prompt tok/s benchmark
+  using Ollama's `eval_count` / `eval_duration`. Three measured
+  data points logged on the Strix Halo reference hardware (Ryzen
+  AI Max+ 395 / Radeon 8060S iGPU): Q3_K_S Vulkan 12.31 tok/s,
+  Q4_K_M Vulkan 9.31 + 9.19 tok/s (two runs), Q3_K_S ROCm
+  10.14 tok/s (older backend snapshot).
+- **`scripts/smoke_test.sh` + `make smoke` / `make smoke-tools`** —
+  server reachable, model present, tools capability, round-trip,
+  token-leakage guard, opt-in tool-call round-trip
+  (`TOOLS_TEST=1`).
+- **Vision section** restructured around llama.cpp / llama-cpp-python
+  (Ollama vision is broken upstream — `qwen35` / `qwen35moe` arch
+  entries present in Go engine, missing from the C++ fallback the
+  Ollama runner switches to when mmproj is attached;
+  [ollama/ollama#15898](https://github.com/ollama/ollama/issues/15898)).
+  `llama-server --mmproj` HTTP path now the lead recommendation.
+- **Bundled GGUF**: `Thanatos-27B.Q4_K_M.gguf` (~17 GB) is the only
+  redistributed quant, picked by HF's Ollama bridge for the default
+  `:latest` tag. Other quants (Q3_K_S, Q5_K_M, Q6_K, safetensors)
+  pulled from upstream on demand via `make build QUANT=...`.
+Bug-fix highlights:
+- `scripts/fetch_mmproj.sh` renamed to `scripts/fetch_vision.sh` —
+  HF's Ollama bridge was filename-pattern-matching `mmproj*` anywhere
+  in the repo and shipping the 2-KB bash script as the
+  `application/vnd.ollama.image.projector` layer, breaking
+  `ollama show` / `ollama run` with `Error: invalid file magic`.
+- `Modelfile` PARAMETER stop directives for `<|im_end|>`,
+  `<|endoftext|>`, `<|im_start|>` — Ollama was only picking up
+  `<|im_end|>` from GGUF metadata, so the model kept generating past
+  `<|endoftext|>` and synthesised a fake user turn.
+- `examples/ollama_chat.py` thinking-trace handling under Ollama
+  0.24 (now reads `message.thinking`, falls back to `<think>`
+  extraction for older builds).
+See the full per-bullet changelog below.
 ### Added
 - `scripts/heal_hf_pull.sh` + `make heal-hf`: heal an already-pulled
   `hf.co/FoolDev/Thanatos-27B:...` tag in-store by rebadging its