Instructions to use FoolDev/Thanatos-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FoolDev/Thanatos-27B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="FoolDev/Thanatos-27B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("FoolDev/Thanatos-27B", dtype="auto")

llama-cpp-python

How to use FoolDev/Thanatos-27B with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="FoolDev/Thanatos-27B",
	filename="Thanatos-27B.Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": [
				{
					"type": "text",
					"text": "Describe this image in one sentence."
				},
				{
					"type": "image_url",
					"image_url": {
						"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
					}
				}
			]
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use FoolDev/Thanatos-27B with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Use Docker

docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M

LM Studio
Jan

vLLM

How to use FoolDev/Thanatos-27B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FoolDev/Thanatos-27B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FoolDev/Thanatos-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M

SGLang

How to use FoolDev/Thanatos-27B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FoolDev/Thanatos-27B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FoolDev/Thanatos-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FoolDev/Thanatos-27B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FoolDev/Thanatos-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Ollama
How to use FoolDev/Thanatos-27B with Ollama:
```
ollama run hf.co/FoolDev/Thanatos-27B:Q4_K_M
```

Unsloth Studio new

How to use FoolDev/Thanatos-27B with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FoolDev/Thanatos-27B to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FoolDev/Thanatos-27B to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for FoolDev/Thanatos-27B to start chatting

Pi new

How to use FoolDev/Thanatos-27B with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "FoolDev/Thanatos-27B:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use FoolDev/Thanatos-27B with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default FoolDev/Thanatos-27B:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use FoolDev/Thanatos-27B with Docker Model Runner:
```
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
```

Lemonade

How to use FoolDev/Thanatos-27B with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull FoolDev/Thanatos-27B:Q4_K_M

Run and chat with the model

lemonade run user.Thanatos-27B-Q4_K_M

List all available models

lemonade list

FoolDev Claude Opus 4.7 commited on 10 days ago

Commit

a4d3b6e

1 Parent(s): c1c4dfd

docs: ground "qwen35 is canonical" in upstream code-level evidence

Browse files

The README "Architecture" section previously asserted that qwen35 is
the canonical load-time label for Qwen 3.6 without showing the work.
Audit of the actual upstream stack found three converging pieces of
evidence — now listed inline:

- Qwen's own HF configs for Qwen/Qwen3.6-27B and
Qwen/Qwen3.6-35B-A3B declare model_type "qwen3_5" /
"qwen3_5_moe" and the Qwen3_5*ForConditionalGeneration arch
classes. No Qwen3_6 class exists in transformers.
- llama.cpp's convert_hf_to_gguf.py maps those classes to
MODEL_ARCH.QWEN35 / QWEN35MOE, so unsloth's GGUFs are
natively qwen35-stamped out of the converter.
- src/models/qwen35.cpp has `case 64: LLM_TYPE_27B` (this model)
and qwen35moe.cpp has `case 40: LLM_TYPE_35B_A3B` (the
Janus-35B sibling base). The arch entries were written to
load Qwen 3.6 weights, not just Qwen 3.5.

Conclusion: qwen35 isn't a stand-in awaiting qwen36 support; it's
already the upstream-canonical name for the whole Qwen 3.5 / 3.6
hybrid SSM + attention family. The "qwen36" name was Foolish-dev's
invention, not an upstream concept.

History paragraph softened from "qwen36 was the architecturally-
honest label" to "version-specific label, on the theory that qwen35
was a loader stand-in" — a theory the audit shows was mistaken.

The rename_arch.py example flipped to qwen36 -> qwen35 (the
direction that actually serves a real user need: legacy v0.6.0-era
recovery). CHANGELOG [Unreleased] picks up a "Changed (upstream-
evidence audit)" bullet with the per-source breakdown.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Files changed (2) hide show

CHANGELOG.md +39 -0
README.md +42 -19

CHANGELOG.md CHANGED Viewed

@@ -61,6 +61,45 @@ and documentation**, not the underlying base model.
   (skips early); load-bundle does a no-op rebadge on a qwen35 bundle
   and runs `ollama create` as normal.
 ### Changed (round-trip residue cleanup)
 - Five doc surfaces still carried the v0.6.0-era "bundle is
   qwen36-stamped, fails today, rebadge to qwen35 first" narrative

   (skips early); load-bundle does a no-op rebadge on a qwen35 bundle
   and runs `ollama create` as normal.
+### Changed (upstream-evidence audit — qwen35 is canonical, not a workaround)
+- Read the actual upstream code paths for Qwen 3.6 loading and
+  found that `qwen35` isn't a stand-in awaiting `qwen36` support;
+  it's the upstream-canonical name for the whole Qwen 3.5 / 3.6
+  hybrid SSM + attention family. Three converging pieces of
+  evidence, now folded into the README "Architecture" section:
+  - **Qwen's own HF configs** for both `Qwen/Qwen3.6-27B` and
+    `Qwen/Qwen3.6-35B-A3B` declare `model_type: "qwen3_5"` /
+    `"qwen3_5_moe"` and the `Qwen3_5*ForConditionalGeneration`
+    arch classes. No `Qwen3_6` arch class exists in `transformers`
+    — Qwen reuses the 3.5 class names for 3.6 weights.
+  - **`convert_hf_to_gguf.py`** in `ggml-org/llama.cpp` registers
+    `Qwen3_5ForCausalLM` → `MODEL_ARCH.QWEN35` and
+    `Qwen3_5MoeForCausalLM` → `MODEL_ARCH.QWEN35MOE`, so the
+    unsloth GGUFs this repo pulls from are natively qwen35-stamped
+    out of the converter.
+  - **`src/models/qwen35.cpp`** has an explicit
+    `case 64: type = LLM_TYPE_27B` branch (this model);
+    **`qwen35moe.cpp`** has `case 40: type = LLM_TYPE_35B_A3B`
+    (the Janus-35B sibling base). The arch entries were written
+    to load Qwen 3.6 weights, not just Qwen 3.5; this is
+    code-level confirmation, not interpretation.
+- Reframed the README "Architecture" section accordingly: the
+  three pieces of evidence are listed inline; the "qwen35 is the
+  canonical load-time label" claim is now grounded rather than
+  asserted; the History paragraph softens the v0.6.0 stamp choice
+  from "architecturally-honest" to "version-specific on the
+  theory that `qwen35` was a loader stand-in awaiting `qwen36`
+  support" — a theory the audit shows was mistaken.
+- Flipped the `scripts/rename_arch.py` example back to qwen36 →
+  qwen35 (the direction that actually serves the only realistic
+  use case: legacy v0.6.0-era recovery). Previous direction
+  (qwen35 → qwen36) made less sense given the audit conclusion.
+- Tooling unchanged. `scripts/rename_arch.py`, `make heal-hf`,
+  `make load-bundle` still exist and still work; they're just
+  documented honestly as legacy recovery + generic flip utility
+  rather than as workarounds awaiting upstream changes that will
+  never come.
 ### Changed (round-trip residue cleanup)
 - Five doc surfaces still carried the v0.6.0-era "bundle is
   qwen36-stamped, fails today, rebadge to qwen35 first" narrative

README.md CHANGED Viewed

@@ -174,37 +174,60 @@ If you want the safetensors for `transformers`, fetch them from [`Qwen/Qwen3.6-2
   current loader compatibility.
 - Multi-token prediction (MTP) head trained for speculative decoding
-**The bundled GGUF declares `general.architecture: 'qwen35'`** — the
-arch entry every released llama.cpp / Ollama loads the Qwen 3.6
-hybrid SSM + attention family under. Upstream `ggml-org/llama.cpp`
-and `ollama/ollama` register the stack as `qwen35` / `qwen35moe`,
-including for actual Qwen 3.6-family weights (the upstream
-`Qwen3.6-35B-A3B_Q8_0` GGUFs load under `qwen35moe`); no `qwen36`
-arch entry exists, and there's no open PR or tracking issue for
-one — `qwen35` is the canonical load-time label.
 `ollama run hf.co/FoolDev/Thanatos-27B` and `llama-server -m
 Thanatos-27B.Q4_K_M.gguf` both load directly on current stock
 loaders.
-**History.** Through v0.6.0 the bundle was stamped `qwen36` — the
-architecturally-honest label, but unloadable on stock loaders.
-Users needed `make load-bundle` or `make heal-hf` to rebadge
-before loading. The stamp was reverted to `qwen35` in commit
-`964e418` to stop forcing the workaround on every fresh pull;
-tensor data was byte-identical across both stamps, only the
 `general.architecture` KV (and namespaced KV keys) flipped. See
 the [CHANGELOG](CHANGELOG.md) entry for the full rationale.
 `scripts/rename_arch.py` is kept in the repo as a generic GGUF
-arch renamer (metadata only, tensors byte-identical) for future
-flips:
 ```bash
 python3 scripts/rename_arch.py \
-    --from-arch qwen35 --to-arch qwen36 \
-    Thanatos-27B.Q4_K_M.gguf \
-    Thanatos-27B.Q4_K_M.qwen36.gguf
 ```
 If you pulled the bundle while it was still qwen36-stamped (v0.6.0

   current loader compatibility.
 - Multi-token prediction (MTP) head trained for speculative decoding
+**The bundled GGUF declares `general.architecture: 'qwen35'`** — not a
+workaround for an unimplemented `qwen36` arch, but the canonical
+upstream label for the entire Qwen 3.5 / 3.6 hybrid SSM + attention
+family. The naming convergence runs through three layers of the
+stack:
+- **Qwen's own HF configs.** `Qwen/Qwen3.6-27B/config.json` declares
+  `"model_type": "qwen3_5"` and
+  `"architectures": ["Qwen3_5ForConditionalGeneration"]`. The MoE
+  sibling `Qwen/Qwen3.6-35B-A3B` declares `"qwen3_5_moe"` /
+  `Qwen3_5MoeForConditionalGeneration`. No `Qwen3_6` arch class
+  exists in `transformers`; Qwen reuses the 3.5 class names.
+- **llama.cpp's converter.** `convert_hf_to_gguf.py` registers
+  `Qwen3_5ForCausalLM` → `MODEL_ARCH.QWEN35` and
+  `Qwen3_5MoeForCausalLM` → `MODEL_ARCH.QWEN35MOE`. The unsloth
+  GGUFs this repo pulls from (`unsloth/Qwen3.6-27B-GGUF`,
+  `unsloth/Qwen3.6-35B-A3B-GGUF`) inherit those stamps.
+- **llama.cpp's model code.** `src/models/qwen35.cpp` has an
+  explicit `case 64: type = LLM_TYPE_27B` branch for this model;
+  `qwen35moe.cpp` has `case 40: type = LLM_TYPE_35B_A3B` for the
+  Janus-35B sibling base. The arch entries were written to load
+  Qwen 3.6 weights, not just Qwen 3.5.
+There is no PR or tracking issue for a `qwen36` arch entry in
+`ggml-org/llama.cpp` or `ollama/ollama` because none is needed —
+`qwen35` already loads the model the upstream code path was
+designed to load.
 `ollama run hf.co/FoolDev/Thanatos-27B` and `llama-server -m
 Thanatos-27B.Q4_K_M.gguf` both load directly on current stock
 loaders.
+**History.** Through v0.6.0 the bundle was stamped `qwen36` as a
+version-specific label, on the theory that `qwen35` was a
+loader-side stand-in awaiting proper Qwen 3.6 support. Stock
+loaders rejected it; users needed `make load-bundle` or
+`make heal-hf` to rebadge before loading. The stamp was reverted
+to `qwen35` in commit `964e418` once daily friction outweighed
+the version-specificity. The upstream-evidence audit captured
+above (post-`72259c1`) then confirmed `qwen35` is in fact the
+canonical family label — not a workaround. Tensor data was
+byte-identical across both stamps; only the
 `general.architecture` KV (and namespaced KV keys) flipped. See
 the [CHANGELOG](CHANGELOG.md) entry for the full rationale.
 `scripts/rename_arch.py` is kept in the repo as a generic GGUF
+arch renamer (metadata only, tensors byte-identical), useful for
+the legacy qwen36 → qwen35 rebadge or any future arch flip:
 ```bash
 python3 scripts/rename_arch.py \
+    --from-arch qwen36 --to-arch qwen35 \
+    Thanatos-27B.Q4_K_M.qwen36.gguf \
+    Thanatos-27B.Q4_K_M.gguf
 ```
 If you pulled the bundle while it was still qwen36-stamped (v0.6.0