Instructions to use FoolDev/Thanatos-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FoolDev/Thanatos-27B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="FoolDev/Thanatos-27B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("FoolDev/Thanatos-27B", dtype="auto")

llama-cpp-python

How to use FoolDev/Thanatos-27B with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="FoolDev/Thanatos-27B",
	filename="Thanatos-27B.Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": [
				{
					"type": "text",
					"text": "Describe this image in one sentence."
				},
				{
					"type": "image_url",
					"image_url": {
						"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
					}
				}
			]
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use FoolDev/Thanatos-27B with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Use Docker

docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M

LM Studio
Jan

vLLM

How to use FoolDev/Thanatos-27B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FoolDev/Thanatos-27B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FoolDev/Thanatos-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M

SGLang

How to use FoolDev/Thanatos-27B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FoolDev/Thanatos-27B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FoolDev/Thanatos-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FoolDev/Thanatos-27B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FoolDev/Thanatos-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Ollama
How to use FoolDev/Thanatos-27B with Ollama:
```
ollama run hf.co/FoolDev/Thanatos-27B:Q4_K_M
```

Unsloth Studio new

How to use FoolDev/Thanatos-27B with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FoolDev/Thanatos-27B to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FoolDev/Thanatos-27B to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for FoolDev/Thanatos-27B to start chatting

Pi new

How to use FoolDev/Thanatos-27B with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "FoolDev/Thanatos-27B:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use FoolDev/Thanatos-27B with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default FoolDev/Thanatos-27B:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use FoolDev/Thanatos-27B with Docker Model Runner:
```
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
```

Lemonade

How to use FoolDev/Thanatos-27B with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull FoolDev/Thanatos-27B:Q4_K_M

Run and chat with the model

lemonade run user.Thanatos-27B-Q4_K_M

List all available models

lemonade list

FoolDev commited on 27 days ago

Commit

6973bea

1 Parent(s): 84d3da6

Lowercase Q3_K_S filename so HF picks Q4_K_M as the default Ollama tag

Browse files

HF's Ollama bridge picks the alphabetically-first GGUF in the repo as
the default :latest tag (verified empirically — see CHANGELOG entry).
The previous filenames sorted byte-order as

Janus-27B.Q3_K_S.gguf < Janus-27B.Q4_K_M.gguf

(uppercase Q in both, then '3' < '4'), so the bare
'ollama run hf.co/FoolDev/janus-27b' pulled the 12 GB Q3_K_S blob —
not the 17 GB Q4_K_M as the README claimed it should.

Renaming the Q3 file to use a lowercase quant suffix
(Janus-27B.q3_k_s.gguf) flips the byte-order: uppercase 'Q' (0x51) <
lowercase 'q' (0x71), so 'Janus-27B.Q4_K_M.gguf' now sorts first and
wins HF's default-pick. The Ollama tag itself remains case-insensitive
per https://huggingface.co/docs/hub/en/ollama so '%Q3_K_S' still
resolves correctly to the renamed file.

Updated:
- Modelfile: comment now mentions the lowercase quant suffix is
intentional and explains why
- README: removed the 'pin the explicit :Q4_K_M tag' warnings; the
bare form now resolves correctly
- examples/README.md, examples/ollama_chat.py: same warning removal
- CHANGELOG: full forensic entry under [Unreleased] / Fixed

Verified bridge sync still passes (check_bridge_sync.py). Will verify
HF API + ollama pull behavior post-push.

Files changed (6) hide show

CHANGELOG.md +23 -0
Janus-27B.Q3_K_S.gguf → Janus-27B.q3_k_s.gguf +0 -0
Modelfile +4 -2
README.md +14 -20
examples/README.md +2 -6
examples/ollama_chat.py +3 -4

CHANGELOG.md CHANGED Viewed

@@ -7,6 +7,29 @@ and documentation**, not the underlying base model.
 ## [Unreleased]
 ### Added
 - `scripts/smoke_test.sh` opt-in tool-call round-trip (gated by
   `TOOLS_TEST=1`). The previous smoke test only verified the `tools`

 ## [Unreleased]
+### Fixed
+- `Janus-27B.Q3_K_S.gguf` renamed to `Janus-27B.q3_k_s.gguf`
+  (lowercase quant suffix). HF's Ollama bridge picks the
+  alphabetically-first GGUF in the repo as the default `:latest` tag,
+  and the previous filenames sorted as
+  `Janus-27B.Q3_K_S.gguf` < `Janus-27B.Q4_K_M.gguf`
+  in byte-order (uppercase 'Q' is 0x51, decimal 51 < 52 between
+  `3` and `4`), so the bare `ollama run hf.co/FoolDev/janus-27b`
+  pulled the 12 GB Q3_K_S blob, not the 17 GB Q4_K_M as the README
+  claimed it should. Lowercasing the Q3 file's quant suffix flips
+  the byte-order: uppercase 'Q' (0x51) < lowercase 'q' (0x71), so
+  `Janus-27B.Q4_K_M.gguf` now sorts first and wins the default
+  selection. The Ollama tag itself remains case-insensitive (per
+  https://huggingface.co/docs/hub/en/ollama), so users typing the
+  conventional `:Q3_K_S` still hit the lowercased file.
+  This is a workaround for an undocumented HF quirk; if HF's GGUF
+  picker switches to a different heuristic in future, this hack may
+  stop being load-bearing. The previous "Changed" entry that
+  recommended the explicit `:Q4_K_M` tag is now obsolete: bare
+  `ollama run hf.co/FoolDev/janus-27b` resolves to Q4_K_M
+  unambiguously after this rename.
 ### Added
 - `scripts/smoke_test.sh` opt-in tool-call round-trip (gated by
   `TOOLS_TEST=1`). The previous smoke test only verified the `tools`

Janus-27B.Q3_K_S.gguf → Janus-27B.q3_k_s.gguf RENAMED Viewed

File without changes

Modelfile CHANGED Viewed

@@ -6,13 +6,15 @@
 # input, or wait for the fix. See the Vision section in README.md.
 #
 # This repo bundles two GGUFs: Janus-27B.Q4_K_M.gguf (~17 GB, default)
-# and Janus-27B.Q3_K_S.gguf (~12 GB, smaller-footprint option). The FROM
 # line below points at the bundled Q4_K_M, so a fresh clone (with LFS
 # smudge enabled) supports the no-script path:
 #
 #     ollama create janus-27b -f Modelfile && ollama run janus-27b
 #
-# To use the smaller quant, edit FROM to ./Janus-27B.Q3_K_S.gguf, or run
 # `make build QUANT=Q3_K_S` which patches FROM in a temp Modelfile copy.
 #
 # Other GGUF sources (use with `make build GGUF_PATH=...`):

 # input, or wait for the fix. See the Vision section in README.md.
 #
 # This repo bundles two GGUFs: Janus-27B.Q4_K_M.gguf (~17 GB, default)
+# and Janus-27B.q3_k_s.gguf (~12 GB, smaller-footprint option; lowercase
+# quant suffix is intentional — keeps it alphabetically after Q4_K_M so
+# HF's Ollama bridge picks Q4_K_M for the bare `:latest` tag). The FROM
 # line below points at the bundled Q4_K_M, so a fresh clone (with LFS
 # smudge enabled) supports the no-script path:
 #
 #     ollama create janus-27b -f Modelfile && ollama run janus-27b
 #
+# To use the smaller quant, edit FROM to ./Janus-27B.q3_k_s.gguf, or run
 # `make build QUANT=Q3_K_S` which patches FROM in a temp Modelfile copy.
 #
 # Other GGUF sources (use with `make build GGUF_PATH=...`):

README.md CHANGED Viewed

@@ -68,16 +68,11 @@ template — HF's Ollama bridge ingests those three files, not
 `Modelfile`):
 ```bash
-ollama run hf.co/FoolDev/janus-27b:Q4_K_M    # ~17 GB, recommended
 ollama run hf.co/FoolDev/janus-27b:Q3_K_S    # ~12 GB, smaller-footprint
 ```
-> **Use the explicit `:Q4_K_M` tag.** HF's Ollama bridge currently picks
-> `Janus-27B.Q3_K_S.gguf` (the alphabetically-first GGUF in the repo) as
-> the default for the bare `ollama run hf.co/FoolDev/janus-27b` form,
-> *not* Q4_K_M. If you don't pin a quant tag explicitly you'll get the
-> 12 GB Q3_K_S blob.
 Or build locally (uses this repo's `Modelfile`, kept in sync with the
 three bridge files) for any quant:
@@ -133,19 +128,20 @@ The 27B is **dense**: every parameter participates in every forward pass. It's s
 | `README.md` | This file |
 This repo ships two GGUFs to back the HF/Ollama "Use this model"
-widget — `Janus-27B.Q4_K_M.gguf` (~17 GB, recommended) and
-`Janus-27B.Q3_K_S.gguf` (~12 GB, smaller-footprint option for 16 GB
 GPUs / unified-memory laptops):
 ```bash
-ollama run hf.co/FoolDev/janus-27b:Q4_K_M    # 17 GB, recommended
 ollama run hf.co/FoolDev/janus-27b:Q3_K_S    # 12 GB, smaller-footprint
 ```
-The bare `ollama run hf.co/FoolDev/janus-27b` form (no quant tag)
-currently resolves to the alphabetically-first GGUF in the repo —
-`Janus-27B.Q3_K_S.gguf` — *not* Q4_K_M. Pin the tag explicitly to get
-the recommended Q4_K_M quant.
 For other quants or local builds, pull from
 [`unsloth/Qwen3.6-27B-GGUF`](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF)
@@ -178,11 +174,9 @@ Two paths:
 ```bash
 # A. Pull straight from HF (uses the bundled GGUF + root-level
-#    template / system / params files; pin a quant tag explicitly):
-ollama run hf.co/FoolDev/janus-27b:Q4_K_M    # 17 GB, recommended
 ollama run hf.co/FoolDev/janus-27b:Q3_K_S    # 12 GB, smaller-footprint
-# (Bare `ollama run hf.co/FoolDev/janus-27b` resolves to Q3_K_S, not
-#  Q4_K_M — HF picks the alphabetically-first GGUF when no tag pinned.)
 # B. Build locally (lets you pick the quant):
 make build                                              # Q4_K_M  -> janus-27b
@@ -214,8 +208,8 @@ local app — point it at this repo and pick a quant.
 | App | How to load this model |
 |---|---|
-| **Ollama** | `ollama run hf.co/FoolDev/janus-27b:Q4_K_M` (or `:Q3_K_S`). Pulls the GGUF + the root-level `template` / `system` / `params` files in one step (HF's Ollama bridge ingests these three files; it does **not** read `Modelfile`). For local builds, `make build` uses `Modelfile`, which is kept in sync. **Pin the quant tag** — bare `hf.co/FoolDev/janus-27b` resolves to Q3_K_S, not Q4_K_M. |
-| **LM Studio** | Search → `FoolDev/janus-27b` → pick `Janus-27B.Q4_K_M.gguf` or `Janus-27B.Q3_K_S.gguf`. Uses the GGUF's embedded jinja chat template (Qwen 3.6 ChatML); set the system prompt manually from the `SYSTEM` block in this repo's `Modelfile`. |
 | **Jan** | Hub → "Import from Hugging Face" → `FoolDev/janus-27b`. Same template behavior as LM Studio. |
 | **llama.cpp** | `hf download FoolDev/janus-27b Janus-27B.Q4_K_M.gguf --local-dir .` then `llama-server -m Janus-27B.Q4_K_M.gguf` (or `llama-cli`, `llama-mtmd-cli` for vision via the upstream `mmproj-F16.gguf`). |
 | **llama-cpp-python** | See `examples/llama_cpp_quickstart.py` (text) and `examples/llama_cpp_vision.py` (image input). |

 `Modelfile`):
 ```bash
+ollama run hf.co/FoolDev/janus-27b           # default, ~17 GB Q4_K_M
+ollama run hf.co/FoolDev/janus-27b:Q4_K_M    # explicit, same blob
 ollama run hf.co/FoolDev/janus-27b:Q3_K_S    # ~12 GB, smaller-footprint
 ```
 Or build locally (uses this repo's `Modelfile`, kept in sync with the
 three bridge files) for any quant:
 | `README.md` | This file |
 This repo ships two GGUFs to back the HF/Ollama "Use this model"
+widget — `Janus-27B.Q4_K_M.gguf` (~17 GB, default) and
+`Janus-27B.q3_k_s.gguf` (~12 GB, smaller-footprint option for 16 GB
 GPUs / unified-memory laptops):
 ```bash
+ollama run hf.co/FoolDev/janus-27b           # default, ~17 GB Q4_K_M
 ollama run hf.co/FoolDev/janus-27b:Q3_K_S    # 12 GB, smaller-footprint
 ```
+The Q3 file is named `q3_k_s` (lowercase) so it sorts alphabetically
+after `Janus-27B.Q4_K_M.gguf`, which is what HF's Ollama bridge picks
+as the default `:latest` tag. The Ollama tag itself is case-insensitive
+(per [HF docs](https://huggingface.co/docs/hub/en/ollama)) so
+`:Q3_K_S` still resolves correctly.
 For other quants or local builds, pull from
 [`unsloth/Qwen3.6-27B-GGUF`](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF)
 ```bash
 # A. Pull straight from HF (uses the bundled GGUF + root-level
+#    template / system / params files):
+ollama run hf.co/FoolDev/janus-27b           # default, ~17 GB Q4_K_M
 ollama run hf.co/FoolDev/janus-27b:Q3_K_S    # 12 GB, smaller-footprint
 # B. Build locally (lets you pick the quant):
 make build                                              # Q4_K_M  -> janus-27b
 | App | How to load this model |
 |---|---|
+| **Ollama** | `ollama run hf.co/FoolDev/janus-27b` (default Q4_K_M) or `:Q3_K_S` for the smaller quant. Pulls the GGUF + the root-level `template` / `system` / `params` files in one step (HF's Ollama bridge ingests these three files; it does **not** read `Modelfile`). For local builds, `make build` uses `Modelfile`, which is kept in sync. |
+| **LM Studio** | Search → `FoolDev/janus-27b` → pick `Janus-27B.Q4_K_M.gguf` or `Janus-27B.q3_k_s.gguf`. Uses the GGUF's embedded jinja chat template (Qwen 3.6 ChatML); set the system prompt manually from the `SYSTEM` block in this repo's `Modelfile`. |
 | **Jan** | Hub → "Import from Hugging Face" → `FoolDev/janus-27b`. Same template behavior as LM Studio. |
 | **llama.cpp** | `hf download FoolDev/janus-27b Janus-27B.Q4_K_M.gguf --local-dir .` then `llama-server -m Janus-27B.Q4_K_M.gguf` (or `llama-cli`, `llama-mtmd-cli` for vision via the upstream `mmproj-F16.gguf`). |
 | **llama-cpp-python** | See `examples/llama_cpp_quickstart.py` (text) and `examples/llama_cpp_vision.py` (image input). |

examples/README.md CHANGED Viewed

@@ -21,17 +21,13 @@ Easiest path — pull straight from HF (gets the bundled Q4_K_M GGUF +
 this repo's Modelfile in one step):
 ```bash
-ollama pull hf.co/FoolDev/janus-27b:Q4_K_M    # 17 GB, recommended
 # or:
 ollama pull hf.co/FoolDev/janus-27b:Q3_K_S    # 12 GB, smaller-footprint
 pip install requests
-MODEL=hf.co/FoolDev/janus-27b:Q4_K_M python ollama_chat.py
 ```
-> Pin the quant tag explicitly. Bare `ollama pull hf.co/FoolDev/janus-27b`
-> resolves to Q3_K_S (HF picks the alphabetically-first GGUF in the
-> repo), not Q4_K_M.
 Or build locally from this repo (uses the bundled `Janus-27B.Q4_K_M.gguf`,
 no edits required):

 this repo's Modelfile in one step):
 ```bash
+ollama pull hf.co/FoolDev/janus-27b           # default, 17 GB Q4_K_M
 # or:
 ollama pull hf.co/FoolDev/janus-27b:Q3_K_S    # 12 GB, smaller-footprint
 pip install requests
+MODEL=hf.co/FoolDev/janus-27b python ollama_chat.py
 ```
 Or build locally from this repo (uses the bundled `Janus-27B.Q4_K_M.gguf`,
 no edits required):

examples/ollama_chat.py CHANGED Viewed

@@ -9,10 +9,9 @@ Prerequisites (pick one):
         # or:
         $ ollama create janus-27b -f ../Modelfile
-    B. Pull straight from HF (pin the quant tag explicitly — bare
-       `hf.co/FoolDev/janus-27b` resolves to Q3_K_S, not Q4_K_M):
-        $ ollama run hf.co/FoolDev/janus-27b:Q4_K_M
-        # then set MODEL=hf.co/FoolDev/janus-27b:Q4_K_M below
 Then:
     $ ollama serve         # usually already running

         # or:
         $ ollama create janus-27b -f ../Modelfile
+    B. Pull straight from HF (default tag is Q4_K_M, ~17 GB):
+        $ ollama run hf.co/FoolDev/janus-27b
+        # then set MODEL=hf.co/FoolDev/janus-27b below
 Then:
     $ ollama serve         # usually already running