Instructions to use FoolDev/Thanatos-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use FoolDev/Thanatos-27B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="FoolDev/Thanatos-27B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("FoolDev/Thanatos-27B", dtype="auto") - llama-cpp-python
How to use FoolDev/Thanatos-27B with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="FoolDev/Thanatos-27B", filename="Thanatos-27B.Q4_K_M.gguf", )
llm.create_chat_completion( messages = [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] ) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- llama.cpp
How to use FoolDev/Thanatos-27B with llama.cpp:
Install from brew
brew install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf FoolDev/Thanatos-27B:Q4_K_M # Run inference directly in the terminal: llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf FoolDev/Thanatos-27B:Q4_K_M # Run inference directly in the terminal: llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf FoolDev/Thanatos-27B:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf FoolDev/Thanatos-27B:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M
Use Docker
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
- LM Studio
- Jan
- vLLM
How to use FoolDev/Thanatos-27B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "FoolDev/Thanatos-27B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FoolDev/Thanatos-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
- SGLang
How to use FoolDev/Thanatos-27B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "FoolDev/Thanatos-27B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FoolDev/Thanatos-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "FoolDev/Thanatos-27B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FoolDev/Thanatos-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Ollama
How to use FoolDev/Thanatos-27B with Ollama:
ollama run hf.co/FoolDev/Thanatos-27B:Q4_K_M
- Unsloth Studio new
How to use FoolDev/Thanatos-27B with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for FoolDev/Thanatos-27B to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for FoolDev/Thanatos-27B to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for FoolDev/Thanatos-27B to start chatting
- Pi new
How to use FoolDev/Thanatos-27B with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "FoolDev/Thanatos-27B:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use FoolDev/Thanatos-27B with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default FoolDev/Thanatos-27B:Q4_K_M
Run Hermes
hermes
- Docker Model Runner
How to use FoolDev/Thanatos-27B with Docker Model Runner:
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
- Lemonade
How to use FoolDev/Thanatos-27B with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull FoolDev/Thanatos-27B:Q4_K_M
Run and chat with the model
lemonade run user.Thanatos-27B-Q4_K_M
List all available models
lemonade list
docs: reframe stale qwen36-stamped narrative as legacy / recovery
Browse filesFive doc surfaces still carried the v0.6.0-era "bundle is
qwen36-stamped, fails today, rebadge first" framing that the earlier
83022eb / 4811e8d collapses missed: the README "What's here" intro
paragraph + load_bundle.sh / heal_hf_pull.sh table cells, the "Local
apps" intro + Ollama cell, the load_bundle.sh and heal_hf_pull.sh
preambles, and the Makefile load-bundle help text. Reframed each to
describe the current qwen35-stamped reality; the qwen36 rebadge code
paths stay, but their docs now say "legacy v0.6.0 recovery, no-op on
fresh pulls". Net -15 lines.
Upstream re-poll (gh search ggml-org/llama.cpp + ollama/ollama for
qwen36, 2026-05-19): still no arch entry, still no open PR or issue
β same answer as the earlier recheck; the qwen35 stamp stands.
CHANGELOG [Unreleased] picks up a "Changed (round-trip residue
cleanup)" bullet with the per-surface breakdown.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- CHANGELOG.md +40 -0
- Makefile +2 -2
- README.md +8 -17
- scripts/heal_hf_pull.sh +14 -19
- scripts/load_bundle.sh +14 -15
|
@@ -61,6 +61,46 @@ and documentation**, not the underlying base model.
|
|
| 61 |
(skips early); load-bundle does a no-op rebadge on a qwen35 bundle
|
| 62 |
and runs `ollama create` as normal.
|
| 63 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
### Added
|
| 65 |
- `scripts/check.sh` now greps for the `VAR="$(cmd 2>/dev/null | filter)"`
|
| 66 |
silent-exit pattern that bit `heal_hf_pull.sh` (commit 385ed94).
|
|
|
|
| 61 |
(skips early); load-bundle does a no-op rebadge on a qwen35 bundle
|
| 62 |
and runs `ollama create` as normal.
|
| 63 |
|
| 64 |
+
### Changed (round-trip residue cleanup)
|
| 65 |
+
- Five doc surfaces still carried the v0.6.0-era "bundle is
|
| 66 |
+
qwen36-stamped, fails today, rebadge to qwen35 first" narrative
|
| 67 |
+
that the earlier `83022eb` / `4811e8d` collapses missed. Reframed
|
| 68 |
+
to describe the current qwen35-stamped reality:
|
| 69 |
+
- **README** "What's here" intro paragraph (the secondary
|
| 70 |
+
TL;DR-style block above Architecture): flipped the
|
| 71 |
+
"qwen36-stamped β fails today, see Heads-up above" example to
|
| 72 |
+
the same one-liner the primary TL;DR uses; dropped the "as a
|
| 73 |
+
working-today fallback while the bundle waits on upstream
|
| 74 |
+
qwen36 support" framing from the Q3_K_S paragraph since
|
| 75 |
+
there's no longer anything to wait on.
|
| 76 |
+
- **README** "Local apps" intro paragraph: removed the entire
|
| 77 |
+
"every row below assumes you've rebadged it to qwen35"
|
| 78 |
+
caveat; trimmed "or to bypass the qwen36 block today" from
|
| 79 |
+
the Ollama table cell.
|
| 80 |
+
- **README** "What's here" table cells for `load_bundle.sh`
|
| 81 |
+
and `heal_hf_pull.sh`: reframed each as primary qwen35 path
|
| 82 |
+
+ legacy qwen36 recovery branch, instead of describing the
|
| 83 |
+
rebadge as mandatory.
|
| 84 |
+
- **`scripts/load_bundle.sh`** preamble: rewritten to describe
|
| 85 |
+
the current qwen35-stamped bundle as the default; the qwen36
|
| 86 |
+
rebadge branch is documented as a fallback for legacy
|
| 87 |
+
pre-`964e418` checkouts.
|
| 88 |
+
- **`scripts/heal_hf_pull.sh`** preamble: reframed as recovery
|
| 89 |
+
for pre-`964e418` v0.6.0-era pulls; clarified that fresh
|
| 90 |
+
pulls don't need it.
|
| 91 |
+
- **Makefile** `load-bundle` help text + example comment: dropped
|
| 92 |
+
the "qwen36-stamped bundle, rebadged to qwen35" framing.
|
| 93 |
+
Net -15 lines. Upstream re-poll (`gh search` against
|
| 94 |
+
`ggml-org/llama.cpp` + `ollama/ollama` for `qwen36` PRs/issues,
|
| 95 |
+
2026-05-19): no qwen36 arch entry, no open PR or tracking issue
|
| 96 |
+
β same answer as the earlier subagent recheck; the
|
| 97 |
+
`qwen35` stamp choice still stands. (Side-finding worth noting
|
| 98 |
+
for a future README update: llama.cpp PR `#22673` merged
|
| 99 |
+
2026-05-16 adds MTP speculative decoding for Qwen3.6, so the
|
| 100 |
+
Architecture section's "MTP head trained for speculative
|
| 101 |
+
decoding" claim now has a usable consumer path β out of scope
|
| 102 |
+
for this cleanup.)
|
| 103 |
+
|
| 104 |
### Added
|
| 105 |
- `scripts/check.sh` now greps for the `VAR="$(cmd 2>/dev/null | filter)"`
|
| 106 |
silent-exit pattern that bit `heal_hf_pull.sh` (commit 385ed94).
|
|
@@ -13,7 +13,7 @@
|
|
| 13 |
# make build # Q4_K_M from unsloth (qwen35-stamped, loads today)
|
| 14 |
# make build QUANT=Q3_K_S # smaller quant
|
| 15 |
# make build GGUF_PATH=~/models/Qwen3.6-27B-Q4_K_M.gguf
|
| 16 |
-
# make load-bundle # this repo's
|
| 17 |
# make smoke
|
| 18 |
# make check
|
| 19 |
# make clean
|
|
@@ -40,7 +40,7 @@ endif
|
|
| 40 |
build: ## Download qwen35-stamped GGUF from unsloth and run 'ollama create' (loads today).
|
| 41 |
GGUF_PATH=$(GGUF_PATH) TAG=$(TAG) ./scripts/build.sh $(QUANT)
|
| 42 |
|
| 43 |
-
load-bundle: ## Load THIS repo's
|
| 44 |
TAG=$(TAG) ./scripts/load_bundle.sh
|
| 45 |
|
| 46 |
heal-hf: ## Heal an already-pulled hf.co/FoolDev/Thanatos-27B tag in-store (rebadge blob + manifest digest).
|
|
|
|
| 13 |
# make build # Q4_K_M from unsloth (qwen35-stamped, loads today)
|
| 14 |
# make build QUANT=Q3_K_S # smaller quant
|
| 15 |
# make build GGUF_PATH=~/models/Qwen3.6-27B-Q4_K_M.gguf
|
| 16 |
+
# make load-bundle # this repo's bundled GGUF -> local Ollama tag (smudge LFS if needed)
|
| 17 |
# make smoke
|
| 18 |
# make check
|
| 19 |
# make clean
|
|
|
|
| 40 |
build: ## Download qwen35-stamped GGUF from unsloth and run 'ollama create' (loads today).
|
| 41 |
GGUF_PATH=$(GGUF_PATH) TAG=$(TAG) ./scripts/build.sh $(QUANT)
|
| 42 |
|
| 43 |
+
load-bundle: ## Load THIS repo's bundled GGUF into a local Ollama tag (smudge LFS + ollama create).
|
| 44 |
TAG=$(TAG) ./scripts/load_bundle.sh
|
| 45 |
|
| 46 |
heal-hf: ## Heal an already-pulled hf.co/FoolDev/Thanatos-27B tag in-store (rebadge blob + manifest digest).
|
|
@@ -122,8 +122,8 @@ The 27B is **dense**: every parameter participates in every forward pass. It's s
|
|
| 122 |
| `template`, `system`, `params` | Used by HF's Ollama bridge when users `ollama run hf.co/FoolDev/Thanatos-27B` directly (the bridge does **not** read `Modelfile` β see [HF Ollama docs](https://huggingface.co/docs/hub/en/ollama)). Mirrors the `Modelfile`'s template / system prompt / sampling params. |
|
| 123 |
| `examples/` | Ready-to-run Python clients for Ollama, Transformers, and llama-cpp-python |
|
| 124 |
| `scripts/build.sh` | Pulls a qwen35-stamped GGUF from `unsloth/Qwen3.6-27B-GGUF` and runs `ollama create` (loads on today's llama.cpp / Ollama; see `make build`) |
|
| 125 |
-
| `scripts/load_bundle.sh` | One-shot path from *this repo's*
|
| 126 |
-
| `scripts/heal_hf_pull.sh` |
|
| 127 |
| `scripts/smoke_test.sh` | Verifies an Ollama daemon + model, runs a round-trip, asserts no chat-template tokens leak into the response. With `TOOLS_TEST=1`, also exercises an end-to-end tool-call round-trip and checks the response shape |
|
| 128 |
| `scripts/bench.sh` | Measures real tok/s using Ollama's `eval_count` / `eval_duration` metadata over a 3-prompt mix (run `make bench`) |
|
| 129 |
| `scripts/fetch_vision.sh` | Pulls the vision projector (`mmproj-F16.gguf`) for llama.cpp (Ollama vision is broken upstream β see [Vision](#vision)). Renamed from `fetch_mmproj.sh` because HF's Ollama bridge auto-indexed the script as a vision projector layer (filename pattern match). |
|
|
@@ -136,17 +136,15 @@ The 27B is **dense**: every parameter participates in every forward pass. It's s
|
|
| 136 |
| `README.md` | This file |
|
| 137 |
|
| 138 |
This repo ships a single GGUF to back the HF/Ollama "Use this model"
|
| 139 |
-
widget β `Thanatos-27B.Q4_K_M.gguf` (~17 GB,
|
| 140 |
|
| 141 |
```bash
|
| 142 |
-
ollama run hf.co/FoolDev/Thanatos-27B # 17 GB Q4_K_M,
|
| 143 |
```
|
| 144 |
|
| 145 |
-
For 16 GB GPUs / unified-memory laptops
|
| 146 |
-
|
| 147 |
-
`
|
| 148 |
-
from `unsloth/Qwen3.6-27B-GGUF` (qwen35-stamped, loads on every
|
| 149 |
-
current llama.cpp / Ollama build) and creates a local `thanatos-27b`
|
| 150 |
Ollama tag. Does not redistribute via this repo.
|
| 151 |
|
| 152 |
For other quants or local builds, pull from
|
|
@@ -258,16 +256,9 @@ python examples/ollama_chat.py # full demo: chat, streaming, tools, OpenAI-
|
|
| 258 |
|
| 259 |
### Local apps
|
| 260 |
|
| 261 |
-
The bundled `Thanatos-27B.Q4_K_M.gguf` is `qwen36`-stamped β every row
|
| 262 |
-
below assumes you've rebadged it to `qwen35` per
|
| 263 |
-
[Architecture](#architecture), or that you're pulling a `qwen35`-stamped
|
| 264 |
-
GGUF from `unsloth/Qwen3.6-27B-GGUF` instead. The "fails today with
|
| 265 |
-
`unknown model architecture: 'qwen36'`" caveat applies to every row
|
| 266 |
-
until that's done.
|
| 267 |
-
|
| 268 |
| App | How to load this model |
|
| 269 |
|---|---|
|
| 270 |
-
| **Ollama** | `ollama run hf.co/FoolDev/Thanatos-27B` (default Q4_K_M). Pulls the GGUF + the root-level `template` / `system` / `params` files in one step (HF's Ollama bridge ingests these three files; it does **not** read `Modelfile`). For other quants,
|
| 271 |
| **LM Studio** | Search β `FoolDev/Thanatos-27B` β pick `Thanatos-27B.Q4_K_M.gguf`. Uses the GGUF's embedded jinja chat template (Qwen 3.6 ChatML); set the system prompt manually from the `SYSTEM` block in this repo's `Modelfile`. |
|
| 272 |
| **Jan** | Hub β "Import from Hugging Face" β `FoolDev/Thanatos-27B`. Same template behavior as LM Studio. |
|
| 273 |
| **llama.cpp** | `hf download FoolDev/Thanatos-27B Thanatos-27B.Q4_K_M.gguf --local-dir .` then `llama-server -m Thanatos-27B.Q4_K_M.gguf` (or `llama-cli`, `llama-mtmd-cli` for vision via the upstream `mmproj-F16.gguf`). |
|
|
|
|
| 122 |
| `template`, `system`, `params` | Used by HF's Ollama bridge when users `ollama run hf.co/FoolDev/Thanatos-27B` directly (the bridge does **not** read `Modelfile` β see [HF Ollama docs](https://huggingface.co/docs/hub/en/ollama)). Mirrors the `Modelfile`'s template / system prompt / sampling params. |
|
| 123 |
| `examples/` | Ready-to-run Python clients for Ollama, Transformers, and llama-cpp-python |
|
| 124 |
| `scripts/build.sh` | Pulls a qwen35-stamped GGUF from `unsloth/Qwen3.6-27B-GGUF` and runs `ollama create` (loads on today's llama.cpp / Ollama; see `make build`) |
|
| 125 |
+
| `scripts/load_bundle.sh` | One-shot path from *this repo's* bundle β loadable local Ollama tag (smudges LFS pointer via `hf download` if needed, runs `ollama create`; see `make load-bundle`). Carries a qwen36 β qwen35 rebadge branch for legacy v0.6.0-era checkouts β no-op on the current qwen35-stamped bundle. |
|
| 126 |
+
| `scripts/heal_hf_pull.sh` | Recovery for users who pulled `hf.co/FoolDev/Thanatos-27B` *before* the qwen35 re-stamp (commits `964e418` / `72259c1`) and still have the qwen36-stamped blob in their local Ollama store: rebadges the blob qwen36 β qwen35 and rewrites the manifest's model-layer digest so the same tag becomes loadable in place. See `make heal-hf`. Idempotent and a no-op on tags already on qwen35 β fresh pulls don't need it. |
|
| 127 |
| `scripts/smoke_test.sh` | Verifies an Ollama daemon + model, runs a round-trip, asserts no chat-template tokens leak into the response. With `TOOLS_TEST=1`, also exercises an end-to-end tool-call round-trip and checks the response shape |
|
| 128 |
| `scripts/bench.sh` | Measures real tok/s using Ollama's `eval_count` / `eval_duration` metadata over a 3-prompt mix (run `make bench`) |
|
| 129 |
| `scripts/fetch_vision.sh` | Pulls the vision projector (`mmproj-F16.gguf`) for llama.cpp (Ollama vision is broken upstream β see [Vision](#vision)). Renamed from `fetch_mmproj.sh` because HF's Ollama bridge auto-indexed the script as a vision projector layer (filename pattern match). |
|
|
|
|
| 136 |
| `README.md` | This file |
|
| 137 |
|
| 138 |
This repo ships a single GGUF to back the HF/Ollama "Use this model"
|
| 139 |
+
widget β `Thanatos-27B.Q4_K_M.gguf` (~17 GB, qwen35-stamped):
|
| 140 |
|
| 141 |
```bash
|
| 142 |
+
ollama run hf.co/FoolDev/Thanatos-27B # 17 GB Q4_K_M, qwen35-stamped, loads on stock Ollama
|
| 143 |
```
|
| 144 |
|
| 145 |
+
For 16 GB GPUs / unified-memory laptops, `make build QUANT=Q3_K_S`
|
| 146 |
+
downloads the smaller ~12 GB Q3_K_S quant from
|
| 147 |
+
`unsloth/Qwen3.6-27B-GGUF` and creates a local `thanatos-27b`
|
|
|
|
|
|
|
| 148 |
Ollama tag. Does not redistribute via this repo.
|
| 149 |
|
| 150 |
For other quants or local builds, pull from
|
|
|
|
| 256 |
|
| 257 |
### Local apps
|
| 258 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 259 |
| App | How to load this model |
|
| 260 |
|---|---|
|
| 261 |
+
| **Ollama** | `ollama run hf.co/FoolDev/Thanatos-27B` (default Q4_K_M). Pulls the GGUF + the root-level `template` / `system` / `params` files in one step (HF's Ollama bridge ingests these three files; it does **not** read `Modelfile`). For other quants, `make build QUANT=Q3_K_S` downloads from unsloth and creates a local Ollama tag using the `Modelfile`, which is kept in sync with the bridge files. |
|
| 262 |
| **LM Studio** | Search β `FoolDev/Thanatos-27B` β pick `Thanatos-27B.Q4_K_M.gguf`. Uses the GGUF's embedded jinja chat template (Qwen 3.6 ChatML); set the system prompt manually from the `SYSTEM` block in this repo's `Modelfile`. |
|
| 263 |
| **Jan** | Hub β "Import from Hugging Face" β `FoolDev/Thanatos-27B`. Same template behavior as LM Studio. |
|
| 264 |
| **llama.cpp** | `hf download FoolDev/Thanatos-27B Thanatos-27B.Q4_K_M.gguf --local-dir .` then `llama-server -m Thanatos-27B.Q4_K_M.gguf` (or `llama-cli`, `llama-mtmd-cli` for vision via the upstream `mmproj-F16.gguf`). |
|
|
@@ -1,27 +1,22 @@
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
-
# Thanatos-27B β heal a
|
| 3 |
-
# is `qwen36`-stamped.
|
|
|
|
| 4 |
#
|
| 5 |
-
#
|
| 6 |
-
#
|
| 7 |
-
#
|
| 8 |
-
#
|
| 9 |
-
# README "Architecture"). `make load-bundle` works around this by
|
| 10 |
-
# building a *separate* local `thanatos-27b` tag from a rebadged copy,
|
| 11 |
-
# but the canonical HF-bridge tag stays broken.
|
| 12 |
#
|
| 13 |
-
#
|
| 14 |
-
#
|
| 15 |
-
#
|
| 16 |
-
#
|
| 17 |
-
# loads.
|
| 18 |
#
|
| 19 |
# Idempotent: a tag already on qwen35 / qwen35moe is left untouched.
|
| 20 |
-
#
|
| 21 |
-
#
|
| 22 |
-
#
|
| 23 |
-
# Once upstream adds the qwen36 arch entry this script (and the
|
| 24 |
-
# whole rebadge dance) can be deleted; the bundle works as-is.
|
| 25 |
#
|
| 26 |
# Usage:
|
| 27 |
# ./scripts/heal_hf_pull.sh # default tag
|
|
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
+
# Thanatos-27B β heal a previously pulled HF-bridge tag whose bundled
|
| 3 |
+
# GGUF is `qwen36`-stamped (legacy v0.6.0-era pulls, before the
|
| 4 |
+
# qwen35 re-stamp in 964e418 / 72259c1).
|
| 5 |
#
|
| 6 |
+
# Fresh pulls of `ollama run hf.co/FoolDev/Thanatos-27B` now get the
|
| 7 |
+
# qwen35-stamped bundle and load directly β this script is the
|
| 8 |
+
# recovery path for users who pulled the qwen36-stamped blob into
|
| 9 |
+
# their local Ollama store and haven't refreshed since.
|
|
|
|
|
|
|
|
|
|
| 10 |
#
|
| 11 |
+
# It rebadges the HF-bridge tag's model blob in-place (qwen36 ->
|
| 12 |
+
# qwen35, metadata-only, byte-identical tensors) and rewrites the
|
| 13 |
+
# manifest's model-layer digest to point at the new blob. After
|
| 14 |
+
# running, the cached `hf.co/FoolDev/Thanatos-27B` tag loads.
|
|
|
|
| 15 |
#
|
| 16 |
# Idempotent: a tag already on qwen35 / qwen35moe is left untouched.
|
| 17 |
+
# The current bundle is qwen35-stamped so this script is a no-op for
|
| 18 |
+
# anyone who pulled after the re-stamp; it stays in the repo for the
|
| 19 |
+
# legacy recovery case.
|
|
|
|
|
|
|
| 20 |
#
|
| 21 |
# Usage:
|
| 22 |
# ./scripts/heal_hf_pull.sh # default tag
|
|
@@ -1,24 +1,23 @@
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
-
# Thanatos-27B β load
|
| 3 |
#
|
| 4 |
-
# The bundled GGUF (Thanatos-27B.Q4_K_M.gguf)
|
| 5 |
-
#
|
| 6 |
-
#
|
| 7 |
-
#
|
| 8 |
-
# `unknown model architecture: 'qwen36'`. This script handles the
|
| 9 |
-
# rebadge workaround end-to-end so users don't have to copy-paste from
|
| 10 |
-
# the README:
|
| 11 |
#
|
| 12 |
# 1. Resolve the bundle. If it's an LFS pointer (cloned without
|
| 13 |
# `git lfs pull`), download the real ~17 GB blob via `hf download`.
|
| 14 |
-
# 2. Inspect `general.architecture`. If
|
| 15 |
-
#
|
| 16 |
-
#
|
| 17 |
-
#
|
|
|
|
|
|
|
| 18 |
#
|
| 19 |
-
#
|
| 20 |
-
#
|
| 21 |
-
#
|
| 22 |
#
|
| 23 |
# Usage:
|
| 24 |
# ./scripts/load_bundle.sh # default tag: thanatos-27b
|
|
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
+
# Thanatos-27B β load this repo's bundle into Ollama as a local tag.
|
| 3 |
#
|
| 4 |
+
# The bundled GGUF (Thanatos-27B.Q4_K_M.gguf) is qwen35-stamped and
|
| 5 |
+
# loads directly on stock llama.cpp / Ollama. This script is the
|
| 6 |
+
# one-shot path from "I just cloned this repo" to "I have a working
|
| 7 |
+
# local Ollama tag":
|
|
|
|
|
|
|
|
|
|
| 8 |
#
|
| 9 |
# 1. Resolve the bundle. If it's an LFS pointer (cloned without
|
| 10 |
# `git lfs pull`), download the real ~17 GB blob via `hf download`.
|
| 11 |
+
# 2. Inspect `general.architecture`. If qwen35 / qwen35moe (current
|
| 12 |
+
# bundle), skip straight to step 3. If qwen36 (legacy v0.6.0-era
|
| 13 |
+
# checkout), rebadge to qwen35 via scripts/rename_arch.py
|
| 14 |
+
# (metadata-only, byte-identical tensors).
|
| 15 |
+
# 3. Run `ollama create <tag> -f <temp Modelfile pointing at the
|
| 16 |
+
# resolved bundle>`.
|
| 17 |
#
|
| 18 |
+
# Useful if you want a bare local tag (`thanatos-27b`) rather than
|
| 19 |
+
# the `hf.co/FoolDev/Thanatos-27B` path. The legacy qwen36 rebadge
|
| 20 |
+
# branch is kept for anyone working from a pre-964e418 checkout.
|
| 21 |
#
|
| 22 |
# Usage:
|
| 23 |
# ./scripts/load_bundle.sh # default tag: thanatos-27b
|