# Thanatos-27B — Ollama wrapper around Qwen 3.6 27B (dense) # # Text + tool calling. Vision via Ollama is currently broken for this # architecture (ollama/ollama#15898 — the qwen35 arch entries are in # Ollama's Go text engine but missing from the C++ llama.cpp fallback # Ollama uses when an mmproj is attached). Use llama.cpp directly for # image input, or wait for the fix. See the Vision section in README.md. # # This repo bundles a single GGUF: Thanatos-27B.Q4_K_M.gguf (~17 GB), # stamped `general.architecture: 'qwen35'` — the upstream-canonical # arch entry every released llama.cpp / Ollama loads under for the # Qwen 3.5 / 3.6 hybrid SSM + attention family. `ollama create # thanatos-27b -f Modelfile && ollama run thanatos-27b` loads it # directly. See README "Architecture" for the full stamp history # (eight flips between qwen35 and qwen36, settled on qwen35 at # `e03e10e` after the 4th qwen36 round trip had its friction # re-tested in a fresh next-day session). # # For other quants (Q3_K_S, Q5_K_M, Q6_K, etc.), `make build QUANT=Q3_K_S` # downloads the chosen quant from unsloth/Qwen3.6-27B-GGUF and patches # FROM in a temp Modelfile copy. The Q3_K_S used to ship in this repo; # it was removed so HF's Ollama bridge picks Q4_K_M as the default # `:latest` tag instead of Q3_K_S (alphabetically-first heuristic). # # Other GGUF sources (use with `make build GGUF_PATH=...`): # https://huggingface.co/unsloth/Qwen3.6-27B-GGUF # https://huggingface.co/rico03/Qwen3.6-27B-Claude-Opus-Reasoning-Distilled-GGUF FROM ./Thanatos-27B.Q4_K_M.gguf # Chat template — Qwen 3.6 ChatML in Ollama Go-template form, with the # tool-calling blocks Ollama's capability detector looks for. Without a # TEMPLATE that references .Tools and .ToolCalls, /api/chat and # /v1/chat/completions reject any request carrying a `tools` array with # ` does not support tools`. Same template as the 35B sibling — # both share the Qwen 3.6 chat format. TEMPLATE """{{- $lastUserIdx := -1 -}} {{- range $idx, $msg := .Messages -}} {{- if eq $msg.Role "user" }}{{ $lastUserIdx = $idx }}{{ end -}} {{- end }} {{- if or .System .Tools }}<|im_start|>system {{ if .System }}{{ .System }} {{ end }} {{- if .Tools }}# Tools You may call one or more functions to assist with the user query. You are provided with function signatures within XML tags: {{- range .Tools }} {"type": "function", "function": {{ .Function }}} {{- end }} For each function call, return a json object with function name and arguments within XML tags: {"name": , "arguments": } {{- end -}}<|im_end|> {{ end }} {{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1 -}} {{- if eq .Role "user" }}<|im_start|>user {{ .Content }}<|im_end|> {{ else if eq .Role "assistant" }}<|im_start|>assistant {{ if (and $.IsThinkSet (and .Thinking (or $last (gt $i $lastUserIdx)))) -}} {{ .Thinking }} {{ end -}} {{ if .Content }}{{ .Content }}{{ end }} {{- if .ToolCalls }} {{- range .ToolCalls }} {"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}} {{- end }} {{- end }}{{ if not $last }}<|im_end|> {{ end }} {{- else if eq .Role "tool" }}<|im_start|>user {{ .Content }} <|im_end|> {{ end }} {{- if and (ne .Role "assistant") $last }}<|im_start|>assistant {{ end }} {{- end }}""" # Sampling tuned for reasoning + general use. See README "Recommended sampling" # for creative/RP alternatives. PARAMETER temperature 0.6 PARAMETER top_p 0.95 PARAMETER top_k 20 PARAMETER repeat_penalty 1.05 PARAMETER num_ctx 16384 # Stop tokens. Without these, Ollama only honors <|im_end|> from the GGUF # metadata; the model occasionally emits <|endoftext|> instead and Ollama # keeps generating past it (synthesising a fake new user turn). Listing # both — plus <|im_start|> as a belt-and-braces guard against the same # loop — keeps responses cleanly terminated. PARAMETER stop "<|im_end|>" PARAMETER stop "<|endoftext|>" PARAMETER stop "<|im_start|>" SYSTEM """You are Thanatos, a precise and capable assistant for reasoning, writing, coding, and long-form dialogue. Behavior rules: - Answer the user's actual request directly. - Be accurate, complete, and structured. - Think before answering, but do not get stuck in repetitive loops or meta-commentary. - If the request is ambiguous or incomplete, state what is missing and make the smallest reasonable assumption needed to continue. - If the user wants creative writing, preserve tone, continuity, and character consistency. - If the user wants analysis or technical help, prefer concrete steps, examples, and decisions over fluff. - Finish with a usable answer, not just planning.""" # Hardware notes # -------------- # Qwen 3.6 27B is *dense* — every parameter participates in every forward pass. # Q4_K_M GGUF is ~17 GB. Practical footprint: # weights mmap ~17 GB # compute graph alloc ~12 GB (smaller than 35B-A3B because dense ≠ MoE) # KV cache @ 16K ctx ~1 GB (with OLLAMA_KV_CACHE_TYPE=q8_0) # total minimum ~30 GB # # Working configurations: # ✓ RTX 3090 / 4090 24 GB — full Q4 offload, ~25-40 tok/s # ✓ RTX 5090 32 GB — full offload at Q5/Q6 quant # ✓ Mac Studio M2/M3 32 GB+ unified — ~15-25 tok/s # ✓ Linux box with 32 GB+ RAM (CPU-only) — ~1-3 tok/s # ⚠ 32 GB unified-memory laptops — borderline at Q4, try # `make build QUANT=Q3_K_S` # (~12 GB) and trim num_ctx # # Measured data points (ASUS ROG Flow Z13 GZ302EA, Ryzen AI Max+ 395 + # Radeon 8060S iGPU, 32 GB unified, gfx1151, OLLAMA_FLASH_ATTENTION=1, # OLLAMA_KV_CACHE_TYPE=q8_0, num_ctx 16384, 3-prompt mix): # Vulkan (OLLAMA_VULKAN=1): # Q3_K_S → 12.31 tok/s aggregate (run 1) # (6182 tokens / 501.9 s; 12.67 / 12.55 / 12.25 short/medium/long) # Q3_K_S → 11.70 tok/s aggregate (run 2, 2026-05-19 evening) # (8009 tokens / 684.0 s; 12.23 / 12.12 / 11.66 short/medium/long) # Second run measured against a `thanatos-27b:latest` (pre-rename) # built via `make build QUANT=Q3_K_S` against the then-current # unsloth/Qwen3.6-27B-GGUF source. Aggregate is 4.9% below # run 1 (within the ±20% noise band) — slightly longer # per-prompt outputs this run (8009 vs 6182 tokens) likely # contribute the difference, plus late-in-session thermal # pressure on the Strix Halo iGPU. # (Heretic v2 base is not benched here yet; rebundle pending.) # Q4_K_M → 9.31 tok/s aggregate (run 1) # (5356 tokens / 574.9 s; 9.48 / 9.43 / 9.28 short/medium/long) # Q4_K_M → 9.19 tok/s aggregate (run 2, 2026-05-19 afternoon) # (6210 tokens / 675.6 s; 9.40 / 9.29 / 9.16 short/medium/long) # Second run measured against the qwen36-stamped HF-bridge tag # after `make heal-hf` rebadged it to qwen35 in store — confirms # the in-place heal produces a model with the same performance # profile as `make load-bundle`. Aggregate is 1.3% below run 1 # (within the ±20% noise band the README hardware section # warns about). # Q4_K_M → 9.32 tok/s aggregate (run 3, 2026-05-19 evening) # (4592 tokens / 492.7 s; 9.49 / 9.44 / 9.28 short/medium/long) # Third run, also against a heal-hf-rebadged qwen36-stamped # HF-bridge tag — this time the 3rd-round-trip bundle from # commit 973d7ef. Aggregate is within 0.1% of run 1's 9.31, # confirming the latest qwen36 -> qwen35 heal yields the same # performance profile as the prior two runs (no regression # from the third stamp flip). # ROCm (older snapshot, kept for backend comparison): # Q3_K_S → 10.14 tok/s aggregate # (8080 tokens / 796.5 s; 10.37 / 10.31 / 10.11 short/medium/long)