Instructions to use FoolDev/Thanatos-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FoolDev/Thanatos-27B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="FoolDev/Thanatos-27B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("FoolDev/Thanatos-27B", dtype="auto")

llama-cpp-python

How to use FoolDev/Thanatos-27B with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="FoolDev/Thanatos-27B",
	filename="Thanatos-27B.Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": [
				{
					"type": "text",
					"text": "Describe this image in one sentence."
				},
				{
					"type": "image_url",
					"image_url": {
						"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
					}
				}
			]
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use FoolDev/Thanatos-27B with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M

Use Docker

docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M

LM Studio
Jan

vLLM

How to use FoolDev/Thanatos-27B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FoolDev/Thanatos-27B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FoolDev/Thanatos-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M

SGLang

How to use FoolDev/Thanatos-27B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FoolDev/Thanatos-27B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FoolDev/Thanatos-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FoolDev/Thanatos-27B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FoolDev/Thanatos-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Ollama
How to use FoolDev/Thanatos-27B with Ollama:
```
ollama run hf.co/FoolDev/Thanatos-27B:Q4_K_M
```

Unsloth Studio new

How to use FoolDev/Thanatos-27B with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FoolDev/Thanatos-27B to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FoolDev/Thanatos-27B to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for FoolDev/Thanatos-27B to start chatting

Pi new

How to use FoolDev/Thanatos-27B with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "FoolDev/Thanatos-27B:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use FoolDev/Thanatos-27B with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf FoolDev/Thanatos-27B:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default FoolDev/Thanatos-27B:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use FoolDev/Thanatos-27B with Docker Model Runner:
```
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
```

Lemonade

How to use FoolDev/Thanatos-27B with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull FoolDev/Thanatos-27B:Q4_K_M

Run and chat with the model

lemonade run user.Thanatos-27B-Q4_K_M

List all available models

lemonade list

FoolDev commited on 26 days ago

Commit

75bbdfe

1 Parent(s): c9ce901

Remove bundled Q3_K_S so HF picks Q4_K_M as the default Ollama tag

Browse files

Resolution to the recurring 'Q4 to widget' issue: with Q3_K_S removed,
Janus-27B.Q4_K_M.gguf is the only GGUF in the repo. HF's Ollama bridge
now defaults to it unambiguously — bare 'ollama run hf.co/FoolDev/janus-27b'
resolves to the 17 GB Q4_K_M blob instead of the 12 GB Q3_K_S blob.

Other approaches we tried first:
- Documenting ':Q4_K_M' explicit tag (kept working but didn't fix the widget)
- Renaming Janus-27B.Q3_K_S.gguf -> Janus-27B.q3_k_s.gguf (broke HF GGUF
detection entirely; reverted in commit 384e186)
- Looking for HF metadata to override default-quant pick (none documented)

Removing Q3_K_S keeps the smaller-footprint workflow available — users
on 16 GB GPUs / 32 GB unified-memory laptops run 'make build QUANT=Q3_K_S'
which downloads Qwen3.6-27B-Q3_K_S.gguf from unsloth/Qwen3.6-27B-GGUF
and creates a local Ollama tag using the bundled Modelfile (kept in
sync with the bridge files, verified by check_bridge_sync.py).

Tradeoff: 12 GB less data shipped via the repo; Q3_K_S users do an
extra ~3 minute download; default Ollama tag now matches what the
README's been recommending all along.

Updated:
- Modelfile: comment block now says 'single bundled GGUF'
- README: TL;DR / Architecture table / What's here / Quick start /
Local apps / Hardware sections all reflect Q4_K_M-only bundling
- examples/README.md + examples/ollama_chat.py: updated Ollama snippets
- CITATION.cff: abstract mentions one bundled GGUF, not two
- CHANGELOG: full forensic entry under [Unreleased] / Removed

Files changed (7) hide show

CHANGELOG.md +19 -0
CITATION.cff +6 -5
Janus-27B.Q3_K_S.gguf +0 -3
Modelfile +10 -7
README.md +24 -32
examples/README.md +4 -7
examples/ollama_chat.py +3 -4

CHANGELOG.md CHANGED Viewed

@@ -7,6 +7,25 @@ and documentation**, not the underlying base model.
 ## [Unreleased]
 ### Investigated (and reverted)
 - Tried renaming `Janus-27B.Q3_K_S.gguf` to `Janus-27B.q3_k_s.gguf`
   (lowercase quant suffix) hoping to flip HF's default-`:latest`-tag

 ## [Unreleased]
+### Removed
+- `Janus-27B.Q3_K_S.gguf` no longer redistributed in this repo.
+  Removing it leaves `Janus-27B.Q4_K_M.gguf` as the only GGUF, which
+  flips HF's Ollama bridge default-tag pick from Q3_K_S to Q4_K_M
+  unambiguously: `ollama run hf.co/FoolDev/janus-27b` now resolves
+  to the 17 GB Q4_K_M blob instead of the 12 GB Q3_K_S blob. This
+  is the resolution to the long-running "Q4 to widget" thread —
+  rename hacks broke HF's GGUF detection (see earlier "Investigated
+  (and reverted)" entry), and HF doesn't expose a documented
+  default-quant metadata field, so the only reliable fix was to
+  remove the conflicting file. Q3_K_S users build it locally with
+  `make build QUANT=Q3_K_S`, which downloads
+  `Qwen3.6-27B-Q3_K_S.gguf` from `unsloth/Qwen3.6-27B-GGUF` and
+  patches the Modelfile FROM line into a temp copy. Tradeoff: 12 GB
+  less data shipped through the repo, smaller-footprint users do an
+  extra ~3 minute download, default Ollama tag now matches the
+  README's recommendation. Updated Modelfile / README / examples /
+  CITATION.cff to reflect Q4_K_M as the only bundled quant.
 ### Investigated (and reverted)
 - Tried renaming `Janus-27B.Q3_K_S.gguf` to `Janus-27B.q3_k_s.gguf`
   (lowercase quant suffix) hoping to flip HF's default-`:latest`-tag

CITATION.cff CHANGED Viewed

@@ -10,11 +10,12 @@ url: "https://huggingface.co/FoolDev/janus-27b"
 abstract: >-
   Janus-27B is a personal repackaging of the dense Qwen 3.6 27B base model
   with Claude Opus 4.7 in the reasoning teacher slot. The repository ships
-  an Ollama Modelfile, sampling defaults, usage examples, and two
-  ready-to-run GGUFs (Q4_K_M ~17 GB and Q3_K_S ~12 GB) so the HF "Use
-  this model" widget surfaces a one-liner Ollama snippet. Other quants
-  and the upstream safetensors (Qwen/Qwen3.6-27B) are pulled from
-  upstream on demand rather than redistributed.
 keywords:
   - qwen
   - qwen3.6

 abstract: >-
   Janus-27B is a personal repackaging of the dense Qwen 3.6 27B base model
   with Claude Opus 4.7 in the reasoning teacher slot. The repository ships
+  an Ollama Modelfile, sampling defaults, usage examples, and a single
+  ready-to-run GGUF (Q4_K_M ~17 GB) so the HF "Use this model" widget
+  surfaces a one-liner Ollama snippet. Other quants (Q3_K_S, Q5_K_M,
+  Q6_K, etc.) and the upstream safetensors (Qwen/Qwen3.6-27B) are
+  pulled from upstream (unsloth/Qwen3.6-27B-GGUF) on demand rather
+  than redistributed.
 keywords:
   - qwen
   - qwen3.6

Janus-27B.Q3_K_S.gguf DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:4afb4abcf0207a484b0d7e92c0421b74e8ce1c7a7250bb9d824b79288da68f20
-size 12358727904

Modelfile CHANGED Viewed

@@ -5,15 +5,17 @@
 # missing the qwen35 arch entries). Use llama.cpp directly for image
 # input, or wait for the fix. See the Vision section in README.md.
 #
-# This repo bundles two GGUFs: Janus-27B.Q4_K_M.gguf (~17 GB, default)
-# and Janus-27B.Q3_K_S.gguf (~12 GB, smaller-footprint option). The FROM
-# line below points at the bundled Q4_K_M, so a fresh clone (with LFS
-# smudge enabled) supports the no-script path:
 #
 #     ollama create janus-27b -f Modelfile && ollama run janus-27b
 #
-# To use the smaller quant, edit FROM to ./Janus-27B.Q3_K_S.gguf, or run
-# `make build QUANT=Q3_K_S` which patches FROM in a temp Modelfile copy.
 #
 # Other GGUF sources (use with `make build GGUF_PATH=...`):
 #     https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
@@ -121,7 +123,8 @@ Behavior rules:
 #   ✓ RTX 5090 32 GB                            — full offload at Q5/Q6 quant
 #   ✓ Mac Studio M2/M3 32 GB+ unified           — ~15-25 tok/s
 #   ✓ Linux box with 32 GB+ RAM (CPU-only)      — ~1-3 tok/s
-#   ⚠ 32 GB unified-memory laptops              — borderline at Q4, try Q3_K_S
 #                                                 (~12 GB) and trim num_ctx
 #
 # Measured data point (ASUS ROG Flow Z13 GZ302EA, Ryzen AI Max+ 395 +

 # missing the qwen35 arch entries). Use llama.cpp directly for image
 # input, or wait for the fix. See the Vision section in README.md.
 #
+# This repo bundles a single GGUF: Janus-27B.Q4_K_M.gguf (~17 GB).
+# The FROM line below points at it, so a fresh clone (with LFS smudge
+# enabled) supports the no-script path:
 #
 #     ollama create janus-27b -f Modelfile && ollama run janus-27b
 #
+# For other quants (Q3_K_S, Q5_K_M, Q6_K, etc.), `make build QUANT=Q3_K_S`
+# downloads the chosen quant from unsloth/Qwen3.6-27B-GGUF and patches
+# FROM in a temp Modelfile copy. The Q3_K_S used to ship in this repo;
+# it was removed so HF's Ollama bridge picks Q4_K_M as the default
+# `:latest` tag instead of Q3_K_S (alphabetically-first heuristic).
 #
 # Other GGUF sources (use with `make build GGUF_PATH=...`):
 #     https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
 #   ✓ RTX 5090 32 GB                            — full offload at Q5/Q6 quant
 #   ✓ Mac Studio M2/M3 32 GB+ unified           — ~15-25 tok/s
 #   ✓ Linux box with 32 GB+ RAM (CPU-only)      — ~1-3 tok/s
+#   ⚠ 32 GB unified-memory laptops              — borderline at Q4, try
+#                                                 `make build QUANT=Q3_K_S`
 #                                                 (~12 GB) and trim num_ctx
 #
 # Measured data point (ASUS ROG Flow Z13 GZ302EA, Ryzen AI Max+ 395 +

README.md CHANGED Viewed

@@ -68,15 +68,12 @@ template — HF's Ollama bridge ingests those three files, not
 `Modelfile`):
 ```bash
-ollama run hf.co/FoolDev/janus-27b:Q4_K_M    # ~17 GB, recommended
-ollama run hf.co/FoolDev/janus-27b:Q3_K_S    # ~12 GB, smaller-footprint
 ```
-> **Use the explicit `:Q4_K_M` tag.** HF's Ollama bridge currently picks
-> `Janus-27B.Q3_K_S.gguf` (the alphabetically-first GGUF in the repo) as
-> the default for the bare `ollama run hf.co/FoolDev/janus-27b` form,
-> *not* Q4_K_M. If you don't pin a quant tag explicitly you'll get the
-> 12 GB Q3_K_S blob.
 Or build locally (uses this repo's `Modelfile`, kept in sync with the
 three bridge files) for any quant:
@@ -104,8 +101,8 @@ The 27B is **dense**: every parameter participates in every forward pass. It's s
 | Active params per token | 27 B | ~3 B |
 | Layers | 64 | 40 |
 | Hidden size | 5120 | 2048 |
-| Q4_K_M GGUF size | ~17 GB | ~19 GB |
-| Q3_K_S GGUF size | ~12 GB | n/a |
 | Min host memory @ Q4 / 8K ctx | ~22 GB | ~38 GB |
 | Multimodal (text path) | Yes | Yes |
 | Multimodal (vision via Ollama) | Broken upstream — see below | Broken upstream |
@@ -133,19 +130,16 @@ The 27B is **dense**: every parameter participates in every forward pass. It's s
 | `README.md` | This file |
 This repo ships two GGUFs to back the HF/Ollama "Use this model"
-widget — `Janus-27B.Q4_K_M.gguf` (~17 GB, recommended) and
-`Janus-27B.Q3_K_S.gguf` (~12 GB, smaller-footprint option for 16 GB
-GPUs / unified-memory laptops):
 ```bash
-ollama run hf.co/FoolDev/janus-27b:Q4_K_M    # 17 GB, recommended
-ollama run hf.co/FoolDev/janus-27b:Q3_K_S    # 12 GB, smaller-footprint
 ```
-The bare `ollama run hf.co/FoolDev/janus-27b` form (no quant tag)
-currently resolves to the alphabetically-first GGUF in the repo —
-`Janus-27B.Q3_K_S.gguf` — *not* Q4_K_M. Pin the tag explicitly to get
-the recommended Q4_K_M quant.
 For other quants or local builds, pull from
 [`unsloth/Qwen3.6-27B-GGUF`](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF)
@@ -177,16 +171,14 @@ If you want the safetensors for `transformers`, fetch them from [`Qwen/Qwen3.6-2
 Two paths:
 ```bash
-# A. Pull straight from HF (uses the bundled GGUF + root-level
-#    template / system / params files; pin a quant tag explicitly):
-ollama run hf.co/FoolDev/janus-27b:Q4_K_M    # 17 GB, recommended
-ollama run hf.co/FoolDev/janus-27b:Q3_K_S    # 12 GB, smaller-footprint
-# (Bare `ollama run hf.co/FoolDev/janus-27b` resolves to Q3_K_S, not
-#  Q4_K_M — HF picks the alphabetically-first GGUF when no tag pinned.)
-# B. Build locally (lets you pick the quant):
 make build                                              # Q4_K_M  -> janus-27b
-make build QUANT=Q3_K_S                                 # smaller quant
 make build GGUF_PATH=~/models/Qwen3.6-27B-Q4_K_M.gguf   # skip download
 ollama run janus-27b
 ```
@@ -209,13 +201,13 @@ python examples/ollama_chat.py      # full demo: chat, streaming, tools, OpenAI-
 ### Local apps
-Both bundled GGUFs (Q4_K_M and Q3_K_S) work in any GGUF-compatible
-local app — point it at this repo and pick a quant.
 | App | How to load this model |
 |---|---|
-| **Ollama** | `ollama run hf.co/FoolDev/janus-27b:Q4_K_M` (or `:Q3_K_S`). Pulls the GGUF + the root-level `template` / `system` / `params` files in one step (HF's Ollama bridge ingests these three files; it does **not** read `Modelfile`). For local builds, `make build` uses `Modelfile`, which is kept in sync. **Pin the quant tag** — bare `hf.co/FoolDev/janus-27b` resolves to Q3_K_S, not Q4_K_M. |
-| **LM Studio** | Search → `FoolDev/janus-27b` → pick `Janus-27B.Q4_K_M.gguf` or `Janus-27B.Q3_K_S.gguf`. Uses the GGUF's embedded jinja chat template (Qwen 3.6 ChatML); set the system prompt manually from the `SYSTEM` block in this repo's `Modelfile`. |
 | **Jan** | Hub → "Import from Hugging Face" → `FoolDev/janus-27b`. Same template behavior as LM Studio. |
 | **llama.cpp** | `hf download FoolDev/janus-27b Janus-27B.Q4_K_M.gguf --local-dir .` then `llama-server -m Janus-27B.Q4_K_M.gguf` (or `llama-cli`, `llama-mtmd-cli` for vision via the upstream `mmproj-F16.gguf`). |
 | **llama-cpp-python** | See `examples/llama_cpp_quickstart.py` (text) and `examples/llama_cpp_vision.py` (image input). |
@@ -325,7 +317,7 @@ The dense 27B is the easier of the two Janus models to deploy.
 | RTX 3090 / 4090 24 GB | Works, full Q4 offload, ~25-40 tok/s |
 | RTX 5090 32 GB | Works, full offload at higher quant (Q5/Q6), ~30-50 tok/s |
 | Mac Studio M2/M3 32 GB+ unified | Works, ~15-25 tok/s |
-| 32 GB unified-memory laptops (Mac M-series, Ryzen AI Max+, etc.) | Borderline at Q4. Drop to Q3_K_S (~12 GB) and trim `num_ctx` for headroom. |
 Most numbers in this table are estimates from comparable models; the
 gradient is right but the absolute values will move ±20% with prompt

 `Modelfile`):
 ```bash
+ollama run hf.co/FoolDev/janus-27b           # ~17 GB Q4_K_M (the only bundled quant)
 ```
+For other quants (Q3_K_S ~12 GB, Q5_K_M ~20 GB, etc.), `make build
+QUANT=Q3_K_S` downloads from `unsloth/Qwen3.6-27B-GGUF` and creates the
+local Ollama tag. See [Quick start](#quick-start) below.
 Or build locally (uses this repo's `Modelfile`, kept in sync with the
 three bridge files) for any quant:
 | Active params per token | 27 B | ~3 B |
 | Layers | 64 | 40 |
 | Hidden size | 5120 | 2048 |
+| Q4_K_M GGUF size | ~17 GB (bundled) | ~19 GB (bundled) |
+| Q3_K_S GGUF size | ~12 GB (build locally via `make build QUANT=Q3_K_S`) | n/a |
 | Min host memory @ Q4 / 8K ctx | ~22 GB | ~38 GB |
 | Multimodal (text path) | Yes | Yes |
 | Multimodal (vision via Ollama) | Broken upstream — see below | Broken upstream |
 | `README.md` | This file |
 This repo ships two GGUFs to back the HF/Ollama "Use this model"
+widget — `Janus-27B.Q4_K_M.gguf` (~17 GB):
 ```bash
+ollama run hf.co/FoolDev/janus-27b           # 17 GB Q4_K_M (only bundled quant)
 ```
+For 16 GB GPUs / unified-memory laptops, `make build QUANT=Q3_K_S`
+downloads the smaller ~12 GB Q3_K_S quant from `unsloth/Qwen3.6-27B-GGUF`
+and creates a local `janus-27b` Ollama tag (does not redistribute via
+this repo).
 For other quants or local builds, pull from
 [`unsloth/Qwen3.6-27B-GGUF`](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF)
 Two paths:
 ```bash
+# A. Pull straight from HF (uses the bundled Q4_K_M + root-level
+#    template / system / params files):
+ollama run hf.co/FoolDev/janus-27b           # 17 GB Q4_K_M (only bundled quant)
+# B. Build locally for a different quant (downloads from unsloth):
 make build                                              # Q4_K_M  -> janus-27b
+make build QUANT=Q3_K_S                                 # 12 GB smaller quant
+make build QUANT=Q5_K_M                                 # 20 GB higher quality
 make build GGUF_PATH=~/models/Qwen3.6-27B-Q4_K_M.gguf   # skip download
 ollama run janus-27b
 ```
 ### Local apps
+The bundled `Janus-27B.Q4_K_M.gguf` works in any GGUF-compatible local
+app — point it at this repo and load.
 | App | How to load this model |
 |---|---|
+| **Ollama** | `ollama run hf.co/FoolDev/janus-27b` (default Q4_K_M). Pulls the GGUF + the root-level `template` / `system` / `params` files in one step (HF's Ollama bridge ingests these three files; it does **not** read `Modelfile`). For other quants, `make build QUANT=Q3_K_S` downloads from unsloth and creates a local Ollama tag using the `Modelfile`, which is kept in sync with the bridge files. |
+| **LM Studio** | Search → `FoolDev/janus-27b` → pick `Janus-27B.Q4_K_M.gguf`. Uses the GGUF's embedded jinja chat template (Qwen 3.6 ChatML); set the system prompt manually from the `SYSTEM` block in this repo's `Modelfile`. |
 | **Jan** | Hub → "Import from Hugging Face" → `FoolDev/janus-27b`. Same template behavior as LM Studio. |
 | **llama.cpp** | `hf download FoolDev/janus-27b Janus-27B.Q4_K_M.gguf --local-dir .` then `llama-server -m Janus-27B.Q4_K_M.gguf` (or `llama-cli`, `llama-mtmd-cli` for vision via the upstream `mmproj-F16.gguf`). |
 | **llama-cpp-python** | See `examples/llama_cpp_quickstart.py` (text) and `examples/llama_cpp_vision.py` (image input). |
 | RTX 3090 / 4090 24 GB | Works, full Q4 offload, ~25-40 tok/s |
 | RTX 5090 32 GB | Works, full offload at higher quant (Q5/Q6), ~30-50 tok/s |
 | Mac Studio M2/M3 32 GB+ unified | Works, ~15-25 tok/s |
+| 32 GB unified-memory laptops (Mac M-series, Ryzen AI Max+, etc.) | Borderline at Q4. `make build QUANT=Q3_K_S` (~12 GB) and trim `num_ctx` for headroom. |
 Most numbers in this table are estimates from comparable models; the
 gradient is right but the absolute values will move ±20% with prompt

examples/README.md CHANGED Viewed

@@ -21,16 +21,13 @@ Easiest path — pull straight from HF (gets the bundled Q4_K_M GGUF +
 this repo's Modelfile in one step):
 ```bash
-ollama pull hf.co/FoolDev/janus-27b:Q4_K_M    # 17 GB, recommended
-# or:
-ollama pull hf.co/FoolDev/janus-27b:Q3_K_S    # 12 GB, smaller-footprint
 pip install requests
-MODEL=hf.co/FoolDev/janus-27b:Q4_K_M python ollama_chat.py
 ```
-> Pin the quant tag explicitly. Bare `ollama pull hf.co/FoolDev/janus-27b`
-> resolves to Q3_K_S (HF picks the alphabetically-first GGUF in the
-> repo), not Q4_K_M.
 Or build locally from this repo (uses the bundled `Janus-27B.Q4_K_M.gguf`,
 no edits required):

 this repo's Modelfile in one step):
 ```bash
+ollama pull hf.co/FoolDev/janus-27b           # 17 GB Q4_K_M (only bundled quant)
 pip install requests
+MODEL=hf.co/FoolDev/janus-27b python ollama_chat.py
 ```
+For the smaller-footprint Q3_K_S (~12 GB) or other quants, build
+locally instead — see the parent repo's `make build QUANT=...` flow.
 Or build locally from this repo (uses the bundled `Janus-27B.Q4_K_M.gguf`,
 no edits required):

examples/ollama_chat.py CHANGED Viewed

@@ -9,10 +9,9 @@ Prerequisites (pick one):
         # or:
         $ ollama create janus-27b -f ../Modelfile
-    B. Pull straight from HF (pin the quant tag explicitly — bare
-       `hf.co/FoolDev/janus-27b` resolves to Q3_K_S, not Q4_K_M):
-        $ ollama run hf.co/FoolDev/janus-27b:Q4_K_M
-        # then set MODEL=hf.co/FoolDev/janus-27b:Q4_K_M below
 Then:
     $ ollama serve         # usually already running

         # or:
         $ ollama create janus-27b -f ../Modelfile
+    B. Pull straight from HF (Q4_K_M is the only bundled quant):
+        $ ollama run hf.co/FoolDev/janus-27b
+        # then set MODEL=hf.co/FoolDev/janus-27b below
 Then:
     $ ollama serve         # usually already running