examples/README.md

# Thanatos-27B examples

Four minimal entry points. Pick the one that matches how you run models.

| File | Backend | When to use |
|---|---|---|
| `ollama_chat.py` | Ollama HTTP API | You already have `ollama serve` running and the `thanatos-27b` model created from the project `Modelfile`. **Text + tool calling** — vision via Ollama is broken upstream for this arch. |
| `transformers_quickstart.py` | Hugging Face Transformers | You want to run the upstream safetensors (`Qwen/Qwen3.6-27B`) on GPU, optionally in 4-bit via bitsandbytes. |
| `llama_cpp_quickstart.py` | llama-cpp-python | You want to invoke a local GGUF directly without a daemon (CI, batch jobs, scripts). Text only. |
| `llama_cpp_vision.py` | llama-cpp-python + mmproj | **Image input.** Loads a text GGUF + `mmproj-F16.gguf` and answers questions about an image. The only working vision path right now. |

All four apply the same Thanatos system prompt and sampling defaults
(`temp=0.6, top_p=0.95, top_k=20, repeat_penalty=1.05`) so behavior should
be consistent across backends modulo quantization noise. The three
non-Ollama scripts set them explicitly; `ollama_chat.py` inherits them
from the `Modelfile` / bridge files.

## Setup

### Ollama

Pull straight from HF (gets the bundled Q4_K_M GGUF + this repo's
root-level `template` / `system` / `params` files via HF's Ollama
bridge):

```bash
ollama pull hf.co/FoolDev/Thanatos-27B           # 17 GB Q4_K_M (only bundled quant)
pip install requests
MODEL=hf.co/FoolDev/Thanatos-27B python ollama_chat.py
```

If you pulled before the latest qwen35 re-stamp (HF commit
`e03e10e`) and still have a qwen36-stamped blob in your local
Ollama store, run `cd .. && make heal-hf` once to rebadge it
in place (qwen36 → qwen35, metadata-only, ~5 s) — the same
tag then loads. Fresh pulls after the re-stamp go straight
through.

For a non-bundled quant (e.g. Q3_K_S ~12 GB, Q5_K_M ~20 GB),
`make build QUANT=...` downloads from `unsloth/Qwen3.6-27B-GGUF`
and creates a local `thanatos-27b` tag:

```bash
cd ..  &&  make build QUANT=Q3_K_S  &&  cd examples
MODEL=thanatos-27b python ollama_chat.py
```

Or build a local tag from this repo's bundled GGUF without going
through the HF pull:

```bash
cd ..  &&  make load-bundle  &&  cd examples
MODEL=thanatos-27b python ollama_chat.py
```

For a quant the repo doesn't bundle (e.g. Q5_K_M), `make build` will
fetch it from `unsloth/Qwen3.6-27B-GGUF` and patch the `Modelfile`
`FROM` line into a temp copy automatically:

```bash
cd ..  &&  make build QUANT=Q5_K_M  &&  cd examples
python ollama_chat.py
```

### Transformers (safetensors)

```bash
pip install --upgrade "transformers>=4.45" accelerate sentencepiece bitsandbytes
python transformers_quickstart.py            # 4-bit, ~16 GB VRAM
python transformers_quickstart.py --no-4bit  # bf16, ~54 GB VRAM
```

### llama-cpp-python (GGUF, no daemon)

```bash
pip install llama-cpp-python  # CPU-only build
python llama_cpp_quickstart.py /path/to/Qwen3.6-27B-Q4_K_M.gguf --gpu-layers 99
```

For GPU offload, rebuild llama-cpp-python with the matching backend — see
the script header for `CMAKE_ARGS` recipes (CUDA, Metal, ROCm/HIP).

### Vision (image input)

```bash
# Pull the projector once (~927 MB):
hf download unsloth/Qwen3.6-27B-GGUF mmproj-F16.gguf --local-dir .

pip install llama-cpp-python pillow
python llama_cpp_vision.py \
  --gguf /path/to/Qwen3.6-27B-Q4_K_M.gguf \
  --mmproj /path/to/mmproj-F16.gguf \
  --image /path/to/photo.jpg \
  --prompt "Describe this image."
```

Why not Ollama? Ollama's Go engine has the `qwen35` / `qwen35moe`
arch entries (text inference works in 0.24+), but the C++ llama.cpp
fallback that Ollama switches to when an mmproj is attached still
lacks them. `ollama create` accepts the dual-`FROM` and `ollama show`
reports `vision` capability, but the first inference call fails with
`error loading model architecture: unknown model architecture:
'qwen35'` (verified empirically against the dense 27B +
`mmproj-F16.gguf`). Tracked in
[ollama/ollama#15898](https://github.com/ollama/ollama/issues/15898).
Until that's fixed, llama.cpp / llama-cpp-python is the working path
for vision.