Add vision support via llama.cpp; document Ollama upstream gap

Vision via Ollama is currently broken for the qwen35/qwen35moe family.
Ollama 0.22's vendored llama.cpp fork is missing the architecture
entries from upstream ggml-org/llama.cpp, so attaching mmproj via
either FROM or ADAPTER returns:
unknown model architecture: 'qwen35moe'
Tracked in ollama/ollama#15898 (and duplicates #14730, #15346). The
fix has not landed.

Rather than ship a Modelfile.vision that doesn't work, this commit:

- Adds examples/llama_cpp_vision.py — uses llama-cpp-python with the
Qwen2.5-VL chat handler and a separate mmproj-F16.gguf. Works today.
- Adds scripts/fetch_mmproj.sh + 'make mmproj' to pull the projector
from unsloth/Qwen3.6-27B-GGUF (~927 MB).
- Updates README:
- Replaces the misleading 'Multimodal: Yes (vision)' comparison row
with a per-loader breakdown.
- Adds a dedicated Vision section explaining the Ollama gap and the
working llama.cpp path.
- Updates Known Limitations.
- Adds explicit text-only headers to Modelfile and Modelfile.z13.
- Updates examples/README.md.
- CHANGELOG: documents this batch and tags 0.4.0 (9ca8700).

CHANGELOG.md

@@ -7,6 +7,30 @@ and documentation**, not the underlying base model.
 
 ## [Unreleased]
 
+### Added
+- `examples/llama_cpp_vision.py` — image-text-to-text via
+  `llama-cpp-python` + a separate `mmproj-F16.gguf`. Currently the only
+  working vision path for Janus-27B (Ollama is broken; see Changed).
+- `scripts/fetch_mmproj.sh` — pulls `mmproj-F16.gguf` (or BF16/F32) from
+  `unsloth/Qwen3.6-27B-GGUF`. Honors `MMPROJ_PATH` override.
+- `Makefile`: new `mmproj` target.
+
+### Changed
+- README: replaced the misleading "Multimodal: Yes (vision)" comparison
+  row with a per-loader breakdown. Added a dedicated **Vision** section
+  documenting:
+  - The Ollama 0.22 architecture gap (`unknown model architecture:
+    'qwen35moe'`) tracked in
+    [ollama/ollama#15898](https://github.com/ollama/ollama/issues/15898).
+  - A working llama.cpp / llama-cpp-python path with the `mmproj`.
+  - Updated Known Limitations entry accordingly.
+- `Modelfile` and `Modelfile.z13`: header comments now state explicitly
+  that they're text-only and link to the Vision section.
+- `examples/README.md`: reflects the new Vision example and explains
+  why Ollama is not the recommended backend for it (yet).
+
+## [0.4.0] - 2026-05-02 — `9ca8700`
+
 ### Added
 - `Makefile` — convenience wrapper. `make help` lists targets:
   `build` / `smoke` / `check` / `hooks` / `clean`. Variables

Makefile

@@ -31,13 +31,15 @@ MODEL ?= $(TAG)
 
 .DEFAULT_GOAL := help
 
-.PHONY: help build smoke check hooks clean
+PRECISION ?= F16
+
+.PHONY: help build smoke check hooks mmproj clean
 
 help:  ## Show this help.
 	@awk 'BEGIN {FS = ":.*##"; printf "Targets:\n"} /^[a-zA-Z_-]+:.*?##/ { printf "  \033[36m%-12s\033[0m %s\n", $$1, $$2 }' $(MAKEFILE_LIST)
 	@echo
 	@echo "Current settings:"
-	@echo "  QUANT=$(QUANT) PROFILE=$(PROFILE) TAG=$(TAG)"
+	@echo "  QUANT=$(QUANT) PROFILE=$(PROFILE) TAG=$(TAG) PRECISION=$(PRECISION)"
 ifdef GGUF_PATH
 	@echo "  GGUF_PATH=$(GGUF_PATH)"
 endif
@@ -48,6 +50,9 @@ build:  ## Download GGUF (if needed) and run 'ollama create'.
 smoke:  ## Verify the model is reachable and round-trips.
 	MODEL=$(MODEL) ./scripts/smoke_test.sh
 
+mmproj:  ## Fetch the vision projector for llama.cpp (Ollama vision is broken upstream).
+	./scripts/fetch_mmproj.sh $(PRECISION)
+
 check:  ## Lint shell + python files; block dot-pattern footgun.
 	./scripts/check.sh
 
@@ -56,6 +61,6 @@ hooks:  ## Install scripts/check.sh as the git pre-commit hook.
 
 clean:  ## Remove local GGUF copies and ephemeral caches in this repo.
 	@echo "[*] removing local GGUFs and ephemeral caches in $$PWD"
-	@rm -f ./Qwen3.6-27B-*.gguf
+	@rm -f ./Qwen3.6-27B-*.gguf ./mmproj-*.gguf
 	@rm -rf ./.cache __pycache__ examples/__pycache__
 	@echo "[+] clean"

Modelfile

@@ -1,5 +1,10 @@
 # Janus-27B — Ollama wrapper around Qwen 3.6 27B (dense)
 #
+# Text-only. Vision via Ollama is currently broken for this architecture
+# (ollama/ollama#15898 — the vendored llama.cpp fork is missing the
+# qwen35 arch entries). Use llama.cpp directly for image input, or wait
+# for the fix. See the Vision section in README.md.
+#
 # This repo does not redistribute weights. Edit the FROM line below to
 # point at a local Qwen 3.6 27B GGUF, then:
 #

Modelfile.z13

@@ -1,5 +1,8 @@
 # Janus-27B — Z13 variant for ASUS ROG Flow Z13 (Ryzen AI Max+ 395, 128 GB)
 #
+# Text-only (same caveat as the default Modelfile — vision via Ollama
+# is broken upstream for qwen35; see README Vision section).
+#
 # This Modelfile is tuned for an iGPU with a shared/unified memory pool.
 # Defaults differ from the main Modelfile in three ways:
 #   1. Smaller context (8K instead of 16K) to keep KV cache slim.

README.md

@@ -74,7 +74,9 @@ The 27B is **dense**: every parameter participates in every forward pass. It's s
 | Q4_K_M GGUF size | ~17 GB | ~19 GB |
 | Q3_K_S GGUF size | ~12 GB | n/a |
 | Min host memory @ Q4 / 8K ctx | ~22 GB | ~38 GB |
-| Multimodal | Yes (vision) | Yes (vision) |
+| Multimodal (text path) | Yes | Yes |
+| Multimodal (vision via Ollama) | Broken upstream — see below | Broken upstream |
+| Multimodal (vision via llama.cpp) | Yes, with mmproj | Yes, with mmproj |
 | Max context | 262 144 | 262 144 |
 
 ## What's here
@@ -87,6 +89,7 @@ The 27B is **dense**: every parameter participates in every forward pass. It's s
 | `examples/` | Ready-to-run Python clients for Ollama, Transformers, and llama-cpp-python |
 | `scripts/build.sh` | One-shot helper: pulls a GGUF and runs `ollama create` for you |
 | `scripts/smoke_test.sh` | Verifies an Ollama daemon + model and runs a round-trip |
+| `scripts/fetch_mmproj.sh` | Pulls the vision projector for llama.cpp (Ollama vision is broken upstream — see [Vision](#vision)) |
 | `scripts/check.sh` | Local lint: `bash -n`, `pyflakes`, `py_compile`, footgun-grep |
 | `scripts/install-hooks.sh` | Installs `check.sh` as a git pre-commit hook |
 | `Makefile` | Convenience wrapper — `make help` lists targets |
@@ -107,7 +110,10 @@ If you want the safetensors for `transformers`, fetch them from [`Qwen/Qwen3.6-2
 - Hidden size 5120, FFN intermediate 17408 (~3.4× ratio)
 - Vocab 248,320 (shared with 35B-A3B sibling)
 - 262 144 native context, extensible to ~1 M with YaRN
-- Vision + video support via upstream `mmproj` (not in this repo)
+- Vision + video supported by the **base architecture** via a separate
+  `mmproj` projector (not redistributed here; pull `mmproj-F16.gguf`
+  from `unsloth/Qwen3.6-27B-GGUF`). See [Vision](#vision) below for
+  current loader compatibility.
 - Multi-token prediction (MTP) head trained for speculative decoding
 
 ## Quick start
@@ -176,6 +182,50 @@ Behavior rules:
 - Finish with a usable answer, not just planning.
 ```
 
+## Vision
+
+The Qwen 3.6 base supports image (and video) input via a separate
+`mmproj` projector. The full multimodal stack is:
+
+```
+Qwen3.6-27B-Q4_K_M.gguf   (~17 GB, the text decoder)
+mmproj-F16.gguf           (~927 MB, the vision projector)
+```
+
+Both files are at
+[`unsloth/Qwen3.6-27B-GGUF`](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF).
+This repo intentionally does not redistribute either.
+
+### Loader compatibility — the honest table
+
+| Loader | Text | Vision (mmproj) | Notes |
+|---|---|---|---|
+| **llama.cpp** (`llama-mtmd-cli`, `llama-server --mmproj`) | ✅ | ✅ | Reference path. Upstream has the `qwen35`/`qwen35moe` arch entries. |
+| **llama-cpp-python** | ✅ | ✅ | See `examples/llama_cpp_vision.py`. |
+| **Ollama 0.22** | ✅ | ❌ | Vendored llama.cpp fork is missing the architecture entries. Attaching `mmproj` via `FROM` *or* `ADAPTER` returns `unknown model architecture: 'qwen35moe'` (and the same for the dense `qwen35`). See [ollama/ollama#15898](https://github.com/ollama/ollama/issues/15898). Will work once that PR lands. |
+| **LM Studio** | ✅ | ✅ (last tested) | Uses upstream llama.cpp directly. |
+
+### Vision via llama.cpp
+
+```bash
+# CLI:
+llama-mtmd-cli \
+  -m Qwen3.6-27B-Q4_K_M.gguf \
+  --mmproj mmproj-F16.gguf \
+  --image photo.jpg \
+  -p "Describe this image."
+
+# Python:
+python examples/llama_cpp_vision.py \
+  --gguf /path/to/Qwen3.6-27B-Q4_K_M.gguf \
+  --mmproj /path/to/mmproj-F16.gguf \
+  --image /path/to/photo.jpg \
+  --prompt "What is in this image?"
+```
+
+Until the Ollama upstream issue is fixed, treat Ollama as **text-only**
+for this model.
+
 ## Hardware requirements
 
 The dense 27B is the easier of the two Janus models to deploy.
@@ -197,7 +247,7 @@ See the [Janus-35B Chat template section](https://huggingface.co/FoolDev/janus#c
 ## Known limitations
 
 - **Slower per token than the 35B-A3B sibling.** Dense 27B beats sparse 35B/3B-active on steps-per-second benchmarks because every parameter contributes; if you optimize for tokens-per-second, the MoE wins.
-- **No mmproj in this release.** Same as 35B — fetch upstream for vision input.
+- **No mmproj in this release**, and **vision via Ollama is broken upstream** (qwen35/qwen35moe arch entries missing from Ollama's vendored llama.cpp fork — see the [Vision](#vision) section). For image input use llama.cpp directly until that's fixed.
 - **Q4_K_M quality loss** is real. Use Q5_K_M or Q6_K if you have the VRAM (~20-22 GB).
 - **No formal evaluation in this card.** Numbers above are estimates.
 

examples/README.md

@@ -4,9 +4,10 @@ Three minimal entry points. Pick the one that matches how you run models.
 
 | File | Backend | When to use |
 |---|---|---|
-| `ollama_chat.py` | Ollama HTTP API | You already have `ollama serve` running and the `janus-27b` model created from the project `Modelfile`. |
+| `ollama_chat.py` | Ollama HTTP API | You already have `ollama serve` running and the `janus-27b` model created from the project `Modelfile`. **Text only** — vision via Ollama is broken upstream for this arch. |
 | `transformers_quickstart.py` | Hugging Face Transformers | You want to run the upstream safetensors (`Qwen/Qwen3.6-27B`) on GPU, optionally in 4-bit via bitsandbytes. |
-| `llama_cpp_quickstart.py` | llama-cpp-python | You want to invoke a local GGUF directly without a daemon (CI, batch jobs, scripts). |
+| `llama_cpp_quickstart.py` | llama-cpp-python | You want to invoke a local GGUF directly without a daemon (CI, batch jobs, scripts). Text only. |
+| `llama_cpp_vision.py` | llama-cpp-python + mmproj | **Image input.** Loads a text GGUF + `mmproj-F16.gguf` and answers questions about an image. The only working vision path right now. |
 
 All three apply the same Janus system prompt and sampling defaults
 (`temp=0.6, top_p=0.95, top_k=20, repeat_penalty=1.05`) so behavior should
@@ -47,3 +48,23 @@ python llama_cpp_quickstart.py /path/to/Qwen3.6-27B-Q4_K_M.gguf --gpu-layers 99
 
 For GPU offload, rebuild llama-cpp-python with the matching backend — see
 the script header for `CMAKE_ARGS` recipes (CUDA, Metal, ROCm/HIP).
+
+### Vision (image input)
+
+```bash
+# Pull the projector once (~927 MB):
+hf download unsloth/Qwen3.6-27B-GGUF mmproj-F16.gguf --local-dir .
+
+pip install llama-cpp-python pillow
+python llama_cpp_vision.py \
+  --gguf /path/to/Qwen3.6-27B-Q4_K_M.gguf \
+  --mmproj /path/to/mmproj-F16.gguf \
+  --image /path/to/photo.jpg \
+  --prompt "Describe this image."
+```
+
+Why not Ollama? Ollama 0.22's vendored llama.cpp is missing the `qwen35`
+architecture entries needed to attach an mmproj — `FROM` and `ADAPTER`
+both fail with `unknown model architecture: 'qwen35moe'`. Tracked in
+[ollama/ollama#15898](https://github.com/ollama/ollama/issues/15898).
+Until that's fixed, llama.cpp / llama-cpp-python is the working path.

examples/llama_cpp_vision.py

@@ -0,0 +1,126 @@
+#!/usr/bin/env python3
+"""
+Janus-27B — vision (image-text-to-text) via llama-cpp-python.
+
+Why this script exists:
+    Ollama 0.22's vendored llama.cpp fork is missing the qwen35/qwen35moe
+    architecture entries needed to attach a separate mmproj projector.
+    Both `FROM mmproj.gguf` and `ADAPTER mmproj.gguf` fail with:
+        unknown model architecture: 'qwen35moe'
+    See ollama/ollama#15898, #14730 (closed as duplicates of #15898 root
+    cause). Until that lands, vision via Ollama is broken for Qwen 3.5 /
+    3.6.
+
+    Upstream ggml-org/llama.cpp **does** have the architecture, so vision
+    works fine via llama.cpp directly. This script uses the python binding.
+
+Install:
+    pip install llama-cpp-python pillow
+    # GPU offload? rebuild with the matching backend:
+    #   CMAKE_ARGS="-DGGML_CUDA=on"   pip install llama-cpp-python --no-binary :all:
+    #   CMAKE_ARGS="-DGGML_METAL=on"  pip install llama-cpp-python --no-binary :all:
+    #   CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install llama-cpp-python --no-binary :all:
+
+Files you need (both from unsloth/Qwen3.6-27B-GGUF):
+    1. A text GGUF (any quant): e.g. Qwen3.6-27B-Q4_K_M.gguf  (~17 GB)
+    2. A vision projector:        mmproj-F16.gguf              (~927 MB)
+
+Usage:
+    python llama_cpp_vision.py \
+        --gguf /path/to/Qwen3.6-27B-Q4_K_M.gguf \
+        --mmproj /path/to/mmproj-F16.gguf \
+        --image  /path/to/photo.jpg \
+        --prompt "What is in this image? Be specific."
+
+    # CLI alternative without python binding (ships with llama.cpp):
+    #   llama-mtmd-cli \
+    #     -m Qwen3.6-27B-Q4_K_M.gguf \
+    #     --mmproj mmproj-F16.gguf \
+    #     --image photo.jpg \
+    #     -p "Describe this image."
+"""
+from __future__ import annotations
+
+import argparse
+import base64
+import sys
+from pathlib import Path
+
+try:
+    from llama_cpp import Llama
+    from llama_cpp.llama_chat_format import Qwen25VLChatHandler
+except ImportError:  # pragma: no cover
+    sys.exit(
+        "Missing llama-cpp-python (>=0.3 with VL handlers).\n"
+        "  pip install --upgrade llama-cpp-python pillow"
+    )
+
+
+JANUS_SYSTEM = (
+    "You are Janus, a precise vision-language assistant. Describe images "
+    "accurately, do not invent details, and ground every claim in the "
+    "pixels you can actually see."
+)
+
+
+def encode_image_data_uri(path: Path) -> str:
+    suffix = path.suffix.lower().lstrip(".")
+    mime = {"jpg": "jpeg", "jpeg": "jpeg", "png": "png", "webp": "webp", "gif": "gif"}.get(suffix, "jpeg")
+    return f"data:image/{mime};base64,{base64.b64encode(path.read_bytes()).decode()}"
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--gguf", required=True, help="Text GGUF (e.g. Qwen3.6-27B-Q4_K_M.gguf).")
+    ap.add_argument("--mmproj", required=True, help="Vision projector GGUF (mmproj-F16.gguf).")
+    ap.add_argument("--image", required=True, help="Image to analyze.")
+    ap.add_argument("--prompt", default="Describe this image in detail.")
+    ap.add_argument("--ctx", type=int, default=8192)
+    ap.add_argument(
+        "--gpu-layers",
+        type=int,
+        default=0,
+        help="Layers to offload to GPU (-1 or 99 = all).",
+    )
+    ap.add_argument("--max-tokens", type=int, default=512)
+    args = ap.parse_args()
+
+    image_path = Path(args.image)
+    if not image_path.exists():
+        sys.exit(f"Image not found: {image_path}")
+
+    # Qwen 2.5 VL chat handler is the closest match shipped with
+    # llama-cpp-python; Qwen 3.5/3.6 vision uses the same projector layout.
+    # If/when llama-cpp-python ships a Qwen3VLChatHandler, swap it in.
+    handler = Qwen25VLChatHandler(clip_model_path=args.mmproj)
+
+    llm = Llama(
+        model_path=args.gguf,
+        chat_handler=handler,
+        n_ctx=args.ctx,
+        n_gpu_layers=args.gpu_layers,
+        verbose=False,
+    )
+
+    out = llm.create_chat_completion(
+        messages=[
+            {"role": "system", "content": JANUS_SYSTEM},
+            {
+                "role": "user",
+                "content": [
+                    {"type": "image_url", "image_url": {"url": encode_image_data_uri(image_path)}},
+                    {"type": "text", "text": args.prompt},
+                ],
+            },
+        ],
+        temperature=0.6,
+        top_p=0.95,
+        top_k=20,
+        repeat_penalty=1.05,
+        max_tokens=args.max_tokens,
+    )
+    print(out["choices"][0]["message"]["content"])
+
+
+if __name__ == "__main__":
+    main()

scripts/fetch_mmproj.sh

@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+# Janus-27B — fetch the vision projector (mmproj) for image input.
+#
+# Why this is separate from build.sh:
+#   build.sh is for the Ollama text path. The mmproj is only useful for
+#   llama.cpp / llama-cpp-python right now, because Ollama's vendored
+#   llama.cpp fork is missing the qwen35 arch entries needed to attach
+#   it (see README Vision section, ollama/ollama#15898).
+#
+# Usage:
+#   ./scripts/fetch_mmproj.sh                    # default: F16, ~927 MB
+#   ./scripts/fetch_mmproj.sh BF16               # ~931 MB
+#   ./scripts/fetch_mmproj.sh F32                # ~1.8 GB
+#
+# Requires: huggingface-cli (or hf).
+set -euo pipefail
+
+PRECISION="${1:-${PRECISION:-F16}}"
+REPO_ID="${REPO_ID:-unsloth/Qwen3.6-27B-GGUF}"
+FILE_NAME="mmproj-${PRECISION}.gguf"
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+DEST="${MMPROJ_PATH:-${ROOT}/${FILE_NAME}}"
+
+echo "[*] repo:      ${REPO_ID}"
+echo "[*] precision: ${PRECISION}"
+echo "[*] file:      ${FILE_NAME}"
+echo "[*] dest:      ${DEST}"
+
+if [[ -f "${DEST}" ]]; then
+    echo "[=] already present at ${DEST}, skipping."
+    exit 0
+fi
+
+HF=""
+if command -v hf >/dev/null 2>&1; then
+    HF="hf"
+elif command -v huggingface-cli >/dev/null 2>&1; then
+    HF="huggingface-cli"
+else
+    echo "[!] Neither 'hf' nor 'huggingface-cli' found." >&2
+    echo "    pip install -U huggingface_hub" >&2
+    exit 1
+fi
+
+DEST_DIR="$(dirname "${DEST}")"
+mkdir -p "${DEST_DIR}"
+
+case "${HF}" in
+    hf)              hf download "${REPO_ID}" "${FILE_NAME}" --local-dir "${DEST_DIR}" ;;
+    huggingface-cli) huggingface-cli download "${REPO_ID}" "${FILE_NAME}" --local-dir "${DEST_DIR}" ;;
+esac
+
+if [[ ! -f "${DEST}" ]]; then
+    echo "[!] download failed: ${DEST} not present." >&2
+    exit 1
+fi
+
+echo
+echo "[+] Done. Use it via:"
+echo "    python ${ROOT}/examples/llama_cpp_vision.py \\"
+echo "        --gguf  /path/to/Qwen3.6-27B-Q4_K_M.gguf \\"
+echo "        --mmproj ${DEST} \\"
+echo "        --image /path/to/photo.jpg \\"
+echo "        --prompt 'Describe this image.'"