Instructions to use jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp with MLX:

# Make sure mlx-vlm is installed
# pip install --upgrade mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load the model
model, processor = load("jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp")
config = load_config("jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp")

# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
prompt = "Describe this image."

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

# Generate output
output = generate(model, processor, formatted_prompt, image)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp"

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Hermes Agent

How to use jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp

Run Hermes

hermes

KAT-Coder-V2.5-Dev-VL-oQ4e-mtp / README.md

jason-schulz

Add oQ4e build of KAT-Coder-V2.5-Dev-VL with restored vision tower and MTP head

deda453 verified 2 days ago

preview code

Raw

History Blame Contribute Delete

6.69 kB

	---
	license: apache-2.0
	language:
	- en
	- zh
	pipeline_tag: image-text-to-text
	library_name: mlx
	tags:
	- code
	- agent
	- agentic-coding
	- moe
	- coding
	- vision
	- mtp
	- speculative-decoding
	- omlx
	- apple-silicon
	base_model:
	- Kwaipilot/KAT-Coder-V2.5-Dev
	- Qwen/Qwen3.6-35B-A3B
	---

	# KAT-Coder-V2.5-Dev-VL-oQ4e-mtp

	[`Kwaipilot/KAT-Coder-V2.5-Dev`](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) with its
	vision tower and MTP head restored from the base model it was fine-tuned from, quantized to
	4-bit for Apple Silicon. 20 GB.

	This is the 4-bit member of a set built from one bf16 master:

	\| level \| size \| HumanEval pass@1 \| MTP acceptance \| tok/cycle \|
	\|---\|---\|---\|---\|---\|
	\| [oQ8e](https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ8e-mtp) \| 36 GB \| 0.8598 \| 74.1% \| 2.27 \|
	\| [oQ6e](https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ6e-mtp) \| 28 GB \| 0.8659 \| 75.5% \| 2.46 \|
	\| [oQ4e](https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp) \| 20 GB \| 0.8963 \| 79.3% \| 2.54 \|

	Read the accuracy column as a tie, not a ranking. All three were measured the same day, same
	settings (full 164 problems, `batch_size=1`, thinking enabled, deterministic profile). The oQ8e
	build scored 146/164 on an earlier run and 141/164 on the run in this table — a 5-problem swing on
	an unchanged model, which is as large as the entire spread between levels. Speculative decoding plus
	Metal batching numerics means "deterministic" is not bit-reproducible here. The honest conclusion is
	that 8, 6, and 4-bit are indistinguishable on this benchmark, not that 4-bit is better.

	> Read this first: what you need to load it.
	>
	> The `oQ4e` quantization is not a proprietary format. oQ writes standard mlx-lm safetensors
	> with a standard per-module quantization config, and portable oQ models load in stock MLX apps.
	>
	> *What makes this repo* oMLX-only is the restored MTP head.** Stock `mlx-vlm` does a strict load
	> and rejects the 42 extra `language_model.mtp.*` tensors:
	> `ValueError: Received 42 parameters not in model`. So as published this needs
	> [oMLX](https://github.com/jundot/omlx) on Apple Silicon — for the MTP tensors, not the quant.
	>
	> Not loadable by llama.cpp, vLLM, or Transformers (MLX safetensors either way).

	## What was done

	\| \| source \|
	\|---\|---\|
	\| text weights \| `Kwaipilot/KAT-Coder-V2.5-Dev`, unmodified (bit-exact; only the MoE expert tensor layout was converted) \|
	\| vision tower (333 tensors) \| `Qwen/Qwen3.6-35B-A3B` \|
	\| MTP head (19 tensors) \| `Qwen/Qwen3.6-35B-A3B` \|
	\| quantization \| oMLX `oQ4` enhanced, `group_size=64`, affine, bf16 scales, 128x512 imatrix \|

	Kwaipilot's open-weight release ships language-model weights only — the vision components and the
	multi-token-prediction head were stripped. Both are recoverable because KAT's `config.json` is
	byte-identical to `Qwen/Qwen3.6-35B-A3B`'s except for one field, and the text architecture is
	unchanged.

	The tensor key set is identical to an oMLX `oQ8e-mtp` build of the stock base model — 2052
	tensors, with the MTP head correctly `switch_mlp`-named.

	## Verification

	Measured on this exact checkpoint:

	- Loads as a VLM. Server log reports `VLMBatchedEngine loaded`, not the silent VLM to LLM
	fallback that yields a vision-less model.
	- Vision. 8-part probe (dense code screenshots, high-entropy strings, UI defect-spotting, mixed
	heading sizes, sparse-glyph sweeps). Transcription is exact except for two failure modes the
	native Qwen3.6 base reproduces as well: the capital `Q` in `sk-7Qv3...` reads as lowercase,
	and a sparse 384x320 @64px image misreads. No regression against the oQ8e build.
	- MTP. 79.3% draft acceptance, 2.54 tokens/cycle over 8 generations.
	- HumanEval. 0.8963 pass@1 (147/164), 722.5s, thinking enabled.

	Caveat on MTP at low bit depths. oMLX warns that quantizing a preserved MTP head can collapse
	draft acceptance toward ~0%. That did not happen here, and the likely reason is that the imatrix was
	recomputed with the MTP head present rather than reused from a pre-graft cache. If you build
	your own level from a stale imatrix, check the `MTP[n] ... accept=` line before trusting the
	speedup — do not assume 4-bit is universally safe for MTP-preserved heads.

	## Reproducing

	Build scripts live in the oQ8e repo: [`build/merge_kat_vl.py`](https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ8e-mtp/blob/main/build/merge_kat_vl.py)
	and [`build/graft_mtp.py`](https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ8e-mtp/blob/main/build/graft_mtp.py), with full instructions in its
	[README](https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ8e-mtp#reproducing). Budget ~146 GB of free disk; peak RAM is about 8 GB.

	Steps 1 and 2 produce the bf16 master. This repo is step 3 with `"oq_level": 4`:

	```bash
	curl -X POST http://127.0.0.1:8899/admin/api/oq/start \
	-H 'Content-Type: application/json' -d '{
	"model_path": "'"$HOME"'/Models/_build/KAT-Coder-V2.5-Dev-VL",
	"oq_level": 4, "enhanced": true, "dtype": "bfloat16", "group_size": 64,
	"preserve_mtp": true }'
	```

	Then set `mtp_enabled` on the new model id — it is per-model and off by default, and a model with
	MTP disabled logs no acceptance line at all, which is easy to misread as a collapsed head.

	## Limitations

	- *oMLX only as published*** — because of the MTP head, not the quantization. Strip the `mtp.*`
	tensors and set `text_config.mtp_num_hidden_layers = 0` and the remainder is a portable oQ
	checkpoint that loads in stock MLX apps; you lose speculative decoding, keep vision and the text
	model.
	- Vision is untrained for this checkpoint. Good for screenshots of code/terminals/UI; unreliable
	for isolated large glyphs and occasionally for single characters in random strings.
	- Set a `max_tokens` cap. Long reasoning excursions stall rather than error in an agent loop.
	- HumanEval is a narrow benchmark. "Indistinguishable on HumanEval" is not "indistinguishable in
	agentic coding use" — if you have a workload that discriminates between these levels, that result
	is more informative than the table above.
	- Chinese/English only, inherited from the base.

	## Credits

	- [Kwaipilot](https://huggingface.co/Kwaipilot) — KAT-Coder-V2.5-Dev
	- [Qwen](https://huggingface.co/Qwen) — Qwen3.6-35B-A3B, source of the vision tower and MTP head
	- [oMLX](https://github.com/jundot/omlx) — quantization and serving
	- The [KAT-Coder community discussions](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev/discussions),
	where the vision and MTP transplants were first proposed and tested

	Apache-2.0, inherited from both parents.