--- license: apache-2.0 language: - en - zh pipeline_tag: image-text-to-text library_name: mlx tags: - code - agent - agentic-coding - moe - coding - vision - mtp - speculative-decoding - omlx - apple-silicon base_model: - Kwaipilot/KAT-Coder-V2.5-Dev - Qwen/Qwen3.6-35B-A3B --- # KAT-Coder-V2.5-Dev-VL-oQ4e-mtp [`Kwaipilot/KAT-Coder-V2.5-Dev`](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) with its **vision tower and MTP head restored** from the base model it was fine-tuned from, quantized to 4-bit for Apple Silicon. **20 GB.** This is the 4-bit member of a set built from one bf16 master: | level | size | HumanEval pass@1 | MTP acceptance | tok/cycle | |---|---|---|---|---| | [oQ8e](https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ8e-mtp) | 36 GB | 0.8598 | 74.1% | 2.27 | | [oQ6e](https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ6e-mtp) | 28 GB | 0.8659 | 75.5% | 2.46 | | [oQ4e](https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp) | 20 GB | 0.8963 | 79.3% | 2.54 | **Read the accuracy column as a tie, not a ranking.** All three were measured the same day, same settings (full 164 problems, `batch_size=1`, thinking enabled, deterministic profile). The oQ8e build scored 146/164 on an earlier run and 141/164 on the run in this table — a 5-problem swing on an unchanged model, which is as large as the entire spread between levels. Speculative decoding plus Metal batching numerics means "deterministic" is not bit-reproducible here. The honest conclusion is that **8, 6, and 4-bit are indistinguishable on this benchmark**, not that 4-bit is better. > **Read this first: what you need to load it.** > > The `oQ4e` quantization is **not** a proprietary format. oQ writes standard mlx-lm safetensors > with a standard per-module quantization config, and portable oQ models load in stock MLX apps. > > **What makes *this repo* oMLX-only is the restored MTP head.** Stock `mlx-vlm` does a strict load > and rejects the 42 extra `language_model.mtp.*` tensors: > `ValueError: Received 42 parameters not in model`. So as published this needs > [oMLX](https://github.com/jundot/omlx) on Apple Silicon — for the MTP tensors, not the quant. > > Not loadable by llama.cpp, vLLM, or Transformers (MLX safetensors either way). ## What was done | | source | |---|---| | text weights | `Kwaipilot/KAT-Coder-V2.5-Dev`, unmodified (bit-exact; only the MoE expert tensor **layout** was converted) | | vision tower (333 tensors) | `Qwen/Qwen3.6-35B-A3B` | | MTP head (19 tensors) | `Qwen/Qwen3.6-35B-A3B` | | quantization | oMLX `oQ4` enhanced, `group_size=64`, affine, bf16 scales, 128x512 imatrix | Kwaipilot's open-weight release ships language-model weights only — the vision components and the multi-token-prediction head were stripped. Both are recoverable because KAT's `config.json` is byte-identical to `Qwen/Qwen3.6-35B-A3B`'s except for one field, and the text architecture is unchanged. The tensor key set is **identical** to an oMLX `oQ8e-mtp` build of the stock base model — 2052 tensors, with the MTP head correctly `switch_mlp`-named. ## Verification Measured on this exact checkpoint: - **Loads as a VLM.** Server log reports `VLMBatchedEngine loaded`, not the silent VLM to LLM fallback that yields a vision-less model. - **Vision.** 8-part probe (dense code screenshots, high-entropy strings, UI defect-spotting, mixed heading sizes, sparse-glyph sweeps). Transcription is exact except for two failure modes the **native Qwen3.6 base reproduces as well**: the capital `Q` in `sk-7Qv3...` reads as lowercase, and a sparse 384x320 @64px image misreads. No regression against the oQ8e build. - **MTP.** 79.3% draft acceptance, 2.54 tokens/cycle over 8 generations. - **HumanEval.** 0.8963 pass@1 (147/164), 722.5s, thinking enabled. **Caveat on MTP at low bit depths.** oMLX warns that quantizing a preserved MTP head can collapse draft acceptance toward ~0%. That did not happen here, and the likely reason is that the imatrix was **recomputed with the MTP head present** rather than reused from a pre-graft cache. If you build your own level from a stale imatrix, check the `MTP[n] ... accept=` line before trusting the speedup — do not assume 4-bit is universally safe for MTP-preserved heads. ## Reproducing Build scripts live in the oQ8e repo: [`build/merge_kat_vl.py`](https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ8e-mtp/blob/main/build/merge_kat_vl.py) and [`build/graft_mtp.py`](https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ8e-mtp/blob/main/build/graft_mtp.py), with full instructions in its [README](https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ8e-mtp#reproducing). Budget ~146 GB of free disk; peak RAM is about 8 GB. Steps 1 and 2 produce the bf16 master. This repo is step 3 with `"oq_level": 4`: ```bash curl -X POST http://127.0.0.1:8899/admin/api/oq/start \ -H 'Content-Type: application/json' -d '{ "model_path": "'"$HOME"'/Models/_build/KAT-Coder-V2.5-Dev-VL", "oq_level": 4, "enhanced": true, "dtype": "bfloat16", "group_size": 64, "preserve_mtp": true }' ``` Then set `mtp_enabled` on the new model id — it is per-model and off by default, and a model with MTP disabled logs no acceptance line at all, which is easy to misread as a collapsed head. ## Limitations - **oMLX only *as published*** — because of the MTP head, not the quantization. Strip the `mtp.*` tensors and set `text_config.mtp_num_hidden_layers = 0` and the remainder is a portable oQ checkpoint that loads in stock MLX apps; you lose speculative decoding, keep vision and the text model. - **Vision is untrained for this checkpoint.** Good for screenshots of code/terminals/UI; unreliable for isolated large glyphs and occasionally for single characters in random strings. - **Set a `max_tokens` cap.** Long reasoning excursions stall rather than error in an agent loop. - HumanEval is a narrow benchmark. "Indistinguishable on HumanEval" is not "indistinguishable in agentic coding use" — if you have a workload that discriminates between these levels, that result is more informative than the table above. - Chinese/English only, inherited from the base. ## Credits - [Kwaipilot](https://huggingface.co/Kwaipilot) — KAT-Coder-V2.5-Dev - [Qwen](https://huggingface.co/Qwen) — Qwen3.6-35B-A3B, source of the vision tower and MTP head - [oMLX](https://github.com/jundot/omlx) — quantization and serving - The [KAT-Coder community discussions](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev/discussions), where the vision and MTP transplants were first proposed and tested Apache-2.0, inherited from both parents.