README.md

---
license: apache-2.0
base_model: Kwaipilot/KAT-Coder-V2.5-Dev
tags:
  - mtp
  - speculative-decoding
  - qwen3_5_moe
  - experiment
---

# KAT-Coder-V2.5-Dev — MTP head (experimental, un-fine-tuned baseline)

[Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) ships with
`mtp_num_hidden_layers: 0` — no MTP/NextN head at all, so no speculative-decoding speedup is
possible out of the box. This repo adds one back by transplanting
[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)'s own trained MTP head onto
KAT-Coder. This is architecturally clean, not a hack: every relevant dimension (hidden_size,
expert count, attention/MoE shapes) matches exactly between the two models, since KAT-Coder is
itself built on the Qwen3.6-35B-A3B backbone.

**This is the naive, un-fine-tuned transplant** — the MTP head's weights are copied directly from
stock Qwen3.6 with zero training on KAT-Coder's own hidden-state distribution. A fine-tuning pass
(freezing KAT-Coder's own 40 layers entirely, training only this ~850M-parameter head) is a
planned follow-up; this repo documents the honest baseline before that.

## What's actually in this repo

Just **one new shard** (`model-00014-of-mtp.safetensors`, 1.69GB) containing the 19 MTP-related
tensors, plus a `config.json`/`model.safetensors.index.json` already merged to reference it
alongside KAT-Coder's own 13 original shards. Nothing else was touched — KAT-Coder's own 13
shards are byte-identical to the original release; this is a drop-in addition, not a full
re-upload of the 65GB checkpoint.

**To use**: put this repo's 3 files into a local copy of
[Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) alongside its
own 13 `model-*-of-00013.safetensors` shards (the `config.json`/index here already reference all
14 shards together).

## Baseline: draft-acceptance rate (before fine-tuning)

Measured via `llama-server --spec-type draft-mtp --spec-draft-n-max 2` on an APEX i-quality quant
(4.67 bpw) built from this checkpoint, 5 real from-scratch coding prompts, `n_predict=400` each:

| prompt | draft tokens | accepted | rate | tok/s |
|---|---:|---:|---:|---:|
| binary search tree (insert/delete/traverse) | 286 | 255 | 89.2% | 105.5 |
| CSV parser (stdlib only) | 114 | 84 | 73.7% | 94.6 |
| retry decorator w/ exponential backoff | 188 | 152 | 80.9% | 100.6 |
| merge two sorted linked lists | 74 | 58 | 78.4% | 97.6 |
| thread-safe LRU cache | 360 | 218 | 60.6% | 85.9 |
| **overall** | **1022** | **767** | **75.1%** | — |

For reference, the same KAT-Coder quant tier **without** any MTP head runs at ~70.8 tok/s solo
decode speed (measured separately, no speculative decoding available at all). So even this
un-fine-tuned transplant is already a real ~20-40% speedup, not a wash — acceptance rate varies
by task in a way that tracks predictability (the BST task, mostly standard tree-traversal
boilerplate, hits 89%; the LRU cache, with genuine locking/eviction logic, is the hardest at 61%).

## Full coding-benchmark validation (real agentic tasks, not just completion speed)

Ran the same two task suites used elsewhere in this project's Qwen3.6-lineage work (an existing
5-task hard tier, plus a new 3-task tier designed to require genuine algorithmic reasoning rather
than spec-transcription) against this MTP-enabled build vs the same quant without MTP, via
`opencode`'s real agentic tool-calling loop (not just raw completion). **Correctness was
identical — 100% on both suites, both with and without MTP** — exactly what the theory predicts:
the MTP head only drafts candidate tokens, the main model always verifies against its own true
computation, so a bad draft just gets rejected and never reaches the output. Speed differed
substantially:

| hard-tier task | no MTP | with MTP | speedup |
|---|---:|---:|---:|
| batch_scheduler | 22.5s | 20.8s | 1.08x |
| buddy_allocator | 123.5s | 63.5s | 1.94x |
| circuit_breaker | 50.7s | 14.0s | 3.62x |
| expr_parser_advanced | 54.7s | 21.1s | 2.59x |
| inventory_fix_and_extend | 16.4s | 12.3s | 1.33x |
| **total** | **267.8s** | **131.7s** | **2.03x** |

| tier-4 task | no MTP | with MTP | speedup |
|---|---:|---:|---:|
| lru_ttl_cache | 20.5s | 25.2s | 0.81x |
| rate_limiter_fix_and_extend | 19.7s | 12.1s | 1.63x |
| weighted_interval_scheduling | 22.5s | 17.4s | 1.29x |
| **total** | **62.7s** | **54.7s** | **1.15x** |

Both suites hit 100% (59/59 and 28/28 test cases respectively) in both configurations. The one
slower result (`lru_ttl_cache`, 0.81x) is most likely ordinary run-to-run variance in how much
iteration that specific attempt needed, not a systematic MTP cost — every other task improved, and
the hard-tier set as a whole is a genuine 2x.

## Attribution

- KAT-Coder base model: **Kwaipilot** — [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) (Apache-2.0).
- MTP head weights: **Qwen** — [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (Apache-2.0), transplanted unmodified.
- Quantization/serving: **llama.cpp** ([ggml-org](https://github.com/ggml-org/llama.cpp)).

Unofficial community experiment; not affiliated with or endorsed by Kwaipilot or Qwen.