--- license: apache-2.0 base_model: Kwaipilot/KAT-Coder-V2.5-Dev tags: - mtp - speculative-decoding - qwen3_5_moe - experiment --- # KAT-Coder-V2.5-Dev — MTP head (experimental, un-fine-tuned baseline) [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) ships with `mtp_num_hidden_layers: 0` — no MTP/NextN head at all, so no speculative-decoding speedup is possible out of the box. This repo adds one back by transplanting [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)'s own trained MTP head onto KAT-Coder. This is architecturally clean, not a hack: every relevant dimension (hidden_size, expert count, attention/MoE shapes) matches exactly between the two models, since KAT-Coder is itself built on the Qwen3.6-35B-A3B backbone. **This is the naive, un-fine-tuned transplant** — the MTP head's weights are copied directly from stock Qwen3.6 with zero training on KAT-Coder's own hidden-state distribution. A fine-tuning pass (freezing KAT-Coder's own 40 layers entirely, training only this ~850M-parameter head) is a planned follow-up; this repo documents the honest baseline before that. ## What's actually in this repo Just **one new shard** (`model-00014-of-mtp.safetensors`, 1.69GB) containing the 19 MTP-related tensors, plus a `config.json`/`model.safetensors.index.json` already merged to reference it alongside KAT-Coder's own 13 original shards. Nothing else was touched — KAT-Coder's own 13 shards are byte-identical to the original release; this is a drop-in addition, not a full re-upload of the 65GB checkpoint. **To use**: put this repo's 3 files into a local copy of [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) alongside its own 13 `model-*-of-00013.safetensors` shards (the `config.json`/index here already reference all 14 shards together). ## Baseline: draft-acceptance rate (before fine-tuning) Measured via `llama-server --spec-type draft-mtp --spec-draft-n-max 2` on an APEX i-quality quant (4.67 bpw) built from this checkpoint, 5 real from-scratch coding prompts, `n_predict=400` each: | prompt | draft tokens | accepted | rate | tok/s | |---|---:|---:|---:|---:| | binary search tree (insert/delete/traverse) | 286 | 255 | 89.2% | 105.5 | | CSV parser (stdlib only) | 114 | 84 | 73.7% | 94.6 | | retry decorator w/ exponential backoff | 188 | 152 | 80.9% | 100.6 | | merge two sorted linked lists | 74 | 58 | 78.4% | 97.6 | | thread-safe LRU cache | 360 | 218 | 60.6% | 85.9 | | **overall** | **1022** | **767** | **75.1%** | — | For reference, the same KAT-Coder quant tier **without** any MTP head runs at ~70.8 tok/s solo decode speed (measured separately, no speculative decoding available at all). So even this un-fine-tuned transplant is already a real ~20-40% speedup, not a wash — acceptance rate varies by task in a way that tracks predictability (the BST task, mostly standard tree-traversal boilerplate, hits 89%; the LRU cache, with genuine locking/eviction logic, is the hardest at 61%). ## Full coding-benchmark validation (real agentic tasks, not just completion speed) Ran the same two task suites used elsewhere in this project's Qwen3.6-lineage work (an existing 5-task hard tier, plus a new 3-task tier designed to require genuine algorithmic reasoning rather than spec-transcription) against this MTP-enabled build vs the same quant without MTP, via `opencode`'s real agentic tool-calling loop (not just raw completion). **Correctness was identical — 100% on both suites, both with and without MTP** — exactly what the theory predicts: the MTP head only drafts candidate tokens, the main model always verifies against its own true computation, so a bad draft just gets rejected and never reaches the output. Speed differed substantially: | hard-tier task | no MTP | with MTP | speedup | |---|---:|---:|---:| | batch_scheduler | 22.5s | 20.8s | 1.08x | | buddy_allocator | 123.5s | 63.5s | 1.94x | | circuit_breaker | 50.7s | 14.0s | 3.62x | | expr_parser_advanced | 54.7s | 21.1s | 2.59x | | inventory_fix_and_extend | 16.4s | 12.3s | 1.33x | | **total** | **267.8s** | **131.7s** | **2.03x** | | tier-4 task | no MTP | with MTP | speedup | |---|---:|---:|---:| | lru_ttl_cache | 20.5s | 25.2s | 0.81x | | rate_limiter_fix_and_extend | 19.7s | 12.1s | 1.63x | | weighted_interval_scheduling | 22.5s | 17.4s | 1.29x | | **total** | **62.7s** | **54.7s** | **1.15x** | Both suites hit 100% (59/59 and 28/28 test cases respectively) in both configurations. The one slower result (`lru_ttl_cache`, 0.81x) is most likely ordinary run-to-run variance in how much iteration that specific attempt needed, not a systematic MTP cost — every other task improved, and the hard-tier set as a whole is a genuine 2x. ## Attribution - KAT-Coder base model: **Kwaipilot** — [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) (Apache-2.0). - MTP head weights: **Qwen** — [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (Apache-2.0), transplanted unmodified. - Quantization/serving: **llama.cpp** ([ggml-org](https://github.com/ggml-org/llama.cpp)). Unofficial community experiment; not affiliated with or endorsed by Kwaipilot or Qwen.