README.md

---
tags:
- qwen
- qwen3_5_moe
- gguf
- mtp
- text-to-text
- conversational
- text-generation-inference
base_model:
- Kwaipilot/KAT-Coder-V2.5-Dev
- unsloth/Qwen3.6-35B-A3B-MTP-GGUF
notes:
- based on GGUF format quantizations by bartowski and unsloth
---

## About

 * WIP, more GGUF quantizations are coming after some tests


What you probably want instead is one of the [APEX][apex] `gguf` builds by [gbuzhf][gbuzhf]: 
 * [gbuzhf/KAT-Coder-V2.5-Dev-APEX-MTP-GGUF][gbuzhf]

However, sometimes you want to see if a more classic quantization will be more suitable for your specific task, or simply want to fit (or use) a specific vram target.
In this case, "one can't go wrong with [bartowski][bartowski] builds (c) reddit" -- except for these are sadly missing any draft models, 
and a 20-30 %% gain in inference speed is not something to easily give up on.

Hence, I'll be adding here some "franken-merges" of original [bartowski builds][bartowski] with 20 MTP layers added to it from [unsloth/Qwen3.6-35B-A3B-MTP-GGUF][unsloth] model 
using a [conversion script] made by [buzz][buzz].

All uploaded models were tested and add a ~20-30 % gain in inference speed compared to original ("ungrafted") models on AMD hardware.

(It is strongly recommended to use at most `q8_0` quantizations for KV cache if your hardware supoorts it, especially if it has a chance to be more performant on it.)


## llama.cpp invocation examples

### llama.cpp invocation examples - Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_XS.bartowski.mtp.q8_k_xl.gguf

 * this fits 24 GiB VRAM due to using quantized KV cache


```sh

llama-server  \
--model Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_XS.bartowski.mtp.q8_k_xl.gguf  \
--no-mmproj  \
--flash-attn on  \
--n-gpu-layers all  \
--n-gpu-layers-draft all  \
--kv-offload  \
--kv-unified  \
--cache-type-k q8_0  \
--cache-type-v q8_0  \
--temperature 1.0  \
--top-k 20  \
--top-p 0.95  \
--min-p 0  \
--chat-template-kwargs '{"enable_thinking":false}'  \
--reasoning-preserve  \
--spec-type draft-mtp  \
--spec-draft-n-max 4  \
--cache-type-k-draft q8_0  \
--cache-type-v-draft q8_0

```

### llama.cpp invocation examples - find optimal mtp length

As for `--spec-draft-n-max 4` setting above -- the exact number would obviously depend on your GPU model. 

You might want to do a log scale search by trying
   * `spec-draft-n-max := 1,2,4,8,...` (unlikely more) while the inference speed grows or stays the same,
   * and then using binary search to narrow down the exact value that works best for you on your typical tasks;
   * for a quick control check, your server log will show something like:
      * nb: showing results for the same model running two consequitive coding tasks
```
[37813] 812.35.434.815 I slot launch_slot_: id  0 | task 56272 | processing task, is_child = 0
[37813] 812.37.748.230 I slot print_timing: id  0 | task 56272 | prompt eval time =     310.31 ms /   177 tokens (    1.75 ms per token,   570.40 tokens per second)
[37813] 812.37.748.234 I slot print_timing: id  0 | task 56272 |        eval time =    2002.97 ms /   400 tokens (    5.01 ms per token,   199.70 tokens per second)
[37813] 812.37.748.234 I slot print_timing: id  0 | task 56272 |       total time =    2313.28 ms /   577 tokens
[37813] 812.37.748.235 I slot print_timing: id  0 | task 56272 |    graphs reused =      52595
[37813] 812.37.748.238 I slot print_timing: id  0 | task 56272 | draft acceptance = 0.95482 (  317 accepted /   332 generated), mean len =  4.82
[37813] 812.37.749.091 I slot      release: id  0 | task 56272 | stop processing: n_tokens = 29120, truncated = 0

[37813] 812.38.114.222 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.988 (> 0.100 thold), f_keep = 1.000
[37813] 812.38.115.131 I slot launch_slot_: id  0 | task 56358 | processing task, is_child = 0
[37813] 812.38.767.224 I slot print_timing: id  0 | task 56358 | prompt eval time =     440.40 ms /   341 tokens (    1.29 ms per token,   774.29 tokens per second)
[37813] 812.38.767.228 I slot print_timing: id  0 | task 56358 |        eval time =     211.57 ms /    20 tokens (   10.58 ms per token,    94.53 tokens per second)
[37813] 812.38.767.228 I slot print_timing: id  0 | task 56358 |       total time =     651.97 ms /   361 tokens
[37813] 812.38.767.229 I slot print_timing: id  0 | task 56358 |    graphs reused =      52603
[37813] 812.38.767.232 I slot print_timing: id  0 | task 56358 | draft acceptance = 0.33333 (   12 accepted /    36 generated), mean len =  2.33

```

As you can see, we could have also tried `--spec-draft-n-max 5` and see if that would improve the benchmarks: basically as long as "mean len" frequently shows values 
which are greater than your `spec-draft-n-max` parameter -- you can still increment it by at least 1 and see if it improves your statistics (and waiting times).

Also, in some scenarios there could be some use for additional settings like `--spec-draft-p-min 0.5`.

I am no expert, but in my tests I had some luck with first determining the optimal `--spec-draft-n-max` value, 
and then seeing if raising up the acceptance min level (and/or the "split" values such as `--spec-draft-p-split`) will improve things a little more.


[apex]: https://github.com/localai-org/apex-quant
[gbuzhf]: https://huggingface.co/gbuzhf/KAT-Coder-V2.5-Dev-APEX-MTP-GGUF
[bartowski]: https://huggingface.co/bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF
[unsloth]: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/blob/main/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf
[buzz]: https://gist.github.com/buzz/1c439684d5e3f36492ae9f64ef7e3f67