README.md

---
license: apache-2.0
base_model: Kwaipilot/KAT-Coder-V2.5-Dev
base_model_relation: quantized
tags:
- gguf
- rocm
- rocmfp4
- strix-halo
- gfx1151
- speculative-decoding
- mtp
pipeline_tag: text-generation
---

# KAT-Coder-V2.5-Dev — ROCmFP4_STRIX + grafted MTP head (GGUF)

A 4-bit `ROCmFP4_STRIX` quant of [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) with the model's **MTP (multi-token prediction) head grafted back in at Q8_0**, so it can self-speculate. Built and measured on an AMD Ryzen AI Max+ 395 (Strix Halo, gfx1151, 128 GB unified memory).

**19.0 GB** — `KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4_STRIX-MTP.gguf`

## ⚠️ Requires a llama.cpp fork — stock llama.cpp cannot read this file

`ROCmFP4` uses GGUF tensor types **100/101**, which upstream llama.cpp does not know. You need [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX). Stock `gguf-py` also can't parse it (use the fork's `gguf-py` for inspection).

If you don't have that fork, this file is not for you — take a Q4_K_M or Q6_K build of the base model instead.

## Serving

```bash
env HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
  llama-server -m KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4_STRIX-MTP.gguf \
  -ngl 999 -fa on -c 32768 --jinja --metrics \
  --spec-type draft-mtp --spec-draft-n-max 1 \
  --reasoning-format deepseek --reasoning-budget 0
```

- **`--spec-draft-n-max 1`** is what was tuned here. The MTP head predicts one token ahead; deeper drafting costs more in verification than it returns on this hardware.
- **`--reasoning-format deepseek --reasoning-budget 0`** to suppress thinking. Do **not** use `--reasoning-format none` if you post-process the output: `none` tells llama.cpp not to parse think tags, so `</think>` is left **inline in `content`** and will corrupt anything that extracts code from the response. With `deepseek`, residue lands in `reasoning_content` instead. Expect ~72 chars of residue there that cannot be driven to zero.

## Measured

Same hardware, `--parallel 1`, greedy, **thinking off** (probe-verified), single runs.

### Quality

| benchmark | score | notes |
|---|---:|---|
| GSM8K (n=500, 5-shot) | **0.952** ±0.010 | identical to the plain STRIX base — MTP costs no quality |
| IFEval prompt_strict (n=500) | 0.764 ±0.019 | inst_strict 0.834 |
| HumanEval (pass@1, 164) | **0.939** | |
| HumanEval+ (pass@1, 164) | **0.884** | evalplus hardened tests |

### Speed

| | plain STRIX | this build (MTP) |
|---|---:|---:|
| decode @1k | 67.6 t/s | **92.5 t/s** (+37%) |
| decode @8k | 63.8 t/s | **87.1 t/s** (+37%) |

## Caveats

- **MTP defeats the prompt cache** (`spec-boundary-mismatch`), so this build suits single-shot work better than long multi-turn chat. If you need prompt-cache reuse, serve the plain STRIX quant without `--spec-type`.
- **Thinking-off numbers only.** Reasoning-on was not benchmarked; the scores above are floors.
- Single runs, greedy, pass@1. No repeats, so treat 1–2 point differences as noise.
- HumanEval base is saturated at this capability level — **HumanEval+ is the more discriminating number**.
- `--spec-type draft-mtp` works with `--parallel > 1` (per-slot draft state is maintained). But `n-max` should come *down* as concurrency rises, since the verify batch scales as `n-max × active slots`.

## How it was built

Quantized from a BF16 conversion of the base model with the base model's own imatrix, using the fork's `llama-quantize` at `Q4_0_ROCMFP4_STRIX` (a selective recipe: `attn_k`/`attn_v` get the quality layout, `attn_q`/`attn_output`/`ffn_*_exps` get the fast layout, `token_embd` gets Q6_K). The MTP head — which ships inside the base model rather than as a separate draft — was then grafted in at Q8_0.

Credit to Kwaipilot for the base model, and to the ROCmFPX fork for the FP4 kernels.