raulvidis/KAT-Coder-V2.5-Dev-ROCmFP4-STRIX-MTP-GGUF

KAT-Coder-V2.5-Dev — ROCmFP4_STRIX + grafted MTP head (GGUF)

A 4-bit ROCmFP4_STRIX quant of Kwaipilot/KAT-Coder-V2.5-Dev with the model's MTP (multi-token prediction) head grafted back in at Q8_0, so it can self-speculate. Built and measured on an AMD Ryzen AI Max+ 395 (Strix Halo, gfx1151, 128 GB unified memory).

19.0 GB — KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4_STRIX-MTP.gguf

⚠️ Requires a llama.cpp fork — stock llama.cpp cannot read this file

ROCmFP4 uses GGUF tensor types 100/101, which upstream llama.cpp does not know. You need charlie12345/ROCmFPX. Stock gguf-py also can't parse it (use the fork's gguf-py for inspection).

If you don't have that fork, this file is not for you — take a Q4_K_M or Q6_K build of the base model instead.

Serving

env HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
  llama-server -m KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4_STRIX-MTP.gguf \
  -ngl 999 -fa on -c 32768 --jinja --metrics \
  --spec-type draft-mtp --spec-draft-n-max 1 \
  --reasoning-format deepseek --reasoning-budget 0

• --spec-draft-n-max 1 is what was tuned here. The MTP head predicts one token ahead; deeper drafting costs more in verification than it returns on this hardware.
• --reasoning-format deepseek --reasoning-budget 0 to suppress thinking. Do not use --reasoning-format none if you post-process the output: none tells llama.cpp not to parse think tags, so </think> is left inline in content and will corrupt anything that extracts code from the response. With deepseek, residue lands in reasoning_content instead. Expect ~72 chars of residue there that cannot be driven to zero.

Measured

Same hardware, --parallel 1, greedy, thinking off (probe-verified), single runs.

Quality

benchmark	score	notes
GSM8K (n=500, 5-shot)	0.952 ±0.010	identical to the plain STRIX base — MTP costs no quality
IFEval prompt_strict (n=500)	0.764 ±0.019	inst_strict 0.834
HumanEval (pass@1, 164)	0.939
HumanEval+ (pass@1, 164)	0.884	evalplus hardened tests

Speed

	plain STRIX	this build (MTP)
decode @1k	67.6 t/s	92.5 t/s (+37%)
decode @8k	63.8 t/s	87.1 t/s (+37%)

Caveats

• MTP defeats the prompt cache (spec-boundary-mismatch), so this build suits single-shot work better than long multi-turn chat. If you need prompt-cache reuse, serve the plain STRIX quant without --spec-type.
• Thinking-off numbers only. Reasoning-on was not benchmarked; the scores above are floors.
• Single runs, greedy, pass@1. No repeats, so treat 1–2 point differences as noise.
• HumanEval base is saturated at this capability level — HumanEval+ is the more discriminating number.
• --spec-type draft-mtp works with --parallel > 1 (per-slot draft state is maintained). But n-max should come down as concurrency rises, since the verify batch scales as n-max × active slots.

How it was built

Quantized from a BF16 conversion of the base model with the base model's own imatrix, using the fork's llama-quantize at Q4_0_ROCMFP4_STRIX (a selective recipe: attn_k/attn_v get the quality layout, attn_q/attn_output/ffn_*_exps get the fast layout, token_embd gets Q6_K). The MTP head — which ships inside the base model rather than as a separate draft — was then grafted in at Q8_0.

Credit to Kwaipilot for the base model, and to the ROCmFPX fork for the FP4 kernels.