--- license: apache-2.0 base_model: Kwaipilot/KAT-Coder-V2.5-Dev base_model_relation: quantized pipeline_tag: text-generation language: - en - zh quantized_by: 1337Hero tags: - gguf - kat-coder - quantized - rocm - amd - rdna4 - gfx1201 - vulkan - moe - code - experimental --- # KAT-Coder-V2.5-Dev — ROCmFP4 GGUF (experimental, AMD RDNA4 / gfx1201) Two experimental 4-bit quantizations of [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev), a 34.66B-parameter MoE coding model (256 experts, 8 active, 256K context, `qwen35moe` architecture). Converted and quantized on a Radeon AI PRO R9700 (`gfx1201`, RDNA4). > [!IMPORTANT] > These files do **not** run on upstream llama.cpp, Ollama, LM Studio, or > vLLM. They use the custom `Q4_0_ROCMFP4` tensor layouts and require a > [ROCmFPX](https://github.com/charlie12345/ROCmFPX) build as described below. > Unsupported runtimes should reject the files; if a tool appears to load one > anyway, do not trust the output. > [!WARNING] > Validation was performed on RDNA4 `gfx1201` only: both files load, generate > coherent output, were throughput-benchmarked, and were measured against the > BF16 source for wikitext-2 perplexity. No Strix Halo testing and **no > code-specific or agentic evaluation** — see [What was not > measured](#what-was-not-measured) before relying on either file. ## Which file? | File | Size | Effective BPW | Wikitext-2 PPL | Pick it if | | --- | ---: | ---: | ---: | --- | | `KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4.gguf` | 21.18 GiB | 5.25 | 6.9182 (+1.38%) | You care about output quality. **Recommended for coding.** | | `KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4_STRIX_LEAN.gguf` | 17.32 GiB | 4.29 | 7.1079 (+4.16%) | You need the smaller file or the extra decode speed. | This is a **real tradeoff, not a clean win for either file.** `STRIX_LEAN` is 18% smaller and 13% faster at decode, but gives up three times as much perplexity against the BF16 source. For a coding model — where a single wrong token breaks a program — the plain `Q4_0_ROCMFP4` is the safer default, and 21.18 GiB still fits a 32 GB card comfortably. Take `STRIX_LEAN` if you are memory-constrained (24 GB cards), or if you are throughput-bound and have validated that the quality holds on your own tasks. Its recipe was tuned on `gfx1151`; nothing about the file format is Strix-specific. ## Why the sizes differ from the nominal BPW Both presets apply tensor-aware routing, and on a 256-expert MoE the expert tensors dominate the file. The routing difference between the two is almost entirely one tensor family: | Tensor | `Q4_0_ROCMFP4` | `Q4_0_ROCMFP4_STRIX_LEAN` | | --- | --- | --- | | `ffn_down_exps` | `q6_K` | `q4_0_rocmfp4_fast` | | `ffn_gate_exps` / `ffn_up_exps` | `q4_0_rocmfp4` | `q4_0_rocmfp4_fast` | | `attn_qkv` | `q5_K` | Strix attn K/V recipe | | `token_embd` | `q6_K` | `q5_K` | That is why `Q4_0_ROCMFP4` lands at 5.25 BPW rather than its nominal 4.50 — `ffn_down_exps` alone is roughly a third of the model's parameters. ## Measured throughput — Radeon AI PRO R9700, `gfx1201` `llama-bench`, `pp512` + `tg128`, 3 repetitions, full offload, FlashAttention on, one model resident at a time on an otherwise idle GPU. | Backend | Quant | Prompt fill `pp512` t/s | Decode `tg128` t/s | | --- | --- | ---: | ---: | | Vulkan0 | **`STRIX_LEAN`** | **3278.14 ± 50.90** | **122.27 ± 1.27** | | Vulkan0 | `Q4_0_ROCMFP4` | 3120.99 ± 17.61 | 107.80 ± 1.02 | | ROCm0 | `STRIX_LEAN` | 2598.84 ± 5.06 | 59.39 ± 0.19 | | ROCm0 | `Q4_0_ROCMFP4` | 1787.51 ± 74.26 | 52.51 ± 0.26 | Two results worth acting on: - **Use Vulkan on this hardware.** Vulkan decodes roughly **2× faster** than HIP/ROCm for both files (122 vs 59 t/s on `STRIX_LEAN`) and also leads on prompt fill. This matches ROCmFPX's own Strix Halo findings. - **`STRIX_LEAN` is the faster file** — +13% decode and +5% prefill on Vulkan, +13% decode and +45% prefill on ROCm — but see the quality section below before choosing it on speed alone. No control quant (Q4_K_M or similar) was benchmarked, so these numbers compare the two ROCmFP4 files against each other, not against ordinary GGUF quants. ## Measured quality — wikitext-2 perplexity `llama-perplexity`, full wikitext-2 test set (580 chunks), `-c 512 -b 512`, FlashAttention on, Vulkan. The BF16 source GGUF was measured on the same host with the same settings, split across three GPUs. | File | BPW | PPL | Δ vs BF16 | | --- | ---: | ---: | ---: | | `KAT-Coder-V2.5-Dev-BF16.gguf` (source) | 16.01 | 6.8237 ± 0.04537 | — | | `Q4_0_ROCMFP4` | 5.25 | 6.9182 ± 0.04607 | **+1.38%** | | `Q4_0_ROCMFP4_STRIX_LEAN` | 4.29 | 7.1079 ± 0.04762 | **+4.16%** | Both quants land where you would expect for their bit budgets, and neither is degenerate. The gap between them is larger than the error bars, so it is a real difference and not measurement noise: `STRIX_LEAN` buys its 18% size reduction with roughly 3× the perplexity cost. Perplexity is a weak proxy for coding ability. It measures next-token prediction on English Wikipedia, not code correctness or tool-call formatting. Treat it as a floor check — it rules out a broken quantization, it does not establish that either file codes as well as the source. ## What was not measured - **Coding ability.** No HumanEval, MBPP, or any code benchmark. Wikitext-2 perplexity was measured (see above), but it does not measure code correctness. - **KL-divergence** against the BF16 source. Perplexity only. - **Agentic and tool-calling behavior**, which is the point of a coding model. Untested. - **Any hardware other than `gfx1201`.** Not tested on Strix Halo, RDNA3, RDNA2, or CPU. - **Long context.** Benchmarked at `pp512`/`tg128`; the model claims 262144. Deep-context behavior and KV-cache pressure are unmeasured. - **Batch > 1 / concurrent requests.** Single-stream only. - **A non-ROCmFPX control quant.** The two files were compared to each other, not to Q4_K_M. ## Required runtime ```bash git clone https://github.com/charlie12345/ROCmFPX.git cd ROCmFPX && git checkout main # built and quantized at commit db6844d env JOBS=16 scripts/build-rdna4.sh # -> build-rdna4/ (gfx1201 auto-detected) ``` On a Navi 48 card (RX 9070, 9070 XT, AI PRO R9700) the script builds `gfx1201` automatically. `gfx1200` builds are **not** interchangeable on these cards. ## Example run ```bash ./build-rdna4/bin/llama-server \ -m KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4.gguf \ -dev Vulkan0 \ -ngl 999 \ -fa on \ -c 32768 \ -b 512 -ub 512 \ --jinja ``` `-dev Vulkan0`, not `ROCm0` — see the benchmark table above. Swap in `-dev ROCm0` only if Vulkan is unavailable on your system. `--jinja` is required — the model ships a chat template with `` reasoning blocks. The model has **no MTP/NextN head** (`mtp_num_hidden_layers = 0` in the source config), so ROCmFPX's self-speculative decoding is not available here. ## Artifacts | Field | `STRIX_LEAN` | `Q4_0_ROCMFP4` | | --- | --- | --- | | Size | 18,597,337,248 bytes | 22,741,457,056 bytes | | Effective BPW | 4.29 | 5.25 | | SHA-256 | `857d39a696d448a9349a000ffdd5811c88761df59ec7d5a67e7ada3ac46a8161` | `f87c3f509c487876dc76d8a7606583faa1a3cbd04407b288c36dd2031b2cd92f` | | Quantization | `Q4_0_ROCMFP4_STRIX_LEAN` | `Q4_0_ROCMFP4` | | Importance matrix | none | none | Source: `KAT-Coder-V2.5-Dev-BF16.gguf`, 69,376,637,408 bytes, converted from the upstream `safetensors` release with ROCmFPX's `convert_hf_to_gguf.py` at commit `db6844d`: ```bash python convert_hf_to_gguf.py /path/to/KAT-Coder-V2.5-Dev \ --outtype bf16 --outfile KAT-Coder-V2.5-Dev-BF16.gguf ``` Quantization (same commit): ```bash ./build-rdna4/bin/llama-quantize \ KAT-Coder-V2.5-Dev-BF16.gguf \ KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4_STRIX_LEAN.gguf \ Q4_0_ROCMFP4_STRIX_LEAN 16 ./build-rdna4/bin/llama-quantize \ KAT-Coder-V2.5-Dev-BF16.gguf \ KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4.gguf \ Q4_0_ROCMFP4 16 ``` Verify after download: ```bash sha256sum -c SHA256SUMS ``` ## Notes on the source model The upstream open-weight release ships **language-model weights only** — the vision and multimodal components described in the model card are not included, and the converted GGUFs contain no multimodal projector. Despite the `Qwen3_5MoeForConditionalGeneration` class name, these are text-only files. ## Limitations - Requires the ROCmFPX fork; no upstream llama.cpp compatibility. - Validated on exactly one `gfx1201` host, batch 1, shallow context. - Quality evidence is wikitext-2 perplexity only; no code or agentic evals. - 34.66B MoE: needs ~18–22 GB for weights plus KV cache. Comfortable on a 32 GB card, tight on 24 GB with meaningful context. ## License and attribution - **Base model:** KAT-Coder-V2.5-Dev, Kwaipilot, Apache-2.0. This repository redistributes a converted and quantized derivative under the same license. - **Format and execution path:** the `Q4_0_ROCMFP4` representations and kernels are the work of the [ROCmFPX](https://github.com/charlie12345/ROCmFPX) project, which builds on [llama.cpp](https://github.com/ggml-org/llama.cpp). - **This repository:** the quantized artifacts only. KAT-Coder and related marks belong to their owners. This community quantization is not affiliated with or endorsed by Kwaipilot, AMD, ROCmFPX, or llama.cpp.