| --- |
| license: apache-2.0 |
| base_model: Kwaipilot/KAT-Coder-V2.5-Dev |
| tags: |
| - qwen3_5_moe |
| - code |
| - mxfp4 |
| - compressed-tensors |
| - vllm |
| - xpu |
| library_name: vllm |
| --- |
| |
| # KAT-Coder-V2.5-Dev-MXFP4 |
|
|
| MXFP4 (`compressed-tensors` `mxfp4-pack-quantized`: E2M1 group-32 symmetric, |
| E8M0 scales) quantization of [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) |
| (35B-A3B MoE, 40 layers / 256 experts / 8 active), built for the |
| **vLLM XPU backend** (`CompressedTensorsW4A4Mxfp4` + `XPUExpertsMxFp4` on Intel Arc / Xe2). |
|
|
| ## Contents |
|
|
| - **Trunk**: all 30,720 routed-expert projections (per-expert gate/up/down) + |
| 150 linear-attention projections, quantized with the inference kernel's own |
| quantizer (`torch.ops.vllm.xpu_mxfp4_quantize` — format match by construction). |
| Everything else (self-attn, norms, shared experts, embed, lm_head) is unchanged BF16. |
| - **MTP draft block** (`mtp.*`, from |
| [SpectreSystems/KAT-Coder-V2.5-Dev-MTP](https://huggingface.co/SpectreSystems/KAT-Coder-V2.5-Dev-MTP)): |
| 768 experts unfused and MXFP4-quantized, 17 support tensors BF16 — ready for |
| `--speculative-config '{"method":"mtp","num_speculative_tokens":2}'`. |
| - `text_config.mtp_num_hidden_layers` is set to 1 (the base checkpoint ships 0, |
| which would build a zero-layer draft). |
| |
| 63,756 tensors / 29 shards / **19.60 GiB**. `SHA256SUMS` included. |
| |
| ## Measured quality of the quantization |
| |
| - In-conversion spot checks (kernel dequantizer, 1-in-64): max mean rel-err 0.134. |
| - Independent post-hoc CPU audit of 2,075 pairs vs fresh BF16: mean 0.116, |
| max 0.157 on a single pair (outlier-group concentrated, median-normal). |
| |
| ## Measured serving (Intel Arc 140V, vLLM 0.30.0+xpu, eager) |
| |
| | Config | 1024/128/1 decode | |
| |---|---| |
| | Eager | ~13–14 tok/s (E-cores; ~19 expected on P-cores) | |
| | MTP K=1 | 23.22 tok/s | |
| | MTP K=2 | 24.82 tok/s | |
| |
| 10/10 golden-suite prompts clean, 0 errors. Example launch: |
| |
| ```bash |
| vllm serve KAT-Coder-V2.5-Dev-MXFP4 --served-model-name KAT-Coder-V2.5-Dev \ |
| --max-model-len 4096 --max-num-seqs 1 --gpu-memory-utilization 0.68 \ |
| --language-model-only --enforce-eager \ |
| --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \ |
| --kv-cache-memory-bytes 500000000 |
| ``` |
| |
| ## Provenance |
| |
| Built with the `quantize_kat_mxfp4.py` / `quantize_kat_mtp_mxfp4.py` / |
| `finalize_kat_mxfp4.py` / `audit_kat_mxfp4.py` pipeline |
| (see [alpha2beta/vllm.xpu](https://github.com/alpha2beta/vllm.xpu)). |
| Base weights: `Kwaipilot/KAT-Coder-V2.5-Dev` (Apache-2.0). |
| MTP weights: `SpectreSystems/KAT-Coder-V2.5-Dev-MTP`. |
| Reference schema: `pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4`. |
|
|
