README.md

metadata

license: apache-2.0
base_model: Kwaipilot/KAT-Coder-V2.5-Dev
tags:
  - qwen3_5_moe
  - code
  - mxfp4
  - compressed-tensors
  - vllm
  - xpu
library_name: vllm

KAT-Coder-V2.5-Dev-MXFP4

MXFP4 (compressed-tensors mxfp4-pack-quantized: E2M1 group-32 symmetric, E8M0 scales) quantization of Kwaipilot/KAT-Coder-V2.5-Dev (35B-A3B MoE, 40 layers / 256 experts / 8 active), built for the vLLM XPU backend (CompressedTensorsW4A4Mxfp4 + XPUExpertsMxFp4 on Intel Arc / Xe2).

Contents

• Trunk: all 30,720 routed-expert projections (per-expert gate/up/down) + 150 linear-attention projections, quantized with the inference kernel's own quantizer (torch.ops.vllm.xpu_mxfp4_quantize — format match by construction). Everything else (self-attn, norms, shared experts, embed, lm_head) is unchanged BF16.
• MTP draft block (mtp.*, from SpectreSystems/KAT-Coder-V2.5-Dev-MTP): 768 experts unfused and MXFP4-quantized, 17 support tensors BF16 — ready for --speculative-config '{"method":"mtp","num_speculative_tokens":2}'.
• text_config.mtp_num_hidden_layers is set to 1 (the base checkpoint ships 0, which would build a zero-layer draft).

63,756 tensors / 29 shards / 19.60 GiB. SHA256SUMS included.

Measured quality of the quantization

• In-conversion spot checks (kernel dequantizer, 1-in-64): max mean rel-err 0.134.
• Independent post-hoc CPU audit of 2,075 pairs vs fresh BF16: mean 0.116, max 0.157 on a single pair (outlier-group concentrated, median-normal).

Measured serving (Intel Arc 140V, vLLM 0.30.0+xpu, eager)

Config	1024/128/1 decode
Eager	~13–14 tok/s (E-cores; ~19 expected on P-cores)
MTP K=1	23.22 tok/s
MTP K=2	24.82 tok/s

10/10 golden-suite prompts clean, 0 errors. Example launch:

vllm serve KAT-Coder-V2.5-Dev-MXFP4 --served-model-name KAT-Coder-V2.5-Dev \
  --max-model-len 4096 --max-num-seqs 1 --gpu-memory-utilization 0.68 \
  --language-model-only --enforce-eager \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --kv-cache-memory-bytes 500000000

Provenance

Built with the quantize_kat_mxfp4.py / quantize_kat_mtp_mxfp4.py / finalize_kat_mxfp4.py / audit_kat_mxfp4.py pipeline (see alpha2beta/vllm.xpu). Base weights: Kwaipilot/KAT-Coder-V2.5-Dev (Apache-2.0). MTP weights: SpectreSystems/KAT-Coder-V2.5-Dev-MTP. Reference schema: pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4.