KAT-Coder-V2.5-Dev-oQ4e-mtp

This model was quantized using oQ (oMLX v0.5.4) mixed-precision quantization.

• Base model: Kwaipilot/KAT-Coder-V2.5-Dev
• MTP head: Qwen/Qwen3.6-35B-A3B (original MTP head grafted)
• Chat template: Qwen Fixed Chat Template (v21) — fixes KV cache optimization, thinking mode control, tool calling, and agent loop issues. Original backed up as chat_template.jinja.bak.

Quantization details

• Model type: qwen3_5_moe
• Bits: 4
• Group size: 64
• Format: MLX safetensors
• MTP: Grafted from Qwen3.6-35B-A3B (bf16 original)
• Calibration: oQ4e (enhanced, imatrix-based)

Environment

• Hardware: M5 MacBook Air 32GB
• Inference Framework: oMLX v0.5.4
• Max Concurrent Requests: 4
• Settings:
  • Thinking: Disabled
  • TurboQuant KV Cache: Enabled (4-bit)
  • Lightning MTP: Enabled (key speed improvement)

Performance Benchmarks

Note: Results are for reference only and may vary depending on hardware, software configuration, and workload.

Single Request Results

Test	TTFT(ms)	TPOT(ms)	pp TPS	tg TPS	E2E(s)	Throughput	Peak Mem
pp1024/tg128	1180.7	19.88	867.3 tok/s	50.7 tok/s	3.723	309.4 tok/s	21.72 GB
pp4096/tg128	4048.1	22.43	1011.8 tok/s	44.9 tok/s	6.926	609.8 tok/s	22.47 GB

Continuous Batching (pp1024 / tg128)

Batch	tg TPS	Speedup	pp TPS	pp TPS/req	TTFT(ms)	E2E(s)
1x	50.7 tok/s	1.00x	867.3 tok/s	867.3 tok/s	1180.7	3.723
2x	61.1 tok/s	1.21x	802.5 tok/s	401.3 tok/s	2552.0	6.743
4x	86.4 tok/s	1.70x	790.3 tok/s	197.6 tok/s	5033.6	11.108

Intelligence Benchmark

Note: Each benchmark round tests only 30 questions. Results are for reference only.

Benchmark	Accuracy	Correct	Total	Time(s)	Think
MMLU	70.0%	21	30	53.4	No
TRUTHFULQA	96.7%	29	30	17.1	No
GSM8K	93.3%	28	30	83	No
MATHQA	43.3%	13	30	49.5	No
HUMANEVAL	90.0%	27	30	125.9	No