Abiray/KAT-Coder-V2.5-Dev-Imatrix-GGUF

KAT-Coder-V2.5-Dev (GGUF Quantizations)

This repository contains Importance-Matrix (imatrix) quantized GGUF files for Kwaipilot/KAT-Coder-V2.5-Dev.

KAT-Coder-V2.5-Dev is an open-weight, post-trained Mixture-of-Experts (MoE) coding agent model featuring 35B total parameters with 3B activated parameters per token, fine-tuned on top of Qwen3.6-35B-A3B.

⚠️ Note: This open-weight release contains only language-model weights and operates as a text-only model. Vision/multimodal components are not included.

---

📦 Provided GGUF Files

All quantizations in this repository were converted using llama.cpp and optimized using an imatrix (Importance Matrix) calibration file to maintain high performance at lower precision levels.

Filename	Size	Description / Recommendation
KAT-Coder-V2.5-Dev-IQ3_XXS.gguf	14.9 GB	Extreme 3-bit compression. Lowest VRAM/RAM requirement.
KAT-Coder-V2.5-Dev-IQ3_XS.gguf	16.2 GB	High-compression 3-bit quant with imatrix tuning.
KAT-Coder-V2.5-Dev-Q3_K_M.gguf	16.2 GB	Standard 3-bit medium quantization.
KAT-Coder-V2.5-Dev-IQ3_M.gguf	16.9 GB	Balanced 3-bit quantization with strong reasoning retention.
KAT-Coder-V2.5-Dev-IQ4_XS.gguf	18.8 GB	Great choice for systems with ~20 GB VRAM/RAM.
KAT-Coder-V2.5-Dev-IQ4_NL.gguf	19.9 GB	Non-linear 4-bit quantization optimized via imatrix.
KAT-Coder-V2.5-Dev-Q4_K_S.gguf	20.6 GB	Small 4-bit quantization.
KAT-Coder-V2.5-Dev-Q4_K_M.gguf	21.4 GB	Recommended. Optimal balance of speed, size, and perplexity for 24GB GPUs.
KAT-Coder-V2.5-Dev-Q5_K_S.gguf	24.2 GB	5-bit small quantization with higher fidelity.
KAT-Coder-V2.5-Dev-Q5_K_M.gguf	25.0 GB	High Quality. Near-lossless output; suitable for 32GB+ systems.
KAT-Coder-V2.5-Dev-Q6_K.gguf	30.1 GB	High-precision 6-bit quant for power users.
KAT-Coder-V2.5-Dev-Q8_0.gguf	36.9 GB	Virtually identical to full 16-bit float precision.

---

🚀 Quickstart & Usage

Running with llama.cpp

Ensure you are using a recent build of llama.cpp that supports Qwen3 / MoE architectures.

CLI Example:

./llama-cli -m KAT-Coder-V2.5-Dev-Q4_K_M.gguf \
    -p "Write a Python function that implements a binary search tree with deletion." \
    -n 4096 \
    -c 32768 \
    --temp 0.7

Launching an OpenAI-Compatible API Server:

./llama-server -m KAT-Coder-V2.5-Dev-Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    -c 262144 \
    -ngl 99

GUI Frontends (LM Studio, KoboldCpp, Jan)

1. Download your preferred .gguf file from the table above.
2. Place the file inside your local model folder (e.g., ~/.cache/lm-studio/models or KoboldCpp directory).
3. Set your context size up to 262,144 tokens (adjust depending on your available system RAM/VRAM).

---

✨ Original Model Highlights

• SOTA Agentic Coding Performance: Through post-training SFT and RL, KAT-Coder-V2.5-Dev achieves state-of-the-art results among models of similar parameter scales on benchmark tasks like SWE-bench Verified (69.40%).
• Reduced Pathological Behaviors: Reinforcement Learning significantly reduced unwanted behaviors, such as abnormal tool labels (-9pp improvement) and single-turn continuous repetitions (reduced to 0%).
• Preserve Thinking Mode: The model supports retaining historical thinking context across multi-turn interactions, improving agent consistency and saving redundant reasoning tokens.

---

📊 Benchmark Performance

The table below shows the official benchmark evaluation results reproduced in-house by the original authors:

Benchmark	KAT-Coder-V2.5-Dev	Qwen3.5-27B	Qwen3.6-35BA3B	Gemma4-31B	Qwen3.5-35BA3B	Ornith-1.0-35B	Gemma4-26BA4B	Qwen3-Coder-30B
SWE-bench Verified	69.40	68.60	64.40	60.60	58.60	55.80	35.80	31.80
SWE-bench Multilingual	63.00	57.67	57.00	49.33	47.67	51.67	27.33	20.67
SWE-bench Pro	45.96	42.13	40.63	32.97	38.03	34.47	9.58	19.84
Terminal-Bench 2.1	41.02	34.84	32.02	32.59	26.12	35.98	20.94	13.50
PinchBench	93.43	90.71	92.21	85.53	88.75	91.62	82.01	72.30
Scicode	44.20	25.58	37.53	33.19	27.73	30.34	30.84	18.27
KAT-Code-Bench	46.21	44.83	42.76	37.93	35.86	33.10	22.06	15.17

---

🔬 Post-Training Details

KAT-Coder-V2.5-Dev follows a two-stage post-training pipeline built on top of Qwen3.6-35B-A3B:

1. Supervised Fine-Tuning (SFT): Fine-tuned on 127K curated agentic and coding examples.
2. Reinforcement Learning (RL):
   • Token-in-Token-out (TITO) Consistency: Eliminates off-policy training discrepancies caused by tokenizer or chat-template changes.
   • Truncated Importance Sampling (TIS): Mitigates policy staleness during asynchronous rollout collection.
   • Reliable Execution Feedback: Built using verified sandbox execution for dense and reliable reward signals.
   • Specific Penalties: Introduced targeted reward penalties against abnormal parallel tool calling (70+ tool calls in one turn), failed calls, and loops.

---

📜 Citation

If you use KAT-Coder-V2.5-Dev or these GGUF quantizations in your work, please cite the technical report:

@misc{katcoder_v25_2026,
  title={{KAT-Coder-V2.5 Technical Report}},
  author={{KwaiKAT Team}},
  year={2026},
  month={July},
  eprint={2607.05471},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={[https://arxiv.org/pdf/2607.05471](https://arxiv.org/pdf/2607.05471)}
}

---

Original model created by the KwaiKAT Team / Kwaipilot. Quantized to GGUF format by Abiray.