README.md

---
library_name: mlx
base_model: Kwaipilot/KAT-Coder-V2.5-Dev
tags:
- mlx
- mlx-vlm
- moe
- coding
- agentic
- swe-bench
- reasoning
- basequant-xl
- kat-coder
pipeline_tag: text-generation
---

# leonsarmiento/KAT-Coder-V2.5-Dev-6bit-XL-mlx

[**KAT-Coder-V2.5-Dev**](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) by [Kwaipilot](https://huggingface.co/Kwaipilot) — a 35B-A3B sparse MoE agentic coding model achieving **69.40 SWE-bench Verified**, quantized for Apple Silicon using the **BaseQuant_XL 6/8-bit** recipe.

KAT-Coder-V2.5-Dev is a post-trained MoE built on Qwen3.6-35B-A3B via two-stage SFT (127K examples) + RL (10 epochs) with token-level consistency (TITO), truncated importance sampling (TIS), and hierarchical reward shaping from harness execution feedback. It ships **text-only weights** (no vision tower) despite the `Qwen3_5MoeForConditionalGeneration` architecture — the MLX conversion uses `mlx_vlm` with `strict=False` to skip the 333 missing `vision_tower.*` parameters.

## Quickstart

```bash
pip install -U mlx-lm
```

```bash
python -m mlx_lm.generate --model leonsarmiento/KAT-Coder-V2.5-Dev-6bit-XL-mlx --max-tokens 256 --temperature 1.0 --top-p 0.95 --prompt "Implement an LRU cache in Python with O(1) get/put."
```

Works with LM Studio. Thinking mode is **on by default** (emits `<think>...</think>`). Pass `enable_thinking=False` for direct answers.

## Quantization Strategy

BaseQuant_XL recipe — precision is allocated by layer importance, not applied uniformly:

| Layers | Bits | Rationale |
|--------|------|-----------|
| `mlp.gate` (router), `shared_expert_gate`, `lm_head`, `shared_expert` | bf16 | Routing decisions and output projection — any quantization noise here causes expert misrouting or output degradation |
| `embed_tokens`, `self_attn`, `linear_attn` | 8-bit | Every-token layers — near-lossless, attention quality preserved |
| `switch_mlp` (routed experts) | 6-bit | Bulk parameters — 256 experts with only 8 active per token; redundancy absorbs quantization noise. 6-bit is the sweet spot for routed experts (higher bits can cause overthinking) |

- **Bits per weight**: ~6.9 · **Total size**: ~28 GB (6 shards) · **Group size**: 64

## Recommended Inference Parameters

### Thinking Mode (default)

| Parameter | Value |
|-----------|-------|
| `temperature` | 1.0 |
| `top_p` | 0.95 |
| `top_k` | 20 |
| `presence_penalty` | 1.5 |
| `reasoning_parser` | qwen3 |

### Instruct Mode (non-thinking)

| Parameter | Value |
|-----------|-------|
| `temperature` | 0.7 |
| `top_p` | 0.8 |
| `top_k` | 20 |
| `presence_penalty` | 1.5 |
| `enable_thinking` | false |

> `preserve_thinking` is supported — set `chat_template_kwargs: {"preserve_thinking": true}` to retain thinking traces from historical messages (beneficial for agentic scenarios).

## Chat Template

The `chat_template.jinja` and `tokenizer_config.json` were updated on **2026-07-30** to match the latest chat template published by Kwaipilot in [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev). This update enforces that system messages must appear at the beginning of the conversation — mid-conversation system messages now raise an exception instead of being silently rendered.

## Model Overview

| Property | Value |
|----------|-------|
| **Architecture** | Qwen3.6 MoE (35B-A3B) — text-only weights |
| **Parameters** | 35B total / ~3B active per token |
| **Experts** | 256 (8 routed + 1 shared) |
| **Context window** | 262,144 tokens native (~1M with YaRN) |
| **Attention** | Hybrid linear + full (full every 4th layer) |
| **Thinking** | On by default (`<think>...</think>`) |
| **Tool calling** | XML-style via `qwen3_coder` parser |
| **License** | Apache-2.0 |

## MLX Benchmarks (n=50, instruct mode)

| Benchmark | KAT-Coder-V2.5-Dev 6bit-XL | Qwen3.6-35B 6bit-XL | Gemma-4 26B 6bit-XL |
|-----------|:---:|:---:|:---:|
| MMLU | 70% | 64% | **76%** |
| MMLU_PRO | 64% | 64% | **82%** |
| ARC_CHALLENGE | **90%** | **90%** | **90%** |
| HUMANEVAL | **92%** | 78% | 98% |
| MBPP | 78% | 78% | **82%** |
| **Average** | **78.8%** | 74.8% | **85.6%** |

KAT-Coder leads Qwen3.6 on 4 of 5 benchmarks — the RL training for agentic coding shows in HUMANEVAL (+14 points). MMLU_PRO trails Gemma-4 significantly (64% vs 82%); KAT-Coder is specialized for SWE, not broad knowledge. Coding is strong but Gemma-4 remains overall champion.

## MATHQA Thinking Efficiency (n=10, seed=42, 8192-token budget)

3-stage methodology: initial run → answer extraction (regex cascade) → re-test failures. Verified accuracy separates sampling/extraction variance from genuine capability gaps.

| Model | Verified Acc | Reason chars/correct | Time/correct |
|-------|:---:|---:|---:|
| Gemma-4 26B 6bit-XL | 10/10 (100%) | 2,420 | 39.8s |
| Qwen3.6-35B 6bit-XL | 10/10 (100%) | 3,562 | **32.1s** |
| Salience-1.5-Pro 6bit-XL | 10/10 (100%) | 5,350 | 47.7s |
| **KAT-Coder-V2.5-Dev 6bit-XL** | 9/10 (90%) | **265** | **7.5s** |

KAT-Coder is the **reasoning efficiency champion** — 9× less reasoning per correct answer than Gemma-4, and 20× less than Salience. Its SWE-focused RL training produces extremely terse, decisive reasoning chains. The one genuine failure (Q636) was an algebra error, not a reasoning deficit. KAT doesn't overthink — it computes, verifies, and answers.

## Preserve Thinking

KAT-Coder is a Qwen3.6 model and supports `preserve_thinking` for agentic multi-turn workflows. Set `chat_template_kwargs: {"preserve_thinking": true}` to retain thinking traces from historical messages — essential for long-horizon SWE tasks where the model needs to recall its own reasoning from prior turns.

**Verification** (number-guessing game, 4 turns): With `preserve_thinking` enabled, KAT stays perfectly consistent — picking a secret number in turn 1 and giving correct higher/lower hints in all subsequent turns (1.4–1.6s each). The model reads its own prior `<think>` blocks rather than re-deriving the answer. Without `preserve_thinking`, the model re-picks a new secret number each turn and reconstructs consistency from visible conversation history — which works for simple games but risks drift in complex agentic tasks.

## Source Model Benchmarks

From Kwaipilot's evaluation (vLLM/SGLang, BF16):

| Benchmark | KAT-Coder-V2.5-Dev | Qwen3.6-35B-A3B (base) |
|-----------|:---:|:---:|
| SWE-bench Verified | **69.40** | 64.40 |
| SWE-bench Multilingual | **63.00** | 57.00 |
| SWE-bench Pro | **45.96** | 40.63 |
| Terminal-Bench 2.1 | **41.02** | 32.02 |
| KAT-Code-Bench | **46.21** | 42.76 |
| Scicode | **44.20** | 37.53 |

Paper: [arXiv:2607.05471](https://arxiv.org/abs/2607.05471).

## Capabilities

- **Agentic coding** — multi-step tool use, repo-scale edits, long-horizon SWE tasks
- **SWE-bench SOTA** — 69.40 Verified at this scale class
- **Deep reasoning** — structured `<think>...</think>` chains with one-token switch to disable
- **Tool calling** — RL-trained for reliable, non-degenerate tool-use patterns

## Flight Simulator — Coded by This Model

A real-world coding benchmark: each model was prompted to write a complete flight simulator from scratch. The resulting code was rendered and recorded.

![flightsimulator](flightsimulator.gif)

KAT-Coder made by Kwaipilot (KwaiKAT Team). MLX quantization by leonsarmiento.