README.md

---
license: apache-2.0
base_model: Kwaipilot/KAT-Coder-V2.5-Dev
base_model_relation: quantized
pipeline_tag: text-generation
library_name: gguf
tags:
  - gguf
  - moe
  - apex
  - quantized
  - imatrix
  - torch-imatrix
  - qwen3_5_moe
  - coding
  - agentic-coding
  - llama.cpp
---

# KAT-Coder-V2.5-Dev — APEX GGUF (torch imatrix)

MoE-aware, mixed-precision **APEX** quantization of
[Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) — an
agentic-coding fine-tune built on the Qwen3.6-35B-A3B backbone (`qwen3_5_moe`: 40 layers,
256 routed + 1 shared expert, hybrid GatedDeltaNet linear-attention + periodic full
attention). Kwaipilot reports SWE-bench Verified 69.40 for this size class (vs
Qwen3.6-35B-A3B's 64.40).

Two architectural notes vs. the stock Qwen3.6 base this is built on:

- **No MTP/NextN head** — this checkpoint ships `mtp_num_hidden_layers: 0`, unlike
  Qwen3.6's fused MoE MTP layer. Nothing to quantize or skip; the head simply isn't there.
- **Text-only release** — the upstream config carries a `vision_config` block (a
  Qwen3-VL-style SigLIP tower) with no corresponding weights in the checkpoint; Kwaipilot's
  own README confirms the vision/multimodal components are not included in this release.
  Conversion here uses the text-only path (`Qwen3_5MoeForCausalLM`), which only ever reads
  the model's `text_config` — irrelevant to output correctness either way, but avoids
  routing through vision-aware conversion code for a component that has no weights.

Checkpoint stores MoE experts as fully separate per-expert tensors
(`mlp.experts.{e}.{gate,up,down}_proj.weight` × 256), fused into the `gate_up_proj`/`down_proj`
layout during imatrix generation and GGUF conversion.

## The `-torch` imatrix — what it is

Same rationale as this project's Qwen3.6 quants (identical `qwen3_5_moe` architecture):
`llama-imatrix` is impractical here because the GatedDeltaNet linear-attention is a serial
state-space recurrence — the imatrix collection callback breaks the GPU path, falling back
to a single CPU thread. `kat-coder.imatrix` is a standard GGUF-format importance matrix
(`in_sum2` + `counts` per tensor), generated with a band-serialized PyTorch forward pass
over a general text/code corpus, bit-compatible with `llama-quantize --imatrix`.

## Sizes

| tier | precision | file size | bits/weight | wikitext PPL | vs bf16 |
|---|---|---|---|---:|---:|
| bf16 (reference) | full | 64.61 GiB | 16.01 | 7.028 | — |
| **APEX-i-quality** | Q6_K/Q4_K/Q3_K by role+band | 19.23 GiB | 4.65 | 7.117 | +1.27% |
| **APEX-dynamic (16GB-card)** | Q6_K/Q3_K/IQ3_XXS/IQ2_XXS by role+band | 12.13 GiB | 2.94 | 7.543 | +7.33% |
| IQ3_XXS (stock, imatrix-guided) | uniform | 12.68 GB | ~2.98 | 7.881 | +12.13% |

All three tiers gated coherent with 3/3 chained tool-calling (see `TOOL_CALLING.md`).

**APEX-dynamic beats the stock IQ3_XXS baseline on both axes at once here** — smaller
(12.13 vs 12.68 GiB) *and* lower PPL (7.543 vs 7.881) — a cleaner result than this
toolkit's Laguna-XS run, where the same method traded size for a small PPL cost. Whether
role/depth-aware allocation beats a well-tuned stock quant appears to depend on the
specific architecture and calibration, not something to assume either way without
measuring — this is the honest result for this model.

**i-quality** is the pick if you're not VRAM-constrained to 16GB — within 1.3% of full
bf16 quality at under a third of the size.

## Structural allocation (dynamic-APEX)

Both APEX tiers use `dynamic_apex.py`'s structural mode: routed experts are bucketed into
edge / near / middle depth-bands (edge = first/last layers, generally most load-bearing)
and allocated decreasing precision toward the middle:

- **i-quality** (19.23 GiB, 4.65 bpw): routed edge → Q6_K, near → Q4_K, middle → Q3_K.
- **APEX-dynamic** (12.13 GiB, 2.94 bpw): routed edge → Q3_K, near → IQ3_XXS, middle → IQ2_XXS.

Attention, shared-expert, and router tensors stay at higher fixed precision (Q6_K–Q8_0)
in both tiers.

## Files

- `kat-coder.imatrix` — the imatrix used for every quant below.
- `KAT-Coder-V2.5-Dev-APEX-i-quality.gguf` (19.23 GiB) — recommended default; best
  quality-per-byte of the tiers here.
- `KAT-Coder-V2.5-Dev-APEX-dynamic.gguf` (12.13 GiB) — role-aware allocation targeting a
  16GB-card budget; also the best-quality option at this size point.
- `KAT-Coder-V2.5-Dev-IQ3_XXS.gguf` (12.68 GB) — stock imatrix-guided baseline, included
  for the size/PPL comparison above.
- `TOOL_CALLING.md` — reproducible chained tool-calling transcript.

## Attribution

- Base model: **Kwaipilot** — [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev).
- APEX recipe & toolkit: **LocalAI** — [localai-org/apex-quant](https://github.com/localai-org/apex-quant).
- Quantization engine: **llama.cpp** ([ggml-org](https://github.com/ggml-org/llama.cpp)).

Unofficial community quantization; not affiliated with or endorsed by Kwaipilot.