README.md

---
license: apache-2.0
base_model: Kwaipilot/KAT-Coder-V2.5-Dev
tags:
- cmf
- moe
- code
- expert-pruning
- rust
pipeline_tag: text-generation
---

# KAT-Coder-V2.5 — CMF coding specialist (expert-defragmented MoE)

**A 34.7B-A3B MoE coder in a single 12.7 GB file — 35% smaller than the
full quantized model, ×1.8 faster on a 24 GB MacBook, +2.8% code
perplexity.** This is [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev)
(Qwen3.6-35B-A3B architecture: 40 layers, 30 of them GatedDeltaNet
linear attention, 256 routed experts top-8 + a shared expert),
quantized to 4-bit tiles and then **physically stripped of the experts
that code generation never routes to**.

| file | size | held-out code ppl | decode, M4 24 GB | prefill |
|---|---:|---:|---:|---:|
| full q4t model | 19.6 GB | 5.058 | 7.6 tok/s | 6.1 tok/s |
| **this file** | **12.7 GB** | 5.198 (+2.8%) | **13.7 tok/s (×1.8)** | **20.0 tok/s (×3.3)** |

The speedup is not a kernel trick: the full model does not fit a 24 GB
machine and pages from disk on every token, while the specialist
resides in memory entirely. On machines with plenty of RAM the two
decode at similar speed and you simply save the 7 GB.

## How to use

[CMF](https://github.com/infosave2007/cmf) is a single-file LLM format
with a small pure-Rust runtime — no torch, no CUDA install, no Python.
Install the CLI (one command, GPU backends included) and run:

```sh
cargo install cortiq-cli   # or a release binary: github.com/infosave2007/cmf/releases

cortiq run KAT-Coder-V2.5-CMF.cmf \
    --prompt "Write a Python function that checks if a number is prime." --max-tokens 300

cortiq serve KAT-Coder-V2.5-CMF.cmf --port 8080   # OpenAI-compatible API + dashboard
cortiq bench KAT-Coder-V2.5-CMF.cmf               # measure on your hardware
```

The tokenizer and chat template are embedded in the file — `run` is a
real chat turn out of the box.

GPU: `CMF_GPU=1` enables the GPU path. On discrete Vulkan/DX12 cards
the entire decode — GatedDeltaNet recurrence, attention, the MoE
router, the on-device top-k expert selection and every selected
expert — executes as **one GPU submit per token** (the full-model
variant of this pipeline decodes at 32.8 tok/s on an RTX 5090 vs 14.4
on its 32-core host CPU). On Apple silicon a runtime probe arbitrates
Metal against CPU per operation and keeps whichever wins. Long
contexts: `--o1 all` converts the 10 softmax-attention layers into a
constant-memory streaming operator (KV+state at 4K context: 238 → 83 MB).

## The technology

MoE expert usage turns out to be **strongly task-conditional**.
Measured on the full KAT-Coder: the top-64 expert sets selected for
code vs for natural-language prose overlap with a Jaccard index of just
**0.25** — near-disjoint working sets — and a code-derived expert mask
captures only ~39% of prose routing mass. A model serving one task
therefore carries hundreds of experts it never routes to.

The pipeline that produced this file (two commands, reproducible with
[`cortiq` ≥ 0.5.27](https://github.com/infosave2007/cmf)):

1. **Record the routing field.** A teacher-forced pass over a
   representative code corpus with `CMF_MOE_STATS=stats.json` records
   per-layer expert-selection frequencies — an empirical routing
   field over the expert lattice (the "B-field" of the CMF patent
   family's claim 12).
2. **Drop what the task never uses.** `cortiq moe-defrag model.cmf
   --stats stats.json --cover 0.95` keeps, per layer, the smallest
   top set of experts reaching 95% of the recorded routing mass
   (mean 158 of 256 per layer here; 3,906 experts dropped in total),
   renumbers the kept experts into a contiguous prefix, slices the
   router's rows to match, and writes a standard `.cmf`. At
   inference the router's softmax renormalizes over the kept set —
   no retraining, weights byte-identical to the quantized originals.

The same restriction can be previewed at runtime without rewriting the
file (`CMF_MOE_MASK=stats.json CMF_MOE_MASK_COVER=0.95`), which is how
the perplexity gate above was measured before the physical cut — the
two are mathematically identical.

**Scope, honestly.** This is a *code* specialist: calibration was
Rust-heavy source code, and quality outside the calibrated task
degrades by design (prose routes to experts that are no longer there).
For general-purpose use, take the full model through
[the step-by-step guide](https://github.com/infosave2007/cmf/blob/master/docs/KAT_CODER.md)
— it covers GGUF → CMF conversion, Vulkan and Metal, and carving your
own specialist for any task with your own corpus.

## Provenance

- Base model: [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) (Apache-2.0)
- Quantized source: [bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF](https://huggingface.co/bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF) (Q4_K_M), imported to CMF q4_tiled by `cortiq import-gguf` (pure Rust; every llama.cpp storage convention undone on import)
- Expert defrag: `cortiq moe-defrag`, cover 0.95, code-calibrated (3,072 tokens of Rust source)
- Runtime: [github.com/infosave2007/cmf](https://github.com/infosave2007/cmf) (Apache-2.0), `cortiq` ≥ 0.5.27

## Ecosystem

- **Engine / converter / format spec**:
  https://github.com/infosave2007/cmf — the pure-Rust runtime this
  file targets (`cargo install cortiq-cli`).
- **[CMF Mobile](https://github.com/infosave2007/cmfmobile)** — a
  Flutter app on the same runtime: local on-device chat, or the phone
  as an OpenAI-compatible server. This 12.7 GB specialist wants a
  desktop's RAM; on phones pick a smaller CMF build (e.g.
  [Bonsai-1.7B](https://huggingface.co/infosave/Bonsai-1.7Bcmf) or
  [Nanbeige 4.2 3B](https://huggingface.co/infosave/Nanbeige4.2-3Bcmf)).