--- license: apache-2.0 language: - en - zh base_model: Kwaipilot/KAT-Coder-V2.5-Dev base_model_relation: quantized pipeline_tag: text-generation tags: - gguf - quantized - imatrix - code - coding - agent - agentic-coding - qwen - qwen3.6 - qwen3.6-35b-a3b - moe - mixture-of-experts - 2-bit - iq2_xxs - q2_k - q8_0 - q36 - quarkstar - vulkan - metal - antirez - radv - bc-250 - conversational --- # KAT-Coder-V2.5-Dev — 11.7 GB Asymmetric GGUF for QuarkStar These GGUF weights are built for [QuarkStar](https://github.com/Ninnix/q36), a small native inference engine specialized for Qwen3.6-35B-A3B. This repository contains an imatrix-calibrated GGUF quantization of [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev), a 35B-A3B Mixture-of-Experts model focused on agentic coding and repository-level software engineering. KAT-Coder-V2.5-Dev is post-trained from [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B). It is designed for tasks such as: * understanding unfamiliar repositories; * locating and fixing bugs; * editing multiple files; * using shell, search and file tools; * running and interpreting tests; * implementing features from issue descriptions; * long-horizon coding-agent workflows. The quantization recipe follows the asymmetric MoE strategy explored by Salvatore Sanfilippo in [DS4](https://github.com/antirez/ds4): aggressively compress the routed experts, which contain most of the model weights, while retaining substantially higher precision for the attention path, shared experts and output tensors. The resulting model is approximately **11.7 GB**, making a 35B-class coding model practical on machines with **16 GB of unified memory**. > This is a quantized derivative. Refer to the upstream KAT-Coder model card and technical report for training details, intended use, benchmark methodology and model limitations. ## Model details | Property | Value | | -------------------------- | --------------------------------------- | | Upstream model | `Kwaipilot/KAT-Coder-V2.5-Dev` | | Foundation model | `Qwen/Qwen3.6-35B-A3B` | | Architecture | Qwen3.6 MoE / HF `qwen3_5_moe` | | Total parameters | Approximately 35B | | Active parameters | Approximately 3B per token | | Transformer layers | 40 | | Routed experts | 256 | | Experts selected per token | 8 | | Maximum upstream context | 262,144 tokens | | Format | GGUF | | Calibration | Importance matrix / imatrix | | Primary use | Coding and agentic software engineering | | Quantized size | Approximately 11.7 GB | The upstream context limit is an architectural maximum. Practical context size depends on available memory, KV-cache precision, runtime buffers and the inference engine. ## Files | File | Size | Quantization | Purpose | | ------------------------------------------------------------------- | ------: | -------------------------------------- | ------------------------------------------ | | `KAT-Coder-V2.5-Dev-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf` | 11.7 GB | Asymmetric `IQ2_XXS` / `Q2_K` / `Q8_0` | Compact local coding and agentic inference | SHA-256: ```text 13e3f7c3f9fc3b6dc85b63e9737dce9ea049f2759db0fb269b0fc43b1b7c0a90 ``` This repository currently contains one language-model GGUF. It does not include a separate multimodal projector or MTP speculative-decoding model. ## Quantization recipe Unlike a uniform 2-bit quantization, this model applies low precision primarily to the routed MoE experts. | Tensor class | Quantization | | -------------------------------------------------------- | ---------------------------- | | Routed expert gate projections | `IQ2_XXS` | | Routed expert up projections | `IQ2_XXS` | | Routed expert down projections | `Q2_K` | | Attention projections | `Q8_0` | | Shared experts | `Q8_0` | | Output head | `Q8_0` | | Routing, normalization, biases and small control tensors | Retained at higher precision | Conceptually: ```text routed gate/up experts -> IQ2_XXS routed down experts -> Q2_K attention projections -> Q8_0 shared experts -> Q8_0 output head -> Q8_0 ``` The routed experts account for most of the model's storage footprint. Compressing them aggressively provides most of the size reduction while preserving higher precision in components that directly control attention, routing and token prediction. The imatrix calibration data is used to make the low-bit expert quantization more sensitive to tensor importance than a purely weight-only conversion. ## Why this quantization? A conventional Q8 GGUF of this model is roughly 37 GB, while the original BF16 weights are roughly 69 GB. This asymmetric release targets a different operating point: * a fully local 35B-class coding model; * approximately 11.7 GB of model weights; * enough remaining memory on a 16 GB unified-memory system for runtime buffers, the operating system and a moderate KV cache; * higher precision for attention, shared experts and output generation; * aggressive compression concentrated in the routed experts. This is especially useful for Mixture-of-Experts models because only a subset of the routed experts is activated for each token. ## QuarkStar / q36 This GGUF targets the Qwen3.6-35B-A3B-compatible tensor layout used by [QuarkStar](https://github.com/Ninnix/q36). QuarkStar is a small native inference engine with Vulkan and Metal backends. It is deliberately specialized rather than being a general-purpose GGUF runner. ### Vulkan ```bash git clone https://github.com/Ninnix/q36 cd q36 make mkdir -p gguf ./download_model.sh kat-coder ./q36 \ -m gguf/KAT-Coder-V2.5-Dev-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf \ --ctx 32768 \ -p "Inspect this repository and explain its architecture." ``` ### Apple Silicon / Metal ```bash make metal ./q36 \ --metal \ -m gguf/KAT-Coder-V2.5-Dev-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf \ --ctx 32768 \ -p "Write a minimal Redis-compatible server in C." ``` ### OpenAI-compatible server ```bash ./q36-server \ -m gguf/KAT-Coder-V2.5-Dev-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf \ --ctx 32768 ``` QuarkStar changes quickly and is intentionally model-specific. Use a revision that supports the KAT-Coder GGUF and its embedded metadata. ## Upstream model * Model: [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) * Technical report: [KAT-Coder-V2.5 Technical Report](https://arxiv.org/abs/2607.05471) * Foundation model: [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) KAT-Coder-V2.5 is trained for autonomous work inside executable repositories rather than only single-turn code generation. The upstream project combines supervised training, reinforcement learning, executable environments, verifiable rewards and multi-teacher on-policy distillation. ## Acknowledgements The asymmetric expert-quantization strategy and runtime direction were inspired by Salvatore Sanfilippo's work on: * [DwarfStar](https://github.com/antirez/ds4) This release also depends on the formats, quantization methods, kernels, tests and engineering knowledge developed by: * [Kwaipilot](https://huggingface.co/Kwaipilot) * [Qwen](https://huggingface.co/Qwen) * [llama.cpp](https://github.com/ggml-org/llama.cpp) * GGML * Salvatore Sanfilippo * Georgi Gerganov * llama.cpp and GGML contributors ## License The upstream KAT-Coder-V2.5-Dev model, its Qwen foundation and these derived GGUF weights are distributed under the **Apache License 2.0**. The QuarkStar / q36 inference-engine source code is distributed separately under the **MIT License**. See the upstream repositories for their complete license text, acceptable-use guidance and attribution requirements: * [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) * [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)