README.md

---
library_name: transformers
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE
pipeline_tag: image-text-to-text
base_model:
- Kwaipilot/KAT-Coder-V2.5-Dev
tags:
- unsloth
- qwen
- qwen3_5
- GGUF
- llama.cpp
- MTP
datasets:
- tristandruyen/calibration_data_v5_rc
metrics:
- perplexity 
---

# KAT-Coder-V2.5-Dev-48Gb
## Overview

This repository hosts two production-ready `Q8_0` quantizations of the exceptional Qwen3.6-35B-A3B fine-tune made by [Kwaipilot](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev/tree/main), specifically optimized to **run within 48 GB of VRAM** (dual RTX 3090s via NVLink). This repository represents a change of objectives in my quantization work. You will find here two models, one made for maximum accuracy and one for multiple servings (up to 4 at 131072 context window each, 2 at 262144). Both were thoroughly quantized using specialized schemas and a high quality iMatrix. 
The weights were calibrated using the great iMatrix file [calibration_data_v5_rc](https://gist.github.com/tristandruyen/9e207a95c7d75ddf37525d353e00659c) to precisely target layer-wise quantization sensitivity. On the tested hardware, this configuration achieves up to **~6,400 t/s prefill** and **~140 t/s decode** throughput. The `Q8_0` format was selected for its optimal speed-to-accuracy ratio, lower-precision variants (e.g., Q6_K) showed a 5 to 10% throughput drop alongside reduced long-context stability. By preserving critical precision in attention and KV cache operations, these quantizations enable stable inference at the model’s maximum context window.

> *Note: This model is NOT compatible with Vision OR MTP.*

## Research & Methodology
### Selective precision Quantization for high-speed inference
#### Q8_0-Plus - the capacity monster

This GGUF holds the accuracy-focused model. Using Q8_0 on the experts and F16 where it trully matters, it achieves the highest perplexity possible on dual 24Gb GPUs. 
The schema identified was :

| Tensor Group | Specific Tensors / Patterns | Format |
| :--- | :--- | :---: |
| **Embeddings & Output** | `token_embd`, `output` | `F16` |
| **Attention** | `attn_qkv`, `attn_q`, `attn_k`, `attn_v`, `attn_gate`, `attn_output` | `F16` |
| **SSM (State Space Model)** | `ssm_alpha`, `ssm_beta`, `ssm_out` | `F16` |
| **Feed-Forward / Shared Experts** | `ffn_gate_inp`, `ffn_gate_inp_shexp`, `ffn_gate_shexp`, `ffn_up_shexp`, `ffn_down_shexp` | `F16` |
| **MoE Experts** | `ffn_gate_up_exps`, `ffn_down_exps` | `Q8_0` |

Therefore, **87.39%** of the model is **`Q8_0`**, 12.39% is **`F16`**.

#### Q8_0-Opt - the agentic harness ready version

This mostly `Q8_0` schema was entirely design to achieve *2 or 4 parallel slots* while retaining as much of the BF16 base model's capacities. The goal was to enable users to use it in any agentic harness and have it used in parallel subagents. This will enable users to reduce time spent waiting for the LLM to calculate prefill.
The schema used was :

| Tensor Group | Specific Tensors / Patterns | Format |
| :--- | :--- | :---: |
| **Output Layer** | `output` | `F16` |
| **Attention** | `attn_output` | `F16` |
| **All Other Tensors** | Embeddings, MoE Experts, SSM, Feed-Forward & Gates | `Q8_0` |

Here, **99.79%** of the model is **`Q8_0`**, and 0.21% is **`F16`**.

### iMatrix Calibration

The model was calibrated using the very well, and rightfully, known calibration data [`calibration_data_v5_rc`](https://gist.github.com/tristandruyen/9e207a95c7d75ddf37525d353e00659c). I tested several iMatrix dataset, including of my own making, but this file achieved the best results. For better coding capacity retaintion, the final 'combined' iMatrix file includes a coding and agentic focused dataset.

### SSM conv1d outlier

While investigating the model for a future fine-tuning project, I came across several reports discussing a potential long-context instability. The issue was notably documented by the Reddit user EvilEngineer and later referenced in the Hugging Face repository for [AEON-7/Ornith-1.0-35B-AEON-Ultimate-Uncensored-BF16](https://huggingface.co/AEON-7/Ornith-1.0-35B-AEON-Ultimate-Uncensored-BF16).
According to these reports, the instability only manifests during very long conversations or extended agentic workloads. It has been linked to an unusually large outlier in the linear_attn.conv1d.weight tensor, whose standard deviation is approximately 60% higher than the median of the corresponding tensors.

This GGUF adds the proposed fix by applying the correction directly to the original Safetensors checkpoint before conversion.

### Fusion of the gate_exps and up_exps tensors

To maximize inference throughput and improve execution stability, this quantization was produced using llama.cpp's --fuse-gate-up-exps option during convert_hf_to_gguf.py.

This optimization fuses the MoE `gate_exps` and `up_exps` tensors into a single tensor, reducing the number of GPU kernel launches and lowering memory traffic during inference. It does not modify the model's behavior or capabilities, but can improve efficiency, particularly on consumer GPU backends.

### Accuracy Analysis

To evaluate the precision loss after the mixed-quantization scheme, I compared the perplexity on Wiki-Text-raw and a custom made dataset, composed of code mainly found on llama.cpp's repo on both versions.

#### Q8_0-Plus :

| Metric | Code | WikiText-Raw |
| ------------------- | -------------------: | -------------------: |
| **Base Mean Perplexity** | **1.990067 ± 0.006650** | **6.870226 ± 0.045554** |
| **Quant Mean Perplexity** | **1.991305 ± 0.006668** | **6.878675 ± 0.045691** |
| **PPL(Q) / PPL(Base)** | **1.000622 ± 0.000195** | **1.001230 ± 0.000330** |
| Cor(ln PPL) | 99.83% | 99.88% |
| **Mean KLD** | **0.003476 ± 0.000058** | **0.005141 ± 0.000047** |
| 99.9% KLD | 0.182350 | 0.205312 |
| 99.0% KLD | 0.047081 | 0.054726 |
| 95.0% KLD | 0.014844 | 0.018677 |
| Median KLD | 0.000138 | 0.001748 |
| Max KLD | 9.155661 | 2.238823 |
| **RMS Δp** | **2.110 ± 0.026 %** | **2.063 ± 0.024 %** |
| **Same Top-p** | **98.609 ± 0.022 %** | **96.895 ± 0.045 %** |

**Key Findings:**
* **Higher Distribution Fidelity:** Demonstrates tight KL-divergence (`0.003476` on Code, `0.005141` on WikiText) and top-p sampling agreement up to **98.61%**, outperforming the `Opt` variant in logit alignment.
* **Near-Lossless Perplexity:** Minimal PPL increase over base (**+0.001238** Code / **+0.008448** WikiText), maintaining baseline reasoning and generation quality.

#### Q8_0-Opt :

| Metric | Code | WikiText-Raw |
| ------------------- | -------------------: | -------------------: |
| **Base Mean Perplexity** | **1.990067 ± 0.006650** | **6.870226 ± 0.045554** |
| **Quant Mean Perplexity** | **1.991062 ± 0.006661** | **6.879145 ± 0.045687** |
| **PPL(Q) / PPL(Base)** | **1.000500 ± 0.000241** | **1.001298 ± 0.000390** |
| Cor(ln PPL) | 99.74% | 99.83% |
| **Mean KLD** | **0.005207 ± 0.000079** | **0.007361 ± 0.000071** |
| 99.9% KLD | 0.256829 | 0.275173 |
| 99.0% KLD | 0.069576 | 0.076752 |
| 95.0% KLD | 0.022196 | 0.026565 |
| Median KLD | 0.000202 | 0.002576 |
| Max KLD | 10.033477 | 2.940986 |
| **RMS Δp** | **2.556 ± 0.030 %** | **2.450 ± 0.028 %** |
| **Same Top-p** | **98.302 ± 0.024 %** | **96.285 ± 0.049 %** |

**Key Findings:**
* **Slightly Tight Code Perplexity:** Achieves a marginally lower PPL ratio on Code (**1.000500**) compared to `Plus`, though with slightly wider tail-end KL-divergence.
* **Robust Quantization:** Maintains strong probability retention (**98.30%** top-p match on Code) while offering an alternative quantization layout.

## Recommended Usage

To replicate the optimal performance (262K context, F16 K-Cache, Multi-GPU) using [`llama.cpp`](https://github.com/ggml-org/llama.cpp), use the following `llama-server` command.
Note the specific use of `--split-mode tensor` and `--tensor-split 1,1` for optimal PCIe bandwidth management across dual RTX 3090s. This command appeared to be the best one I could come across using an NVLink.

*Q8-Plus* :

``` bash
/path/to/llama.cpp/build/bin/llama-server \
    -m /path/to/KAT-Coder-V2.5-Dev-Q8_0-Plus.gguf \
    --split-mode tensor \
    --tensor-split 1,1 \
    --host 0.0.0.0 \
    --port 8080 \
    --ctx-size 262144 \
    --parallel 1 \
    --gpu-layers 999 \
    --cache-type-k f16 \
    --cache-type-v f16 \
    --flash-attn on \
    -b 2048 -ub 2048 \
    --alias KAT-V2.5-Dev \
    --jinja
```

*Q8-Opt* :

``` bash
/path/to/llama.cpp/build/bin/llama-server \
    -m /path/to/KAT-Coder-V2.5-Dev-Q8_0-Opt.gguf \
    --split-mode tensor \
    --tensor-split 1,1 \
    --host 0.0.0.0 \
    --port 8080 \
    --ctx-size 524288 \
    --parallel 2 \
    --gpu-layers 999 \
    --cache-type-k f16 \
    --cache-type-v q8_0 \
    --flash-attn on \
    -b 2048 -ub 2048 \
    --alias Qwen3.6-35b \
    --jinja
```

I also developed a proxy to enable users to select thinking or non-thinking behaviors, apply the recommended sampling parameters AND the `"Preserve Thinking"` option. You may find it on my [GitHub](https://github.com/AlexanderKyng/Qwen3.6-reasoning-toggle-proxy).

## Hardware Requirements
- **Target VRAM:** 48 GB (Tested on 2x NVIDIA RTX 3090 24GB).
- **RAM:** Minimum 32GB system RAM (Prompt caching, system overhead and MMPROJ).
- **Context limit:** The command above loads ~9GB of KV cache across the two GPUs. If you experience OOM (Out of Memory) errors, consider reducing `--ctx-size` or using 8-bit K-cache (`--cache-type-k q8_0`).

## Acknowledgments

This project was made possible thanks to the outstanding tools and contributions from the open-source AI community. Special thanks to:

*   **DKwaipilot**: For their immense work done on this exceptionnal fine-tuned version.
*   **The Qwen Team**: For researching and releasing the exceptional Qwen3.6 architecture.
*   **llama.cpp**: Using the new Tensor split mode, it finally achieves extremelly high performances on dual-GPU setups.
*   **Froggeric**: For the great work done on fixing Qwen3.5 and 3.6 chat_template and the base BF16 model used for this project.
*   **tristandruyen**: For the great calibration_data_v5-rc.txt file.
*   **EvilEngineer**: For their research on the SSM Conv1D tensors outliers.