AlexanderKyng/KAT-Codex-V2.5-Dev-48Gb

KAT-Coder-V2.5-Dev-48Gb

Overview

This repository hosts two production-ready Q8_0 quantizations of the exceptional Qwen3.6-35B-A3B fine-tune made by Kwaipilot, specifically optimized to run within 48 GB of VRAM (dual RTX 3090s via NVLink). This repository represents a change of objectives in my quantization work. You will find here two models, one made for maximum accuracy and one for multiple servings (up to 4 at 131072 context window each, 2 at 262144). Both were thoroughly quantized using specialized schemas and a high quality iMatrix. The weights were calibrated using the great iMatrix file calibration_data_v5_rc to precisely target layer-wise quantization sensitivity. On the tested hardware, this configuration achieves up to ~6,400 t/s prefill and ~140 t/s decode throughput. The Q8_0 format was selected for its optimal speed-to-accuracy ratio, lower-precision variants (e.g., Q6_K) showed a 5 to 10% throughput drop alongside reduced long-context stability. By preserving critical precision in attention and KV cache operations, these quantizations enable stable inference at the model’s maximum context window.

Note: This model is NOT compatible with Vision OR MTP.

Research & Methodology

Selective precision Quantization for high-speed inference

Q8_0-Plus - the capacity monster

This GGUF holds the accuracy-focused model. Using Q8_0 on the experts and F16 where it trully matters, it achieves the highest perplexity possible on dual 24Gb GPUs. The schema identified was :

Tensor Group	Specific Tensors / Patterns	Format
Embeddings & Output	token_embd, output	F16
Attention	attn_qkv, attn_q, attn_k, attn_v, attn_gate, attn_output	F16
SSM (State Space Model)	ssm_alpha, ssm_beta, ssm_out	F16
Feed-Forward / Shared Experts	ffn_gate_inp, ffn_gate_inp_shexp, ffn_gate_shexp, ffn_up_shexp, ffn_down_shexp	F16
MoE Experts	ffn_gate_up_exps, ffn_down_exps	Q8_0

Therefore, 87.39% of the model is Q8_0, 12.39% is F16.

Q8_0-Opt - the agentic harness ready version

This mostly Q8_0 schema was entirely design to achieve 2 or 4 parallel slots while retaining as much of the BF16 base model's capacities. The goal was to enable users to use it in any agentic harness and have it used in parallel subagents. This will enable users to reduce time spent waiting for the LLM to calculate prefill. The schema used was :

Tensor Group	Specific Tensors / Patterns	Format
Output Layer	output	F16
Attention	attn_output	F16
All Other Tensors	Embeddings, MoE Experts, SSM, Feed-Forward & Gates	Q8_0

Here, 99.79% of the model is Q8_0, and 0.21% is F16.

iMatrix Calibration

The model was calibrated using the very well, and rightfully, known calibration data calibration_data_v5_rc. I tested several iMatrix dataset, including of my own making, but this file achieved the best results. For better coding capacity retaintion, the final 'combined' iMatrix file includes a coding and agentic focused dataset.

SSM conv1d outlier

While investigating the model for a future fine-tuning project, I came across several reports discussing a potential long-context instability. The issue was notably documented by the Reddit user EvilEngineer and later referenced in the Hugging Face repository for AEON-7/Ornith-1.0-35B-AEON-Ultimate-Uncensored-BF16. According to these reports, the instability only manifests during very long conversations or extended agentic workloads. It has been linked to an unusually large outlier in the linear_attn.conv1d.weight tensor, whose standard deviation is approximately 60% higher than the median of the corresponding tensors.

This GGUF adds the proposed fix by applying the correction directly to the original Safetensors checkpoint before conversion.

Fusion of the gate_exps and up_exps tensors

To maximize inference throughput and improve execution stability, this quantization was produced using llama.cpp's --fuse-gate-up-exps option during convert_hf_to_gguf.py.

This optimization fuses the MoE gate_exps and up_exps tensors into a single tensor, reducing the number of GPU kernel launches and lowering memory traffic during inference. It does not modify the model's behavior or capabilities, but can improve efficiency, particularly on consumer GPU backends.

Accuracy Analysis

To evaluate the precision loss after the mixed-quantization scheme, I compared the perplexity on Wiki-Text-raw and a custom made dataset, composed of code mainly found on llama.cpp's repo on both versions.

Q8_0-Plus :

Metric	Code	WikiText-Raw
Base Mean Perplexity	1.990067 ± 0.006650	6.870226 ± 0.045554
Quant Mean Perplexity	1.991305 ± 0.006668	6.878675 ± 0.045691
PPL(Q) / PPL(Base)	1.000622 ± 0.000195	1.001230 ± 0.000330
Cor(ln PPL)	99.83%	99.88%
Mean KLD	0.003476 ± 0.000058	0.005141 ± 0.000047
99.9% KLD	0.182350	0.205312
99.0% KLD	0.047081	0.054726
95.0% KLD	0.014844	0.018677
Median KLD	0.000138	0.001748
Max KLD	9.155661	2.238823
RMS Δp	2.110 ± 0.026 %	2.063 ± 0.024 %
Same Top-p	98.609 ± 0.022 %	96.895 ± 0.045 %

Key Findings:

• Higher Distribution Fidelity: Demonstrates tight KL-divergence (0.003476 on Code, 0.005141 on WikiText) and top-p sampling agreement up to 98.61%, outperforming the Opt variant in logit alignment.
• Near-Lossless Perplexity: Minimal PPL increase over base (+0.001238 Code / +0.008448 WikiText), maintaining baseline reasoning and generation quality.

Q8_0-Opt :

Metric	Code	WikiText-Raw
Base Mean Perplexity	1.990067 ± 0.006650	6.870226 ± 0.045554
Quant Mean Perplexity	1.991062 ± 0.006661	6.879145 ± 0.045687
PPL(Q) / PPL(Base)	1.000500 ± 0.000241	1.001298 ± 0.000390
Cor(ln PPL)	99.74%	99.83%
Mean KLD	0.005207 ± 0.000079	0.007361 ± 0.000071
99.9% KLD	0.256829	0.275173
99.0% KLD	0.069576	0.076752
95.0% KLD	0.022196	0.026565
Median KLD	0.000202	0.002576
Max KLD	10.033477	2.940986
RMS Δp	2.556 ± 0.030 %	2.450 ± 0.028 %
Same Top-p	98.302 ± 0.024 %	96.285 ± 0.049 %

Key Findings:

• Slightly Tight Code Perplexity: Achieves a marginally lower PPL ratio on Code (1.000500) compared to Plus, though with slightly wider tail-end KL-divergence.
• Robust Quantization: Maintains strong probability retention (98.30% top-p match on Code) while offering an alternative quantization layout.

Recommended Usage

To replicate the optimal performance (262K context, F16 K-Cache, Multi-GPU) using llama.cpp, use the following llama-server command. Note the specific use of --split-mode tensor and --tensor-split 1,1 for optimal PCIe bandwidth management across dual RTX 3090s. This command appeared to be the best one I could come across using an NVLink.

Q8-Plus :

/path/to/llama.cpp/build/bin/llama-server \
    -m /path/to/KAT-Coder-V2.5-Dev-Q8_0-Plus.gguf \
    --split-mode tensor \
    --tensor-split 1,1 \
    --host 0.0.0.0 \
    --port 8080 \
    --ctx-size 262144 \
    --parallel 1 \
    --gpu-layers 999 \
    --cache-type-k f16 \
    --cache-type-v f16 \
    --flash-attn on \
    -b 2048 -ub 2048 \
    --alias KAT-V2.5-Dev \
    --jinja

Q8-Opt :

/path/to/llama.cpp/build/bin/llama-server \
    -m /path/to/KAT-Coder-V2.5-Dev-Q8_0-Opt.gguf \
    --split-mode tensor \
    --tensor-split 1,1 \
    --host 0.0.0.0 \
    --port 8080 \
    --ctx-size 524288 \
    --parallel 2 \
    --gpu-layers 999 \
    --cache-type-k f16 \
    --cache-type-v q8_0 \
    --flash-attn on \
    -b 2048 -ub 2048 \
    --alias Qwen3.6-35b \
    --jinja

I also developed a proxy to enable users to select thinking or non-thinking behaviors, apply the recommended sampling parameters AND the "Preserve Thinking" option. You may find it on my GitHub.

Hardware Requirements

• Target VRAM: 48 GB (Tested on 2x NVIDIA RTX 3090 24GB).
• RAM: Minimum 32GB system RAM (Prompt caching, system overhead and MMPROJ).
• Context limit: The command above loads ~9GB of KV cache across the two GPUs. If you experience OOM (Out of Memory) errors, consider reducing --ctx-size or using 8-bit K-cache (--cache-type-k q8_0).

Acknowledgments

This project was made possible thanks to the outstanding tools and contributions from the open-source AI community. Special thanks to:

• DKwaipilot: For their immense work done on this exceptionnal fine-tuned version.
• The Qwen Team: For researching and releasing the exceptional Qwen3.6 architecture.
• llama.cpp: Using the new Tensor split mode, it finally achieves extremelly high performances on dual-GPU setups.
• Froggeric: For the great work done on fixing Qwen3.5 and 3.6 chat_template and the base BF16 model used for this project.
• tristandruyen: For the great calibration_data_v5-rc.txt file.
• EvilEngineer: For their research on the SSM Conv1D tensors outliers.