steampunque/Qwen3-Coder-Next-MP-GGUF

Mixed Precision GGUF layer quantization of Qwen3-Coder-Next by Qwen

Original model: https://huggingface.co/Qwen/Qwen3-Coder-Next

The hybrid quant employs different quantization levels on a per layer basis to increase flexibility of trading off performance vs file size. Less parameter bits are used at deep layers and more bits at cortex layers to simulultaneously optimize quantized size and model performance. An extended layer definition E quant Q4_E_H for the model is defined as follows (updated 8/31/2026):

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOSD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q5_K_8666"],[1 ,"Q5_K_6665"],[2 ,"Q4_K_6555"],[3 ,"Q4_K_6555"],
   [4 ,"Q4_K_5555"],[5 ,"Q4_K_5555"],[6 ,"Q4_K_4444"],[7 ,"Q4_K_5554"],
   [8 ,"Q4_K_4444"],[9 ,"Q4_K_4444"],[10,"Q4_K_4444"],[11,"Q4_K_5554"],
   [12,"Q4_K_4444"],[13,"Q4_K_4444"],[14,"Q4_K_4444"],[15,"Q4_K_5554"],
   [16,"Q4_K_4444"],[17,"Q4_K_4444"],[18,"Q4_K_4444"],[19,"Q4_K_5554"],
   [20,"Q4_K_4444"],[21,"Q4_K_4444"],[22,"Q4_K_4444"],[23,"Q4_K_5554"],
   [24,"Q4_K_5554"],[25,"Q4_K_5554"],[26,"Q4_K_5554"],[27,"Q4_K_6554"],
   [28,"Q4_K_5554"],[29,"Q4_K_5554"],[30,"Q4_K_5555"],[31,"Q4_K_6555"],
   [32,"Q4_K_5555"],[33,"Q4_K_5555"],[34,"Q4_K_5555"],[35,"Q4_K_6555"],
   [36,"Q4_K_6555"],[37,"Q4_K_6555"],[38,"Q4_K_6555"],[39,"Q4_K_6555"],
   [40,"Q4_K_6555"],[41,"Q4_K_6555"],[42,"Q5_K_6555"],[43,"Q5_K_6555"],
   [44,"Q5_K_6666"],[45,"Q5_K_6666"],[46,"Q5_K_6668"],[47,"Q6_K_8666"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The layer quants were optimized for 100% success across a small set of code generation test prompts while sized for operation on machines with 48G CPU RAM and one consumer grade GPU (8G VRAM or higher).

Comparison:

Quant	size	PPL	Comment
Q4_K_M	48.5e9	7.6	default embed and output
Q4_E_H	48.6e9	7.7	Q6_K embed Q6_K output extended quant defintions

Usage:

This is a 80B parameter coding optimized moe model with 3B activated parameters. It can be efficiently run by offloading expert tensors to CPU via -ot exps=CPU to open up very large context space on even low VRAM GPUs. The smaller size of the optimally quantized parameters will give an effective boost in CPU processing speed due to reducing the memory BW needed to repeatedly copy them from main memory to SIMD regs.

The model cannot be speculated due to use of some kind of recurrent attention scheme which prohibits it. Later versions of llama.cpp began to support rollback for recurrent models to allow speculation but this feature has not been tested. In this particular case the lack of speculation capability is not severe since experts will have to be run on CPU for most setups which makes speculation impractical anyway. The model was sized at ~48G and should run on a 48G RAM machine. It should be run with all experts offloaded to CPU. Partial offload slows the model down significantly.

Rough performance metrics on a 9900k (128G RAM) and 4070 (12G VRAM)

CPU EXP OFFLOAD	QKV	Context size	gen rate	ot config
all	F16	256K	23	OT="-ot exps=CPU -ngl 99"
all	Q8_0	256k	23	""
4-48	F16	256K	10	OT="-ot blk\.[4-9]|1[0-9]|2[0-9]|3[0-9]|4[0-7].*exps=CPU -ngl 99"

High context performance appears to work verified against a needle in haystack prompt. However, prompt processing is too slow (~70tps) to be practically used on very large prompts without a much stronger CPU or full GPU offload of the model.

Benchmarks:

Code evals for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm.

Download the file from below:

Link	Type	Size/e9 B	Notes
Qwen3-Coder-Next.Q4_E_H.gguf	Q4_E_H	48.6 B	~Q4_K_M size

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040