steampunque/Qwen3-Coder-30B-A3B-Instruct-MP-GGUF

Mixed Precision GGUF layer quantization of Qwen3-Coder-30B-A3B-Instruct by Qwen

Original model: https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct

The hybrid quant employs different quantization levels on a per layer basis to increase flexibility of trading off performance vs file size. Less parameter bits are used at deep layers and more bits at cortex layers to simulultaneously optimize quantized size and model performance. For this file the layer quants are as follows (refreshed on 4/26/2026):

Q4_K_L : Q4_K_M + attn_o = q6_k
Q5_K_L : attn_v = q8_0 attn_o = q6_k ffn_d = q6_k
Q6_K_S : Q6_K

  LAYER_TYPES='[
   [0 ,"Q5_K_S"],[1 ,"Q4_K_L"],[2 ,"Q4_K_M"],[3 ,"Q4_K_S"],[4 ,"Q4_K_M"],[5 ,"Q4_K_S"],[6 ,"Q4_K_M"],[7 ,"Q4_K_S"],
   [8 ,"Q4_K_S"],[9 ,"Q4_K_S"],[10,"Q4_K_S"],[11,"Q4_K_S"],[12,"Q4_K_S"],[13,"Q4_K_S"],[14,"Q4_K_S"],[15,"Q4_K_S"],
   [16,"Q4_K_S"],[17,"Q4_K_S"],[18,"Q4_K_S"],[19,"Q4_K_S"],[20,"Q4_K_M"],[21,"Q4_K_S"],[22,"Q4_K_M"],[23,"Q4_K_S"],
   [24,"Q4_K_M"],[25,"Q4_K_S"],[26,"Q4_K_M"],[27,"Q4_K_S"],[28,"Q4_K_M"],[29,"Q4_K_S"],[30,"Q4_K_M"],[31,"Q4_K_S"],
   [32,"Q4_K_M"],[33,"Q4_K_M"],[34,"Q4_K_M"],[35,"Q4_K_M"],[36,"Q4_K_M"],[37,"Q4_K_M"],[38,"Q4_K_M"],[39,"Q4_K_M"],
   [40,"Q4_K_M"],[41,"Q4_K_M"],[42,"Q4_K_M"],[43,"Q4_K_L"],[44,"Q5_K_S"],[45,"Q5_K_M"],[46,"Q5_K_L"],[47,"Q6_K_S"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

These layer quants were optimized for strong performance across a small set of code gen prompts while sized to approximately Q4_K_M bit efficiency.

A second quant is available using new extended K quant layer definitions which provide more flexibility in configuring the attn_v, attn_o, and ffn_d quant levels:

Extended K (QN_E_H) mixed precision layer quant nomenclature:

QN_K_VOD, Q8_0_VOD
N = {2,3,4,5,6}

VOD = attnV:attnO:ffnD
V,O,D = {0,2,3,4,5,6,8,f,F}

VOD MAP:
2:Q2_K, 3:Q3_K, 4:Q4_K, 5:Q5_K, 6:Q6_K, 8:Q8_0, f:F:F16, 0:F32, default QN_K

   LAYER_TYPES='[
   [0 ,"Q5_K_885"],[1 ,"Q5_K_866"],[2 ,"Q4_K_664"],[3 ,"Q4_K_664"],[4 ,"Q4_K_654"],[5 ,"Q4_K_554"],[6 ,"Q4_K_654"],[7 ,"Q4_K_554"],
   [8 ,"Q4_K_654"],[9 ,"Q4_K_554"],[10,"Q4_K_654"],[11,"Q4_K_554"],[12,"Q4_K_654"],[13,"Q4_K_554"],[14,"Q4_K_654"],[15,"Q4_K_554"],
   [16,"Q4_K_654"],[17,"Q4_K_654"],[18,"Q4_K_654"],[19,"Q4_K_654"],[20,"Q4_K_654"],[21,"Q4_K_654"],[22,"Q4_K_654"],[23,"Q4_K_654"],
   [24,"Q4_K_664"],[25,"Q4_K_654"],[26,"Q4_K_664"],[27,"Q4_K_654"],[28,"Q4_K_664"],[29,"Q4_K_654"],[30,"Q4_K_664"],[31,"Q4_K_654"],
   [32,"Q4_K_664"],[33,"Q4_K_664"],[34,"Q4_K_664"],[35,"Q4_K_664"],[36,"Q4_K_664"],[37,"Q4_K_664"],[38,"Q4_K_664"],[39,"Q4_K_664"],
   [40,"Q4_K_665"],[41,"Q5_K_665"],[42,"Q4_K_665"],[43,"Q5_K_665"],[44,"Q5_K_865"],[45,"Q5_K_885"],[46,"Q5_K_886"],[47,"Q6_K_886"]
   ]'

   FLAGS="--token-embedding-type Q5_K --output-tensor-type Q6_K --layer-types-high"

Comparison:

Quant	size	PPL	Comment
Q4_K_M	18.6e9	9.6	default embed and output
Q4_K_H	18.5e9	9.5	Q6_K embed Q6_K output
Q4_E_H	18.2e9	9.5	Q5_K embed Q6_K output

Usage:

This moe model can be efficiently run by offloading expert tensors to CPU via -ot exps=CPU to open up very large context space. The smaller size of the optimally quantized parameters will give an effective boost in CPU processing speed due to reducing the memory BW needed to repeatedly copy them from main memory to SIMD regs. It can also run fully offloaded on GPU via RPC or high VRAM GPU.

The model can be speculated using Qwen2.5-Coder-0.5B-Instruct if the inference platform can support vocabulary translation between draft and target. Example performance on humaneval first problem prompt using a downstream speculator with fixed draft block length ND:

Config	ND	F16 context size	Q8 context size	gen rate
2 4070, RPC	0	~55k tokens	~100k tokens	104tps
2 4070, RPC	4	~45k tokens	~75k tokens	130tps
1 4070, exp CPU	0	~105k tokens	~200k tokens	22 tps
1 4070, exp CPU	4	~90k tokens	~155k tokens	28 tps

These results show speculation to be of marginal benefit with the model.

The model has a native 256k context window which can be extended to 1M with Yarn as follows:

Set base context for yarn rope scale compute to 262144 (256k), then with a context of N tokens > 256k the rope scale = N / 256k.

Example, if N = 512k then rope scale = 2.0.

Then on model start pass --rope-scaling yarn --yarn-orig-ctx 262144 --rope_scale 2.0 (must be ajusted if kv other than 512k)

Later versions of llama.cpp have a bug which soft caps context length to the training context, effectively disabling yarn context extension. Patch server-context.cpp according to https://github.com/ggml-org/llama.cpp/issues/22140 to fix it.

High context performance appears to work verified against a needle in haystack test at 75k tokens.

Benchmarks:

Code evals for the model (prior to refresh) are given here: https://huggingface.co/spaces/steampunque/benchlm.

Download the file from below:

Link	Type	Size/e9 B	Notes
Qwen3-30B-Coder-A3B-Instruct.Q4_K_H.gguf	Q4_K_H	18.5e9 B	~Q4_K_M size
Qwen3-30B-Coder-A3B-Instruct.Q4_E_H.gguf	Q4_E_H	18.2e9 B	~Q4_K_M size

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040