Benchmarks on 16GB VRAM (RX 9070 XT / Vulkan): Compact is 3.04x faster than the Q4_K_M — and the reason is not the quantization
Thanks for shipping an imatrix quant that actually fits consumer VRAM. I benchmarked this IQ2_XXS against the third-party DogukanUrker/BTL-4-GGUF Q4_K_M on the same box, same settings, and the gap is much larger than I expected. Posting the numbers in case they're useful.
Setup
- GPU: Radeon RX 9070 XT (gfx1201, RDNA4), 16,304 MiB usable VRAM
- Backend: llama.cpp Vulkan, build b10295 (Docker), not ROCm
- CPU: Ryzen 9 3900X (Zen2), 12 threads
--ctx-size 12000 --n-gpu-layers 99 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 -np 1 --no-mmap- Sampling:
temperature 0.0, top_k 1, seed 1234, max_tokens 400, 3 runs each, clean container reload between configurations
Both files load fine as qwen35moe on b10295. No patches needed.
Speed
| Q4_K_M (21.5 GB) | This repo, IQ2_XXS (9.97 GB) | |
|---|---|---|
-ncmoe (MoE layers pushed to CPU) |
12 (required) | 0 |
| VRAM used | 15,537 MiB | 9,990 MiB |
| Prose prompt | 39.80 t/s | 120.84 t/s |
| Repetitive / structured prompt | 39.74 t/s | 121.28 t/s |
The 3.04x is not because IQ2_XXS decodes faster — it's because 9.97 GB fits and 21.5 GB does not. The Q4_K_M has to offload 12 of 40 layers' experts to CPU via -ncmoe, and that offload is the entire cost. On a 16GB card, picking a quant that fits beats every other tuning knob I have.
Note both quants show prose ≈ repetitive. That's expected: BTL-4 has mtp_num_hidden_layers: 0, so there are no MTP tensors and speculative decoding is unavailable (--spec-type draft-mtp fails). On models that do have MTP I normally see a 1.5x spread between these two prompt types; here there's none.
If you must run the Q4_K_M on 16GB: -ncmoe is a peak, not a slope
-ncmoe |
10 | 11 | 12 | 13 | 14 | 16 | 20 | 24 |
|---|---|---|---|---|---|---|---|---|
| t/s | 32.4 | 41.1 | 40.0 | 39.1 | 37.8 | 35.4 | 32.9 | 30.1 |
| VRAM MiB | 16009 | 15969 | 15537 | 15103 | 14597 | 13780 | 11873 | 10076 |
11 is fastest but sits at 15,969 MiB — one step from the cliff. At 10 it spills into GTT and collapses to 32.4 t/s (prompt throughput also drops 242 → 94 t/s). 12 is the setting I'd recommend: it reproduced at 39.99 / 39.96 across sessions and gives up only 2.8%.
Quality: what IQ2_XXS actually costs
Greedy, one shot each. Not scored, just checked for breakage.
| Probe | Q4_K_M | IQ2_XXS |
|---|---|---|
Code (merge_intervals) |
correct | correct |
Tool use (OpenAI-format tools) |
correct, finish_reason=tool_calls |
identical |
| Arithmetic (84*3/2) | 126 + check | 126 + check |
| "Return ONLY valid JSON" | wrapped it in a ```json fence | clean JSON — better |
| Language adherence, 6 JA prompts | 0/6 drift | 0/6 drift |
| Explain Japan's rainy season | correct | wrong (reversed front movement; counted one anticyclone as two) |
| First 10 digits of pi | 3.141592653 (10, correct) | 3.14159265 (9, wrong count) |
| 3rd longest river in Japan | Tone (wrong; it's Ishikari) | Tenryu (further off) |
Agentic behaviour survives IQ2_XXS intact. Tool calling, code, instruction following, arithmetic — no measurable degradation, and instruction following was actually cleaner. What degrades is fine-grained factual precision and digit-level accuracy.
Two caveats so nobody over-reads this:
- Some errors are the model, not the quant. Asked which two anticyclones bound the Japanese rainy-season front, both quants got it wrong (Q4_K_M said "Azores High", IQ2_XXS said "Atlantic High"; the answer is Okhotsk and Pacific). Always re-test on a higher quant before blaming quantization.
- IQ2_XXS once answered a Japanese question in Chinese, but that did not reproduce across a 6-prompt language-adherence probe. One-off, not a systematic flaw — reporting it only for completeness.
Two asks
- An mmproj GGUF would be very welcome. The base model is
Qwen3_5MoeForConditionalGenerationwith vision, but no mmproj has been published for either quant, so vision is untestable from GGUF today. - At IQ2_XXS there are still 6.3 GB of VRAM free on a 16GB card at ctx 12000. Given
context_length: 262144, a note in the model card about how far context can practically be pushed would help.
Limits of these numbers
These are temperature=0 benchmark figures, not production throughput — real sampling will be slower. All conclusions are Vulkan / RDNA4 specific; I do not claim they carry to CUDA, Metal, or ROCm. In my experience some of these results invert across backends (KV quantization costs ~0% on Vulkan but ~35% on Metal), so please re-measure on your own hardware before relying on them.