ubergarm

/

Qwen3-Coder-Next-GGUF

like 8

Definitely a fast fully offloaded model, seems like it is good enough for some uses too!

Is there a problem with the IK_Llama.cpp version with this model? Am I doing something wrong?
I have a relatively weak configuration: an i5 12400f processor, 96GB of RAM, and a 4070 with 12GB of VRAM.
However, I am having great success using this model with llama.cpp.
Many people praise the IK version of llama for its speed, and I decided to try it with your quantized versions.
The speed is very disappointing.
I am running the Qwen Next Coder model using llama.cpp with the following commands:

llama-server.exe --model G:\LlamaModels\Qwen3-Coder-Next-MXFP4_MOE.gguf --port {PORT} --ctx-size 128000 --fit on --fit-target 512 --fit-ctx 16384 --batch-size 512 --mlock --host 0.0.0.0 --jinja --temp 1.0 --min-p 0.01 --top-p 0.95 --top-k 40

llama-server.exe --model G:\LlamaModels\Qwen3-Coder-Next-UD-Q8_K_XL-00001-of-00003.gguf --port {PORT} --ctx-size 128000 --fit on --fit-target 512 --fit-ctx 16384 --batch-size 512 --mlock --host 0.0.0.0 --jinja --temp 1.0 --min-p 0.01 --top-p 0.95 --top-k 40

I get 20 tokens per second with the first one and 15 tokens per second with the second.
However, the model itself is quite good. The MXFP4 version only starts to slow down noticeably when the filled context exceeds 100,000 tokens.

IK version:

llama-server.exe --model G:\LlamaModels\Qwen3-Coder-Next-IQ4_KSS.gguf --port ${PORT} --ctx-size 128000 -ub 2048 -b 2048 --threads 1 --no-mmap -sm graph --merge-qkv -ger -fa on --n-gpu-layers 99 --jinja -ot ".ffn_.*_exps.=CPU" --host 0.0.0.0 --jinja --temp 1.0 --min-p 0.01 --top-p 0.95 --top-k 40

I get 3-4 tokens per second.
Could the issue be with -ot ".ffn_.*_exps.=CPU"?

@aldubl

Is there a problem with the IK_Llama.cpp version with this model?

No, i've tested it fine just today with some of the latest speed boosts:

Am I doing something wrong?

Yes. You're running on CPU and only giving it 1 thread for starters. There is no -fit on for ik, you will need to manually allocate additional routed expert layers to make better use of VRAM. Check this guide: https://gist.github.com/DocShotgun/a02a4c0c0a57e43ff4f038b46ca66ae0

Rough command will be someting like:

llama-server.exe --model G:\LlamaModels\Qwen3-Coder-Next-IQ4_KSS.gguf --port ${PORT} --ctx-size 128000 -ub 2048 -b 2048 --threads 6 --no-mmap -ger --merge-qkv -ger -fa on --n-gpu-layers 99 --jinja  -ot "blk\.(0|1|2|3)\.ffn_(gate|up|down)_exps.*=CUDA0" --cpu-moe --host 0.0.0.0 --jinja --temp 1.0 --min-p 0.01 --top-p 0.95 --top-k 40 

Add more layers until it OOms your vram. I removed -sm graph as that is only for 2x or more GPUs

Keep us posted, good luck!

Keep us posted, good luck!

Thank you very much! You're awesome, man. With your team's efforts, the generation speed has increased to 17.55 t/s, which is lower than llama.cpp, but the prompt processing speed has reached an impressive 423.93 t/s!
Thank you again, especially for the article. I hope it will help me deepen my understanding. I'll continue experimenting now.

@aldubl

Thank you very much! You're awesome, man. With your team's efforts,

You're welcome! Glad you're getting better results now!

prompt processing speed has reached an impressive 423.93 t/s!

If you're cpu has lscpu | grep avx512_vnni you can get better PP as well on ik_llama.cpp. If you're focusing on PP you can increase batch sizes too e.g. try out -ub 4096 -b 4096 which takes more VRAM but often gives better PP in configurations like yours.

To get back some TG you could possibly add another layer to GPU offload e.g. -ot "blk\.(0|1|2|3|4)\.ffn_(gate|up|down)_exps.*=CUDA0" but its all trade-offs on context length etc. You can also try -ctk q8_0 -ctv q8_0 to use less VRAM on kv-cache to fit more layers to improve TG etc. Yes have fun experimenting!

do you have plan for q8 or bf16 ?

@shewin

No, in general the bf16 and q8_0 produced by all quantizers should be the same assuming they use the mainline llama.cpp convert_hf_to_gguf.py script on the original bf16 safetensors and use a --pure q8_0 with no imatrix as is the standard. Because my public repo quota for hf is limited, I avoid uploading pure models available elsewhere.

tl;dr; i recommend you skip the bf16 and grab this Q8_0: https://huggingface.co/ggml-org/Qwen3-Coder-Next-GGUF/tree/main

Also, new Qwen models coming in now: https://huggingface.co/ubergarm/Qwen3.5-122B-A10B-GGUF