Instructions to use ubergarm/Qwen3-Coder-Next-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ubergarm/Qwen3-Coder-Next-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="ubergarm/Qwen3-Coder-Next-GGUF",
	filename="Qwen3-Coder-Next-IQ1_KT.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use ubergarm/Qwen3-Coder-Next-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0
# Run inference directly in the terminal:
llama-cli -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0
# Run inference directly in the terminal:
llama-cli -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0
# Run inference directly in the terminal:
./llama-cli -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Use Docker

docker model run hf.co/ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

LM Studio
Jan

vLLM

How to use ubergarm/Qwen3-Coder-Next-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ubergarm/Qwen3-Coder-Next-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ubergarm/Qwen3-Coder-Next-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Ollama
How to use ubergarm/Qwen3-Coder-Next-GGUF with Ollama:
```
ollama run hf.co/ubergarm/Qwen3-Coder-Next-GGUF:Q4_0
```

Unsloth Studio

How to use ubergarm/Qwen3-Coder-Next-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ubergarm/Qwen3-Coder-Next-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ubergarm/Qwen3-Coder-Next-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ubergarm/Qwen3-Coder-Next-GGUF to start chatting

How to use ubergarm/Qwen3-Coder-Next-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ubergarm/Qwen3-Coder-Next-GGUF:Q4_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use ubergarm/Qwen3-Coder-Next-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Run Hermes

hermes

Docker Model Runner
How to use ubergarm/Qwen3-Coder-Next-GGUF with Docker Model Runner:
```
docker model run hf.co/ubergarm/Qwen3-Coder-Next-GGUF:Q4_0
```

Lemonade

How to use ubergarm/Qwen3-Coder-Next-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ubergarm/Qwen3-Coder-Next-GGUF:Q4_0

Run and chat with the model

lemonade run user.Qwen3-Coder-Next-GGUF-Q4_0

List all available models

lemonade list

Testing IQ4_KSS

by shewin - opened Feb 23

Discussion

shewin

Feb 23

Tensor blk.47.ffn_down_exps.weight buffer type overriden to CPU
llm_load_tensors: offloading 48 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 49/49 layers to GPU
llm_load_tensors: CPU buffer size = 37920.00 MiB
llm_load_tensors: CUDA_Host buffer size = 245.75 MiB
llm_load_tensors: CUDA0 buffer size = 2156.71 MiB
...................................................................................................~ggml_backend_cuda_context: have 0 graphs
.
===================================== llama_init_from_model: f16
llama_init_from_model: n_ctx = 200192
llama_init_from_model: n_batch = 7096
llama_init_from_model: n_ubatch = 7096
llama_init_from_model: flash_attn = 1
llama_init_from_model: attn_max_b = 2048
llama_init_from_model: fused_moe = 1
llama_init_from_model: grouped er = 1
llama_init_from_model: fused_up_gate = 1
llama_init_from_model: fused_mmad = 1
llama_init_from_model: rope_cache = 0
llama_init_from_model: graph_reuse = 1
llama_init_from_model: k_cache_hadam = 0
llama_init_from_model: split_mode_graph_scheduling = 0
llama_init_from_model: reduce_type = f16
llama_init_from_model: sched_async = 0
llama_init_from_model: ser = -1, 0
llama_init_from_model: freq_base = 5000000.0
llama_init_from_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 4767.38 MiB
llama_init_from_model: KV self size = 4692.00 MiB, K (f16): 2346.00 MiB, V (f16): 2346.00 MiB
llama_init_from_model: CUDA_Host output buffer size = 0.58 MiB
llama_init_from_model: CUDA0 compute buffer size = 4197.09 MiB
llama_init_from_model: CUDA_Host compute buffer size = 2768.16 MiB
llama_init_from_model: graph nodes = 101374
llama_init_from_model: graph splits = 98
llama_init_from_model: enabling only_active_experts scheduling

main: n_kv_max = 200192, n_batch = 7096, n_ubatch = 7096, flash_attn = 1, n_gpu_layers = 99, n_threads = 101, n_threads_batch = 101

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
7096	1774	0	7.529	942.43	33.214	53.41
7096	1774	7096	7.167	990.06	36.596	48.48
7096	1774	14192	7.136	994.39	37.186	47.71
7096	1774	21288	7.144	993.26	37.861	46.86
7096	1774	28384	7.220	982.84	38.111	46.55
7096	1774	35480	7.305	971.38	38.395	46.20

shewin

Feb 23

ubergarm

Owner Feb 23

Definitely a fast fully offloaded model, seems like it is good enough for some uses too!

aldubl

Feb 23

•

edited Feb 23

Is there a problem with the IK_Llama.cpp version with this model? Am I doing something wrong?
I have a relatively weak configuration: an i5 12400f processor, 96GB of RAM, and a 4070 with 12GB of VRAM.
However, I am having great success using this model with llama.cpp.
Many people praise the IK version of llama for its speed, and I decided to try it with your quantized versions.
The speed is very disappointing.
I am running the Qwen Next Coder model using llama.cpp with the following commands:

llama-server.exe --model G:\LlamaModels\Qwen3-Coder-Next-MXFP4_MOE.gguf --port {PORT} --ctx-size 128000 --fit on --fit-target 512 --fit-ctx 16384 --batch-size 512 --mlock --host 0.0.0.0 --jinja --temp 1.0 --min-p 0.01 --top-p 0.95 --top-k 40

llama-server.exe --model G:\LlamaModels\Qwen3-Coder-Next-UD-Q8_K_XL-00001-of-00003.gguf --port {PORT} --ctx-size 128000 --fit on --fit-target 512 --fit-ctx 16384 --batch-size 512 --mlock --host 0.0.0.0 --jinja --temp 1.0 --min-p 0.01 --top-p 0.95 --top-k 40

I get 20 tokens per second with the first one and 15 tokens per second with the second.
However, the model itself is quite good. The MXFP4 version only starts to slow down noticeably when the filled context exceeds 100,000 tokens.

IK version:

llama-server.exe --model G:\LlamaModels\Qwen3-Coder-Next-IQ4_KSS.gguf --port ${PORT} --ctx-size 128000 -ub 2048 -b 2048 --threads 1 --no-mmap -sm graph --merge-qkv -ger -fa on --n-gpu-layers 99 --jinja -ot ".ffn_.*_exps.=CPU" --host 0.0.0.0 --jinja --temp 1.0 --min-p 0.01 --top-p 0.95 --top-k 40

I get 3-4 tokens per second.
Could the issue be with -ot ".ffn_.*_exps.=CPU"?

ubergarm

Owner Feb 23

@aldubl

Is there a problem with the IK_Llama.cpp version with this model?

No, i've tested it fine just today with some of the latest speed boosts:

Am I doing something wrong?

Yes. You're running on CPU and only giving it 1 thread for starters. There is no -fit on for ik, you will need to manually allocate additional routed expert layers to make better use of VRAM. Check this guide: https://gist.github.com/DocShotgun/a02a4c0c0a57e43ff4f038b46ca66ae0

Rough command will be someting like:

llama-server.exe --model G:\LlamaModels\Qwen3-Coder-Next-IQ4_KSS.gguf --port ${PORT} --ctx-size 128000 -ub 2048 -b 2048 --threads 6 --no-mmap -ger --merge-qkv -ger -fa on --n-gpu-layers 99 --jinja  -ot "blk\.(0|1|2|3)\.ffn_(gate|up|down)_exps.*=CUDA0" --cpu-moe --host 0.0.0.0 --jinja --temp 1.0 --min-p 0.01 --top-p 0.95 --top-k 40

Add more layers until it OOms your vram. I removed -sm graph as that is only for 2x or more GPUs

Keep us posted, good luck!

aldubl

Feb 23

Keep us posted, good luck!

Thank you very much! You're awesome, man. With your team's efforts, the generation speed has increased to 17.55 t/s, which is lower than llama.cpp, but the prompt processing speed has reached an impressive 423.93 t/s!
Thank you again, especially for the article. I hope it will help me deepen my understanding. I'll continue experimenting now.

ubergarm

Owner Feb 23

@aldubl

Thank you very much! You're awesome, man. With your team's efforts,

You're welcome! Glad you're getting better results now!

prompt processing speed has reached an impressive 423.93 t/s!

If you're cpu has lscpu | grep avx512_vnni you can get better PP as well on ik_llama.cpp. If you're focusing on PP you can increase batch sizes too e.g. try out -ub 4096 -b 4096 which takes more VRAM but often gives better PP in configurations like yours.

To get back some TG you could possibly add another layer to GPU offload e.g. -ot "blk\.(0|1|2|3|4)\.ffn_(gate|up|down)_exps.*=CUDA0" but its all trade-offs on context length etc. You can also try -ctk q8_0 -ctv q8_0 to use less VRAM on kv-cache to fit more layers to improve TG etc. Yes have fun experimenting!

shewin

Feb 24

do you have plan for q8 or bf16 ?

ubergarm

Owner Feb 24

•

edited Feb 24

@shewin

No, in general the bf16 and q8_0 produced by all quantizers should be the same assuming they use the mainline llama.cpp convert_hf_to_gguf.py script on the original bf16 safetensors and use a --pure q8_0 with no imatrix as is the standard. Because my public repo quota for hf is limited, I avoid uploading pure models available elsewhere.

tl;dr; i recommend you skip the bf16 and grab this Q8_0: https://huggingface.co/ggml-org/Qwen3-Coder-Next-GGUF/tree/main

Also, new Qwen models coming in now: https://huggingface.co/ubergarm/Qwen3.5-122B-A10B-GGUF

shewin

Feb 27

Q8_0:

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment