OxCoder-9B i1 GGUF with embedded MTP draft
This repository provides GGUF files of mradermacher/OxCoder-9B-i1-GGUF
with an embedded MTP (Multi-Token Prediction) block, ready for speculative
decoding in llama.cpp with --spec-type draft-mtp.
OxCoder-9B (OrionLLM/OxCoder-9B) is a Qwen3.5-9B-based coding model that ships without MTP tensors. The MTP block was transplanted (copied tensor by tensor, with no training) from a file of a4lg/Qwen3.5-9B-MTP-ONLY-GGUF, itself extracted from the original Qwen3.5-9B.
What this is (and is not)
- Main model: the imatrix quants
OxCoder-9B.i1-*.ggufby mradermacher, copied tensor by tensor, unchanged. - MTP block (
blk.32.*): taken from a file of a4lg/Qwen3.5-9B-MTP-ONLY-GGUF, an MTP-only subset of Qwen/Qwen3.5-9B. The quant of the donor file used for each file is listed in the table under "Files". - The MTP head was trained on Qwen3.5-9B, not on OxCoder-9B. It still drafts well for this model (acceptance ~0.6โ0.7 in my tests), but it is not an OxCoder-specific draft head.
- In the grafted files the MTP block shares
token_embdandoutputwith OxCoder, so each file is only slightly larger than the original quant and uses less VRAM than loading a separate draft model.
Files
| File | Quant main | Quant MTP | Size |
|---|---|---|---|
| OxCoder-9B.i1-MTP-IQ1_S.gguf | i1-IQ1_S | Q4_K_M | 2.7 GiB (~2.9 GB) |
| OxCoder-9B.i1-MTP-IQ1_M.gguf | i1-IQ1_M | Q4_K_M | 2.8 GiB (~3.0 GB) |
| OxCoder-9B.i1-MTP-IQ2_XXS.gguf | i1-IQ2_XXS | Q4_K_M | 3.0 GiB (~3.3 GB) |
| OxCoder-9B.i1-MTP-IQ2_XS.gguf | i1-IQ2_XS | Q4_K_M | 3.2 GiB (~3.4 GB) |
| OxCoder-9B.i1-MTP-IQ2_S.gguf | i1-IQ2_S | Q4_K_M | 3.3 GiB (~3.6 GB) |
| OxCoder-9B.i1-MTP-IQ2_M.gguf | i1-IQ2_M | Q4_K_M | 3.5 GiB (~3.8 GB) |
| OxCoder-9B.i1-MTP-Q2_K_S.gguf | i1-Q2_K_S | Q4_K_M | 3.6 GiB (~3.9 GB) |
| OxCoder-9B.i1-MTP-Q2_K.gguf | i1-Q2_K | Q4_K_M | 3.7 GiB (~4.0 GB) |
| OxCoder-9B.i1-MTP-IQ3_XXS.gguf | i1-IQ3_XXS | Q4_K_M | 3.8 GiB (~4.1 GB) |
| OxCoder-9B.i1-MTP-IQ3_XS.gguf | i1-IQ3_XS | Q4_K_M | 4.1 GiB (~4.4 GB) |
| OxCoder-9B.i1-MTP-Q3_K_S.gguf | i1-Q3_K_S | Q4_K_M | 4.1 GiB (~4.4 GB) |
| OxCoder-9B.i1-MTP-IQ3_S.gguf | i1-IQ3_S | Q4_K_M | 4.2 GiB (~4.5 GB) |
| OxCoder-9B.i1-MTP-IQ3_M.gguf | i1-IQ3_M | Q4_K_M | 4.3 GiB (~4.6 GB) |
| OxCoder-9B.i1-MTP-Q3_K_M.gguf | i1-Q3_K_M | Q4_K_M | 4.5 GiB (~4.8 GB) |
| OxCoder-9B.i1-MTP-Q3_K_L.gguf | i1-Q3_K_L | Q4_K_M | 4.7 GiB (~5.1 GB) |
| OxCoder-9B.i1-MTP-IQ4_XS.gguf | i1-IQ4_XS | Q4_K_M | 5.0 GiB (~5.4 GB) |
| OxCoder-9B.i1-MTP-Q4_0.gguf | i1-Q4_0 | Q4_K_M | 5.1 GiB (~5.5 GB) |
| OxCoder-9B.i1-MTP-Q4_K_S.gguf | i1-Q4_K_S | Q4_K_M | 5.1 GiB (~5.5 GB) |
| OxCoder-9B.i1-MTP-IQ4_NL.gguf | i1-IQ4_NL | Q4_K_M | 5.2 GiB (~5.6 GB) |
| OxCoder-9B.i1-MTP-Q4_K_M.gguf | i1-Q4_K_M | Q4_K_M | 5.4 GiB (~5.8 GB) |
| OxCoder-9B.i1-MTP-Q4_1.gguf | i1-Q4_1 | Q4_K_M | 5.6 GiB (~6.0 GB) |
| OxCoder-9B.i1-MTP-Q5_K_S.gguf | i1-Q5_K_S | Q5_K_M | 6.0 GiB (~6.5 GB) |
| OxCoder-9B.i1-MTP-Q5_K_M.gguf | i1-Q5_K_M | Q5_K_M | 6.2 GiB (~6.7 GB) |
| OxCoder-9B.i1-MTP-Q6_K.gguf | i1-Q6_K | Q6_K | 7.1 GiB (~7.6 GB) |
The MTP block adds about 0.2 GiB.
Vision projectors (mmproj) are included. They were taken from mradermacher/OxCoder-9B-GGUF and were trained for this model.
Usage
Requires a llama.cpp build with MTP draft support (--spec-type draft-mtp).
# Replace <QUANT> with the quant of the file you downloaded
llama-server \
-m OxCoder-9B.i1-MTP-<QUANT>.gguf \
-ngl auto -c 0 -fit on -fitc 32768 -fa on -np 1 \
-ctk q8_0 -ctv q8_0 \
--temp 0.6 --top-k 20 --min-p 0.0 \
--spec-type draft-mtp --spec-draft-n-max 2
Use the grafted file as the main model. Do not pass it again with
--model-draft.
Notes:
- In llama.cpp
--spec-draft-n-maxdefaults to 3. In my tests 2 was consistently a little better than 3; values of 1 or 4 and above were slower (see below). - On 8 GB of VRAM, keep the default micro-batch (
-ub 512). With-ub 1024or-ub 2048, generation with MTP dropped to ~22 t/s, probably because the larger compute buffer left less VRAM for the model layers under-fit on.
How it was made
The MTP block was transplanted with
convert.py
by @buzz, following "Method 2" of the
a4lg README:
# ./convert.py TARGET DONOR OUTPUT
python3 convert.py \
OxCoder-9B.i1-<QUANT>.gguf \
Qwen3.5-9B-MTP-ONLY-<DONOR_QUANT>.gguf \
OxCoder-9B.i1-MTP-<QUANT>.gguf
The script as published did not work for the qwen35 architecture.
It raises block_count from 32 to 33 but copies per-layer array keys
unchanged, so llama.cpp refused the output with:
key qwen35.attention.recurrent_layers has wrong array length; expected 33, got 32
The donor file does not carry that key, so I extended per-layer arrays
in the loop that writes the target's KVs, appending False for the MTP
block (it is a full-attention layer: attn_q/k/v/output, no ssm_*):
contents = field.contents()
# Per-layer arrays have one entry per block: add an entry for the MTP block
if len(contents) == target_block_count:
src_field = source_reader.get_field(key)
if src_field is not None and len(src_field.contents()) == source_block_count:
extra = list(src_field.contents())[target_block_count:]
elif key == f"{arch}.attention.recurrent_layers":
# Donor lacks the key; the MTP block uses full attention
extra = [False]
else:
extra = []
contents = list(contents) + extra
if sub_type == GGUFValueType.BOOL:
contents = [int(x) for x in contents]
write_array_value(fout, sub_type, contents)
Resulting metadata:
qwen35.block_count = 33
qwen35.nextn_predict_layers = 1
qwen35.attention.recurrent_layers = [... 33 entries, last one False]
Updated script convert.py.
Limitations
- Only the quants listed under "Files" were grafted and tested.
- Only tested with the llama.cpp Vulkan backend on one AMD GPU.
- Speculative decoding with MTP does not apply to image input.
License and Copyright
All GGUF files in this repository are distributed under the Apache License, Version 2.0, the license of both Qwen/Qwen3.5-9B and OrionLLM/OxCoder-9B. A copy of the license is included in the LICENSE file.
Copyright 2026 Alibaba Cloud (Qwen3.5-9B, including the MTP head)
Modification notice: the files were produced by mechanically combining tensors from the two GGUF files listed above and adjusting the metadata described in "How it was made". No weights were changed or trained.
This README file is licensed under the terms of CC-BY-4.0. It is partly based on the README of a4lg/Qwen3.5-9B-MTP-ONLY-GGUF (CC-BY-4.0, Copyright 2026 a4lg).
Copyright 2026 Daniel Dias Rodrigues (Nerun).
Credits
- Qwen team / Alibaba Cloud โ Qwen3.5-9B and its MTP head
- OrionLLM โ OxCoder-9B
- mradermacher โ imatrix GGUF quants of OxCoder-9B
- a4lg โ MTP-only GGUF subset of Qwen3.5-9B
- @buzz โ
convert.pyMTP transplant script
Links: Sources
- Original model: OrionLLM/OxCoder-9B
- Main quants: mradermacher/OxCoder-9B-i1-GGUF
- Vision projectors: mradermacher/OxCoder-9B-GGUF
- MTP donor: a4lg/Qwen3.5-9B-MTP-ONLY-GGUF
- Base of the MTP head: Qwen/Qwen3.5-9B
- Graft script: buzz/convert.py
- Graft script (fixed): nerun/convert.py
- Downloads last month
- 6,933
1-bit
2-bit
3-bit
4-bit
5-bit
6-bit
Model tree for nerun/OxCoder-9B-i1-MTP-GGUF
Base model
Qwen/Qwen3.5-9B-Base