kingjones777/Mellum2-12B-A2.5B-Instruct-ROCmFP4-GGUF

⚠️ STOCK llama.cpp WILL NOT LOAD THIS MODEL

The Mellum architecture is not merged upstream. Ignore the auto-generated "Use this model" commands above — use the ROCmFPX patch in patches/.

🚀 96.92 tok/s on AMD Ryzen AI MAX+ 395 (gfx1151 / Strix Halo) — 6.49 GiB, 1.12 GiB smaller and 7.2% faster than Q4_K_M.

✅ The patch you need is in this repo

patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch — applies to charlie12345/ROCmFPX at commit 2809dc5, verified with git apply --check.

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout 2809dc5
git apply patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch
cmake -B build -S . -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

Full build notes, per-architecture details and licence: patches/README.md in this repo.

⚠️ If you add files under src/models/, re-run cmake -B build -S . — the models/*.cpp GLOB is configure-time, so cmake --build alone will not link them.

---

Mellum2-12B-A2.5B-Instruct — ROCmFP4 (tier 102 COHERENT) GGUF

A 4-bit ROCmFP4 quantization of JetBrains/Mellum2-12B-A2.5B-Instruct, built for AMD gfx1151 (Ryzen AI MAX+ 395 / Strix Halo), with the LM head and token embeddings held at Q6_K.

File	Mellum2-12B-A2.5B-Instruct-Q4_0_ROCMFP4_COHERENT.gguf
Size	6.4907 GiB (6,969,373,344 bytes)
BPW	4.59
ftype	Q4_0_ROCMFP4_COHERENT (102)
Source	BF16 GGUF (22.64 GiB) — lossless source, not a requantization
sha256	161d23aa5dd6813e348cdcbf6873beb9c1cded3b56d211379429dcaa373fc43e

Smaller and faster than Q4_K_M on the target hardware — see below.

---

⛔ REQUIRES A PATCHED llama.cpp — STOCK WILL NOT LOAD THIS

mellum is not in mainline llama.cpp. Support is open in PR #23966 ("model: add Mellum architecture", Xarbirus; branch Xarbirus/llama.cpp:mellum2), unmerged at time of writing. The ROCmFP4 quant types additionally require a fork that implements them — upstream has no Q4_0_ROCMFP4_*.

⚠️ strings is not a capability check

Our build's libllama.so contained the literal string mellum and still failed with unknown model architecture: 'mellum'. The string lives in a name table; the loader is separate code. Grepping the binary tells you nothing — attempt the load.

---

All quant variants

All measured on one box, one binary (Ryzen AI MAX+ 395, gfx1151, ROCm 7.2.4), median of 3, warm-up discarded — so these rows are directly comparable.

variant	ftype	size	bpw	decode (median)	range
4-bit COHERENT	102	6.49 GiB	4.59	104.99	104.96 – 105.73
8-bit AGENT	115	11.88 GiB	8.39	74.93	74.93 – 74.97
8-bit plain	111	11.70 GiB	8.27	72.76	72.61 – 72.79

Repos: 4-bit · 8-bit AGENT · 8-bit plain

AGENT is faster here — 74.93 vs 72.76, ranges disjoint (+3.0%). Both 8-bit builds are well below the 4-bit build's 104.99 tok/s; they exist for accuracy headroom, not speed.

On AGENT generally: it keeps more tensors at true Q8_0 instead of the packed 8-bit type. That raises MTP draft acceptance on models which have an MTP head (measured +6.2% on Qwen3.8-27B). Mellum2 has no MTP head, so there is nothing for the extra precision to feed and the two 8-bit builds differ only marginally — in either direction.

Measured results

Ryzen AI MAX+ 395 (gfx1151, 128 GB unified, ROCm 7.2.4), -ngl 99 -c 4096 -fa on.

build	size	17×23	capital of Japan	days in 2024	decode
this build	6.4907 GiB	✅ 391	✅ Tokyo	✅ 366	96.92 tok/s
Q4_K_M	7.6063 GiB	✅	✅	✅	90.37 tok/s
BF16 (source)	22.6423 GiB	—	—	—	—

+7.2% decode over Q4_K_M while 1.12 GiB smaller.

Per-tensor types (audited in the finished file, 339 tensors)

tensor class	type
output.weight (LM head)	Q6_K
token_embd.weight	Q6_K
ffn_gate_inp router (28)	F32
norms (113)	F32
experts, attention projections	4-bit

tie_word_embeddings is false on this model, so a real output.weight exists and both --output-tensor-type and --token-embedding-type apply. (On a tied model --output-tensor-type is a silent no-op — worth checking before you trust it.)

Mellum2 has no shared experts and no SSM/conv state, so the protections that matter for hybrid architectures do not apply here. Its layer_types alternate sliding_attention ×3 → full_attention (n_swa = 1024), and the loader honours that pattern per layer.

---

What was NOT measured

• No perplexity run, and no quality A/B against Q4_K_M or BF16. The checks above are memorized-fact prompts — necessary but not sufficient; a damaged model can pass them.
• No code-generation benchmark. This is a coding model and we did not evaluate it as one.
• No long-context testing (the model supports 131,072; nothing was run near it).
• No tool-calling evaluation.
• Speed figures are single measurements per build on one machine, not medians of repeated runs.

---

Model

MellumForCausalLM / mellum. 28 layers · hidden 2304 · vocab 98,304 · 64 experts, 8 active · moe_intermediate_size 896 · sliding/full attention interval 4 · context 131,072 · tie_word_embeddings: false.

Base model licence: Apache-2.0 (inherited). All credit for the model itself goes to JetBrains.