kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF

⚠️ STOCK llama.cpp WILL NOT LOAD THIS MODEL

bailingmoe3 is not merged upstream, and Ling-3.0-tiny additionally needs the Q-LoRA attention path (q_lora_rank: 256) that flash-era builds lack. Ignore the auto-generated "Use this model" commands above — use the patch in patches/.

🚀 101.08 tok/s on AMD Ryzen AI MAX+ 395 (gfx1151 / Strix Halo) — 4.30 GiB, smaller than Q4_K_M (4.49 GiB). Verified on two independent machines.

✅ The patch you need is in this repo

patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch — applies to charlie12345/ROCmFPX at commit 2809dc5, verified with git apply --check.

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout 2809dc5
git apply patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch
cmake -B build -S . -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

Full build notes, per-architecture details and licence: patches/README.md in this repo.

⚠️ If you add files under src/models/, re-run cmake -B build -S . — the models/*.cpp GLOB is configure-time, so cmake --build alone will not link them.

---

Ling-3.0-tiny — ROCmFP4 (tier 102 COHERENT) GGUF

A 4-bit ROCmFP4 quantization of inclusionAI/Ling-3.0-tiny, built for AMD gfx1151 (Ryzen AI MAX+ 395 / Strix Halo) with per-tensor protection of the LM head, token embeddings, shared experts, router, and the hybrid model's recurrent state.

File	Ling-3.0-tiny-Q4_0_ROCMFP4_COHERENT.gguf
Size	4.2987 GiB (4,615,656,288 bytes)
BPW	4.676
ftype	Q4_0_ROCMFP4_COHERENT (102)
Source	BF16 GGUF (14.72 GiB) — lossless source, not a requantization
sha256	fd9af463569509aca718f6ccacab388c74d8abe108511d5b6e9c00e3243b42b4

---

⛔ REQUIRES A PATCHED llama.cpp — STOCK WILL NOT LOAD THIS

Two independent reasons, both unavoidable:

1. bailingmoe3 is not in upstream llama.cpp. Support is still open in PR #26608 (unmerged at time of writing).
2. Ling-3.0-tiny needs the Q-LoRA attention path. Its config sets q_lora_rank: 256 (q_a_proj → q_a_layernorm → q_b_proj). Several existing bailingmoe3 implementations were written against Ling-3.0-flash, which has q_lora_rank: null and therefore no query compression. On such a build, every GGUF of tiny fails — including the BF16 and Q4_K_M ones — typically at missing tensor 'blk.0.ssm_f.weight', before the Q-LoRA gap is even reached.

You need a build with both bailingmoe3 and its Q-LoRA path. The reference implementation is the branch behind PR #26608 (aetherbird/llama.cpp, branch bailingmoe3-support). The ROCmFP4 quant types additionally require a fork that implements them; upstream llama.cpp does not have Q4_0_ROCMFP4_*.

If your build loads Ling-3.0-flash but not tiny, you are missing the Q-LoRA path specifically.

⚠️ strings is not a capability check

We tested a second gfx1151 machine whose libllama.so contained bailingmoe3 (60 matches), ssm_f_a, and attn_q_a — it looked fully capable. It still failed with the exact same missing tensor 'blk.0.ssm_f.weight'.

Those symbols live in the tensor-name table. The fallback logic that maps ssm_f → ssm_f_a, and the Q-LoRA branch itself, are separate code. Grepping the binary tells you nothing — attempt the load.

---

All quant variants

Three builds of this model, all measured in one session on one box with one binary (Ryzen AI MAX+ 395, gfx1151, ROCm 7.2.4, ROCmFPX-2809dc5) — so these rows are directly comparable. Median of 3, warm-up discarded, otherwise-idle box.

variant	ftype	size	bpw	decode (median)	range	repo
4-bit COHERENT	102	4.30 GiB	4.67	104.04	104.00 – 104.24	Ling-3.0-tiny-ROCmFP4-GGUF
8-bit AGENT	115	7.72 GiB	8.40	88.82	88.80 – 88.83	Ling-3.0-tiny-ROCmFPX-Q8_0-AGENT-GGUF
8-bit plain	111	7.62 GiB	8.28	89.51	89.48 – 89.51	Ling-3.0-tiny-ROCmFPX-Q8_0-GGUF

⚠️ The 4-bit build is faster (104.04 vs ~89 tok/s) and 44% smaller. These 8-bit builds exist for accuracy headroom, not speed — pick them only if you need the extra precision.

What AGENT actually changes: it keeps far more tensors at true Q8_0 instead of the packed 8-bit type — measured in these files, 135 tensors vs 2 tensors. On models with an MTP draft head that raises draft acceptance and wins ~6%; these two models have no MTP head, and here the two 8-bit builds are within noise of each other.

Measured results

Verified on two independent gfx1151 machines, using the model's official sampling (temperature 0.6, top_p 0.95, top_k 20).

	machine A	machine B
SoC	Ryzen AI MAX+ 395 (gfx1151)	Ryzen AI MAX+ 395 (gfx1151)
memory	128 GB unified	125 GB unified
ROCm	7.2.4	7.13.0
flags	-ngl 99 -c 4096 -fa on	-ngl 999 -c 32768 -fa on -fit off --no-mmap
loads	✅ arch=bailingmoe3, 526 tensors	✅
17 × 23	✅ 391	✅ 391
capital of Japan	✅ Tokyo	✅ Tokyo
days in 2024	✅ 366	✅ 366
reasoning separation	✅ clean, in reasoning_content	✅
decode speed	97.64 tok/s	101.08 tok/s

⭐ The build is portable across ROCm minor versions

The binaries were compiled against ROCm 7.2.4 and run unmodified on a ROCm 7.13.0 host — all 9 Q4_0_ROCMFP4_* quant types still enumerated, model loads, 101 tok/s. Both hosts are gfx1151. Useful if you have one build box and several Strix Halo machines: you can copy llama-server + lib*.so* rather than rebuilding per host (set LD_LIBRARY_PATH to the directory you copied them into).

Serving configuration that works

Long-running deployment on machine B (systemd, always-hot):

-dev ROCm0 -ngl 999 -fa on -c 32768 -fit off -np 1 --no-mmap --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20

plus LimitMEMLOCK=infinity, HSA_OVERRIDE_GFX_VERSION=11.5.1, GGML_HIP_ENABLE_UNIFIED_MEMORY=1.

Verified on the running process, not just at launch: memlock unlimited (the 8 MB default will hobble the model), n_ctx = 32768 actually granted — --fit is on by default and can silently shrink context or push tensors to CPU, so -fit off and then confirm the number — and zero "tensor override to CPU" lines in the log.

Per-tensor protection (audited in the finished file)

tensor class	type
output.weight (LM head)	Q6_K
token_embd.weight	Q6_K
*_shexp shared experts (69)	Q8_0
ffn_gate_inp router (23)	F32
ssm_a, ssm_dt.bias (36)	F32
ssm_conv1d_{q,k,v} (54)	F32
norms (79)	F32
routed experts, attention projections	4-bit

Why this matters. Tier _STRIX (105) protects attention K/V but not the LM head — on this model's 157,184-token vocabulary that leaves every logit passing through a 4-bit tensor. Tier 102 COHERENT carries Q6_K token embeddings, and the head/shared-expert protections above were applied explicitly. Shared experts matter because they are dense — they process every token, so their error is systematic rather than averaged across the 128 routed experts.

The recurrent/linear-attention state (ssm_a, ssm_dt, conv1d) is kept at F32: these are float32 in the source model, and quantizing hybrid state is a known way to produce a model that loads, runs, and emits fluent nonsense.

Size comparison (same source, same machine)

build	size
BF16	14.72 GiB
Q4_K_M	4.4926 GiB
this build	4.2987 GiB

---

What was NOT measured

Stated plainly so you can judge fitness for your use case:

• No perplexity run, and no quality A/B against Q4_K_M or BF16. The correctness checks above are memorized-fact prompts — they are necessary but not sufficient, and a damaged model can pass them.
• No long-context testing. All generations were short. The 32,768-token context was granted and confirmed at load on machine B, but nothing exercised rope/KV behaviour at depth, and nothing was run near the model's 131,072 ceiling.
• No tool-calling evaluation.
• MTP / speculative decoding untested — Ling-3.0-tiny has num_nextn_predict_layers: 0, so it has no MTP layer to exercise.

---

Model

BailingMoeV3ForCausalLM / bailing_hybrid, GGUF arch bailingmoe3. 24 layers in a 3:1 stack of KDA (Kimi Delta Attention, 18 layers) and MLA (Multi-head Latent Attention, 6 layers) · hidden 1536 · 128 routed experts, 8 active · shared experts · vocab 157,184 · q_lora_rank 256 · kv_lora_rank 512 · context 131,072.

Base model licence: MIT (inherited). All credit for the model itself goes to inclusionAI.