README.md

---
license: mit
base_model: inclusionAI/Ling-3.0-tiny
base_model_relation: quantized
tags:
  - rocmfpx
  - ai-max-395
  - ryzen-ai-max-395
  - amd
  - gguf
  - llama.cpp
  - rocm
  - gfx1151
  - strix-halo
  - bailingmoe3
  - moe
  - hybrid-linear-attention
language:
  - en
pipeline_tag: text-generation
---
> ### ⚠️ STOCK `llama.cpp` WILL NOT LOAD THIS MODEL
> `bailingmoe3` is not merged upstream, and Ling-3.0-**tiny** additionally needs the
> **Q-LoRA** attention path (`q_lora_rank: 256`) that flash-era builds lack. Ignore the
> auto-generated "Use this model" commands above — use the patch in `patches/`.
>
> 🚀 **101.08 tok/s** on AMD Ryzen AI MAX+ 395 (gfx1151 / Strix Halo) —
> **4.30 GiB**, smaller than Q4_K_M (4.49 GiB). Verified on two independent machines.


## ✅ The patch you need is in this repo

`patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch` — applies to
[`charlie12345/ROCmFPX`](https://github.com/charlie12345/ROCmFPX) at commit **`2809dc5`**,
verified with `git apply --check`.

```bash
git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout 2809dc5
git apply patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch
cmake -B build -S . -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)
```

Full build notes, per-architecture details and licence: **`patches/README.md`** in this repo.

⚠️ If you add files under `src/models/`, re-run `cmake -B build -S .` — the `models/*.cpp` GLOB
is configure-time, so `cmake --build` alone will not link them.

---

# Ling-3.0-tiny — ROCmFP4 (tier 102 `COHERENT`) GGUF

A 4-bit **ROCmFP4** quantization of [`inclusionAI/Ling-3.0-tiny`](https://huggingface.co/inclusionAI/Ling-3.0-tiny),
built for **AMD gfx1151 (Ryzen AI MAX+ 395 / Strix Halo)** with per-tensor protection of the
LM head, token embeddings, shared experts, router, and the hybrid model's recurrent state.

| | |
|---|---|
| **File** | `Ling-3.0-tiny-Q4_0_ROCMFP4_COHERENT.gguf` |
| **Size** | **4.2987 GiB** (4,615,656,288 bytes) |
| **BPW** | **4.676** |
| **ftype** | `Q4_0_ROCMFP4_COHERENT` (102) |
| **Source** | BF16 GGUF (14.72 GiB) — lossless source, not a requantization |
| **sha256** | `fd9af463569509aca718f6ccacab388c74d8abe108511d5b6e9c00e3243b42b4` |

---

## ⛔ REQUIRES A PATCHED llama.cpp — STOCK WILL NOT LOAD THIS

Two independent reasons, both unavoidable:

1. **`bailingmoe3` is not in upstream llama.cpp.** Support is still open in
   [PR #26608](https://github.com/ggml-org/llama.cpp/pull/26608) (unmerged at time of writing).
2. **Ling-3.0-tiny needs the Q-LoRA attention path.** Its config sets `q_lora_rank: 256`
   (`q_a_proj → q_a_layernorm → q_b_proj`). Several existing `bailingmoe3` implementations were
   written against **Ling-3.0-flash**, which has `q_lora_rank: null` and therefore no query
   compression. On such a build, **every** GGUF of tiny fails — including the BF16 and Q4_K_M
   ones — typically at `missing tensor 'blk.0.ssm_f.weight'`, *before* the Q-LoRA gap is even
   reached.

You need a build with **both** `bailingmoe3` **and** its Q-LoRA path. The reference
implementation is the branch behind PR #26608
(`aetherbird/llama.cpp`, branch `bailingmoe3-support`). The ROCmFP4 quant types additionally
require a fork that implements them; upstream llama.cpp does not have `Q4_0_ROCMFP4_*`.

If your build loads Ling-3.0-**flash** but not tiny, you are missing the Q-LoRA path specifically.

### ⚠️ `strings` is not a capability check

We tested a second gfx1151 machine whose `libllama.so` contained **`bailingmoe3` (60 matches),
`ssm_f_a`, and `attn_q_a`** — it looked fully capable. It still failed with the exact same
`missing tensor 'blk.0.ssm_f.weight'`.

Those symbols live in the tensor-**name** table. The **fallback logic** that maps `ssm_f` →
`ssm_f_a`, and the Q-LoRA branch itself, are separate code. **Grepping the binary tells you
nothing — attempt the load.**

---

## All quant variants

Three builds of this model, **all measured in one session on one box with one binary**
(Ryzen AI MAX+ 395, gfx1151, ROCm 7.2.4, `ROCmFPX-2809dc5`) — so these rows are directly
comparable. Median of 3, warm-up discarded, otherwise-idle box.

| variant | ftype | size | bpw | decode (median) | range | repo |
|---|---|---|---|---|---|---|
| **4-bit COHERENT** | 102 | 4.30 GiB | 4.67 | **104.04** | 104.00 – 104.24 | [Ling-3.0-tiny-ROCmFP4-GGUF](https://huggingface.co/kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF) |
| **8-bit AGENT** | 115 | 7.72 GiB | 8.40 | 88.82 | 88.80 – 88.83 | [Ling-3.0-tiny-ROCmFPX-Q8_0-AGENT-GGUF](https://huggingface.co/kingjones777/Ling-3.0-tiny-ROCmFPX-Q8_0-AGENT-GGUF) |
| **8-bit plain** | 111 | 7.62 GiB | 8.28 | 89.51 | 89.48 – 89.51 | [Ling-3.0-tiny-ROCmFPX-Q8_0-GGUF](https://huggingface.co/kingjones777/Ling-3.0-tiny-ROCmFPX-Q8_0-GGUF) |

⚠️ **The 4-bit build is faster (104.04 vs ~89 tok/s) and 44% smaller.** These 8-bit builds exist for accuracy headroom, not speed — pick them only if you need the extra precision.

**What `AGENT` actually changes:** it keeps far more tensors at true `Q8_0` instead of the
packed 8-bit type — measured in these files, **135 tensors vs 2 tensors**. On models with an
MTP draft head that raises draft acceptance and wins ~6%; **these two models have no MTP head**,
and here the two 8-bit builds are within noise of each other.

## Measured results

Verified on **two independent gfx1151 machines**, using the model's **official sampling**
(`temperature 0.6`, `top_p 0.95`, `top_k 20`).

| | machine A | machine B |
|---|---|---|
| SoC | Ryzen AI MAX+ 395 (gfx1151) | Ryzen AI MAX+ 395 (gfx1151) |
| memory | 128 GB unified | 125 GB unified |
| ROCm | **7.2.4** | **7.13.0** |
| flags | `-ngl 99 -c 4096 -fa on` | `-ngl 999 -c 32768 -fa on -fit off --no-mmap` |
| loads | ✅ `arch=bailingmoe3`, 526 tensors | ✅ |
| 17 × 23 | ✅ `391` | ✅ `391` |
| capital of Japan | ✅ `Tokyo` | ✅ `Tokyo` |
| days in 2024 | ✅ `366` | ✅ `366` |
| reasoning separation | ✅ clean, in `reasoning_content` | ✅ |
| **decode speed** | **97.64 tok/s** | **101.08 tok/s** |

### ⭐ The build is portable across ROCm minor versions

The binaries were compiled against **ROCm 7.2.4** and run unmodified on a **ROCm 7.13.0** host —
all 9 `Q4_0_ROCMFP4_*` quant types still enumerated, model loads, 101 tok/s. Both hosts are
gfx1151. Useful if you have one build box and several Strix Halo machines: you can copy
`llama-server` + `lib*.so*` rather than rebuilding per host (set `LD_LIBRARY_PATH` to the
directory you copied them into).

### Serving configuration that works

Long-running deployment on machine B (systemd, always-hot):

```
-dev ROCm0 -ngl 999 -fa on -c 32768 -fit off -np 1 --no-mmap --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20
```
plus `LimitMEMLOCK=infinity`, `HSA_OVERRIDE_GFX_VERSION=11.5.1`,
`GGML_HIP_ENABLE_UNIFIED_MEMORY=1`.

Verified on the running process, not just at launch: **memlock unlimited** (the 8 MB default
will hobble the model), **`n_ctx = 32768` actually granted** — `--fit` is on by default and can
silently shrink context or push tensors to CPU, so `-fit off` and then confirm the number — and
**zero** "tensor override to CPU" lines in the log.

### Per-tensor protection (audited in the finished file)

| tensor class | type |
|---|---|
| `output.weight` (LM head) | **Q6_K** |
| `token_embd.weight` | **Q6_K** |
| `*_shexp` shared experts (69) | **Q8_0** |
| `ffn_gate_inp` router (23) | **F32** |
| `ssm_a`, `ssm_dt.bias` (36) | **F32** |
| `ssm_conv1d_{q,k,v}` (54) | **F32** |
| norms (79) | **F32** |
| routed experts, attention projections | 4-bit |

**Why this matters.** Tier `_STRIX` (105) protects attention K/V but *not* the LM head — on this
model's **157,184-token vocabulary** that leaves every logit passing through a 4-bit tensor.
Tier `102 COHERENT` carries Q6_K token embeddings, and the head/shared-expert protections above
were applied explicitly. Shared experts matter because they are **dense** — they process every
token, so their error is systematic rather than averaged across the 128 routed experts.

The recurrent/linear-attention state (`ssm_a`, `ssm_dt`, `conv1d`) is kept at F32: these are
float32 in the source model, and quantizing hybrid state is a known way to produce a model that
loads, runs, and emits fluent nonsense.

### Size comparison (same source, same machine)

| build | size |
|---|---|
| BF16 | 14.72 GiB |
| Q4_K_M | 4.4926 GiB |
| **this build** | **4.2987 GiB** |

---

## What was NOT measured

Stated plainly so you can judge fitness for your use case:

- **No perplexity run**, and **no quality A/B against Q4_K_M or BF16.** The correctness checks
  above are memorized-fact prompts — they are necessary but not sufficient, and a damaged model
  can pass them.
- **No long-context testing.** All generations were short. The 32,768-token context was granted
  and confirmed at load on machine B, but nothing exercised rope/KV *behaviour* at depth, and
  nothing was run near the model's 131,072 ceiling.
- **No tool-calling evaluation.**
- **MTP / speculative decoding untested** — Ling-3.0-tiny has `num_nextn_predict_layers: 0`,
  so it has no MTP layer to exercise.

---

## Model

`BailingMoeV3ForCausalLM` / `bailing_hybrid`, GGUF arch `bailingmoe3`.
24 layers in a 3:1 stack of **KDA** (Kimi Delta Attention, 18 layers) and **MLA**
(Multi-head Latent Attention, 6 layers) · hidden 1536 · **128 routed experts, 8 active** ·
shared experts · vocab 157,184 · `q_lora_rank 256` · `kv_lora_rank 512` · context 131,072.

Base model licence: MIT (inherited). All credit for the model itself goes to
[inclusionAI](https://huggingface.co/inclusionAI).