README.md

---
base_model: XiaomiMiMo/MiMo-V2.5
base_model_relation: quantized
pipeline_tag: text-generation
library_name: gguf
tags:
  - gguf
  - rocm
  - rocmfp4
  - rocmfpx
  - strix-halo
  - gfx1151
  - moe
---

# MiMo-V2.5 — ROCmFP4 / ROCmFPX GGUF (Strix Halo / gfx1151)

ROCmFP4 and ROCmFPX quantizations of [MiMo-V2.5](https://huggingface.co/XiaomiMiMo/MiMo-V2.5)
(310B total / 15B active MoE), built for **AMD Strix Halo (Ryzen AI Max+ 395,
gfx1151)** with the [ROCmFPX](https://github.com/charlie12345/ROCmFPX) fork of
llama.cpp.

There are two ways to run this model on Strix Halo, and they want different files:

| you have | use | why |
| --- | --- | --- |
| **two 128 GB boxes** + USB4/Thunderbolt | `ROCmFP4-FAST/` | best quality; 153 GiB does not fit one node |
| **one 128 GB box** | `Q2_0_ROCMFPX-COHERENT/` | 90.9 GiB fits a single node — and is **+37% faster at decode** than the two-node setup |

The single-node option being *faster* is not a mistake. llama.cpp's RPC layer split
runs the nodes **sequentially**, so a second box buys **capacity, not speed**. If the
model fits on one node, one node wins.

> **Note on quality.** The `ROCmFP4` and `ROCmFP4_FAST` variants are *not*
> imatrix-calibrated; Unsloth's `UD-Q4_K_XL` is, and is tensor-aware on top of that.
> They win on **speed and size** — benchmark both if marginal output quality matters
> more to you than tok/s.
>
> `Q2_0_ROCMFPX-COHERENT` **is** imatrix-calibrated (it has to be — see below). But
> it is 2.5 bpw against FAST's 4.26, and that reduction is real. It passes coherence
> checks on code, factual recall, arithmetic and JSON, but **no pass@1 evaluation has
> been run**, so treat the quality gap as *unquantified rather than absent*.

## ⚠️ Requirements — stock llama.cpp will not load these

`Q4_0_ROCMFP4` and `Q4_0_ROCMFP4_FAST` are quantization types defined by the
**[ROCmFPX](https://github.com/charlie12345/ROCmFPX)** fork. Upstream llama.cpp,
Ollama, LM Studio and every downstream that vendors mainline ggml **cannot read
these files** — you will get an unknown-ggml-type error, not a slow model. You
must build the fork.

They also target **gfx1151** specifically (Ryzen AI Max+ 395 / Strix Halo). The
formats are built around that hardware's dequant path; on other GPUs, expect
either a build failure or no benefit.

### Build

```bash
git clone https://github.com/charlie12345/ROCmFPX.git
cd ROCmFPX
git checkout 3edc3d31ee5ebcea47fd7e0f42c89767bb4245db   # the commit these were built and tested with

cmake -S . -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_HIP=ON \
    -DGGML_RPC=ON \
    -DGGML_HIP_FORCE_MMQ=ON \
    -DGGML_HIP_ROCWMMA_FATTN=OFF \
    -DGGML_VULKAN=OFF -DGGML_CUDA=OFF \
    -DCMAKE_HIP_ARCHITECTURES=gfx1151 \
    -DGPU_TARGETS=gfx1151 \
    -DLLAMA_BUILD_SERVER=ON \
    -DLLAMA_BUILD_WEBUI=OFF -DLLAMA_USE_PREBUILT_WEBUI=OFF \
    -DLLAMA_BUILD_TESTS=OFF -DGGML_BUILD_TESTS=OFF

cmake --build build -j "$(nproc)" --target \
    llama-cli llama-server llama-bench llama-quantize rpc-server
```

`-DGGML_RPC=ON` is what gives you `rpc-server` and the `RPC0` device — required
for the two-node split. Built against **ROCm 6.4**.

**Use the ROCm backend, not Vulkan.** RADV imposes a per-buffer allocation ceiling
that a model this size runs straight into; `-dev Vulkan0` will OOM where `ROCm0`
works fine.

### Making the memory available

A single Strix Halo box defaults to a GPU carve-out far below what these need.
Either set `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1` in the environment, or raise the
GTT limit on the kernel command line (what we run):

```
amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856
```

With the cmdline set, `UNIFIED_MEMORY` is no longer needed. We measured no
throughput difference between the two approaches — the cmdline is just less
fragile.

### ⚡ Force the GPU power level — worth 14% decode

**This is the single highest-value host setting, and it is easy to miss.**

```bash
echo high | sudo tee /sys/class/drm/card*/device/power_dpm_force_performance_level
```

| `power_dpm_force_performance_level` | pp512 | tg128 |
| --- | ---: | ---: |
| `auto` (default) | 322.87 | 20.94 |
| **`high`** | **329.43** | **23.87** |

**+14% decode, +2% prefill, from one sysfs write.** The DPM governor never ramps
to peak on MoE decode — the workload is bursty and low-occupancy, so `auto` reads
it as near-idle and leaves the GPU and fabric below spec. Effective memory
bandwidth goes from 167 to 190.7 GB/s (65% -> 75% of the 256 GB/s theoretical).
DRAM is at its rated 8000 MT/s either way; this is purely a clock-governor effect.

The gain is specific to **single-stream** decode. Under concurrency the GPU is
already loaded enough that `auto` ramps by itself — aggregate throughput at C8 is
unchanged.

Transparent hugepages were tested alongside this and make **no difference**
(21.24 with THP alone vs 20.94 baseline), despite the 153 GiB working set. Not
worth the system-wide side effects.

It resets on reboot. To persist:

```ini
# /etc/systemd/system/amdgpu-perf-high.service
[Unit]
Description=Force amdgpu DPM to high
After=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'for c in /sys/class/drm/card*/device/power_dpm_force_performance_level; do echo high > "$c"; done'
ExecStop=/bin/sh -c 'for c in /sys/class/drm/card*/device/power_dpm_force_performance_level; do echo auto > "$c"; done'

[Install]
WantedBy=multi-user.target
```

Apply it on **every** node — the RPC peer's clocks matter just as much as the
head node's.

## Variants

| variant | effective bpw | size | shards | fits 1 node? | folder |
| --- | ---: | ---: | ---: | :---: | --- |
| **`Q4_0_ROCMFP4_FAST`** ← best quality | 4.26 | **153.47 GiB** (164,795,562,720 B) | 4 | no | `ROCmFP4-FAST/` |
| **`Q2_0_ROCMFPX`** ← fastest, single-node | 2.52 | **90.92 GiB** (97,626,242,048 B) | 3 | **yes** | `Q2_0_ROCMFPX-COHERENT/` |
| `Q4_0_ROCMFP4` | 5.24 | 189.02 GiB (202,969,561,824 B) | 5 | no | `ROCmFP4/` |

For reference, `unsloth/MiMo-V2.5-GGUF UD-Q4_K_XL` is **178.44 GiB** at 4.95 bpw —
`ROCmFP4_FAST` is **14% smaller**.

> **`ROCmFP4_FAST` dominates `ROCmFP4` on every axis we measured** — smaller,
> faster prefill, faster decode. Unless you specifically want the higher-precision
> tensors, take FAST. `ROCmFP4` is published for completeness.

**A note on the bpw labels.** `llama-quantize` advertises `Q4_0_ROCMFP4` as
"4.50 bpw", but measured against the 309.77 B parameter count the real figure is
**5.24 bpw** — the recipe promotes several tensor classes (e.g. `ffn_gate` → `q5_K`)
rather than quantizing everything to ROCmFP4. `Q4_0_ROCMFP4_FAST` measures 4.26 bpw
against its 4.25 label, so that one is honest. This is why the "non-fast" build
ends up *larger* than `UD-Q4_K_XL` rather than smaller.

All were quantized from the BF16 GGUF (`unsloth/MiMo-V2.5-GGUF`, 14 shards,
619,638,702,336 bytes, verified byte-exact).

The 4-bit variants need no special handling:

```bash
llama-quantize MiMo-V2.5-BF16-00001-of-00014.gguf \
    MiMo-V2.5-ROCmFP4-FAST.gguf Q4_0_ROCMFP4_FAST 16
```

### `Q2_0_ROCMFPX` needs two extra flags or it produces garbage

This is the important part of this section. A plain `Q2_0_ROCMFPX` run — no imatrix,
default tensor types — yields a model that loads, benchmarks at a healthy 34.97 t/s,
and emits **this**:

```
，问题，1.问题，1.问题，10000000000000000000000000000000000000...
```

Fluent-looking throughput, zero usable output. Both of the following are required:

```bash
llama-quantize \
    --imatrix MiMo-V2.5-imatrix.gguf \
    --token-embedding-type q6_k \
    --output-tensor-type q6_k \
    MiMo-V2.5-BF16-00001-of-00014.gguf \
    MiMo-V2.5-Q2_0_ROCMFPX-COHERENT.gguf Q2_0_ROCMFPX 16
```

- **`--imatrix`** — sub-3-bit quantization needs importance-weighted rounding. The
  proof is that mainline `IQ2_XXS` stays coherent at **2.06 bpw**, *lower* than our
  2.5, because it is imatrix-aware. Uniform rounding survives 4 bpw; it does not
  survive 2.5. The imatrix used here is
  [bartowski's](https://huggingface.co/bartowski/MiMo-V2.5-GGUF) (`MiMo-V2.5-imatrix.gguf`,
  287 entries over 818 chunks).
- **`--token-embedding-type q6_k --output-tensor-type q6_k`** — the embedding table
  (152,576 tokens) and LM head are the most precision-sensitive tensors in the model,
  and the imatrix does not cover them (`did not find weights for token_embd.weight`).
  Crush them to 2.5 bpw and nearby token vectors collapse into one another, which is
  exactly the wrong-language-token + repetition signature above. Cost: **+0.3 GB on a
  97 GB file.**

ROCmFPX's own naming corroborates this: `Q4_0_ROCMFP4_COHERENT` is defined as
"ROCmFP4 **+ Q6_K token embeddings**". Nobody names a variant *coherent* unless the
alternative was incoherent. `Q2_0_ROCMFPX` ships with no such provision, hence the
`-COHERENT` suffix on this folder.

## Measured performance

### Test setup

Two Beelink GTR 9 Pro (Ryzen AI Max+ 395, gfx1151, 128 GB unified) linked by a
single USB4 cable. The 4-bit variants are layer-split across both with llama.cpp RPC;
**`Q2_0_ROCMFPX` runs on one box with no RPC at all** (`-ngl 999`, no `-dev`), which is
the whole reason it decodes faster. **All numbers are single-stream** unless the
concurrency section says otherwise.

| | |
| --- | --- |
| engine | ROCmFPX (llama.cpp fork), ROCm 6.4, HIP backend |
| OS / kernel | Ubuntu 24.04, mainline 6.18.6 |
| topology | node2 = head, node1 = `rpc-server` over Thunderbolt (`192.168.2.1:50052`) |
| GPU carve-out | 126976 MiB via `amdgpu.gttsize` / `ttm.pages_limit` kernel cmdline |
| transport | TCP over `thunderbolt0` (RDMA measured, no difference) |

Benchmark command:

```bash
llama-bench -m <model.gguf> \
    -rpc 192.168.2.1:50052 -dev ROCm0/RPC0 \
    -ngl 999 -fa 1 -mmp 0 -r 1 -p 512 -n 128
```

`-mmp 0` (no mmap) is **required** — with mmap the working set thrashes against
the 128 GB of RAM and never converges. Note `llama-bench` wants `-dev` entries
separated by `/`, while `llama-cli` wants `,`.

### Single-stream results

All rows below are measured with `power_dpm_force_performance_level=high` on both
nodes (see Requirements) — without it every number drops 9–14%.

| variant | nodes | size | prefill (pp512) | decode (tg128) |
| --- | :---: | ---: | ---: | ---: |
| **`Q2_0_ROCMFPX`** | **1** | 90.92 GiB | 241.05 t/s | **32.53 t/s** |
| **`Q4_0_ROCMFP4_FAST`** | 2 | 153.47 GiB | **328.73 t/s** | 23.80 t/s |
| _(reference)_ `UD-Q4_K_XL` | 2 | 178.44 GiB | 339.31 t/s | 17.16 t/s |
| `Q4_0_ROCMFP4` _(at dpm=auto)_ | 2 | 189.02 GiB | 248.10 t/s | 16.80 t/s |

**`Q2_0_ROCMFPX` on one node decodes 36.7% faster than `ROCmFP4_FAST` on two, for
26.7% less prefill.** Two effects compound: fewer bytes to stream per token (2.52 vs
4.26 bpw), and no RPC layer split — which runs the nodes *sequentially*, so it costs
decode latency rather than saving it. Prefill is the opposite case: it is
compute-bound and genuinely benefits from a second GPU, which is why the two-node
rows win there.

Q2 error bars over `-r 2`: pp512 ±0.75, tg128 ±0.08.

**`ROCmFP4_FAST` is +38.7% decode over `UD-Q4_K_XL` for −3.1% prefill, while being
14% smaller.** The decode gain far exceeds what the size reduction alone predicts
(4.26 vs 4.95 bpw) — the single-scale layout also dequantizes more cheaply on
gfx1151.

Both were re-measured at `dpm=high` so the comparison is like-for-like. Worth
noting `UD-Q4_K_XL` gains only ~9% from that tuning where `ROCmFP4_FAST` gains 14%,
consistent with the FAST dequant path being more clock-sensitive. At the old
`dpm=auto` default the gap read as +33%.

`ROCmFP4` (non-fast) has not been re-measured at `dpm=high`; its row is from
`dpm=auto` and is not comparable to the two above. It was the weakest of the three
on throughput at equal settings and is published only for completeness.

Reproducibility: `ROCmFP4_FAST` at `dpm=high` was measured three times —
329.43/23.87, 329.14/23.89, 328.73/23.80 — a 0.2% spread on pp512 and 0.4% on
tg128.

### Decode vs. context depth

`Q4_0_ROCMFP4_FAST`, with `power_dpm_force_performance_level=high`:

| depth | pp512 | tg128 |
| ---: | ---: | ---: |
| 0 | 328.73 | **23.80** |
| 8192 | 286.70 | **23.00** |
| 32768 | 214.53 | **21.72** |

Decode is remarkably flat with context — a consequence of the 9-full/39-SWA
attention split, where only 9 layers grow with depth. Prefill decays normally.

For reference, the same curve at the default `dpm=auto` was 315.5/21.07,
267.1/20.68 and 197.7/19.53 — the tuning is worth ~11–13% of decode at every
depth.

**`Q2_0_ROCMFPX` on a single node**, same benchmark, `dpm=high`, `-r 2`:

| depth | pp512 | tg128 | vs FAST 2-node decode |
| ---: | ---: | ---: | ---: |
| 0 | 241.05 ± 0.75 | **32.53 ± 0.08** | **+36.7%** |
| 8192 | 220.14 ± 1.73 | **29.19 ± 0.67** | **+26.9%** |
| 32768 | 177.29 ± 1.66 | **28.64 ± 0.14** | **+31.9%** |

**Q2 on one node beats FAST on two at decode at every depth**, and loses at prefill at
every depth. Its decode curve is also flatter past 8k (29.19 → 28.64, −1.9% over a 4×
context increase) — same 9-full/39-SWA attention structure as the 4-bit variants, with
less weight traffic per token on top.

Pick by workload: **long prompts / short answers → two-node FAST** (prefill wins).
**Short prompts / long answers, or interactive chat → single-node Q2** (decode wins,
and on one machine).

### Aggregate throughput under concurrency

Total tokens/s across all streams, 16 distinct prompts so that slots cannot share
a prefix-cache hit and inflate the result (`--parallel 8`, `-c 32768`).

| variant | C4 | C6 | C8 |
| --- | ---: | ---: | ---: |
| **`Q4_0_ROCMFP4_FAST`** (dpm=high) | **33.20 t/s** | **34.51 t/s** | 33.57 t/s |
| `Q4_0_ROCMFP4_FAST` (dpm=auto) | 31.93 t/s | 32.61 t/s | 33.92 t/s |
| `Q4_0_ROCMFP4` (dpm=auto) | 24.96 t/s | 26.37 t/s | 28.32 t/s |
| _(reference)_ `UD-Q4_K_XL` (dpm=auto) | 25.81 t/s | 29.19 t/s | 31.72 t/s |

**Aggregate throughput plateaus around 33–35 t/s** and the `dpm=high` tuning barely
helps here — under concurrency the GPU is already busy enough that the governor
ramps on its own. That is the mirror image of the single-stream case, where forcing
`high` is worth 14%.

`ROCmFP4_FAST` is fastest at every concurrency level, but **its margin over
`UD-Q4_K_XL` shrinks as concurrency rises** — the FAST layout's advantage is in
memory bandwidth and dequant cost, which dominate single-stream decode; as batch
size grows the workload shifts toward expert scatter and compute, where the quants
converge.

MiMo-V2.5 is a 256-expert top-8 MoE, the least favourable case for batch
amortization: the number of distinct experts touched at batch B grows as
`256·(1−(1−8/256)^B)`, so more expert weights must be read as concurrency rises
instead of being amortized across the batch. This is why aggregate throughput
scales so weakly — `ROCmFP4_FAST` gains only 6% going from 4 streams to 8, and
per-stream latency roughly halves over that range (7.98 → 4.24 t/s).

## Usage

### Single node — `Q2_0_ROCMFPX` (fastest decode)

```bash
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 \
llama-server -m Q2_0_ROCMFPX-COHERENT/MiMo-V2.5-Q2_0_ROCMFPX-COHERENT-00001-of-00003.gguf \
    -ngl 999 -fa 1 --no-mmap -ctk f16 -ctv f16 \
    -c 32768 --parallel 8 --jinja --host 0.0.0.0 \
    --temp 1.0 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.05
```

No `--rpc`, no `-dev`. `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1` is required — 90.92 GiB
exceeds the hard VRAM cap even though it fits the 128 GB unified pool.

### Two nodes — `Q4_0_ROCMFP4_FAST` (best quality)

```bash
# node1
rpc-server -H 0.0.0.0 -p 50052

# node2
llama-server -m ROCmFP4-FAST/MiMo-V2.5-ROCmFP4-FAST-00001-of-00004.gguf \
    --rpc 192.168.2.1:50052 -dev ROCm0,RPC0 \
    -ngl 999 -fa 1 --no-mmap -ctk f16 -ctv f16 \
    -c 32768 --parallel 8 --jinja --host 0.0.0.0 \
    --temp 1.0 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.05
```

Point llama.cpp at the **first shard**; it pulls in the rest automatically.

### Sampling — use the model's own recommendation

Xiaomi's model card specifies **`temperature=1.0`, `top_p=0.95`** for local
deployment. `--repeat-penalty 1.05` and `--min-p 0.0` come from the community
configuration in [llama.cpp #23074](https://github.com/ggml-org/llama.cpp/issues/23074).

Use `--jinja` with the model's built-in chat template. No corrected template exists
or is needed — neither Unsloth nor bartowski overrides it. You may see
`Expected iterable or object type in for loop: got None` in the log during
reasoning-format detection; it is harmless and also appears on runs that work.

**Note a MiMo-level quirk, not a quantization artifact:** MiMo-V2.5 can enter runaway
chain-of-thought — "extremely long CoT (sometimes for hundreds of thousands of tokens)
with no progress or tool calls" (#23074), reproduced on **Q8_0**, so it is not caused
by low-bit quantization. Cap it with `--reasoning-budget`, and pass
`"chat_template_kwargs": {"enable_thinking": false}` per-request to disable thinking
entirely. Thinking costs ~0.5% of decode *rate* — its real cost is the extra tokens.

Mind the flag spelling — `llama-server` takes `--rpc` and comma-separated `-dev`,
while `llama-bench` takes `-rpc` and **slash**-separated `-dev` (`ROCm0/RPC0`).
Passing the wrong one makes the server exit immediately with
`error: invalid argument`.

### Things worth knowing

- **Keep KV cache at f16 for speed.** Measured on this exact model and split:

  | `-ctk` / `-ctv` | tg128 @d0 | tg128 @32k |
  | --- | ---: | ---: |
  | **f16 / f16** | **20.71** | **19.44** |
  | q8_0 / q8_0 | 20.26 | 17.13 |
  | q8_0 / q4_0 | 20.17 | 17.24 |
  | q4_0 / q4_0 | 20.04 | 17.22 |

  The penalty is ~2% at zero depth but **13% at 32k** — ROCm dequant costs more
  than the bandwidth it saves, and the gap widens as the cache fills. Note the
  three quantized configs are indistinguishable: the cost comes from quantizing
  at all, not from how aggressively.

  **But quantizing buys context.** f16 KV is 22.5 KiB/token here; `q4_0/q4_0` is
  ~6.6 KiB — about **3.4× the KV pool for that 13%**. If maximum context matters
  more to you than decode speed, `-ctk q4_0 -ctv q4_0` is the trade.

- **Prefer the USB4/Thunderbolt link over ethernet for the RPC hop.** Against a
  switched 1 GbE path we measured −5% decode and −9.5% prefill. The causes differ:
  decode is latency-bound (llama.cpp RPC does ~3–4 round trips per token, so hop
  latency multiplies), while prefill is bandwidth-bound (4 MB of activations per
  512-token chunk). A *switched* 10 GbE link fixes the prefill half but not the
  decode half; a direct point-to-point cable fixes both.
- **Speculative decoding does not pay off on this model.** Measured on this exact
  build and split, with `--temp 0 --repeat-penalty 1.0`:

  | workload | no speculation | DFlash n=1 | DFlash n=2 | DFlash n=4 |
  | --- | ---: | ---: | ---: | ---: |
  | structured JSON | **21.00** | 14.87 | 13.80 | 10.02 |
  | Rust code | **21.18** | 13.00 | 11.96 | 8.94 |
  | narrative prose | **21.21** | 12.19 | 10.72 | 8.16 |

  DFlash costs 30–60%, and gets worse the deeper you draft. **The cause is
  acceptance.** Mean accept length is 1.52 / 1.32 / 1.23 (JSON / code / prose),
  and per-position acceptance on JSON runs 0.500, 0.150, 0.011, 0.000 — the
  drafter is right about half the time on token 1 and essentially never by token 3.

  A DFlash step costs **2.15×** a normal decode step here, so break-even needs
  accept length ≥ 2.15. Reference implementations on other engines report 3.78 on
  this same model and drafter, which would be ~1.76× — so the headroom is real,
  it is just not reachable from this engine. An independent DFlash draft GGUF for
  MiMo-V2.5-Pro on `ik_llama.cpp` reports the same shape (54.6–60.4% acceptance,
  55.6–59.4 t/s drafted vs 59.9–60.8 undrafted — also a net loss).

  Measured with **thinking off**; enabling it changes JSON and prose by ~0 and
  costs code about 16%.

  MTP is unavailable (llama.cpp issue #23924 closed `not_planned`, though these
  GGUFs do carry the `blk.48-50.nextn.*` tensors), and no EAGLE3 drafter has been
  published for MiMo-V2.5.

- **If you do experiment with DFlash over an RPC split**, the target's LM head and
  token embeddings must be pinned to the local device or it aborts at load in
  `ggml_backend_sched_backend_id_from_cur`:
  `-ot "output\.weight=ROCm0" -ot "token_embd\.weight=ROCm0" -devd ROCm0`
- KV geometry: MiMo-V2.5 has 9 full-attention + 39 sliding-window layers, so
  only the 9 full layers scale with context — about **22.5 KiB/token**.

## Speculative decoding (MTP) — present, but do not expect a win

MiMo-V2.5 ships its own **MTP / NextN head** (`model_mtp.safetensors` upstream →
`blk.48/49/50.nextn.*` in GGUF, 3 NextN layers on top of 48 trunk layers). All files
here preserve those tensors, so they are available if your build can use them.

**Using them requires engine support that is not yet in any release.**
[llama.cpp PR #26228](https://github.com/ggml-org/llama.cpp/pull/26228) adds `mimo2`
MTP draft support and is still open; the ROCmFPX fork does not carry it either. Without
it the nextn tensors are simply ignored (harmless — the files still load and run
normally on a stock build; verified).

Two warnings, both measured on this hardware, because the intuitive expectations are
wrong in both directions:

**1. Across a two-node RPC split, MTP is a 33% *loss* — even at 100% acceptance.**

| 2-node `ROCmFP4_FAST`, `-dev RPC0,ROCm0` | decode | acceptance |
| --- | ---: | --- |
| no MTP (control) | **23.55 t/s** | — |
| MTP, `n-max 1` | 15.87 t/s | **1.000** (27/27) |

Acceptance was *perfect* and it still lost a third of throughput. The deficit is
per-draft-step cross-device synchronisation, not draft quality — which means **no
better drafter can fix it.** EAGLE3 would be worse still: it needs three hidden-state
taps from layers spread across the trunk, so on a layer split some taps are always
remote, whereas MTP needs only the final pre-norm hidden state.

If you do try it, note the load-time crash and its fix: with `-dev ROCm0,RPC0` the
trailing nextn blocks land on the *remote* node, and reading the hidden state back
aborts inside `ggml_backend_rpc_buffer_get_tensor` during startup. **Flip the device
order to `-dev RPC0,ROCm0`** — layers are assigned to `-dev` entries in order, so the
last device named gets the trailing blocks. `-ot` moves *weights* but **not** the KV
cache; pinning weights alone does not fix it.

**2. On a single node, whether MTP helps depends on the quant.**

| single node | base | best MTP | verdict |
| --- | ---: | ---: | --- |
| `IQ2_XXS` (third-party, 2.06 bpw) | 20.75 t/s | **25.81 t/s** (+24%) | worth it |
| `Q2_0_ROCMFPX` (this repo) | **35.57 t/s** | 31.50 t/s (−11%) | not worth it |

Two compounding reasons MTP loses on `Q2_0_ROCMFPX`: the base is already fast, so
there is less fixed overhead to hide, and **the nextn head is itself quantized to
2.5 bpw** — mean acceptance length falls to 1.39–1.75 here versus 1.95 on `IQ2_XXS`.
Lifting `token_embd`/`output` to Q6_K does not help the drafter; the nextn tensors
stay at 2.5 bpw.

Two counter-intuitive tuning notes if you experiment: **higher `--spec-draft-p-min`
is better** (0.75 → 31.50 vs 0.10 → 30.26; fewer, more confident drafts waste less
verify work), and **`--spec-draft-n-max 1` is optimal** — depth ≥2 loses badly
(22.19 t/s at n=2). Deeper drafts widen the verify batch, and on a 256-expert top-8
MoE a wider batch activates *more experts*, so weight traffic grows faster than
accepted tokens. Mean acceptance length keeps rising with depth while throughput
falls, so acceptance alone is a misleading metric here.

## License

Inherits the license of the base model,
[XiaomiMiMo/MiMo-V2.5](https://huggingface.co/XiaomiMiMo/MiMo-V2.5). Quantization
adds no additional restrictions.

## Credits

- [XiaomiMiMo](https://huggingface.co/XiaomiMiMo) — MiMo-V2.5
- [unsloth](https://huggingface.co/unsloth) — BF16 GGUF conversion used as the source
- [ROCmFPX](https://github.com/charlie12345/ROCmFPX) — the ROCmFP4 quant formats