HelloSun
docs: add PROMPT_REPRODUCE.md (step-by-step task brief to reproduce this port + experiment project); fix diff stat in file list; link prompt in README
7263337 | license: gpl-3.0 | |
| library_name: llama.cpp | |
| tags: | |
| - openpangu | |
| - llama.cpp | |
| - moe | |
| - gguf | |
| - cpu-inference | |
| # openPangu-2.0-Flash support for llama.cpp v0.7.0 | |
| This repository contains a patch that adds support for **openPangu-2.0-Flash** | |
| (92B MoE, ~6B active parameters, 512K context) to the latest llama.cpp release | |
| **v0.7.0** (commit `620d2225cc436ab0b001e3200e590e3b286ef9ab`, 2026-10-15). | |
| The support was ported from | |
| [mrexodia/llama.cpp-openPangu-2.0-Flash](https://github.com/mrexodia/llama.cpp-openPangu-2.0-Flash) | |
| (which tracks an older llama.cpp base) and rebased on top of v0.7.0. | |
| Files: | |
| - `llama.cpp-openpangu-v0.7.0.diff` — the full patch (48 files, +3340 / −368) | |
| - `patch.sh` — downloads the pinned llama.cpp commit, applies the patch and | |
| prints build/run instructions | |
| - `PROMPT_REPRODUCE.md` — task brief for reproducing this project with a | |
| coding agent (port + A/B experiment protocol + validation + upload) | |
| - `LICENSE` — GPL-3.0 (this project) with the references/licenses of the | |
| upstream projects it is built on | |
| ## 中文說明 | |
| 本 repo 是將 openPangu-2.0-Flash（92B MoE、約 6B 激活參數、512K 上下文） | |
| 的 llama.cpp 支援，從 mrexodia 的 fork 移植到最新的 llama.cpp **v0.7.0** 上。 | |
| 用法： | |
| 1. 執行 `./patch.sh`（會自動下載 v0.7.0 原始碼並套用 patch） | |
| 2. `cd llama.cpp-620d2225cc436ab0b001e3200e590e3b286ef9ab` | |
| 3. `cmake -B build -S . -DCMAKE_BUILD_TYPE=Release && cmake --build build -j` | |
| 4. 下載 GGUF 模型（見下方連結）後即可執行 `llama-cli` / `llama-server` | |
| ## What is included | |
| - The `openpangu-v2` architecture: absorbed MLA attention, DSA sparse attention | |
| (lightning-indexer top-2048 selection on the global layers), per-layer | |
| sliding-window attention via a hybrid iSWA cache, 4-stream manifold | |
| hyper-connections (mHC), MoME causal convolutions with recurrent cross-batch | |
| state, and 128 learned cache-resident attention sinks per layer | |
| - Fused CPU/CUDA kernels for the mHC mixing chain (`GGML_OP_SINKHORN`, | |
| `GGML_OP_HC_MIX`), with automatic fallback to the unfused ops on backends | |
| without the kernels | |
| - Gather-based sparse attention at decode: DSA layers attend only the top-2048 | |
| selected tokens (plus sinks), with a fused indexer-scoring kernel | |
| (`GGML_OP_DSA_SCORE`) | |
| - Fused DSA indexer scoring at prefill (tiled kernel that skips the multi-GB | |
| scoring intermediate) plus a radix-select top-k for `GGML_OP_TOP_K` | |
| - Multi-token-prediction self-speculative decoding (`--spec-type draft-mtp`, | |
| with the split MTP draft GGUF) | |
| - Chat support: tool calling and ` thinking` reasoning parsing for the openPangu | |
| template | |
| - HF → GGUF conversion (`conversion/pangu.py`) and a base/MTP split tool | |
| (`conversion/split_pangu_mtp.py`) | |
| Ported and adapted to v0.7.0 APIs, plus fixes on top of the original fork: | |
| - adapted to the v0.7.0 common chat parser API (`build_chat_peg_parser`), | |
| per-layer `n_ff_exp()`, and the new `build_attn_mha(..., n_kv_max, ...)` | |
| signature | |
| - CPU AMX fix: the AMX weight packer only implements 2D layouts; batched | |
| (3D/4D) quantized weights — e.g. the MLA `wk_b`/`wv_b` tensors — were | |
| corrupted when routed through the AMX extra buffer, producing gibberish | |
| output. The AMX backend now rejects batched tensors and they run on the | |
| plain CPU path | |
| - the absorbed-MLA `wk_b`/`wv_b` inputs are kept contiguous so they are also | |
| correct with weight repacking on other backends | |
| ## Requirements | |
| - A C/C++ build environment (CMake ≥ 3.14, C++17 compiler) | |
| - No GPU required (CPU-only works); CUDA/ Metal/ Vulkan also supported | |
| - Get the GGUF models at | |
| [mrexodia/openPangu-2.0-Flash-GGUF](https://huggingface.co/mrexodia/openPangu-2.0-Flash-GGUF): | |
| - `openPangu-2.0-Flash-base-Q3_K_M.gguf` — 40.7 GiB | |
| - `openPangu-2.0-Flash-base-Q4_K_M.gguf` — 51.5 GiB | |
| - `openPangu-2.0-Flash-base-Q8_0.gguf` — 90.8 GiB | |
| - `openPangu-2.0-Flash-base-BF16.gguf` — 170.7 GiB | |
| - `openPangu-2.0-Flash-mtp-Q8_0.gguf` — 9.2 GiB (optional MTP head) | |
| ## Quick start | |
| ```sh | |
| ./patch.sh | |
| cd llama.cpp-620d2225cc436ab0b001e3200e590e3b286ef9ab | |
| cmake -B build -S . -DCMAKE_BUILD_TYPE=Release | |
| cmake --build build -j --target llama-cli llama-server | |
| ``` | |
| CUDA build: | |
| ```sh | |
| cmake -B build -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON | |
| ``` | |
| ## Run | |
| ```sh | |
| ./build/bin/llama-cli -m openPangu-2.0-Flash-base-Q3_K_M.gguf -c 65536 --jinja -t $(nproc) | |
| ./build/bin/llama-server -m openPangu-2.0-Flash-base-Q3_K_M.gguf \ | |
| -c 65536 --jinja --host 0.0.0.0 --port 8080 | |
| ``` | |
| Context scales up to `-c 524288`; the compressed MLA KV cache needs only ~12 GB | |
| at the full 512K. | |
| ### Multi-token prediction (self-speculative decoding) | |
| ```sh | |
| ./build/bin/llama-cli -m openPangu-2.0-Flash-base-Q3_K_M.gguf \ | |
| --spec-draft-model openPangu-2.0-Flash-mtp-Q8_0.gguf \ | |
| --spec-type draft-mtp -c 65536 --jinja | |
| ``` | |
| Notes: | |
| - The MTP draft adds ~10 GB of RAM usage on top of the base model | |
| - On CPUs with AMX, weight repacking (`--repack`, enabled by default) costs | |
| ~16 GiB of extra RAM; if the combined footprint does not fit, run with `-nr` | |
| (no repack) | |
| - On CPU-only setups the draft overhead can make generation slower than plain | |
| decoding — measure on your hardware (`llama-bench` does not support draft | |
| models; use `llama-cli` and compare the reported t/s) | |
| ## Test environment（測試環境） | |
| All performance numbers below were measured on a real machine: | |
| - **CPU**: Intel Xeon Platinum 8559C (Emerald Rapids, supports AMX); the test | |
| container is capped at **16 vCPUs** (host has 2 sockets × 48 cores × 2 SMT) | |
| - **RAM**: 2.0 TiB host; the test environment enforces a **64 GiB per-process | |
| memory cap**, which is why all testing used the 40.7 GiB `Q3_K_M` quant | |
| - **GPU**: none — CPU-only inference, 16 threads (`-t 16`) | |
| - **OS/toolchain**: Linux, gcc 14.2.0, CMake 3.31.6 | |
| - **Model**: `openPangu-2.0-Flash-base-Q3_K_M.gguf` (40.72 GiB, 91.6 B params) | |
| Measured resident memory of `llama-cli` with this model (`-c 512`, `-n 1`): | |
| | configuration | peak RSS | | |
| |---|---| | |
| | default (AMX weight repacking enabled) | ~57.1 GiB | | |
| | `-nr` (no repack) | ~41.2 GiB | | |
| | `-nr` + MTP draft (Q8_0) | ~52 GiB | | |
| | default + MTP draft (Q8_0) | exceeds 64 GiB — could not run under the test cap | | |
| ## Tests | |
| - full `ctest` suite (61 tests): **61/61 pass** — including | |
| `test-generate-models` and `test-llama-context`, which now cover | |
| openpangu-v2 as a first-class arch (synthetic model fixtures, KV-cache | |
| cases) and round-trip its K-sink-prefix metadata through the model saver | |
| - `test-backend-ops -b CPU -o SINKHORN -o HC_MIX -o DSA_SCORE -o TOP_K`: | |
| 534/534 tests pass | |
| - `test-chat`, `test-chat-template`, `test-chat-peg-parser`, | |
| `test-chat-auto-parser`: all pass | |
| ## Comparison with the original fork | |
| Measured against the original | |
| [mrexodia/llama.cpp-openPangu-2.0-Flash](https://github.com/mrexodia/llama.cpp-openPangu-2.0-Flash) | |
| (HEAD `76c92350b`, llama.cpp base `6f4f53f2b` from 2026-06-29), same machine, | |
| same model, same flags. | |
| ### Compatibility on this machine (AMX CPU) | |
| | | original fork | this port (v0.7.0) | | |
| |---|---|---| | |
| | default settings | **aborts** — `GGML_ASSERT(*cur_backend_id != -1)` during `sched_reserve` (AMX weight repacking) | runs | | |
| | `-nr` (weight repacking disabled) | runs | runs | | |
| The original fork cannot be used with default settings on a CPU with AMX. | |
| Its `llama-bench` has no option to disable repacking at all — the original | |
| numbers below required a local source edit. This port fixes the underlying | |
| AMX packing bug (the AMX packer silently corrupted batched/3D weights). | |
| ### llama-bench, `Q3_K_M`, `-t 16 -r 3` (2026-10-16) | |
| Both builds with weight repacking disabled (`-nr`, the only common | |
| runnable configuration on this machine): | |
| | test | original fork | this port | change | | |
| |------|--------------:|----------:|-------:| | |
| | pp128 | 41.51 ± 2.21 | 60.47 ± 4.63 | **+46 %** | | |
| | pp512 | 42.62 ± 0.83 | 64.75 ± 2.26 | **+52 %** | | |
| | tg32 | 11.65 ± 0.19 | 11.52 ± 0.35 | ≈ | | |
| | tg128 | 11.47 ± 0.21 | 11.82 ± 0.72 | ≈ | | |
| This port with the default AMX weight repacking (original fork cannot run | |
| this configuration): | |
| | test | this port (default) | | |
| |------|--------------------:| | |
| | pp128 | 56.84 ± 1.32 | | |
| | pp512 | 62.65 ± 0.54 | | |
| | tg32 | 11.94 ± 0.71 | | |
| | tg128 | 12.02 ± 0.30 | | |
| Reading these numbers: | |
| - Prefill is ~1.5× faster than the original fork, mostly from the v0.7.0 | |
| base (3.5 months of upstream CPU optimizations). Generation is on par. | |
| - On this CPU, AMX weight repacking is performance-neutral and costs | |
| ~16 GiB of extra RAM, so `-nr` is usually the better choice. | |
| - The host is shared and measurements vary between runs (see the ±). | |
| ### Real inference (llama-cli end-to-end) | |
| Actual generation with a real prompt (3,271 tokens of English text), | |
| `llama-cli -c 8192 -n 128 -t 16 --no-warmup -st`, two runs each | |
| (prompt eval / generation speed as reported by llama-cli): | |
| | configuration | prompt eval | generation | | |
| |---------------|------------:|-----------:| | |
| | original fork, `-nr` | 29.7 / 28.9 t/s | 6.1 / 6.1 t/s | | |
| | this port, `-nr` | 41.0 / 41.0 t/s | 6.0 / 6.0 t/s | | |
| | this port, default (AMX repack) | 37.7 / 38.1 t/s | 5.6 / 5.5 t/s | | |
| - real prompt processing is ~**+40 %** faster than the original fork | |
| (+29 % in the default AMX repack configuration) | |
| - generation with a ~3K-token context is ~6 t/s in all configurations | |
| (the DSA indexer adds per-token work over the cached context, so this is | |
| lower than the tg128 llama-bench numbers on a fresh 128-token run) | |
| - run-to-run variance on this host is small for these end-to-end runs | |
| ### MTP (self-speculative decoding) | |
| Same prompt, `llama-cli -n 128 -nr`: | |
| | | generation | | |
| |---|---| | |
| | original fork, MTP draft | 4.9 t/s | | |
| | this port, MTP draft | 4.7 t/s | | |
| | either fork, plain decoding (`llama-bench` tg128) | 11.5–12.0 t/s | | |
| On CPU the MTP draft head is a net loss in both forks; it is included for | |
| completeness and may pay off on GPUs where the draft forward pass is cheap. | |
| ## CPU optimization experiments（CPU 優化實驗記錄） | |
| Upstream CPU-speedup candidates (open pull requests of llama.cpp) are | |
| benchmarked one by one on the machine described above. Because this shared | |
| host is noisy, each candidate is measured with an **alternating A/B | |
| protocol**: two binaries built from the same source tree (baseline vs | |
| baseline + candidate patch) are run interleaved in the same session | |
| (`llama-bench -t 16 -r 3 -p 512 -n 128`, default settings, 3–8 rounds each); | |
| the table reports the medians over all rounds. Every number is | |
| **incremental**: each candidate is compared against the **previous merged | |
| state**, not against the original port. | |
| > Methodology fix (2026-10-16): a first attempt measured `-nr` via the | |
| > `LLAMA_ARG_REPACK=0` env var. `llama-bench` ignores that variable (it has | |
| > its own parser; only common-arg tools such as `llama-cli` / | |
| > `llama-perplexity` honor it), so those runs were silently default-mode. | |
| > The `-nr` entry for #27590 was re-measured with `llama-bench --repack 0`, | |
| > the flag the tool actually supports. | |
| Only candidates that win beyond the run-to-run noise are merged into the | |
| patch (✅ 有幫助); candidates without a real gain are recorded here with | |
| their data and are **not** merged (❌ 沒幫助). | |
| | candidate | content | verdict | pp512 t/s (median) | tg128 t/s (median) | | |
| |---|---|---:|---:|---:| | |
| | upstream PR [#27024](https://github.com/ggml-org/llama.cpp/pull/27024) by [@jinzihao](https://github.com/jinzihao) | ggml-cpu/amx: fix the (dead) `block_q8_K` VNNI quantization and enable the VNNI path | ✅ 有幫助 — merged | 57.25 → 59.20 (**+3.4 %**) | 10.94 → 11.24 (**+2.7 %**) | | |
| | upstream PR [#22525](https://github.com/ggml-org/llama.cpp/pull/22525) by [@HyeongiJeon](https://github.com/HyeongiJeon) | ggml-cpu: optimize `ggml_gemm_q4_K_8x8_q8_K` interleaving/staging for AVX-512 (and AVX2) | ✅ 有幫助（prefill）— merged | 55.78 → 57.42 (**+2.9 %**) | 10.70 → 10.87 (+1.5 %, neutral) | | |
| | upstream PR [#19706](https://github.com/ggml-org/llama.cpp/pull/19706) by [@Manogna-Sree](https://github.com/Manogna-Sree) | Q6_K block interleaving for x86 SIMD (AVX-512/AVX2) | ❌ 沒幫助 — not merged | 58.16 → 58.34 (+0.3 %) | 11.06 → 11.07 (+0.1 %) | | |
| | upstream PR [#27590](https://github.com/ggml-org/llama.cpp/pull/27590) by [@SongXiaoXi](https://github.com/SongXiaoXi) | ggml-cpu: add AVX-512/VNNI paths for Q5_K/Q6_K dot products | ✅ 有幫助（`-nr`）— merged | `-nr`: 65.91 → 66.35 (+0.7 %) | `-nr`: 11.16 → 11.32 (**+1.4 %**) | | |
| | upstream PR [#18698](https://github.com/ggml-org/llama.cpp/pull/18698) by [@zzjianhui](https://github.com/zzjianhui) | ggml-cpu: repack weights in parallel (OpenMP, NUMA-local thread placement) | ❌ 沒幫助（推論退化）— not merged | 56.93 → 54.58 (**−4.1 %**) | 10.67 → 10.45 (**−2.1 %**) | | |
| | draft PR [#27032](https://github.com/ggml-org/llama.cpp/pull/27032) by [@lexasub](https://github.com/lexasub) | CPU optimizations (GET_ROWS+ADD fusion, `get_rows` fast paths, tokenizer, KV-cell rework) — only the ggml-kernel subset is portable onto this tree | ❌ 沒幫助（不適用於本模型）— not merged | 56.64 → 56.88 (+0.4 %) | 10.82 → 10.73 (−0.8 %) | | |
| | draft PR [#27396](https://github.com/ggml-org/llama.cpp/pull/27396) by [@lexasub](https://github.com/lexasub) | speculative draft-model prefill (zero-copy embedding, event-based sync) for EAGLE3/DFlash/GLM4-style draft heads | ❌ 沒幫助（不適用；未移植）— not merged | — | — | | |
| Round-by-round for #27024 (candidate vs baseline, t/s): | |
| | round | pp512 | tg128 | | |
| |---|---:|---:| | |
| | 1 | 57.66 vs 57.69 | 11.09 vs 10.57 | | |
| | 2 | 59.20 vs 57.25 | 11.05 vs 10.94 | | |
| | 3 | 58.66 vs 56.57 | 11.47 vs 11.27 | | |
| | 4 | 62.13 vs 55.67 | 11.62 vs 10.83 | | |
| | 5 | 61.76 vs 65.98 | 11.24 vs 12.01 | | |
| The candidate won 4 of 5 pairwise rounds on both metrics; the single losing | |
| round (r5) is a transient quiet window on the host (both absolute values | |
| spike upward). Median deltas stay positive (+3.4 % / +2.7 %), so the change | |
| is merged. All values t/s; measured 2026-10-16. | |
| Round-by-round for #22525 (candidate vs baseline, t/s), default settings. | |
| Note: `Q3_K_M` does contain 134 `Q4_K` tensors (the MoE expert weights), | |
| so the Q4_K GEMM path this PR optimizes is actually exercised: | |
| | round | pp512 | tg128 | | |
| |---|---:|---:| | |
| | 1 | 56.69 vs 57.18 | 10.97 vs 10.97 | | |
| | 2 | 57.30 vs 57.15 | 10.88 vs 10.78 | | |
| | 3 | 57.30 vs 55.90 | 10.30 vs 10.94 | | |
| | 4 | 56.26 vs 55.66 | 10.85 vs 9.51 | | |
| | 5 | 57.91 vs 55.07 | 10.55 vs 10.62 | | |
| | 6 | 57.55 vs 56.81 | 10.60 vs 10.19 | | |
| | 7 | 57.84 vs 55.52 | 11.04 vs 10.43 | | |
| | 8 | 59.24 vs 55.06 | 10.97 vs 11.05 | | |
| The candidate won 6 of 8 pairwise rounds on prefill (1 loss, 1 tie within | |
| 0.5 %); decode is neutral (4 W / 3 L / 1 T). The PR only changes the AVX-512 | |
| interleaving/staging of the Q4_K GEMM (the prefill path), so the measured | |
| profile matches the expectation. Merged. Measured 2026-10-16. | |
| Round-by-round for #19706 (candidate vs baseline, t/s), default settings: | |
| | round | pp512 | tg128 | | |
| |---|---:|---:| | |
| | 1 | 58.42 vs 53.67 | 11.67 vs 11.45 | | |
| | 2 | 59.46 vs 58.68 | 12.04 vs 10.84 | | |
| | 3 | 54.57 vs 60.04 | 10.54 vs 11.82 | | |
| | 4 | 62.32 vs 57.06 | 10.95 vs 11.16 | | |
| | 5 | 57.66 vs 58.59 | 11.10 vs 10.86 | | |
| | 6 | 57.26 vs 57.92 | 11.05 vs 11.17 | | |
| | 7 | 59.71 vs 56.95 | 10.98 vs 10.96 | | |
| | 8 | 58.25 vs 58.41 | 11.11 vs 10.83 | | |
| **Verdict ❌ 沒幫助 (no measurable gain).** Medians differ by +0.3 % | |
| (prefill) and +0.1 % (decode), with a 4 W / 3 L / 1 T split on both — pure | |
| noise. The base `Q3_K_M` model contains only **one** single `Q6_K` tensor, | |
| so a faster Q6_K kernel has almost no weight to act on. The PR is *not* | |
| merged; the data is recorded here for the record. Measured 2026-10-16. | |
| Round-by-round for #27590 (candidate vs baseline, t/s), `-nr` mode | |
| (`llama-bench --repack 0`). The Q5_K/Q6_K dots this PR changes are used on | |
| the plain CPU path selected by `-nr`; with default repacking those weights go | |
| through the repack gemv/gemm kernels instead (default-mode incremental | |
| measured ~neutral, within noise): | |
| | round | pp512 | tg128 | | |
| |---|---:|---:| | |
| | 1 | 67.88 vs 67.80 | 11.17 vs 11.15 | | |
| | 2 | 66.08 vs 66.46 | 11.46 vs 11.18 | | |
| | 3 | 66.01 vs 65.34 | 11.32 vs 11.19 | | |
| | 4 | 66.63 vs 65.36 | 11.33 vs 11.12 | | |
| **Verdict ✅ 有幫助 (`-nr` 模式, +0.7 % prefill / +1.4 % decode medians; | |
| decode won 4 of 4 rounds).** In this model the PR affects Q5_K (2 tensors, | |
| the dense `ffn_down` of layers 0–1) and Q6_K (1 tensor — `output.weight`, | |
| the 388 M-param head that is read on every token), so the consistent decode | |
| win matches the expectation. One caveat: the upstream PR is *closed but never | |
| merged* (only an automated AI-content flag, no review), so it received extra | |
| local validation before merging: | |
| - perplexity on a fixed corpus: 1.0354 → 1.0354 (identical) | |
| - `test-backend-ops -b CPU -o MUL_MAT -p 'type=q5_K|type=q6_K'` passes | |
| Merged. Re-measured 2026-10-16 with the correct `--repack 0` flag (see | |
| methodology note above). | |
| Round-by-round for #18698 (candidate vs baseline = current HEAD i.e. all | |
| merged patches, t/s), default settings. This PR repacks the quantized | |
| weights in parallel with OpenMP and pins each repack thread NUMA-local. | |
| (Ported by hand: the year-old PR conflicts with the current tree and depends | |
| on the never-merged PR #18239.) | |
| | round | pp512 | tg128 | | |
| |---|---:|---:| | |
| | 1 | 54.24 vs 56.40 | 10.48 vs 10.35 | | |
| | 2 | 55.50 vs 57.92 | 10.37 vs 10.67 | | |
| | 3 | 54.58 vs 56.93 | 10.45 vs 10.87 | | |
| **Verdict ❌ 沒幫助 (clear prefill regression, not merged).** Medians: | |
| prefill −4.1 % (0 W / 3 L) and decode −2.1 % (1 W / 2 L); an earlier 5-round | |
| block against the same effective baseline agreed on prefill (−2.7 %, | |
| 1 W / 4 L) with noisy decode. Side effect worth recording: the parallel | |
| repack cuts model **load** wall time (load + tiny bench, warm page cache) | |
| from 24 s (2/2 runs) to 10 s (2/2 runs) — but the inference phase itself | |
| gets slower, so per the merge rule (real tok/s gains) the candidate is | |
| **not** merged. Correctness: perplexity 1.0361 → 1.0361 (identical), i.e. | |
| the parallel repack produces identical weights; the regression is presumably | |
| a NUMA page-placement effect, not a data error. Measured 2026-10-16. | |
| Round-by-round for draft #27032 (candidate vs baseline = current HEAD, t/s), | |
| default settings. The draft bundles several optimizations (GET_ROWS+ADD | |
| fusion, `get_rows` fast paths, tokenizer, KV-cell rework). The KV/batch and | |
| tokenizer parts are either **not portable** (rooted in a pre-refactor | |
| KV-cell design that conflicts with this tree) or not measurable with this | |
| protocol; only the disjoint ggml CPU-kernel subset could be ported and | |
| tested: | |
| | round | pp512 | tg128 | | |
| |---|---:|---:| | |
| | 1 | 57.49 vs 56.64 | 10.73 vs 10.82 | | |
| | 2 | 56.88 vs 58.68 | 10.85 vs 10.75 | | |
| | 3 | 55.71 vs 56.50 | 10.35 vs 10.95 | | |
| **Verdict ❌ 沒幫助 (not applicable to this model, not merged).** ≈±1 % on | |
| both metrics, PPL identical (1.0361). The reason follows from the graph and | |
| kernel dispatch: (1) the GET_ROWS+ADD fusion needs an ADD node immediately | |
| after a GET_ROWS node — this model's graph has no such pattern; (2) the | |
| optimized Q4_K dequant path is not used (`token_embd` is Q3_K); (3) the | |
| remaining `get_rows` cache-hit path only helps repeated 1-row gathers, which | |
| this workload barely has. Recorded for the record. Measured 2026-10-16. | |
| Recorded verdict for draft #27396 (not ported, not measured): the draft | |
| optimizes the *speculative draft-model prefill* path (zero-copy embedding, | |
| event-based sync) for EAGLE3 / DFlash / GLM4-style draft heads, and it | |
| touches core files (`llama-batch`, `llama-context`, `llama-graph`, ...) that | |
| have diverged from its base — 7 of 17 files conflict and a faithful port | |
| would require redesigning the speculative prefill plumbing. This CPU setup | |
| does not use speculative decoding at all — the bundled MTP head measured | |
| net-negative on CPU (see the inference section above) — so the draft has no | |
| applicable surface here. Recorded 2026-10-16. | |
| ### Combined effect of the merged set | |
| Interleaved A/B of the shipped tree (original port + all merged candidates) | |
| against the original port, same session, 3 rounds each, medians: | |
| | config | pp512 | tg128 | | |
| |---|---:|---:| | |
| | default, original port → shipped | 55.65 → 56.30 (**+1.2 %**, won 3/3) | 10.50 → 10.86 (**+3.4 %**, won 3/3) | | |
| | `-nr`, original port → shipped | 60.96 → 64.87 (**+6.4 %**, won 3/3) | 10.20 → 10.26 (+0.6 %, noisy) | | |
| The per-mechanism incremental numbers in the sections above are the | |
| authoritative records; this table is the same data rolled up against the | |
| original-port baseline in a single session. Note the separate, much larger | |
| `-nr` vs default finding below. | |
| ### `-nr` vs default on this machine | |
| Side observation from the re-anchoring session: on this AMX machine the | |
| plain-CPU (`-nr`) path is clearly faster than the default repack path. | |
| HEAD (all merged patches): pp512 56.93 → 66.35 (**+16.6 %**) and tg128 | |
| 10.67 → 11.32 (**+6.1 %**) with `--repack 0` (3–4 interleaved rounds each, | |
| 2026-10-16). The repack kernels used by default have no AMX path, while the | |
| plain Q4_K×Q8_K kernels use AMX (boosted further by #27024). On AMX-class | |
| machines it is worth testing `-nr` for ~+15 % prefill. | |
| ## License | |
| This project (the patch set, `patch.sh` and documentation) is licensed | |
| under **GPL-3.0** — see [`LICENSE`](LICENSE). The patch is a derivative of | |
| llama.cpp; the MIT-licensed portions remain under their original licenses, | |
| with the references listed in the `LICENSE` header and repeated here: | |
| - [llama.cpp](https://github.com/ggml-org/llama.cpp) — MIT — © ggml-org | |
| contributors (base commit `620d2225cc436ab0b001e3200e590e3b286ef9ab`) | |
| - [mrexodia/llama.cpp-openPangu-2.0-Flash](https://github.com/mrexodia/llama.cpp-openPangu-2.0-Flash) | |
| — MIT — original openPangu-2.0-Flash port | |
| - upstream PR [#27024](https://github.com/ggml-org/llama.cpp/pull/27024) | |
| by [@jinzihao](https://github.com/jinzihao) — MIT (merged) | |
| - upstream PR [#22525](https://github.com/ggml-org/llama.cpp/pull/22525) | |
| by [@HyeongiJeon](https://github.com/HyeongiJeon) — MIT (merged) | |
| - upstream PR [#27590](https://github.com/ggml-org/llama.cpp/pull/27590) | |
| by [@SongXiaoXi](https://github.com/SongXiaoXi) — MIT (merged; closed | |
| upstream unmerged, verified locally) | |
| - [ggml](https://github.com/ggml-org/ggml) — MIT (bundled with llama.cpp) | |