experiments: fix -nr methodology (llama-bench needs --repack 0), re-measure #27590 (-nr +0.7%/+1.4%); record #27032 (kernels subset, no gain, not merged) and corrected #18698 incremental (-4.1%/-2.1%, not merged); note -nr vs default on this machine

README.md

@@ -248,7 +248,16 @@ host is noisy, each candidate is measured with an **alternating A/B
 protocol**: two binaries built from the same source tree (baseline vs
 baseline + candidate patch) are run interleaved in the same session
 (`llama-bench -t 16 -r 3 -p 512 -n 128`, default settings, 3–8 rounds each);
-the table reports the medians over all rounds.
+the table reports the medians over all rounds. Every number is
+**incremental**: each candidate is compared against the **previous merged
+state**, not against the original port.
+
+> Methodology fix (2026-10-16): a first attempt measured `-nr` via the
+> `LLAMA_ARG_REPACK=0` env var. `llama-bench` ignores that variable (it has
+> its own parser; only common-arg tools such as `llama-cli` /
+> `llama-perplexity` honor it), so those runs were silently default-mode.
+> The `-nr` entry for #27590 was re-measured with `llama-bench --repack 0`,
+> the flag the tool actually supports.
 
 Only candidates that win beyond the run-to-run noise are merged into the
 patch (✅ 有幫助); candidates without a real gain are recorded here with
@@ -259,8 +268,9 @@ their data and are **not** merged (❌ 沒幫助).
 | upstream PR [#27024](https://github.com/ggml-org/llama.cpp/pull/27024) by [@jinzihao](https://github.com/jinzihao) | ggml-cpu/amx: fix the (dead) `block_q8_K` VNNI quantization and enable the VNNI path | ✅ 有幫助 — merged | 57.25 → 59.20 (**+3.4 %**) | 10.94 → 11.24 (**+2.7 %**) |
 | upstream PR [#22525](https://github.com/ggml-org/llama.cpp/pull/22525) by [@HyeongiJeon](https://github.com/HyeongiJeon) | ggml-cpu: optimize `ggml_gemm_q4_K_8x8_q8_K` interleaving/staging for AVX-512 (and AVX2) | ✅ 有幫助（prefill）— merged | 55.78 → 57.42 (**+2.9 %**) | 10.70 → 10.87 (+1.5 %, neutral) |
 | upstream PR [#19706](https://github.com/ggml-org/llama.cpp/pull/19706) by [@Manogna-Sree](https://github.com/Manogna-Sree) | Q6_K block interleaving for x86 SIMD (AVX-512/AVX2) | ❌ 沒幫助 — not merged | 58.16 → 58.34 (+0.3 %) | 11.06 → 11.07 (+0.1 %) |
-| upstream PR [#27590](https://github.com/ggml-org/llama.cpp/pull/27590) by [@SongXiaoXi](https://github.com/SongXiaoXi) | ggml-cpu: add AVX-512/VNNI paths for Q5_K/Q6_K dot products | ✅ 有幫助（`-nr`）— merged | `-nr`: 56.41 → 58.03 (**+2.9 %**) | `-nr`: 10.85 → 11.12 (**+2.5 %**) |
-| upstream PR [#18698](https://github.com/ggml-org/llama.cpp/pull/18698) by [@zzjianhui](https://github.com/zzjianhui) | ggml-cpu: repack weights in parallel (OpenMP, NUMA-local thread placement) | ❌ 沒幫助（推論略退化）— not merged | 55.56 → 54.42 (**−2.1 %**) | 10.31 → 10.05 (**−2.5 %**) |
+| upstream PR [#27590](https://github.com/ggml-org/llama.cpp/pull/27590) by [@SongXiaoXi](https://github.com/SongXiaoXi) | ggml-cpu: add AVX-512/VNNI paths for Q5_K/Q6_K dot products | ✅ 有幫助（`-nr`）— merged | `-nr`: 65.91 → 66.35 (+0.7 %) | `-nr`: 11.16 → 11.32 (**+1.4 %**) |
+| upstream PR [#18698](https://github.com/ggml-org/llama.cpp/pull/18698) by [@zzjianhui](https://github.com/zzjianhui) | ggml-cpu: repack weights in parallel (OpenMP, NUMA-local thread placement) | ❌ 沒幫助（推論退化）— not merged | 56.93 → 54.58 (**−4.1 %**) | 10.67 → 10.45 (**−2.1 %**) |
+| draft PR [#27032](https://github.com/ggml-org/llama.cpp/pull/27032) by [@lexasub](https://github.com/lexasub) | CPU optimizations (GET_ROWS+ADD fusion, `get_rows` fast paths, tokenizer, KV-cell rework) — only the ggml-kernel subset is portable onto this tree | ❌ 沒幫助（不適用於本模型）— not merged | 56.64 → 56.88 (+0.4 %) | 10.82 → 10.73 (−0.8 %) |
 
 Round-by-round for #27024 (candidate vs baseline, t/s):
 
@@ -316,59 +326,88 @@ noise. The base `Q3_K_M` model contains only **one** single `Q6_K` tensor,
 so a faster Q6_K kernel has almost no weight to act on. The PR is *not*
 merged; the data is recorded here for the record. Measured 2026-10-16.
 
-Round-by-round for #27590 (candidate vs baseline, t/s), `-nr` mode. The
-Q5_K/Q6_K dots this PR changes are used on the plain CPU path selected by
-`-nr`; with default repacking those weights go through the repack gemv/gemm
-kernels instead, and default mode indeed measured neutral
-(pp512 −1.7 % / tg128 +1.0 %, within noise):
+Round-by-round for #27590 (candidate vs baseline, t/s), `-nr` mode
+(`llama-bench --repack 0`). The Q5_K/Q6_K dots this PR changes are used on
+the plain CPU path selected by `-nr`; with default repacking those weights go
+through the repack gemv/gemm kernels instead (default-mode incremental
+measured ~neutral, within noise):
 
 | round | pp512 | tg128 |
 |---|---:|---:|
-| 1 | 58.03 vs 58.58 | 11.65 vs 11.57 |
-| 2 | 57.50 vs 56.41 | 11.57 vs 10.61 |
-| 3 | 58.12 vs 56.24 | 11.67 vs 10.85 |
-| 4 | 56.88 vs 53.06 | 10.77 vs 9.01 |
-| 5 | 58.79 vs 56.58 | 11.12 vs 10.52 |
-| 6 | 59.25 vs 56.38 | 11.01 vs 11.14 |
-| 7 | 56.81 vs 58.01 | 10.91 vs 11.03 |
-
-**Verdict ✅ 有幫助 (`-nr` 模式, +2.9 % prefill / +2.5 % decode medians,
-5 of 7 rounds won on both).** In this model the PR affects Q5_K (2 tensors,
+| 1 | 67.88 vs 67.80 | 11.17 vs 11.15 |
+| 2 | 66.08 vs 66.46 | 11.46 vs 11.18 |
+| 3 | 66.01 vs 65.34 | 11.32 vs 11.19 |
+| 4 | 66.63 vs 65.36 | 11.33 vs 11.12 |
+
+**Verdict ✅ 有幫助 (`-nr` 模式, +0.7 % prefill / +1.4 % decode medians;
+decode won 4 of 4 rounds).** In this model the PR affects Q5_K (2 tensors,
 the dense `ffn_down` of layers 0–1) and Q6_K (1 tensor — `output.weight`,
-the 388 M-param head that is read on every token). One caveat: the upstream
-PR is *closed but never merged* (only an automated AI-content flag, no
-review), so it received extra local validation before merging:
+the 388 M-param head that is read on every token), so the consistent decode
+win matches the expectation. One caveat: the upstream PR is *closed but never
+merged* (only an automated AI-content flag, no review), so it received extra
+local validation before merging:
 
 - perplexity on a fixed corpus: 1.0354 → 1.0354 (identical)
 - `test-backend-ops -b CPU -o MUL_MAT -p 'type=q5_K|type=q6_K'` passes
 
-Merged. Measured 2026-10-16.
+Merged. Re-measured 2026-10-16 with the correct `--repack 0` flag (see
+methodology note above).
 
-Round-by-round for #18698 (candidate vs baseline, t/s), default settings.
-This PR repacks the quantized weights in parallel with OpenMP and pins each
-repack thread NUMA-local. (Ported by hand: the year-old PR conflicts with the
-current tree and depends on the never-merged PR #18239.)
+Round-by-round for #18698 (candidate vs baseline = current HEAD i.e. all
+merged patches, t/s), default settings. This PR repacks the quantized
+weights in parallel with OpenMP and pins each repack thread NUMA-local.
+(Ported by hand: the year-old PR conflicts with the current tree and depends
+on the never-merged PR #18239.)
 
 | round | pp512 | tg128 |
 |---|---:|---:|
-| 1 | 49.98 vs 53.95 | 9.66 vs 10.24 |
-| 2 | 53.62 vs 55.84 | 10.05 vs 9.81 |
-| 3 | 55.62 vs 55.29 | 9.78 vs 10.38 |
-| 4 | 53.57 vs 55.07 | 10.05 vs 10.02 |
-| 5 | 51.86 vs 56.78 | 8.98 vs 10.52 |
-| 6 | 56.96 vs 54.54 | 11.16 vs 10.05 |
-| 7 | 56.67 vs 57.68 | 11.66 vs 11.40 |
-| 8 | 55.23 vs 57.70 | 11.25 vs 11.24 |
-
-**Verdict ❌ 沒幫助 (slightly slower, not merged).** Medians: prefill −2.1 %
-(2 W / 6 L) and decode −2.5 % (3 W / 3 L / 2 T). Side effect worth recording:
-the parallel repack cuts model **load** wall time (load + tiny bench, warm
-page cache) from 24 s (2/2 runs) to 10 s (2/2 runs) — but the inference phase
-itself gains nothing, so per the merge rule (real tok/s gains) the candidate
-is **not** merged. Correctness: perplexity 1.0361 → 1.0361 (identical), i.e.
+| 1 | 54.24 vs 56.40 | 10.48 vs 10.35 |
+| 2 | 55.50 vs 57.92 | 10.37 vs 10.67 |
+| 3 | 54.58 vs 56.93 | 10.45 vs 10.87 |
+
+**Verdict ❌ 沒幫助 (clear prefill regression, not merged).** Medians:
+prefill −4.1 % (0 W / 3 L) and decode −2.1 % (1 W / 2 L); an earlier 5-round
+block against the same effective baseline agreed on prefill (−2.7 %,
+1 W / 4 L) with noisy decode. Side effect worth recording: the parallel
+repack cuts model **load** wall time (load + tiny bench, warm page cache)
+from 24 s (2/2 runs) to 10 s (2/2 runs) — but the inference phase itself
+gets slower, so per the merge rule (real tok/s gains) the candidate is
+**not** merged. Correctness: perplexity 1.0361 → 1.0361 (identical), i.e.
 the parallel repack produces identical weights; the regression is presumably
 a NUMA page-placement effect, not a data error. Measured 2026-10-16.
 
+Round-by-round for draft #27032 (candidate vs baseline = current HEAD, t/s),
+default settings. The draft bundles several optimizations (GET_ROWS+ADD
+fusion, `get_rows` fast paths, tokenizer, KV-cell rework). The KV/batch and
+tokenizer parts are either **not portable** (rooted in a pre-refactor
+KV-cell design that conflicts with this tree) or not measurable with this
+protocol; only the disjoint ggml CPU-kernel subset could be ported and
+tested:
+
+| round | pp512 | tg128 |
+|---|---:|---:|
+| 1 | 57.49 vs 56.64 | 10.73 vs 10.82 |
+| 2 | 56.88 vs 58.68 | 10.85 vs 10.75 |
+| 3 | 55.71 vs 56.50 | 10.35 vs 10.95 |
+
+**Verdict ❌ 沒幫助 (not applicable to this model, not merged).** ≈±1 % on
+both metrics, PPL identical (1.0361). The reason follows from the graph and
+kernel dispatch: (1) the GET_ROWS+ADD fusion needs an ADD node immediately
+after a GET_ROWS node — this model's graph has no such pattern; (2) the
+optimized Q4_K dequant path is not used (`token_embd` is Q3_K); (3) the
+remaining `get_rows` cache-hit path only helps repeated 1-row gathers, which
+this workload barely has. Recorded for the record. Measured 2026-10-16.
+
+### `-nr` vs default on this machine
+
+Side observation from the re-anchoring session: on this AMX machine the
+plain-CPU (`-nr`) path is clearly faster than the default repack path.
+HEAD (all merged patches): pp512 56.93 → 66.35 (**+16.6 %**) and tg128
+10.67 → 11.32 (**+6.1 %**) with `--repack 0` (3–4 interleaved rounds each,
+2026-10-16). The repack kernels used by default have no AMX path, while the
+plain Q4_K×Q8_K kernels use AMX (boosted further by #27024). On AMX-class
+machines it is worth testing `-nr` for ~+15 % prefill.
+
 ## License
 
 This project (the patch set, `patch.sh` and documentation) is licensed