# Experiment_1 / opt: model- and CPU-specific decode kernels

Target: openPangu-VL-7B-Q4_K_M.gguf (unchanged file), Intel Xeon 8559C (AVX512-VNNI), text only.
Stock llama.cpp (1268edac5 + pangu_vl patch) uses only AVX2 (256-bit) for Q4_K x Q8_K and Q6_K x Q8_K
dot products on x86. This experiment adds AVX512-VNNI versions of those two kernels (maddubs + dpwssd,
512-bit, software prefetch). The weight mix of this model is 73.5% Q4_K (3.62 GB) and 26.5% Q6_K (1.30 GB).

## Diagnosis (16 threads, CPUs 0-15)
| measurement | GB/s |
|---|---:|
| raw sequential read (synthetic) | 184-190 |
| stock Q4_K GEMV micro-benchmark | 93-98 |
| stock end-to-end decode (20.0 t/s x 4.93 GB) | 98 |
=> the kernel, not memory bandwidth, was the limit.

## Kernel micro-benchmark (40 matrices 4096x12800, 16 threads, best of 5, interleaved)
| type | stock GB/s | VNNI GB/s | gain | max rel. error vs stock |
|---|---:|---:|---:|---:|
| Q4_K | 98 | 136 | +39% | 1.3e-7 |
| Q6_K | 108 | 158 | +46% | 1.3e-7 |

## End-to-end llama-bench (-t 16 -fa on --repack 0, taskset 0-15, 3 interleaved rounds x r=3)
| build | pp512 t/s | tg128 t/s |
|---|---|---|
| stock | 240.1 / 239.4 / 225.8 (median 239.4) | 20.2 / 19.9 / 20.0 (median 20.0) |
| Q4_K + Q6_K VNNI | 233.7 / 234.1 / 237.0 (median 234.1) | 27.4 / 27.9 / 27.8 (median 27.8) |
| change | -2% (noise; prefill uses another kernel) | +39% |
Q4_K-only intermediate build (ab.jsonl): tg 19.9 -> 24.9 t/s (+25%).
CLI generation check (opt build, greedy): Generation 27.2 t/s, coherent Chinese output (gen_opt.log).

## Accuracy
llama-perplexity --kl-divergence, 8 chunks x 512 tokens (4096 positions), base = stock logits:
- optimized vs stock: Same top p = 100.000%, max KLD 5.5e-5, max Delta p 0.004%.
- stock vs stock control: identical values (the reported numbers are dominated by quantization of the
  stored base logits; kernel differences are below the resolution of this test).
- Greedy free-running text is not bit-identical to stock (first difference at char ~21), but stock
  itself diverges from stock when only -fa is toggled (first difference at char ~35). Near-tie tokens
  flip under ~1e-7 relative numerical noise; this is expected and not a defect.

## Caveats
- Shared host: absolute numbers move +-15% between time windows; every comparison above was interleaved.
- Under heavy neighbour load the 16-thread (full-quota) runs can stall because of cgroup throttling.
- Remaining headroom: 137 GB/s achieved vs ~185 GB/s raw read. Prefill (pp) not optimized.
- The patch is published as openpangu-vl-cpu-vnni.patch in HelloSun/openPangu-VL-7B-Q4KM-gguf (opt-in: CPU_VNNI=1).
- Reproduced from scratch with the published build script (verify2_ab.jsonl): stock tg 15.1-15.4 vs VNNI 21.6-22.5 t/s
  (busier host window, ratio ~1.44x).


## Update: prefetch tuning (final)
Standalone kernel harness (kbench_*.txt) showed compute alone sustains ~232 GB/s (L1-resident) but streaming was
latency-bound; prefetching ~9 KB ahead into L2 (T1) raised Q4_K from 141 to ~182 GB/s (raw read ceiling ~197).
Final patch: Q4_K prefetch 64 blocks, Q6_K 44 blocks, _MM_HINT_T1.
| window | stock tg128 | VNNI tg128 | ratio |
|---|---|---|---|
| clean (dev build, ab3.jsonl) | 20.1 / 20.1 / 20.1 | 32.4 / 32.6 / 26.4 | 1.61x |
| busy (fresh build with published script, verify3_ab.jsonl) | 15.3 / 15.3 / 15.4 | 25.0 / 24.5 / 24.0 | 1.60x |
Per-token profile (prof_decode.txt, clean window): Q4_K matmul 19.2 ms (188 GB/s), Q6_K 7.1 ms (183 GB/s),
barrier wait ~2.7 ms, other ops ~1.4 ms. 32.4 t/s = ~160 GB/s = 82-86% of the 185-195 GB/s read ceiling.
Tried and rejected: guided (small tail chunk) scheduling (no net change, 30.4 -> 30.5 ms/token);
transparent hugepages (raw read ceiling at most +3%, hugepage_bw.txt).
KL (verify3_kld.txt): Same top p 100%, max dp 0.004%. CLI (verify3_cli.txt): correct answer.