Upload opt/summary.md with huggingface_hub
Browse files- opt/summary.md +44 -0
opt/summary.md
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Experiment_1 / opt: model- and CPU-specific decode kernels
|
| 2 |
+
|
| 3 |
+
Target: openPangu-VL-7B-Q4_K_M.gguf (unchanged file), Intel Xeon 8559C (AVX512-VNNI), text only.
|
| 4 |
+
Stock llama.cpp (1268edac5 + pangu_vl patch) uses only AVX2 (256-bit) for Q4_K x Q8_K and Q6_K x Q8_K
|
| 5 |
+
dot products on x86. This experiment adds AVX512-VNNI versions of those two kernels (maddubs + dpwssd,
|
| 6 |
+
512-bit, software prefetch). The weight mix of this model is 73.5% Q4_K (3.62 GB) and 26.5% Q6_K (1.30 GB).
|
| 7 |
+
|
| 8 |
+
## Diagnosis (16 threads, CPUs 0-15)
|
| 9 |
+
| measurement | GB/s |
|
| 10 |
+
|---|---:|
|
| 11 |
+
| raw sequential read (synthetic) | 184-190 |
|
| 12 |
+
| stock Q4_K GEMV micro-benchmark | 93-98 |
|
| 13 |
+
| stock end-to-end decode (20.0 t/s x 4.93 GB) | 98 |
|
| 14 |
+
=> the kernel, not memory bandwidth, was the limit.
|
| 15 |
+
|
| 16 |
+
## Kernel micro-benchmark (40 matrices 4096x12800, 16 threads, best of 5, interleaved)
|
| 17 |
+
| type | stock GB/s | VNNI GB/s | gain | max rel. error vs stock |
|
| 18 |
+
|---|---:|---:|---:|---:|
|
| 19 |
+
| Q4_K | 98 | 136 | +39% | 1.3e-7 |
|
| 20 |
+
| Q6_K | 108 | 158 | +46% | 1.3e-7 |
|
| 21 |
+
|
| 22 |
+
## End-to-end llama-bench (-t 16 -fa on --repack 0, taskset 0-15, 3 interleaved rounds x r=3)
|
| 23 |
+
| build | pp512 t/s | tg128 t/s |
|
| 24 |
+
|---|---|---|
|
| 25 |
+
| stock | 240.1 / 239.4 / 225.8 (median 239.4) | 20.2 / 19.9 / 20.0 (median 20.0) |
|
| 26 |
+
| Q4_K + Q6_K VNNI | 233.7 / 234.1 / 237.0 (median 234.1) | 27.4 / 27.9 / 27.8 (median 27.8) |
|
| 27 |
+
| change | -2% (noise; prefill uses another kernel) | +39% |
|
| 28 |
+
Q4_K-only intermediate build (ab.jsonl): tg 19.9 -> 24.9 t/s (+25%).
|
| 29 |
+
CLI generation check (opt build, greedy): Generation 27.2 t/s, coherent Chinese output (gen_opt.log).
|
| 30 |
+
|
| 31 |
+
## Accuracy
|
| 32 |
+
llama-perplexity --kl-divergence, 8 chunks x 512 tokens (4096 positions), base = stock logits:
|
| 33 |
+
- optimized vs stock: Same top p = 100.000%, max KLD 5.5e-5, max Delta p 0.004%.
|
| 34 |
+
- stock vs stock control: identical values (the reported numbers are dominated by quantization of the
|
| 35 |
+
stored base logits; kernel differences are below the resolution of this test).
|
| 36 |
+
- Greedy free-running text is not bit-identical to stock (first difference at char ~21), but stock
|
| 37 |
+
itself diverges from stock when only -fa is toggled (first difference at char ~35). Near-tie tokens
|
| 38 |
+
flip under ~1e-7 relative numerical noise; this is expected and not a defect.
|
| 39 |
+
|
| 40 |
+
## Caveats
|
| 41 |
+
- Shared host: absolute numbers move +-15% between time windows; every comparison above was interleaved.
|
| 42 |
+
- Under heavy neighbour load the 16-thread (full-quota) runs can stall because of cgroup throttling.
|
| 43 |
+
- Remaining headroom: 137 GB/s achieved vs ~185 GB/s raw read. Prefill (pp) not optimized.
|
| 44 |
+
- Source changes are NOT uploaded (data only); patch is kept locally.
|