Experiment_1: openPangu-VL-7B-Q4_K_M, text only, CPU
Model file: openPangu-VL-7B-Q4_K_M.gguf (unchanged, no re-quantization) CPU: Intel Xeon Platinum 8559C, cgroup quota 16 CPUs, shared host (run-to-run variance about +-15%) Runtime: llama.cpp 1268edac5 + openpangu-vl-llama.cpp.patch, CLI, 128 generated tokens, pinned to CPUs 0-15
Default vs optimized (llama-cli, real generation)
| Config | Prefill t/s | Decode t/s |
|---|---|---|
| default (no flags, 96 threads) | 73.4 | 5.6 |
| optimized (-t 16 -fa on, F16 KV, no-repack) | 147.4 | 19.4 |
| speedup | ~2.0x | ~3.5x |
What actually caused the gain
| Config | Prefill t/s | Decode t/s |
|---|---|---|
| -t 16 only | 147.7 | 18.3 |
| -t 16 + FA + no-repack | 147.4 | 19.4 |
The gain comes from using 16 threads. FA, repack on/off and the AMX build differ within noise (same config repeated: prefill 119-148 t/s). AMX gave no reliable speedup.
Thread scaling (llama-bench, repack on, AMX build)
| threads | pp512 | tg128 |
|---|---|---|
| 1 | 16.28 | 1.65 |
| 4 | 57.46 | 6.11 |
| 8 | 125.15 | 11.20 |
| 12 | 189.09 | 14.66 |
| 16 | 208.93 | 18.31 |
| 20 | 156.98 | 9.95 |
| 24 | 163.43 | 8.90 |
More than 16 threads degrades sharply (CPU quota throttling).
Long context decode (tg64): 2048 ctx 16.63, 4096 ctx 14.88, 8192 ctx 13.44 t/s.
Raw data: amx_threads.jsonl
Comparison with the HF model-card numbers (same machine type, llama-bench pp512 / tg)
Interleaved A/B, 3 rounds, taskset 0-15, same host load (results/interleaved.jsonl):
| Config | pp512 t/s | tg t/s |
|---|---|---|
| default flags (96 threads, repack on) | 185 / 191 / 191 (median 190.5) | 5.7 / 5.7 / 5.6 (median 5.7) |
| model-card config (-t 16 -fa on -ctk/-ctv f16 --repack 0) | 225.6 / 229.5 / 233.7 (median 229.5) | 16.8 / 17.1 / 17.3 (median 17.1) |
| model card as published | 212 | 14.5 |
- The model-card config IS the optimized config; I found nothing faster than it. Differences between my 230/17 and the published 212/14.5 are within host-load variance (+-15%), not an improvement.
- Gain over default flags: prefill about 1.2x, decode about 3.0x (decode gain is all from -t 16).
- The same command at a different time gave 108 / 4.1 (readme_exact.jsonl) while host load was 54: results on this shared host depend heavily on neighbours. Always compare interleaved.
- CLI prefill numbers (73-148 t/s) use a ~19-token prompt and are not comparable to pp512.