| # Experiment_1: openPangu-VL-7B-Q4_K_M, text only, CPU |
| |
| Model file: openPangu-VL-7B-Q4_K_M.gguf (unchanged, no re-quantization) |
| CPU: Intel Xeon Platinum 8559C, cgroup quota 16 CPUs, shared host (run-to-run variance about +-15%) |
| Runtime: llama.cpp 1268edac5 + openpangu-vl-llama.cpp.patch, CLI, 128 generated tokens, pinned to CPUs 0-15 |
| |
| ## Default vs optimized (llama-cli, real generation) |
| | Config | Prefill t/s | Decode t/s | |
| |---|---:|---:| |
| | default (no flags, 96 threads) | 73.4 | 5.6 | |
| | optimized (-t 16 -fa on, F16 KV, no-repack) | 147.4 | 19.4 | |
| | speedup | ~2.0x | ~3.5x | |
| |
| ## What actually caused the gain |
| | Config | Prefill t/s | Decode t/s | |
| |---|---:|---:| |
| | -t 16 only | 147.7 | 18.3 | |
| | -t 16 + FA + no-repack | 147.4 | 19.4 | |
| |
| The gain comes from using 16 threads. FA, repack on/off and the AMX build differ within noise |
| (same config repeated: prefill 119-148 t/s). AMX gave no reliable speedup. |
| |
| ## Thread scaling (llama-bench, repack on, AMX build) |
| | threads | pp512 | tg128 | |
| |---:|---:|---:| |
| | 1 | 16.28 | 1.65 | |
| | 4 | 57.46 | 6.11 | |
| | 8 | 125.15 | 11.20 | |
| | 12 | 189.09 | 14.66 | |
| | 16 | 208.93 | 18.31 | |
| | 20 | 156.98 | 9.95 | |
| | 24 | 163.43 | 8.90 | |
| |
| More than 16 threads degrades sharply (CPU quota throttling). |
| |
| Long context decode (tg64): 2048 ctx 16.63, 4096 ctx 14.88, 8192 ctx 13.44 t/s. |
| |
| Raw data: amx_threads.jsonl |
|
|
| ## Comparison with the HF model-card numbers (same machine type, llama-bench pp512 / tg) |
| Interleaved A/B, 3 rounds, taskset 0-15, same host load (results/interleaved.jsonl): |
|
|
| | Config | pp512 t/s | tg t/s | |
| |---|---:|---:| |
| | default flags (96 threads, repack on) | 185 / 191 / 191 (median 190.5) | 5.7 / 5.7 / 5.6 (median 5.7) | |
| | model-card config (-t 16 -fa on -ctk/-ctv f16 --repack 0) | 225.6 / 229.5 / 233.7 (median 229.5) | 16.8 / 17.1 / 17.3 (median 17.1) | |
| | model card as published | 212 | 14.5 | |
|
|
| - The model-card config IS the optimized config; I found nothing faster than it. Differences between |
| my 230/17 and the published 212/14.5 are within host-load variance (+-15%), not an improvement. |
| - Gain over default flags: prefill about 1.2x, decode about 3.0x (decode gain is all from -t 16). |
| - The same command at a different time gave 108 / 4.1 (readme_exact.jsonl) while host load was 54: |
| results on this shared host depend heavily on neighbours. Always compare interleaved. |
| - CLI prefill numbers (73-148 t/s) use a ~19-token prompt and are not comparable to pp512. |
| |
