File size: 2,428 Bytes
4e64528 3e2e988 4e64528 3e2e988 4e64528 3e2e988 4e64528 3e2e988 4e64528 3e2e988 4e64528 3e2e988 4e64528 3e2e988 4e64528 3fb76ae | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | # Experiment_1: openPangu-VL-7B-Q4_K_M, text only, CPU
Model file: openPangu-VL-7B-Q4_K_M.gguf (unchanged, no re-quantization)
CPU: Intel Xeon Platinum 8559C, cgroup quota 16 CPUs, shared host (run-to-run variance about +-15%)
Runtime: llama.cpp 1268edac5 + openpangu-vl-llama.cpp.patch, CLI, 128 generated tokens, pinned to CPUs 0-15
## Default vs optimized (llama-cli, real generation)
| Config | Prefill t/s | Decode t/s |
|---|---:|---:|
| default (no flags, 96 threads) | 73.4 | 5.6 |
| optimized (-t 16 -fa on, F16 KV, no-repack) | 147.4 | 19.4 |
| speedup | ~2.0x | ~3.5x |
## What actually caused the gain
| Config | Prefill t/s | Decode t/s |
|---|---:|---:|
| -t 16 only | 147.7 | 18.3 |
| -t 16 + FA + no-repack | 147.4 | 19.4 |
The gain comes from using 16 threads. FA, repack on/off and the AMX build differ within noise
(same config repeated: prefill 119-148 t/s). AMX gave no reliable speedup.
## Thread scaling (llama-bench, repack on, AMX build)
| threads | pp512 | tg128 |
|---:|---:|---:|
| 1 | 16.28 | 1.65 |
| 4 | 57.46 | 6.11 |
| 8 | 125.15 | 11.20 |
| 12 | 189.09 | 14.66 |
| 16 | 208.93 | 18.31 |
| 20 | 156.98 | 9.95 |
| 24 | 163.43 | 8.90 |
More than 16 threads degrades sharply (CPU quota throttling).
Long context decode (tg64): 2048 ctx 16.63, 4096 ctx 14.88, 8192 ctx 13.44 t/s.
Raw data: amx_threads.jsonl
## Comparison with the HF model-card numbers (same machine type, llama-bench pp512 / tg)
Interleaved A/B, 3 rounds, taskset 0-15, same host load (results/interleaved.jsonl):
| Config | pp512 t/s | tg t/s |
|---|---:|---:|
| default flags (96 threads, repack on) | 185 / 191 / 191 (median 190.5) | 5.7 / 5.7 / 5.6 (median 5.7) |
| model-card config (-t 16 -fa on -ctk/-ctv f16 --repack 0) | 225.6 / 229.5 / 233.7 (median 229.5) | 16.8 / 17.1 / 17.3 (median 17.1) |
| model card as published | 212 | 14.5 |
- The model-card config IS the optimized config; I found nothing faster than it. Differences between
my 230/17 and the published 212/14.5 are within host-load variance (+-15%), not an improvement.
- Gain over default flags: prefill about 1.2x, decode about 3.0x (decode gain is all from -t 16).
- The same command at a different time gave 108 / 4.1 (readme_exact.jsonl) while host load was 54:
results on this shared host depend heavily on neighbours. Always compare interleaved.
- CLI prefill numbers (73-148 t/s) use a ~19-token prompt and are not comparable to pp512.
|