Upload report.md with huggingface_hub
Browse files
report.md
CHANGED
|
@@ -36,3 +36,19 @@ More than 16 threads degrades sharply (CPU quota throttling).
|
|
| 36 |
Long context decode (tg64): 2048 ctx 16.63, 4096 ctx 14.88, 8192 ctx 13.44 t/s.
|
| 37 |
|
| 38 |
Raw data: amx_threads.jsonl
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
Long context decode (tg64): 2048 ctx 16.63, 4096 ctx 14.88, 8192 ctx 13.44 t/s.
|
| 37 |
|
| 38 |
Raw data: amx_threads.jsonl
|
| 39 |
+
|
| 40 |
+
## Comparison with the HF model-card numbers (same machine type, llama-bench pp512 / tg)
|
| 41 |
+
Interleaved A/B, 3 rounds, taskset 0-15, same host load (results/interleaved.jsonl):
|
| 42 |
+
|
| 43 |
+
| Config | pp512 t/s | tg t/s |
|
| 44 |
+
|---|---:|---:|
|
| 45 |
+
| default flags (96 threads, repack on) | 185 / 191 / 191 (median 190.5) | 5.7 / 5.7 / 5.6 (median 5.7) |
|
| 46 |
+
| model-card config (-t 16 -fa on -ctk/-ctv f16 --repack 0) | 225.6 / 229.5 / 233.7 (median 229.5) | 16.8 / 17.1 / 17.3 (median 17.1) |
|
| 47 |
+
| model card as published | 212 | 14.5 |
|
| 48 |
+
|
| 49 |
+
- The model-card config IS the optimized config; I found nothing faster than it. Differences between
|
| 50 |
+
my 230/17 and the published 212/14.5 are within host-load variance (+-15%), not an improvement.
|
| 51 |
+
- Gain over default flags: prefill about 1.2x, decode about 3.0x (decode gain is all from -t 16).
|
| 52 |
+
- The same command at a different time gave 108 / 4.1 (readme_exact.jsonl) while host load was 54:
|
| 53 |
+
results on this shared host depend heavily on neighbours. Always compare interleaved.
|
| 54 |
+
- CLI prefill numbers (73-148 t/s) use a ~19-token prompt and are not comparable to pp512.
|