README: add benchmark comparison vs original fork (compatibility, llama-bench, MTP)
Browse files
README.md
CHANGED
|
@@ -145,3 +145,65 @@ Measured resident memory of `llama-cli` with this model (`-c 512`, `-n 1`):
|
|
| 145 |
534/534 tests pass
|
| 146 |
- `test-chat`, `test-chat-template`, `test-chat-peg-parser`,
|
| 147 |
`test-chat-auto-parser`: all pass
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
534/534 tests pass
|
| 146 |
- `test-chat`, `test-chat-template`, `test-chat-peg-parser`,
|
| 147 |
`test-chat-auto-parser`: all pass
|
| 148 |
+
|
| 149 |
+
## Comparison with the original fork
|
| 150 |
+
|
| 151 |
+
Measured against the original
|
| 152 |
+
[mrexodia/llama.cpp-openPangu-2.0-Flash](https://github.com/mrexodia/llama.cpp-openPangu-2.0-Flash)
|
| 153 |
+
(HEAD `76c92350b`, llama.cpp base `6f4f53f2b` from 2026-06-29), same machine,
|
| 154 |
+
same model, same flags.
|
| 155 |
+
|
| 156 |
+
### Compatibility on this machine (AMX CPU)
|
| 157 |
+
|
| 158 |
+
| | original fork | this port (v0.7.0) |
|
| 159 |
+
|---|---|---|
|
| 160 |
+
| default settings | **aborts** — `GGML_ASSERT(*cur_backend_id != -1)` during `sched_reserve` (AMX weight repacking) | runs |
|
| 161 |
+
| `-nr` (weight repacking disabled) | runs | runs |
|
| 162 |
+
|
| 163 |
+
The original fork cannot be used with default settings on a CPU with AMX.
|
| 164 |
+
Its `llama-bench` has no option to disable repacking at all — the original
|
| 165 |
+
numbers below required a local source edit. This port fixes the underlying
|
| 166 |
+
AMX packing bug (the AMX packer silently corrupted batched/3D weights).
|
| 167 |
+
|
| 168 |
+
### llama-bench, `Q3_K_M`, `-t 16 -r 3` (2026-10-16)
|
| 169 |
+
|
| 170 |
+
Both builds with weight repacking disabled (`-nr`, the only common
|
| 171 |
+
runnable configuration on this machine):
|
| 172 |
+
|
| 173 |
+
| test | original fork | this port | change |
|
| 174 |
+
|------|--------------:|----------:|-------:|
|
| 175 |
+
| pp128 | 41.51 ± 2.21 | 60.47 ± 4.63 | **+46 %** |
|
| 176 |
+
| pp512 | 42.62 ± 0.83 | 64.75 ± 2.26 | **+52 %** |
|
| 177 |
+
| tg32 | 11.65 ± 0.19 | 11.52 ± 0.35 | ≈ |
|
| 178 |
+
| tg128 | 11.47 ± 0.21 | 11.82 ± 0.72 | ≈ |
|
| 179 |
+
|
| 180 |
+
This port with the default AMX weight repacking (original fork cannot run
|
| 181 |
+
this configuration):
|
| 182 |
+
|
| 183 |
+
| test | this port (default) |
|
| 184 |
+
|------|--------------------:|
|
| 185 |
+
| pp128 | 56.84 ± 1.32 |
|
| 186 |
+
| pp512 | 62.65 ± 0.54 |
|
| 187 |
+
| tg32 | 11.94 ± 0.71 |
|
| 188 |
+
| tg128 | 12.02 ± 0.30 |
|
| 189 |
+
|
| 190 |
+
Reading these numbers:
|
| 191 |
+
|
| 192 |
+
- Prefill is ~1.5× faster than the original fork, mostly from the v0.7.0
|
| 193 |
+
base (3.5 months of upstream CPU optimizations). Generation is on par.
|
| 194 |
+
- On this CPU, AMX weight repacking is performance-neutral and costs
|
| 195 |
+
~16 GiB of extra RAM, so `-nr` is usually the better choice.
|
| 196 |
+
- The host is shared and measurements vary between runs (see the ±).
|
| 197 |
+
|
| 198 |
+
### MTP (self-speculative decoding)
|
| 199 |
+
|
| 200 |
+
Same prompt, `llama-cli -n 128 -nr`:
|
| 201 |
+
|
| 202 |
+
| | generation |
|
| 203 |
+
|---|---|
|
| 204 |
+
| original fork, MTP draft | 4.9 t/s |
|
| 205 |
+
| this port, MTP draft | 4.7 t/s |
|
| 206 |
+
| either fork, plain decoding (`llama-bench` tg128) | 11.5–12.0 t/s |
|
| 207 |
+
|
| 208 |
+
On CPU the MTP draft head is a net loss in both forks; it is included for
|
| 209 |
+
completeness and may pay off on GPUs where the draft forward pass is cheap.
|