HelloSun commited on
Commit
ad53eb4
·
1 Parent(s): c1fa642

README: add benchmark comparison vs original fork (compatibility, llama-bench, MTP)

Browse files
Files changed (1) hide show
  1. README.md +62 -0
README.md CHANGED
@@ -145,3 +145,65 @@ Measured resident memory of `llama-cli` with this model (`-c 512`, `-n 1`):
145
  534/534 tests pass
146
  - `test-chat`, `test-chat-template`, `test-chat-peg-parser`,
147
  `test-chat-auto-parser`: all pass
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
145
  534/534 tests pass
146
  - `test-chat`, `test-chat-template`, `test-chat-peg-parser`,
147
  `test-chat-auto-parser`: all pass
148
+
149
+ ## Comparison with the original fork
150
+
151
+ Measured against the original
152
+ [mrexodia/llama.cpp-openPangu-2.0-Flash](https://github.com/mrexodia/llama.cpp-openPangu-2.0-Flash)
153
+ (HEAD `76c92350b`, llama.cpp base `6f4f53f2b` from 2026-06-29), same machine,
154
+ same model, same flags.
155
+
156
+ ### Compatibility on this machine (AMX CPU)
157
+
158
+ | | original fork | this port (v0.7.0) |
159
+ |---|---|---|
160
+ | default settings | **aborts** — `GGML_ASSERT(*cur_backend_id != -1)` during `sched_reserve` (AMX weight repacking) | runs |
161
+ | `-nr` (weight repacking disabled) | runs | runs |
162
+
163
+ The original fork cannot be used with default settings on a CPU with AMX.
164
+ Its `llama-bench` has no option to disable repacking at all — the original
165
+ numbers below required a local source edit. This port fixes the underlying
166
+ AMX packing bug (the AMX packer silently corrupted batched/3D weights).
167
+
168
+ ### llama-bench, `Q3_K_M`, `-t 16 -r 3` (2026-10-16)
169
+
170
+ Both builds with weight repacking disabled (`-nr`, the only common
171
+ runnable configuration on this machine):
172
+
173
+ | test | original fork | this port | change |
174
+ |------|--------------:|----------:|-------:|
175
+ | pp128 | 41.51 ± 2.21 | 60.47 ± 4.63 | **+46 %** |
176
+ | pp512 | 42.62 ± 0.83 | 64.75 ± 2.26 | **+52 %** |
177
+ | tg32 | 11.65 ± 0.19 | 11.52 ± 0.35 | ≈ |
178
+ | tg128 | 11.47 ± 0.21 | 11.82 ± 0.72 | ≈ |
179
+
180
+ This port with the default AMX weight repacking (original fork cannot run
181
+ this configuration):
182
+
183
+ | test | this port (default) |
184
+ |------|--------------------:|
185
+ | pp128 | 56.84 ± 1.32 |
186
+ | pp512 | 62.65 ± 0.54 |
187
+ | tg32 | 11.94 ± 0.71 |
188
+ | tg128 | 12.02 ± 0.30 |
189
+
190
+ Reading these numbers:
191
+
192
+ - Prefill is ~1.5× faster than the original fork, mostly from the v0.7.0
193
+ base (3.5 months of upstream CPU optimizations). Generation is on par.
194
+ - On this CPU, AMX weight repacking is performance-neutral and costs
195
+ ~16 GiB of extra RAM, so `-nr` is usually the better choice.
196
+ - The host is shared and measurements vary between runs (see the ±).
197
+
198
+ ### MTP (self-speculative decoding)
199
+
200
+ Same prompt, `llama-cli -n 128 -nr`:
201
+
202
+ | | generation |
203
+ |---|---|
204
+ | original fork, MTP draft | 4.9 t/s |
205
+ | this port, MTP draft | 4.7 t/s |
206
+ | either fork, plain decoding (`llama-bench` tg128) | 11.5–12.0 t/s |
207
+
208
+ On CPU the MTP draft head is a net loss in both forks; it is included for
209
+ completeness and may pay off on GPUs where the draft forward pass is cheap.