Add measured on-device speed table; note wi8 runtime requirement
#11
by mlboydaisuke - opened
README.md
CHANGED
|
@@ -113,3 +113,23 @@ Released under the **Apache-2.0 License**, consistent with the upstream [openbmb
|
|
| 113 |
}
|
| 114 |
```
|
| 115 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 113 |
}
|
| 114 |
```
|
| 115 |
|
| 116 |
+
## Performance (Apple M4 Max, measured)
|
| 117 |
+
|
| 118 |
+
Measured with the LiteRT-LM CLI: `litert-lm benchmark -p 256 -d 256 --runs 3 --cache no`
|
| 119 |
+
(litert-lm 0.15.0 / 0.16.0, noted per row) on an idle Apple M4 Max (macOS); 256 prefill / 256 decode tokens, 3 iterations
|
| 120 |
+
averaged by the tool. A desktop reference point — phone-side figures vary by SoC and backend.
|
| 121 |
+
|
| 122 |
+
| File | Backend | Prefill (tokens/s) | Decode (tokens/s) | Time-to-first-token (s) |
|
| 123 |
+
|---|---|---|---|---|
|
| 124 |
+
| `minicpm_wi4b32_wi8_afp32` | CPU | 533 | 102.8 | 0.49 |
|
| 125 |
+
| `minicpm_wi4b32_wi8_afp32` | GPU | 5,128 | 238.9 | 0.05 |
|
| 126 |
+
| `minicpm_wi4b32_wi8_afp32_gpu_opt` | CPU | 517 | 101.2 | 0.51 |
|
| 127 |
+
| `minicpm_wi4b32_wi8_afp32_gpu_opt` | GPU | 5,516 | 283.1 | 0.05 |
|
| 128 |
+
| `MiniCPM5-1B_dynamic_wi8_afp32` * | CPU | 375 | 62.0 | 0.70 |
|
| 129 |
+
| `MiniCPM5-1B_dynamic_wi8_afp32` * | GPU | 4,459 | 262.8 | 0.06 |
|
| 130 |
+
|
| 131 |
+
\* The `dynamic_wi8_afp32` file requires **litert-lm >= 0.16** (its rows above are 0.16.0;
|
| 132 |
+
the other files were measured on 0.15.0, where they also run). On 0.15 the wi8 file fails
|
| 133 |
+
to load with `INVALID_ARGUMENT: Unsupported model type` — worth a note for anyone pinning
|
| 134 |
+
the runtime. The `gpu_opt` build delivers its intended GPU gain: +18% decode over the
|
| 135 |
+
non-opt int4 file on this machine.
|