Add measured performance section (M4 Max)
#1
by mlboydaisuke - opened
README.md
CHANGED
|
@@ -25,4 +25,28 @@ This repository contains LiteRT-LM variant of [Qwen/Qwen2.5-Coder-3B-Instruct](h
|
|
| 25 |
|
| 26 |
## Integration
|
| 27 |
|
| 28 |
-
Ready to integrate this into your product? Get started in the [LiteRT-LM documentation](https://ai.google.dev/edge/litert-lm/overview).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
## Integration
|
| 27 |
|
| 28 |
+
Ready to integrate this into your product? Get started in the [LiteRT-LM documentation](https://ai.google.dev/edge/litert-lm/overview).
|
| 29 |
+
|
| 30 |
+
## Performance (measured)
|
| 31 |
+
|
| 32 |
+
### Apple M4 Max
|
| 33 |
+
|
| 34 |
+
Measured with the LiteRT-LM CLI: `litert-lm benchmark -p 256 -d 256 --runs 3 --cache no`
|
| 35 |
+
(litert-lm 0.15.0) on an idle Apple M4 Max (macOS); 256 prefill / 256 decode tokens, 3 iterations
|
| 36 |
+
averaged by the tool. A desktop reference point — phone-side figures vary by SoC and backend.
|
| 37 |
+
|
| 38 |
+
| Backend | Prefill (tokens/s) | Decode (tokens/s) | Time-to-first-token (s) |
|
| 39 |
+
|---|---|---|---|
|
| 40 |
+
| CPU | 121 | 26.7 | 2.16 |
|
| 41 |
+
| GPU | 1,320 | 77.8 | 0.21 |
|
| 42 |
+
|
| 43 |
+
### Galaxy S26 — GPU vs CPU (litert-lm 0.16.0)
|
| 44 |
+
|
| 45 |
+
Measured on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16) with `litert_lm_advanced_main` from the litert-lm v0.16.0 release; the GPU backend is OpenCL (`LITERT_CL`). One fixed 205-token prompt text (223 tokens under this tokenizer), `--benchmark`. Two runs per backend taken back-to-back — cells show the range. Peak RSS is the process VmHWM. Before quoting, the same file was run on each backend with a real prompt: both backends produced a correct text answer.
|
| 46 |
+
|
| 47 |
+
| Backend | Prefill (223 tok) | Decode | Time-to-first-token | Init | Peak RSS |
|
| 48 |
+
|---|---|---|---|---|---|
|
| 49 |
+
| **GPU (OpenCL)** | **403–423 tok/s** | **16.3–16.5 tok/s** | **0.59–0.61 s** | 4.3–4.9 s | **864 MB** |
|
| 50 |
+
| CPU (XNNPACK) | 159–187 tok/s | 12.1–13.6 tok/s | 1.26–1.49 s | 3.6–4.6 s | 3934 MB |
|
| 51 |
+
|
| 52 |
+
The GPU takes the whole graph — decode 1603/1603 ops and prefill 1452/1452 on `LITERT_CL`. It wins prefill 2.2–2.7× and decode 1.2–1.4×, and peaks 4.6× lower (864 against 3934 MB) — for a 3.4 GB int8 bundle the RSS difference is the practical headline.
|