mlboydaisuke commited on
Commit
cb4b749
Β·
verified Β·
1 Parent(s): f2d4d75

Add measured Hexagon NPU figures and the JIT reproduction steps

Browse files
Files changed (1) hide show
  1. README.md +64 -0
README.md CHANGED
@@ -94,6 +94,70 @@ Measured on a **Pixel 8a** (Tensor G3, Android 16) with the standard TFLite [`be
94
 
95
  **The two GPU rows are different runtimes, not a contradiction.** The `LITERT_CL` figure is the one recorded when this model shipped, taken through LiteRT's own `CompiledModel` accelerator β€” the path the Kotlin sample app and the LiteRT API use. The `TfLiteGpuDelegateV2` figure is the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. They agree on how much of the graph the GPU takes; they disagree on speed, and the classic delegate is the slower of the two here. Read the `TfLiteGpuDelegateV2` row as a reproducible floor, not as this model's speed on LiteRT.
96
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
97
  ## License
98
 
99
  Apache-2.0 (weights: cpoisson/plantnet300k-resnet18). PlantNet-300K code: BSD-2-Clause
 
94
 
95
  **The two GPU rows are different runtimes, not a contradiction.** The `LITERT_CL` figure is the one recorded when this model shipped, taken through LiteRT's own `CompiledModel` accelerator β€” the path the Kotlin sample app and the LiteRT API use. The `TfLiteGpuDelegateV2` figure is the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. They agree on how much of the graph the GPU takes; they disagree on speed, and the classic delegate is the slower of the two here. Read the `TfLiteGpuDelegateV2` row as a reproducible floor, not as this model's speed on LiteRT.
96
 
97
+ ## Snapdragon NPU (Hexagon)
98
+
99
+ This file runs on the Qualcomm Hexagon NPU **as published** β€” no conversion and no
100
+ pre-compiled artifact. LiteRT compiles it on the device and caches the result.
101
+
102
+ Measured on a physical **Samsung Galaxy S26** (Snapdragon 8 Elite Gen 5 / SM8850,
103
+ Hexagon v81) with LiteRT `CompiledModel` 2.2.0 β€” 5 warm-up runs then 50 timed runs, one
104
+ accelerator per process, every row taken at device thermal status `NONE`.
105
+
106
+ | Compute unit | Inference (median / min) | Load | Start headroom |
107
+ |---|---|---|---|
108
+ | NPU (Hexagon) β€” first launch | 0.90 ms / 0.88 ms | 586 ms | 0.59 |
109
+ | NPU (Hexagon) β€” cached | 0.90 ms / 0.89 ms | **110 ms** | 0.59 |
110
+ | GPU (Adreno) | 3.49 ms / 3.17 ms | 464 ms | 0.59 |
111
+
112
+ The NPU is **3.9x faster** on inference here (0.90 ms against 3.49 ms). The first launch pays once for on-device compilation; every launch after that
113
+ loads in 110 ms against 464 ms for the GPU (4.2x), because the GPU rebuilds its
114
+ shaders each time. The file is fp16 and needs no int8 quantization to reach the NPU.
115
+
116
+ ### Running it on the NPU
117
+
118
+ Put these in `jniLibs/arm64-v8a/`. **None of them are distributed from this repository** β€”
119
+ the first two come from Google, the rest from Qualcomm's own SDK:
120
+
121
+ | Library | Source |
122
+ |---|---|
123
+ | `libLiteRtDispatch_Qualcomm.so`, `libLiteRtCompilerPlugin_Qualcomm.so` | `litert_npu_runtime_libraries_jit.zip`, a release asset of [google-ai-edge/LiteRT](https://github.com/google-ai-edge/LiteRT/releases) |
124
+ | `libQnnHtp.so`, `libQnnSystem.so`, `libQnnHtpV81Stub.so`, `libQnnHtpV81Skel.so`, `libQnnHtpPrepare.so`, `libQnnIr.so`, `libQnnSaver.so` | Qualcomm QAIRT β€” the same zip ships `fetch_qualcomm_library.sh`, which downloads the SDK and copies them for you |
125
+
126
+ Pick the runtime matching the device's Hexagon version: SM8550 β†’ v73, SM8650 β†’ v75,
127
+ SM8750 β†’ v79, SM8850 β†’ v81.
128
+
129
+ ```kotlin
130
+ val env = Environment.create(
131
+ context,
132
+ mapOf(
133
+ Environment.Option.DispatchLibraryDir to context.applicationInfo.nativeLibraryDir,
134
+ // Required for on-device compilation. Without it the model silently runs on CPU.
135
+ Environment.Option.CompilerPluginLibraryDir to context.applicationInfo.nativeLibraryDir,
136
+ ),
137
+ )
138
+ val options = CompiledModel.Options(Accelerator.NPU).apply {
139
+ qualcommOptions = CompiledModel.QualcommOptions(
140
+ htpPerformanceMode = CompiledModel.QualcommOptions.HtpPerformanceMode.BURST
141
+ )
142
+ }
143
+ val model = CompiledModel.create(context.assets, "plantnet.tflite", options, env)
144
+ ```
145
+
146
+ Build settings: `useLegacyPackaging = true` under `packaging { jniLibs { … } }`, so the
147
+ DSP can open the skel from a real path, and Kotlin **2.3+** for LiteRT 2.2.0's metadata.
148
+
149
+ > **Every NPU failure here is silent.** There is no error when the NPU is unavailable β€”
150
+ > you get a plausible CPU number instead. Confirm from logcat which delegate took the
151
+ > graph: `Replacing 1 out of 1 node(s) with delegate (DispatchDelegate)` is the NPU,
152
+ > while `... (TfLiteXNNPackDelegate)` is the CPU. A missing library is reported only as
153
+ > a `W`-level `dlopen failed` line under a generic `No compiler plugin found` summary.
154
+
155
+ **On the conditions.** Thermal headroom is reported as measured, where 1.0 is the
156
+ throttling threshold. All rows were taken at a comparable headroom and compare directly;
157
+ figures taken at a different headroom will differ. Each accelerator ran in its own
158
+ process, because LiteRT's `Environment` is shared within one and the first model load
159
+ fixes the options for every later one.
160
+
161
  ## License
162
 
163
  Apache-2.0 (weights: cpoisson/plantnet300k-resnet18). PlantNet-300K code: BSD-2-Clause