mlboydaisuke commited on
Commit
83a0ef1
Β·
verified Β·
1 Parent(s): b68a15e

Add measured Hexagon NPU figures and the JIT reproduction steps

Browse files
Files changed (1) hide show
  1. README.md +64 -0
README.md CHANGED
@@ -96,6 +96,70 @@ Measured on a **Pixel 8a** (Tensor G3, Android 16) with the standard TFLite [`be
96
 
97
  **The two GPU rows are different runtimes, not a contradiction.** The `LITERT_CL` figure is the one recorded when this model shipped, taken through LiteRT's own `CompiledModel` accelerator β€” the path the Kotlin sample app and the LiteRT API use. The `TfLiteGpuDelegateV2` figure is the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. They agree on how much of the graph the GPU takes; they disagree on speed, and the classic delegate is the slower of the two here. Read the `TfLiteGpuDelegateV2` row as a reproducible floor, not as this model's speed on LiteRT.
98
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
99
  ## License
100
 
101
  MIT (6DRepNet / thohemp). Trained on 300W-LP.
 
96
 
97
  **The two GPU rows are different runtimes, not a contradiction.** The `LITERT_CL` figure is the one recorded when this model shipped, taken through LiteRT's own `CompiledModel` accelerator β€” the path the Kotlin sample app and the LiteRT API use. The `TfLiteGpuDelegateV2` figure is the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. They agree on how much of the graph the GPU takes; they disagree on speed, and the classic delegate is the slower of the two here. Read the `TfLiteGpuDelegateV2` row as a reproducible floor, not as this model's speed on LiteRT.
98
 
99
+ ## Snapdragon NPU (Hexagon)
100
+
101
+ This file runs on the Qualcomm Hexagon NPU **as published** β€” no conversion and no
102
+ pre-compiled artifact. LiteRT compiles it on the device and caches the result.
103
+
104
+ Measured on a physical **Samsung Galaxy S26** (Snapdragon 8 Elite Gen 5 / SM8850,
105
+ Hexagon v81) with LiteRT `CompiledModel` 2.2.0 β€” 5 warm-up runs then 50 timed runs, one
106
+ accelerator per process, every row taken at device thermal status `NONE`.
107
+
108
+ | Compute unit | Inference (median / min) | Load | Start headroom |
109
+ |---|---|---|---|
110
+ | NPU (Hexagon) β€” first launch | 1.75 ms / 1.68 ms | 1287 ms | 0.60 |
111
+ | NPU (Hexagon) β€” cached | 1.74 ms / 1.71 ms | **258 ms** | 0.60 |
112
+ | GPU (Adreno) | 8.53 ms / 8.18 ms | 636 ms | 0.60 |
113
+
114
+ The NPU is **4.9x faster** on inference here (1.74 ms against 8.53 ms). The first launch pays once for on-device compilation; every launch after that
115
+ loads in 258 ms against 636 ms for the GPU (2.5x), because the GPU rebuilds its
116
+ shaders each time. The file is fp16 and needs no int8 quantization to reach the NPU.
117
+
118
+ ### Running it on the NPU
119
+
120
+ Put these in `jniLibs/arm64-v8a/`. **None of them are distributed from this repository** β€”
121
+ the first two come from Google, the rest from Qualcomm's own SDK:
122
+
123
+ | Library | Source |
124
+ |---|---|
125
+ | `libLiteRtDispatch_Qualcomm.so`, `libLiteRtCompilerPlugin_Qualcomm.so` | `litert_npu_runtime_libraries_jit.zip`, a release asset of [google-ai-edge/LiteRT](https://github.com/google-ai-edge/LiteRT/releases) |
126
+ | `libQnnHtp.so`, `libQnnSystem.so`, `libQnnHtpV81Stub.so`, `libQnnHtpV81Skel.so`, `libQnnHtpPrepare.so`, `libQnnIr.so`, `libQnnSaver.so` | Qualcomm QAIRT β€” the same zip ships `fetch_qualcomm_library.sh`, which downloads the SDK and copies them for you |
127
+
128
+ Pick the runtime matching the device's Hexagon version: SM8550 β†’ v73, SM8650 β†’ v75,
129
+ SM8750 β†’ v79, SM8850 β†’ v81.
130
+
131
+ ```kotlin
132
+ val env = Environment.create(
133
+ context,
134
+ mapOf(
135
+ Environment.Option.DispatchLibraryDir to context.applicationInfo.nativeLibraryDir,
136
+ // Required for on-device compilation. Without it the model silently runs on CPU.
137
+ Environment.Option.CompilerPluginLibraryDir to context.applicationInfo.nativeLibraryDir,
138
+ ),
139
+ )
140
+ val options = CompiledModel.Options(Accelerator.NPU).apply {
141
+ qualcommOptions = CompiledModel.QualcommOptions(
142
+ htpPerformanceMode = CompiledModel.QualcommOptions.HtpPerformanceMode.BURST
143
+ )
144
+ }
145
+ val model = CompiledModel.create(context.assets, "6drepnet.tflite", options, env)
146
+ ```
147
+
148
+ Build settings: `useLegacyPackaging = true` under `packaging { jniLibs { … } }`, so the
149
+ DSP can open the skel from a real path, and Kotlin **2.3+** for LiteRT 2.2.0's metadata.
150
+
151
+ > **Every NPU failure here is silent.** There is no error when the NPU is unavailable β€”
152
+ > you get a plausible CPU number instead. Confirm from logcat which delegate took the
153
+ > graph: `Replacing 1 out of 1 node(s) with delegate (DispatchDelegate)` is the NPU,
154
+ > while `... (TfLiteXNNPackDelegate)` is the CPU. A missing library is reported only as
155
+ > a `W`-level `dlopen failed` line under a generic `No compiler plugin found` summary.
156
+
157
+ **On the conditions.** Thermal headroom is reported as measured, where 1.0 is the
158
+ throttling threshold. All rows were taken at a comparable headroom and compare directly;
159
+ figures taken at a different headroom will differ. Each accelerator ran in its own
160
+ process, because LiteRT's `Environment` is shared within one and the first model load
161
+ fixes the options for every later one.
162
+
163
  ## License
164
 
165
  MIT (6DRepNet / thohemp). Trained on 300W-LP.