card: correct fp16 and regional-slicing claims; add measured benchmarks
Browse files
README.md
CHANGED
|
@@ -52,11 +52,28 @@ and `embeddings [batch, 1280]`.
|
|
| 52 |
|
| 53 |
| precision | best for |
|
| 54 |
|---|---|
|
| 55 |
-
| `_fp32` |
|
| 56 |
-
| `_fp16` | half the
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
|
| 58 |
int8-arm is not shipped for v3.0: the backbone is Conv-dominated and int8-arm only
|
| 59 |
-
quantizes MatMul/Gemm weights, so it barely helps here
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
## Full model
|
| 62 |
|
|
@@ -69,8 +86,11 @@ quantizes MatMul/Gemm weights, so it barely helps here (fp16 is the size lever).
|
|
| 69 |
## Why regional models
|
| 70 |
|
| 71 |
The three-head classifier is ~80% of the model's parameters, so restricting it to
|
| 72 |
-
a region's species set
|
| 73 |
-
|
|
|
|
|
|
|
|
|
|
| 74 |
**numerically identical** to the full model on the species it keeps (max abs
|
| 75 |
difference vs the full model < 1e-6); the only change is that out-of-region
|
| 76 |
species are not emitted, which at a fixed monitoring location is exactly what you
|
|
|
|
| 52 |
|
| 53 |
| precision | best for |
|
| 54 |
|---|---|
|
| 55 |
+
| `_fp32` | **run this everywhere**: CPU (ORT + OpenVINO), CUDA/TensorRT, Intel GPU |
|
| 56 |
+
| `_fp16` | half the download; GPU speed win. Not a CPU win (see below) |
|
| 57 |
+
|
| 58 |
+
**FP16 is a download-size lever, not a CPU one.** ONNX Runtime upconverts FP16
|
| 59 |
+
weights at inference, so on CPU (x86 and ARM alike) FP16 is equal to or slower
|
| 60 |
+
than FP32, and on a Raspberry Pi 4 it uses *more* RAM than FP32. For low-RAM CPU
|
| 61 |
+
devices the right choice is a **regional FP32** model, not FP16.
|
| 62 |
|
| 63 |
int8-arm is not shipped for v3.0: the backbone is Conv-dominated and int8-arm only
|
| 64 |
+
quantizes MatMul/Gemm weights, so it barely helps here.
|
| 65 |
+
|
| 66 |
+
### Measured (batch 1, 5 s window)
|
| 67 |
+
|
| 68 |
+
| device | full fp32 | regional fp32 (nordic) | peak RAM full / regional |
|
| 69 |
+
|---|---:|---:|---:|
|
| 70 |
+
| RPi5, OpenVINO | 168 ms | **128 ms** | - |
|
| 71 |
+
| RPi5, ORT 4 threads | 363 ms | 253 ms | 685 MB / **229 MB** |
|
| 72 |
+
| RPi4B, ORT 4 threads | 874 ms | 743 ms | 688 MB / **236 MB** |
|
| 73 |
+
| x86 i7-1260P, ORT 4 threads | 70 ms | 59 ms | - |
|
| 74 |
+
|
| 75 |
+
OpenVINO is ~2.2x faster than ONNX Runtime on a Raspberry Pi 5. Batching helps
|
| 76 |
+
only on a GPU (~1.5x at batch 8, FP16); on CPU use batch 1.
|
| 77 |
|
| 78 |
## Full model
|
| 79 |
|
|
|
|
| 86 |
## Why regional models
|
| 87 |
|
| 88 |
The three-head classifier is ~80% of the model's parameters, so restricting it to
|
| 89 |
+
a region's species set cuts memory a lot (**~67% less RAM**: 685 MB -> 229 MB on a
|
| 90 |
+
Raspberry Pi 5) and latency modestly (**15-30%**). The parameters live in the
|
| 91 |
+
heads, but the convolutional backbone still does most of the arithmetic, so the
|
| 92 |
+
headline win is footprint rather than speed. That is exactly what an always-on
|
| 93 |
+
detector on a phone or single-board computer needs. Each tile is
|
| 94 |
**numerically identical** to the full model on the species it keeps (max abs
|
| 95 |
difference vs the full model < 1e-6); the only change is that out-of-region
|
| 96 |
species are not emitted, which at a fixed monitoring location is exactly what you
|