tphakala commited on
Commit
166f77e
·
verified ·
1 Parent(s): 999ce99

card: correct fp16 and regional-slicing claims; add measured benchmarks

Browse files
Files changed (1) hide show
  1. README.md +25 -5
README.md CHANGED
@@ -52,11 +52,28 @@ and `embeddings [batch, 1280]`.
52
 
53
  | precision | best for |
54
  |---|---|
55
- | `_fp32` | OpenVINO (x86 + Intel GPU), CUDA/TensorRT, Intel CPU |
56
- | `_fp16` | half the size; RPi 5 / CUDA / anywhere with FP16 |
 
 
 
 
 
57
 
58
  int8-arm is not shipped for v3.0: the backbone is Conv-dominated and int8-arm only
59
- quantizes MatMul/Gemm weights, so it barely helps here (fp16 is the size lever).
 
 
 
 
 
 
 
 
 
 
 
 
60
 
61
  ## Full model
62
 
@@ -69,8 +86,11 @@ quantizes MatMul/Gemm weights, so it barely helps here (fp16 is the size lever).
69
  ## Why regional models
70
 
71
  The three-head classifier is ~80% of the model's parameters, so restricting it to
72
- a region's species set shrinks both RAM and per-inference compute a lot, ideal for
73
- always-on detection on phones and single-board computers. Each tile is
 
 
 
74
  **numerically identical** to the full model on the species it keeps (max abs
75
  difference vs the full model < 1e-6); the only change is that out-of-region
76
  species are not emitted, which at a fixed monitoring location is exactly what you
 
52
 
53
  | precision | best for |
54
  |---|---|
55
+ | `_fp32` | **run this everywhere**: CPU (ORT + OpenVINO), CUDA/TensorRT, Intel GPU |
56
+ | `_fp16` | half the download; GPU speed win. Not a CPU win (see below) |
57
+
58
+ **FP16 is a download-size lever, not a CPU one.** ONNX Runtime upconverts FP16
59
+ weights at inference, so on CPU (x86 and ARM alike) FP16 is equal to or slower
60
+ than FP32, and on a Raspberry Pi 4 it uses *more* RAM than FP32. For low-RAM CPU
61
+ devices the right choice is a **regional FP32** model, not FP16.
62
 
63
  int8-arm is not shipped for v3.0: the backbone is Conv-dominated and int8-arm only
64
+ quantizes MatMul/Gemm weights, so it barely helps here.
65
+
66
+ ### Measured (batch 1, 5 s window)
67
+
68
+ | device | full fp32 | regional fp32 (nordic) | peak RAM full / regional |
69
+ |---|---:|---:|---:|
70
+ | RPi5, OpenVINO | 168 ms | **128 ms** | - |
71
+ | RPi5, ORT 4 threads | 363 ms | 253 ms | 685 MB / **229 MB** |
72
+ | RPi4B, ORT 4 threads | 874 ms | 743 ms | 688 MB / **236 MB** |
73
+ | x86 i7-1260P, ORT 4 threads | 70 ms | 59 ms | - |
74
+
75
+ OpenVINO is ~2.2x faster than ONNX Runtime on a Raspberry Pi 5. Batching helps
76
+ only on a GPU (~1.5x at batch 8, FP16); on CPU use batch 1.
77
 
78
  ## Full model
79
 
 
86
  ## Why regional models
87
 
88
  The three-head classifier is ~80% of the model's parameters, so restricting it to
89
+ a region's species set cuts memory a lot (**~67% less RAM**: 685 MB -> 229 MB on a
90
+ Raspberry Pi 5) and latency modestly (**15-30%**). The parameters live in the
91
+ heads, but the convolutional backbone still does most of the arithmetic, so the
92
+ headline win is footprint rather than speed. That is exactly what an always-on
93
+ detector on a phone or single-board computer needs. Each tile is
94
  **numerically identical** to the full model on the species it keeps (max abs
95
  difference vs the full model < 1e-6); the only change is that out-of-region
96
  species are not emitted, which at a fixed monitoring location is exactly what you