docs: add the RaCo-ALIKED ONNX/TensorRT matcher exports
Browse files
README.md
CHANGED
|
@@ -41,3 +41,45 @@ fraction of the latency. ICCV 2023.
|
|
| 41 |
year = {2023}
|
| 42 |
}
|
| 43 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
year = {2023}
|
| 42 |
}
|
| 43 |
```
|
| 44 |
+
|
| 45 |
+
## ONNX / TensorRT exports for vision-rt
|
| 46 |
+
|
| 47 |
+
Alongside the PyTorch checkpoints above, this repo holds the **LightGlue+ matcher for
|
| 48 |
+
RaCo-ALIKED features** as a standalone ONNX graph plus prebuilt TensorRT engines, used by
|
| 49 |
+
[`vision-rt`](https://github.com/kornia/vision-rt)'s `vrt-lightglue` crate. These are the
|
| 50 |
+
ONNX/TensorRT form of `raco_aliked_lightglue.pth` — not a new set of weights.
|
| 51 |
+
|
| 52 |
+
```
|
| 53 |
+
normalized_keypoints (2P,1,K,2) f32 long-edge normalised
|
| 54 |
+
descriptors (2P,1,K,128) f32 L2-normalised
|
| 55 |
+
-> matches0 (P,K) i32 index into image 1, or -1 if unmatched
|
| 56 |
+
-> mscores0 (P,K) f32 match confidence in [0,1]
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
The matching extractor half lives in
|
| 60 |
+
[`kornia/raco-aliked`](https://huggingface.co/kornia/raco-aliked); both halves must come
|
| 61 |
+
from the same `kN` export, since `K` is baked into each.
|
| 62 |
+
|
| 63 |
+
**Why a split graph.** Upstream
|
| 64 |
+
([fabio-sim/LightGlue-ONNX](https://github.com/fabio-sim/LightGlue-ONNX)) publishes only a
|
| 65 |
+
fused extractor+matcher graph, which can match only the two images handed to it in a
|
| 66 |
+
single forward pass. Splitting the matcher out means descriptors extracted at *any* time —
|
| 67 |
+
from a map, a relocalization database, a keyframe store — can be matched against a live
|
| 68 |
+
frame. Cutting upstream of LightGlue's `NonZero` compaction also removes the graph's only
|
| 69 |
+
data-dependent shapes and its int64 output.
|
| 70 |
+
|
| 71 |
+
**Matching cost is O(K²)**, so unlike the extractor it gets rapidly worse with K. On a
|
| 72 |
+
Jetson Orin Nano (MAXN_SUPER, TRT 10.3.0.30, fp16, one pair): **7.9 ms** at k512,
|
| 73 |
+
**21.6 ms** at k1024, **126.5 ms** at k3072. Extraction moves the other way (the RaCo
|
| 74 |
+
ranker is bypassed at K≥3072), so pick k512 if you match every frame and k3072 if
|
| 75 |
+
extraction dominates.
|
| 76 |
+
|
| 77 |
+
**Engines are machine-locked** to the exact TensorRT version and GPU architecture they were
|
| 78 |
+
built for (`trt10.3.0.30`, `sm87` — JetPack 6 on Orin), and to the shape profile they were
|
| 79 |
+
built at. Elsewhere, build from the ONNX; `vrt-hub` does it automatically.
|
| 80 |
+
|
| 81 |
+
**Licences.** The ONNX/engine files are derived works combining LightGlue (Apache-2.0),
|
| 82 |
+
RaCo (Apache-2.0) and ALIKED (**BSD-3-Clause**) weights. BSD-3-Clause requires its
|
| 83 |
+
attribution to be reproduced in redistributions in binary form, which these files are — see
|
| 84 |
+
[`LICENSE-NOTICE.md`](LICENSE-NOTICE.md). That notice is absent from the upstream export
|
| 85 |
+
repo and is reproduced deliberately.
|