README.md

---
license: cc-by-4.0
tags:
- onnx
- stereo
- stereo-matching
- computer-vision
- viame
---

# VIAME Stereo Template Matcher

Epipolar template matching for stereo correspondence, exported from
[VIAME](https://github.com/VIAME/VIAME) as a single ONNX graph. Given a set of points in
the left image of a calibrated stereo pair, it returns the matching points in the right
image, plus match scores.

This is VIAME's stereo measurement **method 1**
(`epipolar_template_matching`): for each source point it walks the epipolar curve implied
by the calibration, sampling candidate depths, and picks the best NCC (`TM_CCOEFF_NORMED`)
template match. Pair it with two-view triangulation to measure real-world lengths — the
usual application is measuring fish in stereo camera rigs.

**There are no learned weights.** The graph is pure geometry plus normalized
cross-correlation, which is why it is 89 KB. Nothing here was trained, and there is no
training data or evaluation set.

## Files

| File | Size | Notes |
| --- | --- | --- |
| `stereo_match.onnx` | 89 KB | opset 18, IR 8, float32 throughout |

## Baked-in constants

Two parameters are **frozen into the graph at export time** and are not runtime inputs:

| Constant | Value |
| --- | --- |
| `template_size` | 13 (13×13 NCC patch) |
| `num_samples` | 5000 (depth samples along the epipolar curve) |

These match `configs/pipelines/interactive_stereo_template.conf` in VIAME, which is what
the DIVE desktop interactive stereo service loads. Note they are **not** the export
script's own defaults (25 / 5000) — re-exporting without explicit flags produces a model
that loads identically but matches differently. To reproduce this exact file:

```bash
python plugins/onnx/export_stereo_mapping.py --model match \
    --out stereo_match.onnx --template-size 13 --num-samples 5000
```

## Inputs

All float32. The world frame is the **left camera**, so the left camera is normally
`R_left = I`, `t_left = 0`, and the right camera carries the rig's relative pose.
Matrices are row-major.

| Name | Shape | Meaning |
| --- | --- | --- |
| `left_gray` | `[H, W]` | Left image, grayscale, 0–255 |
| `right_gray` | `[Hr, Wr]` | Right image, grayscale, 0–255 |
| `points_left` | `[P, 2]` | Source points `(x, y)` in left-image pixels |
| `K_left` | `[3, 3]` | Left intrinsics |
| `dist_left` | `[8]` | Left distortion `[k1, k2, p1, p2, k3, k4, k5, k6]`, zero-padded |
| `R_left` | `[3, 3]` | Left rotation (identity in the usual convention) |
| `t_left` | `[3]` | Left translation (zero in the usual convention) |
| `K_right` | `[3, 3]` | Right intrinsics |
| `dist_right` | `[8]` | Right distortion |
| `R_right` | `[3, 3]` | Rig rotation, left → right |
| `t_right` | `[3]` | Rig translation, in calibration units |
| `min_depth` | scalar | Near bound of the depth search |
| `max_depth` | scalar | Far bound of the depth search |

Grayscale must use BT.601 luma (`0.299 R + 0.587 G + 0.114 B`) to match the OpenCV
`BGR2GRAY` the reference implementation uses.

## Outputs

| Name | Shape | Meaning |
| --- | --- | --- |
| `right_points` | `[P, 2]` | Matched points in right-image pixels |
| `best_score` | `[P]` | Best NCC score |
| `second_score` | `[P]` | Best NCC score outside a `template_size` neighborhood of the winner |

## Acceptance thresholds

The graph deliberately does **not** apply a score threshold — it returns the best match
unconditionally, so the host decides what to accept. VIAME's two reference hosts disagree,
and both are defensible:

| Host | Threshold | Uniqueness ratio |
| --- | --- | --- |
| `interactive_stereo_template.conf` (DIVE desktop) | 0.5 | none |
| `plugins/onnx/run_epipolar_onnx.py` | 0.2 | 0.85 |

The uniqueness test, where used, rejects a match when
`second_score / best_score > ratio` — i.e. the winner was not clearly better than an
unrelated candidate elsewhere on the curve. Useful on repetitive texture.

## Search range

The graph takes a **depth** range, but disparity is usually the more natural way to think
about it. Convert with:

```
min_depth = fx * baseline / max_disparity
max_depth = fx * baseline / min_disparity
```

where `fx = K_left[0][0]` and `baseline = ||t_right||`.

A disparity range of **2–300 px** matches the DIVE desktop interactive stereo config, but
this is scene-dependent: VIAME's batch measurement pipelines ship 7–724 for other rigs. Too
wide invites false matches; too narrow misses the target entirely. Calibrate it to how far
the same object actually shifts between your two cameras.

## Usage

### Python

```python
import numpy as np, onnxruntime as ort

sess = ort.InferenceSession("stereo_match.onnx", providers=["CPUExecutionProvider"])
out = sess.run(None, {
    "left_gray":   left.astype(np.float32),      # [H, W], 0-255
    "right_gray":  right.astype(np.float32),
    "points_left": np.array([[330.4, 234.8]], np.float32),
    "K_left": K1, "dist_left": d1, "R_left": np.eye(3, dtype=np.float32),
    "t_left": np.zeros(3, np.float32),
    "K_right": K2, "dist_right": d2, "R_right": R, "t_right": T,
    "min_depth": np.float32(fx * baseline / 300),
    "max_depth": np.float32(fx * baseline / 2),
})
right_points, best_score, second_score = out
accepted = best_score >= 0.5
```

### Browser / Node (onnxruntime-web)

The graph is small and CPU-only, so it runs comfortably in a browser via the WASM
execution provider — this is how [DIVE](https://github.com/Kitware/dive) warps a detection
from one camera to the other with no backend. A complete client-side implementation lives
in `client/dive-common/use/stereo/` (see
[Kitware/dive#1709](https://github.com/Kitware/dive/pull/1709)), covering calibration
parsing, the search-range conversion above, and two-view triangulation.

### Not a transformers.js model

This is a bespoke geometry graph, not a transformer. It has no `config.json`, no tokenizer
or image processor, and no architecture in the transformers.js registry, so `pipeline()` /
`AutoModel` will not load it. Use onnxruntime (or onnxruntime-web) directly.

## Swapping left and right

The graph always matches *left → right*. To warp a point annotated on the right camera,
invert the rig instead of swapping the inputs: the new world frame is the old right camera,
so `R' = Rᵀ` and `T' = -Rᵀ·T`, with the intrinsics and distortion swapped between sides.

## Accuracy

Validated against the VIAME C++ / Python reference implementation to roughly a quarter
pixel. Note the exporter's own verification reports a sub-pixel `right_points` difference
between eager PyTorch and onnxruntime (about 0.04 px at `template_size=13`) caused by
ties between equally-scoring epipolar candidates; scores match to ~1e-8.

## License and provenance

CC-BY-4.0. Produced by `plugins/onnx/export_stereo_mapping.py` in
[VIAME](https://github.com/VIAME/VIAME), whose core infrastructure is BSD-3-Clause.