calibration head: numbers, recommended weights
Browse files
README.md
CHANGED
|
@@ -35,8 +35,7 @@ https://github.com/kalman17/mcvo (`mcvo/`).
|
|
| 35 |
|
| 36 |
| Method | Labels | Params | Peak GPU mem | Latency | Rotation Sintel / TUM / KITTI | Heading KITTI (zero-shot) | Heading Sintel / TUM | Focal error Sintel / TUM / KITTI |
|
| 37 |
|---|:---:|---:|---:|---:|---:|---:|---:|---:|
|
| 38 |
-
| **This model (
|
| 39 |
-
| **+ MCT calibration ([`anycam-mct`](https://huggingface.co/thekman17/anycam-mct)), back-to-back** | none | 500 M | ≈2.0 GiB | 236 ms | as above | as above | as above | 20.7 % / 12.9 % / 20.4 % |
|
| 40 |
| π³ | GT | 959 M | 5.5 GiB | 171 ms | 0.22° / 0.26° / 0.11° | 2.2° | 27° / 34° | 25.2 % / 7.6 % / 28.9 % |
|
| 41 |
| VGGT-1B | GT | 1257 M | 7.0 GiB | 203 ms | 0.28° / 0.32° / 0.12° | 4.6° | 38° / 37° | 34.0 % / 25.8 % / 37.1 % |
|
| 42 |
| Depth Anything 3 (Giant) | GT | 1690 M | 9.7 GiB | 600 ms | 0.19° / 0.27° / 0.09° | 1.3° | 19° / 32° | 24.4 % / 4.6 % / 15.8 % |
|
|
@@ -45,16 +44,19 @@ https://github.com/kalman17/mcvo (`mcvo/`).
|
|
| 45 |
|
| 46 |
5–14× lower peak memory and 2–8× lower latency than the billion-parameter supervised models, 5–11× below the self-supervised pipelines, no labels at any stage — at roughly twice their rotation error, competitive heading on driving video, near-chance heading on small-baseline indoor video. Not the cheapest VO in existence: patch-based SLAM-style systems (DPVO, ~60 fps / ~4–5 GB, GT-trained) and classical ORB-SLAM are faster per frame, and small photometric self-supervised pose nets are cheaper still but far less accurate; among image-only feed-forward multi-frame transformers at this rotation accuracy, this is the smallest and the only self-supervised one. Measurement: one process per model, same GPU, identical windows, CUDA-synchronised, warm-up excluded; `experiments/bench_latency.py`, `honest_benchmarks/latency_summary.json` in the GitHub repo.
|
| 47 |
|
|
|
|
|
|
|
| 48 |
**Limitations, stated plainly.** Translation direction on small-baseline indoor video (Sintel,
|
| 49 |
TUM-RGBD) is near chance and did not improve under longer context, teacher distillation, an
|
| 50 |
epipolar consistency loss, or motion-rich additional training data; it is treated as a
|
| 51 |
structural limit of flow-reprojection self-supervision when parallax is tiny. Trajectory-level
|
| 52 |
-
ATE trails AnyCam's long-context inference (Sintel 0.18 vs 0.10).
|
| 53 |
-
|
| 54 |
|
| 55 |
## Files
|
| 56 |
|
| 57 |
-
- `
|
|
|
|
| 58 |
`mcvo.model.MCVO(**args)` from the GitHub repo, see `experiments/honest_benchmark.py::MCVOModel`.
|
| 59 |
|
| 60 |
## Attribution
|
|
|
|
| 35 |
|
| 36 |
| Method | Labels | Params | Peak GPU mem | Latency | Rotation Sintel / TUM / KITTI | Heading KITTI (zero-shot) | Heading Sintel / TUM | Focal error Sintel / TUM / KITTI |
|
| 37 |
|---|:---:|---:|---:|---:|---:|---:|---:|---:|
|
| 38 |
+
| **This model (`mcvo_e3_calib.pt`: pose + intrinsics)** | none | 154 M (67 M trained) | **0.69 GiB** | **75 ms** | 0.46° / 0.89° / 0.19° | 7.0° | 81° / 90° | 21.8 % / 13.3 % / 42.4 % |
|
|
|
|
| 39 |
| π³ | GT | 959 M | 5.5 GiB | 171 ms | 0.22° / 0.26° / 0.11° | 2.2° | 27° / 34° | 25.2 % / 7.6 % / 28.9 % |
|
| 40 |
| VGGT-1B | GT | 1257 M | 7.0 GiB | 203 ms | 0.28° / 0.32° / 0.12° | 4.6° | 38° / 37° | 34.0 % / 25.8 % / 37.1 % |
|
| 41 |
| Depth Anything 3 (Giant) | GT | 1690 M | 9.7 GiB | 600 ms | 0.19° / 0.27° / 0.09° | 1.3° | 19° / 32° | 24.4 % / 4.6 % / 15.8 % |
|
|
|
|
| 44 |
|
| 45 |
5–14× lower peak memory and 2–8× lower latency than the billion-parameter supervised models, 5–11× below the self-supervised pipelines, no labels at any stage — at roughly twice their rotation error, competitive heading on driving video, near-chance heading on small-baseline indoor video. Not the cheapest VO in existence: patch-based SLAM-style systems (DPVO, ~60 fps / ~4–5 GB, GT-trained) and classical ORB-SLAM are faster per frame, and small photometric self-supervised pose nets are cheaper still but far less accurate; among image-only feed-forward multi-frame transformers at this rotation accuracy, this is the smallest and the only self-supervised one. Measurement: one process per model, same GPU, identical windows, CUDA-synchronised, warm-up excluded; `experiments/bench_latency.py`, `honest_benchmarks/latency_summary.json` in the GitHub repo.
|
| 46 |
|
| 47 |
+
**Calibration head (new).** `mcvo_e3_calib.pt` adds a 1.9k-parameter linear head on the per-frame camera token that predicts focal length and principal point; it was distilled, with everything else frozen, from the cached AnyCalib per-frame intrinsics that already serve as a training-time teacher — so the pose output is *identical* to `mcvo_e3.pt` and cost is unchanged. Focal error (median, square 336 crops): Sintel 21.8 %, TUM-RGBD 13.3 %, KITTI 42.4 % — teacher level (AnyCalib: 20.1 / 11.2 / 18.4 %) on the kind of footage it was trained on, clearly worse than the teacher on KITTI's narrow-FOV driving crops, which lie outside the training corpus' focal range. A joint fine-tune (one epoch) gave 27.3 / 10.2 / 34.4 % and slightly disturbed pose; not released.
|
| 48 |
+
|
| 49 |
**Limitations, stated plainly.** Translation direction on small-baseline indoor video (Sintel,
|
| 50 |
TUM-RGBD) is near chance and did not improve under longer context, teacher distillation, an
|
| 51 |
epipolar consistency loss, or motion-rich additional training data; it is treated as a
|
| 52 |
structural limit of flow-reprojection self-supervision when parallax is tiny. Trajectory-level
|
| 53 |
+
ATE trails AnyCam's long-context inference (Sintel 0.18 vs 0.10). Intrinsics come from a distilled head that is teacher-level in-domain and weak on
|
| 54 |
+
narrow-FOV driving crops; for calibration-critical use pair with `thekman17/anycam-mct` or AnyCalib.
|
| 55 |
|
| 56 |
## Files
|
| 57 |
|
| 58 |
+
- `mcvo_e3_calib.pt` — **recommended**: E3 weights + calibration head (pose identical to `mcvo_e3.pt`, plus intrinsics)
|
| 59 |
+
- `mcvo_e3.pt` — pose only; weights (`model_state_dict`) + training `args`; load with
|
| 60 |
`mcvo.model.MCVO(**args)` from the GitHub repo, see `experiments/honest_benchmark.py::MCVOModel`.
|
| 61 |
|
| 62 |
## Attribution
|