thekman17 commited on
Commit
139bb75
·
verified ·
1 Parent(s): 20b0e51

calibration head: numbers, recommended weights

Browse files
Files changed (1) hide show
  1. README.md +7 -5
README.md CHANGED
@@ -35,8 +35,7 @@ https://github.com/kalman17/mcvo (`mcvo/`).
35
 
36
  | Method | Labels | Params | Peak GPU mem | Latency | Rotation Sintel / TUM / KITTI | Heading KITTI (zero-shot) | Heading Sintel / TUM | Focal error Sintel / TUM / KITTI |
37
  |---|:---:|---:|---:|---:|---:|---:|---:|---:|
38
- | **This model (image-only VO)** | none | 154 M (67 M trained) | **0.69 GiB** | **75 ms** | 0.46° / 0.89° / 0.19° | 7.0° | 81° / 90° | — |
39
- | **+ MCT calibration ([`anycam-mct`](https://huggingface.co/thekman17/anycam-mct)), back-to-back** | none | 500 M | ≈2.0 GiB | 236 ms | as above | as above | as above | 20.7 % / 12.9 % / 20.4 % |
40
  | π³ | GT | 959 M | 5.5 GiB | 171 ms | 0.22° / 0.26° / 0.11° | 2.2° | 27° / 34° | 25.2 % / 7.6 % / 28.9 % |
41
  | VGGT-1B | GT | 1257 M | 7.0 GiB | 203 ms | 0.28° / 0.32° / 0.12° | 4.6° | 38° / 37° | 34.0 % / 25.8 % / 37.1 % |
42
  | Depth Anything 3 (Giant) | GT | 1690 M | 9.7 GiB | 600 ms | 0.19° / 0.27° / 0.09° | 1.3° | 19° / 32° | 24.4 % / 4.6 % / 15.8 % |
@@ -45,16 +44,19 @@ https://github.com/kalman17/mcvo (`mcvo/`).
45
 
46
  5–14× lower peak memory and 2–8× lower latency than the billion-parameter supervised models, 5–11× below the self-supervised pipelines, no labels at any stage — at roughly twice their rotation error, competitive heading on driving video, near-chance heading on small-baseline indoor video. Not the cheapest VO in existence: patch-based SLAM-style systems (DPVO, ~60 fps / ~4–5 GB, GT-trained) and classical ORB-SLAM are faster per frame, and small photometric self-supervised pose nets are cheaper still but far less accurate; among image-only feed-forward multi-frame transformers at this rotation accuracy, this is the smallest and the only self-supervised one. Measurement: one process per model, same GPU, identical windows, CUDA-synchronised, warm-up excluded; `experiments/bench_latency.py`, `honest_benchmarks/latency_summary.json` in the GitHub repo.
47
 
 
 
48
  **Limitations, stated plainly.** Translation direction on small-baseline indoor video (Sintel,
49
  TUM-RGBD) is near chance and did not improve under longer context, teacher distillation, an
50
  epipolar consistency loss, or motion-rich additional training data; it is treated as a
51
  structural limit of flow-reprojection self-supervision when parallax is tiny. Trajectory-level
52
- ATE trails AnyCam's long-context inference (Sintel 0.18 vs 0.10). No intrinsics head — pair
53
- with a calibration model (e.g. `thekman17/anycam-mct`) if focal length is needed.
54
 
55
  ## Files
56
 
57
- - `mcvo_e3.pt` — weights (`model_state_dict`) + training `args`; load with
 
58
  `mcvo.model.MCVO(**args)` from the GitHub repo, see `experiments/honest_benchmark.py::MCVOModel`.
59
 
60
  ## Attribution
 
35
 
36
  | Method | Labels | Params | Peak GPU mem | Latency | Rotation Sintel / TUM / KITTI | Heading KITTI (zero-shot) | Heading Sintel / TUM | Focal error Sintel / TUM / KITTI |
37
  |---|:---:|---:|---:|---:|---:|---:|---:|---:|
38
+ | **This model (`mcvo_e3_calib.pt`: pose + intrinsics)** | none | 154 M (67 M trained) | **0.69 GiB** | **75 ms** | 0.46° / 0.89° / 0.19° | 7.0° | 81° / 90° | 21.8 % / 13.3 % / 42.4 % |
 
39
  | π³ | GT | 959 M | 5.5 GiB | 171 ms | 0.22° / 0.26° / 0.11° | 2.2° | 27° / 34° | 25.2 % / 7.6 % / 28.9 % |
40
  | VGGT-1B | GT | 1257 M | 7.0 GiB | 203 ms | 0.28° / 0.32° / 0.12° | 4.6° | 38° / 37° | 34.0 % / 25.8 % / 37.1 % |
41
  | Depth Anything 3 (Giant) | GT | 1690 M | 9.7 GiB | 600 ms | 0.19° / 0.27° / 0.09° | 1.3° | 19° / 32° | 24.4 % / 4.6 % / 15.8 % |
 
44
 
45
  5–14× lower peak memory and 2–8× lower latency than the billion-parameter supervised models, 5–11× below the self-supervised pipelines, no labels at any stage — at roughly twice their rotation error, competitive heading on driving video, near-chance heading on small-baseline indoor video. Not the cheapest VO in existence: patch-based SLAM-style systems (DPVO, ~60 fps / ~4–5 GB, GT-trained) and classical ORB-SLAM are faster per frame, and small photometric self-supervised pose nets are cheaper still but far less accurate; among image-only feed-forward multi-frame transformers at this rotation accuracy, this is the smallest and the only self-supervised one. Measurement: one process per model, same GPU, identical windows, CUDA-synchronised, warm-up excluded; `experiments/bench_latency.py`, `honest_benchmarks/latency_summary.json` in the GitHub repo.
46
 
47
+ **Calibration head (new).** `mcvo_e3_calib.pt` adds a 1.9k-parameter linear head on the per-frame camera token that predicts focal length and principal point; it was distilled, with everything else frozen, from the cached AnyCalib per-frame intrinsics that already serve as a training-time teacher — so the pose output is *identical* to `mcvo_e3.pt` and cost is unchanged. Focal error (median, square 336 crops): Sintel 21.8 %, TUM-RGBD 13.3 %, KITTI 42.4 % — teacher level (AnyCalib: 20.1 / 11.2 / 18.4 %) on the kind of footage it was trained on, clearly worse than the teacher on KITTI's narrow-FOV driving crops, which lie outside the training corpus' focal range. A joint fine-tune (one epoch) gave 27.3 / 10.2 / 34.4 % and slightly disturbed pose; not released.
48
+
49
  **Limitations, stated plainly.** Translation direction on small-baseline indoor video (Sintel,
50
  TUM-RGBD) is near chance and did not improve under longer context, teacher distillation, an
51
  epipolar consistency loss, or motion-rich additional training data; it is treated as a
52
  structural limit of flow-reprojection self-supervision when parallax is tiny. Trajectory-level
53
+ ATE trails AnyCam's long-context inference (Sintel 0.18 vs 0.10). Intrinsics come from a distilled head that is teacher-level in-domain and weak on
54
+ narrow-FOV driving crops; for calibration-critical use pair with `thekman17/anycam-mct` or AnyCalib.
55
 
56
  ## Files
57
 
58
+ - `mcvo_e3_calib.pt` — **recommended**: E3 weights + calibration head (pose identical to `mcvo_e3.pt`, plus intrinsics)
59
+ - `mcvo_e3.pt` — pose only; weights (`model_state_dict`) + training `args`; load with
60
  `mcvo.model.MCVO(**args)` from the GitHub repo, see `experiments/honest_benchmark.py::MCVOModel`.
61
 
62
  ## Attribution