Unreasonable Confidence from Simulator Misspecification: Directional Robustness on the SPEED+ Sim-to-Real Benchmark
Abstract
A simple monocular pose estimator trained on synthetic spacecraft images shows poor real-world transfer, with overconfident pose errors and uneven gains from backbone adaptation and photometric augmentation under different lighting.
Vision-based navigation systems for orbit must be trained almost entirely on synthetic imagery, because real labelled images of a given spacecraft in flight rarely exist. Benchmarks like SPEED+ are built for that situation: the training set comes from a simulator, and the test sets are real images of a satellite mockup under two kinds of lighting. That simulator approximates how light reflects off mylar and aluminium, and like every scientific simulator it is misspecified. Its error is radiometric, since it misses the specular highlights, hard shadows and sensor saturation common in real images. We probe the resulting synthetic-to-real gap, the central risk for such systems, with a deliberately simple monocular pose estimator. In a 2x2 study of backbone adaptation and photometric augmentation, trained on synthetic images and tested on both real hardware-in-the-loop domains, we find three things. (1) The model does not know it is failing: regions conformally calibrated to 90% on synthetic data cover only 9-15% of rotation errors on sunlamp and 21-38% on lightbox, so a wrong pose carries the confidence of a correct one. (2) Transfer is asymmetric: scaling data and features improved synthetic rotation error from 1.65 to 1.03 rad but moved the harshly lit sunlamp domain by only 0.05 rad, under twice its seed s.d., whereas adapting the backbone's last stage carried 63-83% of its extra synthetic gain to both real domains. (3) Once the backbone adapts, augmentation helps only in the direction it models: our photometric stack improved the diffusely lit lightbox domain from 1.17 to 0.99 rad on every seed but moved sunlamp no more than seed-to-seed variation, while with the backbone frozen it helped sunlamp more. Every post-review experiment was pre-registered, and the code, per-image predictions and pre-registration record are at https://github.com/lokindoo/spacecraft-pose-estimation.
Community
TL;DR: A spacecraft pose estimator trained only on rendered images (SPEED+) keeps its synthetic-calibrated confidence on real images, while its accuracy collapses. 90% split-conformal regions cover only 9-15% of rotation errors under harsh sunlamp lighting, and 21-38% under diffuse lightbox lighting. Test-time BatchNorm adaptation lowers the error but lifts sunlamp coverage only to 11-20%.
Two more findings from the 2×2 study (backbone frozen vs. last stage trained × photometric augmentation, three seeds per cell):
- Transfer is asymmetric. With the backbone frozen, more data and features cut synthetic rotation error by 0.62 rad but move sunlamp by 0.05 rad, within seed noise. Training the last stage carries 63-83% of its extra gain to both real domains.
- Augmentation is directional. Once the backbone adapts, the augmentation helps only in the direction it models: lightbox improves on every seed (1.17 → 0.99 rad), while sunlamp moves no more than seed-to-seed variation. With the backbone frozen, the order reverses.
This is the extended version of the Sim2Science @ NeurIPS 2026 workshop paper. The repo has per-image predictions for all runs and a script that checks every number in the paper.
Code: https://github.com/lokindoo/spacecraft-pose-estimation
Weights (CC BY 4.0): https://zenodo.org/records/24140385
Happy to answer any questions, especially from anyone working on uncertainty under sim-to-real shift.
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper