--- license: apache-2.0 language: - en --- # SRPose: Two-view Relative Pose Estimation with Sparse Keypoints **SRPose**: A **S**parse keypoint-based framework for **R**elative **Pose** estimation between two views in both camera-to-world and object-to-camera scenarios. | Reference | Query | Ground Truth | |:--------:|:---------:|:--------:| | ![](assets/figures/scene5_vis_0.png) | ![](assets/figures/scene5_vis_1.png) | ![](assets/figures/scene5_vis_gt.png) | | ![](assets/figures/obj_vis_reference_labeled.png) | ![](assets/figures/obj_vis_query.png) |![](assets/figures/obj_vis_gt.png)| ## [Project page](https://frickyinn.github.io/srpose/) | [arXiv](https://arxiv.org/abs/2407.08199) ## Setup Please first intall PyTorch according to [here](https://pytorch.org/get-started/locally/), then install other dependencies using pip: ``` cd SRPose pip install -r requirements.txt ``` ## Evaluation 1. Download pretrained models [here](https://drive.google.com/drive/folders/1bBlds3UX7-XDCevbIl4bnnywvWzzP5nN) for evaluation. 2. Create new folders: ``` mkdir checkpoints & mkdir data ``` 3. Organize the downloaded checkpoints like this: ``` SRPose |-- checkpoints |-- ho3d.ckpt |-- linemod.ckpt |-- mapfree.ckpt |-- matterport.ckpt |-- megadepth.ckpt `-- scannet.ckpt ... ``` ### Matterport 1. Download Matterport dataset [here](https://github.com/jinlinyi/SparsePlanes/blob/main/docs/data.md), only `mp3d_planercnn_json.zip` and `rgb.zip` are required. 2. Unzip and organize the downloaded files: ``` mkdir data/mp3d mkdir data/mp3d/mp3d_planercnn_json & mkdir data/mp3d/rgb unzip /mp3d_planercnn_json.zip -d data/mp3d/mp3d_planercnn_json unzip /rgb.zip -d data/mp3d/rgb ``` 3. The resulted directory tree should be like this: ``` SRPose |-- data |-- mp3d |-- mp3d_planercnn_json | |-- cached_set_test.json | |-- cached_set_train.json | `-- cached_set_val.json `-- rgb |-- 17DRP5sb8fy ... ... ... ``` 4. Evaluate with the following command: ``` python eval.py configs/matterport.yaml checkpoints/matterport.ckpt ``` ### ScanNet & MegaDepth 1. Download and organize the ScanNet-1500 and MegaDepth-1500 test sets according to the [LoFTR Training Script](https://github.com/zju3dv/LoFTR/blob/master/docs/TRAINING.md). Note that only the test sets and the dataset indices are required. 2. The resulted directory tree should be: ``` SRPose |-- data |-- scannet | |-- index | |-- test | `-- train (optional) |-- megadepth |-- index |-- test `-- train (optional) ... ... ``` 3. Evaluate with the following commands: ``` python eval.py configs/scannet.yaml checkpoints/scannet.ckpt python eval.py configs/megadepth.yaml checkpoints/megedepth.ckpt ``` ### HO3D 1. Download HO3D (version 3) dataset [here](https://www.tugraz.at/institute/icg/research/team-lepetit/research-projects/hand-object-3d-pose-annotation/), `HO3D_v3.zip` and `HO3D_v3_segmentations_rendered.zip` are required. 2. Unzip and organize the downloaded files: ``` mkdir data/ho3d unzip /HO3D_v3.zip -d data/ho3d unzip /HO3D_v3_segmentations_rendered.zip -d data/ho3d ``` 3. Evaluate with the following commands: ``` python eval.py configs/ho3d.yaml checkpoints/ho3d.ckpt python eval_add_reproj.py configs/ho3d.yaml checkpoints/ho3d.ckpt ``` ### Linemod 1. Download Linemod dataset [here](https://bop.felk.cvut.cz/datasets/) or run the following commands: ``` cd data export SRC=https://bop.felk.cvut.cz/media/data/bop_datasets wget $SRC/lm_base.zip # Base archive with dataset info, camera parameters, etc. wget $SRC/lm_models.zip # 3D object models. wget $SRC/lm_test_all.zip # All test images ("_bop19" for a subset used in the BOP Challenge 2019/2020). wget $SRC/lm_train_pbr.zip # PBR training images (rendered with BlenderProc4BOP). unzip lm_base.zip # Contains folder "lm". unzip lm_models.zip -d lm # Unpacks to "lm". unzip lm_test_all.zip -d lm # Unpacks to "lm". unzip lm_train_pbr.zip -d lm # Unpacks to "lm". ``` 2. Evaluate with the following commands: ``` python eval.py configs/linemod.yaml checkpoints/linemod.ckpt python eval_add_reproj.py configs/linemod.yaml checkpoints/linemod.ckpt ``` ### Niantic 1. Download Niantic dataset [here](https://research.nianticlabs.com/mapfree-reloc-benchmark/dataset). 2. Unzip and organize the downloaded files: ``` mkdir data/mapfree unzip /train.zip -d data/mapfree unzip /val.zip -d data/mapfree unzip /test.zip -d data/mapfree ``` 3. The ground truth of the test set is not publicly available, but you can run the following command to produce a new submission file and submit it on the [project page](https://research.nianticlabs.com/mapfree-reloc-benchmark/submit) for evaluation: ``` python eval_add_reproj.py configs/mapfree.yaml checkpoints/mapfree.ckpt ``` You should be able to find a `new_submission.zip` in `SRPose/assets/` afterwards, or you can submit the already produced file `SRPose/assets/mapfree_submission.zip` instead. ## Training Download and organize the datasets following [Evaluation](#evaluation), then run the following command for training: ``` python train.py configs/.yaml ``` Please refer to the `.yaml` files in `SRPose/configs/` for detailed configurations. ## Baselines We also offer two publicly available matcher-based baselines, [LightGlue](https://github.com/cvg/LightGlue) and [LoFTR](https://github.com/zju3dv/LoFTR), for evaluation and comparison. Just run the following commands: ``` # For Matterport, ScanNet and MegaDepth python eval_baselines.py configs/.yaml lightglue python eval_baselines.py configs/.yaml loftr # For HO3D and Linemod python eval_baselines.py configs/.yaml lightglue --resize 640 --depth python eval_baselines.py configs/.yaml loftr --resize 640 --depth ``` The `--resize xx` option controls the larger dimension of cropped target object images that will be resized to. The `--depth` option controls whether the depth maps will be used to obtain scaled pose estimation. ## Acknowledgements In this repository, we have used codes from the following repositories. We thank all the authors for sharing great codes. - [LightGlue](https://github.com/cvg/LightGlue) - [LoFTR](https://github.com/zju3dv/LoFTR) - [8point](https://github.com/crockwell/rel_pose) - [SparsePlanes](https://github.com/jinlinyi/SparsePlanes/tree/main) - [Map-free](https://github.com/nianticlabs/map-free-reloc/tree/main) ## Citation ``` @inproceedings{yin2024srpose, title={SRPose: Two-view Relative Pose Estimation with Sparse Keypoints}, author={Yin, Rui and Zhang, Yulun and Pan, Zherong and Zhu, Jianjun and Wang, Cheng and Jia, Biao}, booktitle={ECCV}, year={2024} } ```