| license: mit | |
| datasets: | |
| - allenai/objaverse | |
| - allenai/objaverse-xl | |
| - qizekun/OrienText300K | |
| - tiange/Cap3D | |
| language: | |
| - en | |
| # PointSO | |
| Paper_link: https://arxiv.org/abs/2502.13143 | |
| <br> | |
| Project Page: https://qizekun.github.io/sofar/ | |
| <br> | |
| Github Code: https://github.com/qizekun/SoFar | |
| We introduce the concept of semantic orientation, representing the object orientation condition on open vocabulary language. | |
| The model is part of [SoFar](https://github.com/qizekun/SoFar), a 6-DoF system for spatial reasoning and robotic manipulation. | |
| The released weights is on [Huggingface PointSO](https://huggingface.co/qizekun/PointSO), and the code is in the [orientation](https://github.com/qizekun/SoFar/tree/main/orientation) folder. | |
| ## Pretrain | |
| Download the PointMAE as initialization. | |
| ```bash | |
| wget https://github.com/Pang-Yatian/Point-MAE/releases/download/main/pretrain.pth -P orientation/ | |
| ``` | |
| Perpare the OrienText300K dataset following [DATASET.md](https://github.com/qizekun/SoFar/blob/main/datasets/DATASET.md). | |
| ```bash | |
| cd orientation | |
| sh train_ddp.sh | |
| ``` | |
| ## Finetune | |
| Perpare the Open6DOR finetuning dataset following [DATASET.md](https://github.com/qizekun/SoFar/blob/main/datasets/DATASET.md). | |
| The dataset is generated from isaac sim with different assets from Open6DOR. | |
| Finetune PointSO will significantly improve the performance on Open6DOR rotation track & 6-DoF track. | |
| We recommend using this version of the model for the Open6DOR V2 evaluation. | |
| ```bash | |
| cd orientation | |
| sh train_ddp_ft.sh | |
| ``` |