| license: mit | |
| task_categories: | |
| - object-detection | |
| language: | |
| - en | |
| viewer: false | |
| <p align="center"> | |
| <h1 align="center">Segment Any 3D Object with Language</h1> | |
| <p align="center"> | |
| <a href="https://github.com/0nandon">Seungjun Lee</a><sup>*</sup></span> · | |
| <a href="https://yuyangzhao.com">Yuyang Zhao</a><sup>*</sup> · | |
| <a href="https://www.comp.nus.edu.sg/~leegh/">Gim Hee Lee</a> <br> | |
| National University of Singapore<br> | |
| <sup>*</sup>equal contribution | |
| </p> | |
| <h2 align="center">ICLR 2025</h2> | |
| <h3 align="center"><a href="https://github.com/CVRP-SOLE/SOLE">Code</a> | <a href="https://arxiv.org/abs/2404.02157">Paper</a> | <a href="https://cvrp-sole.github.io">Project Page</a> </h3> | |
| </p> | |
| <p align="center"> | |
| <a href=""> | |
| <img src="https://github.com/CVRP-SOLE/CVRP-SOLE.github.io/blob/main/static/images/teaser.png?raw=true" alt="Logo" width="100%"> | |
| </a> | |
| </p> | |
| <p align="center"> | |
| <strong>SOLE</strong> is highly generalizable and can segment corresponding instances with various language instructions, including but not limited to visual questions, attributes description, and functional description. | |
| </p> | |
| <br> | |
| In this repository, we provide the preprocessed data and official weights to train and evaluate our SOLE model. The provided data are listed below: | |
| ### Pre-processed data | |
| - Pre-processed 3D data | |
| * <a href="https://huggingface.co/datasets/onandon/SOLE/tree/main/processed/scannet">ScanNet</a> | |
| * <a href="https://huggingface.co/datasets/onandon/SOLE/tree/main/processed/scannet200">ScanNet200</a> | |
| * <a href="https://huggingface.co/datasets/onandon/SOLE/tree/main/processed/replica">Replica</a> | |
| - Precomputed per-point CLIP features | |
| * <a href="https://huggingface.co/datasets/onandon/SOLE/tree/main/openseg/scannet">ScanNet</a> | |
| * <a href="https://huggingface.co/datasets/onandon/SOLE/tree/main/openseg/replica">Replica</a> | |
| - Precomputed features of MCA and MEA | |
| * ScanNet : <a href="https://huggingface.co/datasets/onandon/SOLE/tree/main/scannet_mca">MCA</a>, <a href="https://huggingface.co/datasets/onandon/SOLE/tree/main/scannet_mea">MEA</a> | |
| * ScanNet200 : <a href="https://huggingface.co/datasets/onandon/SOLE/tree/main/scannet200_mca">MCA</a>, <a href="https://huggingface.co/datasets/onandon/SOLE/tree/main/scannet200_mea">MEA</a> | |
| ### Weights | |
| - Backbone weights | |
| * <a href="https://huggingface.co/datasets/onandon/SOLE/blob/main/backbone_scannet.ckpt">Backbone weights for ScanNet</a> | |
| * <a href="https://huggingface.co/datasets/onandon/SOLE/blob/main/backbone_scannet200.ckpt">Backbone weights for ScanNet200</a> | |
| - Official weights | |
| * <a href="https://huggingface.co/datasets/onandon/SOLE/blob/main/scannet.ckpt">Offical weights of SOLE for ScanNet</a> | |
| * <a href="https://huggingface.co/datasets/onandon/SOLE/blob/main/scannet200.ckpt">Official weights of SOLE for ScanNet200</a> | |
| * <a href="https://huggingface.co/datasets/onandon/SOLE/blob/main/replica.ckpt">Official weights of SOLE for Replica</a> | |
| ## Data Configuration | |
| ### Pre-processed Data | |
| We first download the official <a href="http://www.scan-net.org">ScanNet</a> dataset and subsequently process it with running `datasets/preprocessing/scannet_preprocessing.py` script in the <a href="">SOLE github repository</a>. If you want to process the data by yourself, 1) first download the official <a href="http://www.scan-net.org">ScanNet</a> dataset and 2) run the command below: | |
| ``` | |
| python -m datasets.preprocessing.scannet_preprocessing preprocess \ | |
| --data_dir="PATH_TO_RAW_SCANNET_DATASET" \ | |
| --save_dir="data/processed/scannet" \ | |
| --git_repo="PATH_TO_SCANNET_GIT_REPO" \ | |
| --scannet200=false/true | |
| ``` | |
| ### Pre-computed per-point CLIP features | |
| This files provide pointwise CLIP features computed with following <a href="">OpenScene</a>. Each file consists of one tensor which has the shape of: | |
| ``` | |
| torch.Size([N, 768]) | |
| ``` | |
| where N is the number of points in one scene. | |
| ### Pre-computed MCA features | |
| This files provide generated captions for each instance and CLIP text features of corresponding caption. | |
| Example of the MCA file is: | |
| ```javascript | |
| { | |
| "captions" : ["a wooden desk with the monitor", ...], // list of generated captions for every instances in a scene | |
| "features" : [torch.Size([1, 768]), ...], // list of CLIP text features for generated captions | |
| } | |
| ``` | |
| ### Pre-computed MEA features | |
| This files provide CLIP text features of the entity-level captions for each instance. Example of the MEA file is: | |
| ```javascript | |
| { | |
| "guide" : torch.Size([K, 768]), // CLIP text features of entity-level captions for every K instances | |
| } | |
| ``` | |
| where K is the number of intances in one scene. | |
| ## Citation | |
| If you find our code or paper useful, please cite | |
| ```bibtex | |
| @article{lee2024segment, | |
| title = {Segment Any 3D Object with Language}, | |
| author = {Lee, Seungjun and Zhao, Yuyang and Lee, Gim Hee}, | |
| year = {2024}, | |
| journal = {arXiv preprint arXiv:2404.02157}, | |
| } | |
| ``` |
Xet Storage Details
- Size:
- 4.93 kB
- Xet hash:
- 634f02b6088af1b3c63384e6112211de7fe1fb7369b9a8e45e43b432444e199e
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.