| ## Dataset |
|
|
| To train our affordance segmentation model, we use two types of data: |
| * **General Segmentation Data**: This follows [LISA](https://github.com/dvlab-research/LISA). |
| * **Affordance Segmentation Data**: This is a large-scale dataset that we collect. |
|
|
| ### General Segmentation Data |
| These data is organized as follows: |
| ``` |
| ./data/ |
| ├── lisa_data |
| │ ├── ade20k |
| │ ├── coco |
| │ ├── cocostuff |
| │ ├── llava_dataset |
| │ ├── mapillary |
| │ ├── reason_seg |
| │ ├── refer_seg |
| │ ├── vlpart |
| ``` |
|
|
| ### Affordance Segmentation Data |
|
|
| We employ images from HANDAL, Open-X, GraspNet, EgoObjects, and RLBench in our affordance segmentation task. |
|
|
| The HANDAL data is downloaded and organized according to its official [repo](https://github.com/NVlabs/HANDAL). |
| Other data can be downloaded from the [Hugging Face](https://huggingface.co/datasets/Dongming97/RAGNet). |
|
|
| The training data is organized as follows: |
| ``` |
| ./data/ |
| ├── openx_train.pkl |
| ├── graspnet_train.pkl |
| ├── egoobjects_train.pkl |
| ├── rlbench_train.pkl |
| ├── handal_hard_reasoning_train.pkl |
| ├── egoobjects_easy_reasoning_train.pkl |
| ├── egoobjects_hard_reasoning_train.pkl |
| ├── HANDAL |
| │ ├── without_depth |
| │ ├── handal_dataset_adjustable_wrenches |
| │ ├── handal_dataset_combinational_wrenches |
| │ ├── handal_dataset_fixed_joint_pliers |
| │ ├── ... |
| ├── openx |
| │ ├── images |
| │ ├── fractal20220817_data |
| │ ├── bridge |
| │ ├── masks |
| │ ├── fractal20220817_data |
| │ ├── bridge |
| ├── graspnet |
| │ ├── images |
| │ ├── masks |
| │ ├── test_seen |
| │ ├── test_novel |
| ├── egoobjects |
| │ ├── images |
| │ ├── masks |
| ├── rlbench |
| │ ├── images |
| │ ├── masks |
| ├── 3doi |
| │ ├── images |
| │ ├── masks |
| ``` |
|
|
| The evaluation data is also in the same dictory, but with the `*_eval.pkl` files instead of `*_train.pkl`. |
|
|
| ``` |
| ./data/ |
| ├── handal_mini_val.pkl |
| ├── graspnet_test_seen_val.pkl |
| ├── graspnet_test_novel_val.pkl |
| ├── 3doi_val.pkl |
| ├── handal_easy_reasoning_val.pkl |
| ├── handal_hard_reasoning_val.pkl |
| ├── 3doi_easy_reasoning_val.pkl |
| ``` |
|
|
| You can use the following script to confirm if data is organized correctly: |
| ```bash |
| python data_curation/check_dataset.py |
| ``` |
|
|
| ### About data curation |
| 1. **SAM2**: We use SAM2 to generate affordance mask if the dataset provides box annotation. |
| 2. **Florence-2 + SAM2**: We use Florence-2 to generate the initial segmentation masks of some complete objects, and then refine them with SAM2. Please see [Florence-2+SAM2](https://github.com/IDEA-Research/Grounded-SAM-2). |
| 3. **VLPart + SAM2**: We use VLPart to generate box of object part, and then refine them with SAM2. We refer to [VLPart](https://github.com/facebookresearch/VLPart). |
| We provide our inference demo scripts in `data_curation/build_vlpart.py` and `data_curation/vlpart_sam2_tracking.py`. |
| 4. **Reasoning Instruction**: We provide two example scripts to generate reasoning instructions for the affordance segmentation task: |
| - `data_curation/prompt_generation_handal_easy_reasoning.py`: This script generates easy reasoning instructions for the HANDAL dataset. |
| - `data_curation/prompt_generation_handal_hard_reasoning.py`: This script generates hard reasoning instructions for the HANDAL dataset. |
