# Dataset Preparation Scripts in `datasets/` follow the conventions of Detectron2 and prior open-vocabulary segmentation work. A dataset can be used by accessing [DatasetCatalog](https://detectron2.readthedocs.io/modules/data.html#detectron2.data.DatasetCatalog) for its data, or [MetadataCatalog](https://detectron2.readthedocs.io/modules/data.html#detectron2.data.MetadataCatalog) for its metadata (class names, etc). This document explains how to setup the builtin datasets so they can be used by the above APIs. [Use Custom Datasets](https://detectron2.readthedocs.io/tutorials/datasets.html) gives a deeper dive on how to use `DatasetCatalog` and `MetadataCatalog`, and how to add new datasets to them. ## Environment Variables Create a new directory `data` to store all the datasets. Set your dataset root directory via `DETECTRON2_DATASETS` before training or evaluation: ```bash export DETECTRON2_DATASETS=/path/to/datasets ``` ## Training Dataset ### COCO (Panoptic + RefCOCOg) Our setup follows the instructions from [X-Decoder](https://github.com/microsoft/X-Decoder/blob/main/asset/DATASET.md) and [Mask2Fomer](https://github.com/facebookresearch/Mask2Former/tree/main/datasets). Prepare `panoptic_train2017`, `panoptic_semseg_train2017` exactly the same as [Mask2Fomer](https://github.com/facebookresearch/Mask2Former/tree/main/datasets). ``` coco/ annotations/ instances_{train,val}2017.jso panoptic_{train,val}2017.json {train,val}2017/ # image files that are mentioned in the corresponding json panoptic_{train,val}2017/ # png annotations panoptic_semseg_{train,val}2017/ # generated by the script mentioned below ``` Install panopticapi: ```bash pip install git+https://github.com/cocodataset/panopticapi.git ``` Then, run `python datasets/prepare_coco_semantic_annos_from_panoptic_annos.py` from the MaskFormer repo, to extract semantic annotations from panoptic annotations (only used for evaluation). Download additional annotations and put them inside `coco/annotations/`: ```bash # coco panoptic wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/panoptic_train2017_filtrefgumdval_filtvlp.json # refcocog valid wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/refcocog_umd_val.json # refcocog train: download from Google Drive https://drive.google.com/file/d/1DQhgTo7B4E-8IIh5fGOlrQVmh4x14mJL/view?usp=sharing ``` ## Evaluation Datasets ### ADE20K-150 Our setup follows the instructions from [Mask2Fomer](https://github.com/facebookresearch/Mask2Former/tree/main/datasets). The scripts mentioned below can be found in the [Mask2Fomer](https://github.com/facebookresearch/Mask2Former/tree/main/datasets) repo. **Expected dataset structure for [ADE20k](http://sceneparsing.csail.mit.edu/):** ``` ADEChallengeData2016/ images/ annotations/ objectInfo150.txt # download instance annotation annotations_instance/ # generated by prepare_ade20k_sem_seg.py annotations_detectron2/ # below are generated by prepare_ade20k_pan_seg.py ade20k_panoptic_{train,val}.json ade20k_panoptic_{train,val}/ # below are generated by prepare_ade20k_ins_seg.py ade20k_instance_{train,val}.json ``` The directory `annotations_detectron2` is generated by running `python datasets/prepare_ade20k_sem_seg.py`. Download the instance annotation from http://sceneparsing.csail.mit.edu/: ``` wget ``` Then, run `python datasets/prepare_ade20k_pan_seg.py`, to combine semantic and instance annotations for panoptic annotations. And run `python datasets/prepare_ade20k_ins_seg.py`, to extract instance annotations in COCO format. ### ADE20K full Our setup follows the instructions from [OV-Seg](https://github.com/facebookresearch/ov-seg/blob/main/datasets/DATASETS.md). Download here: https://www.kaggle.com/datasets/sssunyy/ade20k/data **Expected dataset structure for [ADE20k-Full (ADE20K-847)](https://github.com/CSAILVision/ADE20K#download):** ``` ADE20K_2021_17_01/ images/ index_ade20k.pkl objects.txt # below are generated images_detectron2/ annotations_detectron2/ ``` The directories `images_detectron2` and `annotations_detectron2` are generated by running `python datasets/prepare_ade20k_full_sem_seg.py` (script can be found in [OV-Seg](https://github.com/facebookresearch/ov-seg/blob/main/datasets/DATASETS.md)). ### PASCAL-Context (PC459 / PC59) and VOC 2012 Our setup follows the instructions from [APE](https://github.com/shenyunhang/APE/tree/main/datasets). Obtain VOC2010 and VOC2012 from the [Pascal VOC website](http://host.robots.ox.ac.uk/pascal/VOC/). **Expected dataset structure for [PC459 and PC59](https://cs.stanford.edu/~roozbeh/pascal-context/):** ``` $DETECTRON2_DATASETS/ VOCdevkit/ VOC2010/ Annotations/ ImageSets/ JPEGImages/ SegmentationClass/ SegmentationObject/ # below are from trainval/ labels.txt 59_labels.txt # pascalcontext_val.txt # # below are generated annotations_detectron2/ pc459_val/ pc59_val ``` It starts with a tar file `VOCtrainval_03-May-2010.tar`. Extract the file `tar xf VOCtrainval_03-May-2010.tar`. You may want to download the 5K validation set [here](https://drive.google.com/file/d/1BCbiOKtLvozjVnlTJX51koIveUZHCcUh/view?usp=sharing). The directory `annotations_detectron2` is generated by running (script from [APE](https://github.com/shenyunhang/APE/tree/main/datasets)) ``` python datasets/prepare_pascal_context.py ``` **Expected dataset structure for [VOC](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/):** ``` $DETECTRON2_DATASETS/ VOCdevkit/ VOC2012/ Annotations/ ImageSets/ JPEGImages/ SegmentationClass/ SegmentationObject/ SegmentationClassAug/ # # below are generated images_detectron2/ annotations_detectron2/ val/ ``` It starts with a tar file `VOCtrainval_11-May-2012.tar`. The directories `images_detectron2` and `annotations_detectron2` are generated by running (script from [APE](https://github.com/shenyunhang/APE/tree/main/datasets)) ``` python datasets/prepare_voc_sem_seg.py ``` ## SUN RGB-D (SUN-37) Follow https://github.com/chrischoy/SUN_RGBD to download and extract the dataset: ```bash wget http://cvgl.stanford.edu/data2/sun_rgbd.tgz tar -xzf sun_rgbd.tgz ``` ## ScanNet Use the official download script: https://kaldir.vc.in.tum.de/scannet/download-scannet.py Convert to EfficientPS format following https://github.com/TUTvision/ScanNet-EfficientPS and run: ```bash python tools/scannet_train_val_to_efficientps.py \ -s /path/to/scannet_frames_25k \ -t /path/to/ScanNet/Tasks/Benchmark/scannetv2_train.txt \ -v /path/to/ScanNet/Tasks/Benchmark/scannetv2_val.txt \ -o /path/to/output \ -sc /path/to/ScanNet \ -pn /path/to/panopticapi ```