DATASETS.md

# Dataset Preparation

Scripts in `datasets/` follow the conventions of Detectron2 and prior open-vocabulary segmentation work. A dataset can be used by accessing [DatasetCatalog](https://detectron2.readthedocs.io/modules/data.html#detectron2.data.DatasetCatalog) for its data, or [MetadataCatalog](https://detectron2.readthedocs.io/modules/data.html#detectron2.data.MetadataCatalog) for its metadata (class names, etc). This document explains how to setup the builtin datasets so they can be used by the above APIs. [Use Custom Datasets](https://detectron2.readthedocs.io/tutorials/datasets.html) gives a deeper dive on how to use `DatasetCatalog` and `MetadataCatalog`, and how to add new datasets to them.

## Environment Variables

Create a new directory `data` to store all the datasets. Set your dataset root directory via `DETECTRON2_DATASETS` before training or evaluation:

```bash
export DETECTRON2_DATASETS=/path/to/datasets
```

## Training Dataset

### COCO (Panoptic + RefCOCOg)

Our setup follows the instructions from [X-Decoder](https://github.com/microsoft/X-Decoder/blob/main/asset/DATASET.md) and  [Mask2Fomer](https://github.com/facebookresearch/Mask2Former/tree/main/datasets).

Prepare `panoptic_train2017`, `panoptic_semseg_train2017` exactly the same as [Mask2Fomer](https://github.com/facebookresearch/Mask2Former/tree/main/datasets).

```
coco/
  annotations/
    instances_{train,val}2017.jso
    panoptic_{train,val}2017.json
  {train,val}2017/
    # image files that are mentioned in the corresponding json
  panoptic_{train,val}2017/  # png annotations
  panoptic_semseg_{train,val}2017/  # generated by the script mentioned below
```

Install panopticapi:
```bash
pip install git+https://github.com/cocodataset/panopticapi.git
```

Then, run `python datasets/prepare_coco_semantic_annos_from_panoptic_annos.py` from the MaskFormer repo, to extract semantic annotations from panoptic annotations (only used for evaluation).

Download additional annotations and put them inside `coco/annotations/`:

```bash
# coco panoptic
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/panoptic_train2017_filtrefgumdval_filtvlp.json
# refcocog valid
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/refcocog_umd_val.json
# refcocog train: download from Google Drive
https://drive.google.com/file/d/1DQhgTo7B4E-8IIh5fGOlrQVmh4x14mJL/view?usp=sharing
```

## Evaluation Datasets

### ADE20K-150

Our setup follows the instructions from [Mask2Fomer](https://github.com/facebookresearch/Mask2Former/tree/main/datasets). The scripts mentioned below can be found in the  [Mask2Fomer](https://github.com/facebookresearch/Mask2Former/tree/main/datasets) repo.

**Expected dataset structure for [ADE20k](http://sceneparsing.csail.mit.edu/):**

```
ADEChallengeData2016/
  images/
  annotations/
  objectInfo150.txt
  # download instance annotation
  annotations_instance/
  # generated by prepare_ade20k_sem_seg.py
  annotations_detectron2/
  # below are generated by prepare_ade20k_pan_seg.py
  ade20k_panoptic_{train,val}.json
  ade20k_panoptic_{train,val}/
  # below are generated by prepare_ade20k_ins_seg.py
  ade20k_instance_{train,val}.json
```

The directory `annotations_detectron2` is generated by running `python datasets/prepare_ade20k_sem_seg.py`.

Download the instance annotation from http://sceneparsing.csail.mit.edu/:

```
wget <http://sceneparsing.csail.mit.edu/data/ChallengeData2017/annotations_instance.tar>
```

Then, run `python datasets/prepare_ade20k_pan_seg.py`, to combine semantic and instance annotations for panoptic annotations.

And run `python datasets/prepare_ade20k_ins_seg.py`, to extract instance annotations in COCO format.

### ADE20K full 

Our setup follows the instructions from [OV-Seg](https://github.com/facebookresearch/ov-seg/blob/main/datasets/DATASETS.md).

Download here: https://www.kaggle.com/datasets/sssunyy/ade20k/data

**Expected dataset structure for [ADE20k-Full (ADE20K-847)](https://github.com/CSAILVision/ADE20K#download):**

```
ADE20K_2021_17_01/
  images/
  index_ade20k.pkl
  objects.txt
  # below are generated
  images_detectron2/
  annotations_detectron2/
```

The directories `images_detectron2` and `annotations_detectron2` are generated by running `python datasets/prepare_ade20k_full_sem_seg.py` (script can be found in  [OV-Seg](https://github.com/facebookresearch/ov-seg/blob/main/datasets/DATASETS.md)).

### PASCAL-Context (PC459 / PC59) and VOC 2012

Our setup follows the instructions from [APE](https://github.com/shenyunhang/APE/tree/main/datasets).

Obtain VOC2010 and VOC2012 from the [Pascal VOC website](http://host.robots.ox.ac.uk/pascal/VOC/).

**Expected dataset structure for [PC459 and PC59](https://cs.stanford.edu/~roozbeh/pascal-context/):**

```
$DETECTRON2_DATASETS/
  VOCdevkit/
    VOC2010/
      Annotations/
      ImageSets/
      JPEGImages/
      SegmentationClass/
      SegmentationObject/
      # below are from <https://www.cs.stanford.edu/~roozbeh/pascal-context/trainval.tar.gz>
      trainval/
      labels.txt
      59_labels.txt # <https://www.cs.stanford.edu/~roozbeh/pascal-context/59_labels.txt>
      pascalcontext_val.txt # <https://drive.google.com/file/d/1BCbiOKtLvozjVnlTJX51koIveUZHCcUh/view?usp=sharing>
      # below are generated
      annotations_detectron2/
        pc459_val/
        pc59_val
```

It starts with a tar file `VOCtrainval_03-May-2010.tar`. Extract the file `tar xf VOCtrainval_03-May-2010.tar`. You may want to download the 5K validation set [here](https://drive.google.com/file/d/1BCbiOKtLvozjVnlTJX51koIveUZHCcUh/view?usp=sharing).

The directory `annotations_detectron2` is generated by running (script from [APE](https://github.com/shenyunhang/APE/tree/main/datasets))

```
python datasets/prepare_pascal_context.py
```

**Expected dataset structure for [VOC](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/):**

```
$DETECTRON2_DATASETS/
  VOCdevkit/
    VOC2012/
      Annotations/
      ImageSets/
      JPEGImages/
      SegmentationClass/
      SegmentationObject/
      SegmentationClassAug/ # <https://github.com/kazuto1011/deeplab-pytorch/blob/master/data/datasets/voc12/README.md>
      # below are generated
      images_detectron2/
      annotations_detectron2/
        val/
```

It starts with a tar file `VOCtrainval_11-May-2012.tar`.

The directories `images_detectron2` and `annotations_detectron2` are generated by running  (script from [APE](https://github.com/shenyunhang/APE/tree/main/datasets))

```
python datasets/prepare_voc_sem_seg.py
```

## SUN RGB-D (SUN-37)
Follow https://github.com/chrischoy/SUN_RGBD to download and extract the dataset:
```bash
wget http://cvgl.stanford.edu/data2/sun_rgbd.tgz
tar -xzf sun_rgbd.tgz
```

## ScanNet

Use the official download script: https://kaldir.vc.in.tum.de/scannet/download-scannet.py

Convert to EfficientPS format following https://github.com/TUTvision/ScanNet-EfficientPS and run:

```bash
python tools/scannet_train_val_to_efficientps.py \
    -s /path/to/scannet_frames_25k \
    -t /path/to/ScanNet/Tasks/Benchmark/scannetv2_train.txt \
    -v /path/to/ScanNet/Tasks/Benchmark/scannetv2_val.txt \
    -o /path/to/output \
    -sc /path/to/ScanNet \
    -pn /path/to/panopticapi
```