g3doc/setup/cityscapes.md

# Run DeepLab2 on Cityscapes dataset

This page walks through the steps required to generate
[Cityscapes](https://www.cityscapes-dataset.com/) data for DeepLab2. DeepLab2
uses sharded TFRecords for efficient processing of the data.

## Prework

Before running any Deeplab2 scripts, the user should 1. register on the
Cityscapes dataset [website](https://www.cityscapes-dataset.com) to download the
dataset (gtFine_trainvaltest.zip and leftImg8bit_trainvaltest.zip). 2. install
cityscapesscripts via pip: `bash # This will install the cityscapes scripts and
its stand-alone tools. pip install cityscapesscripts`

1.  run the tools provided by Cityscapes to generate the training groundtruth.
    See sample commandlines below:

```bash
  # Set CITYSCAPES_DATASET to your dataset root.

  # Create train ID label images.
  CITYSCAPES_DATASET='.' csCreateTrainIdLabelImgs

  # To generate panoptic groundtruth, run the following command.
  CITYSCAPES_DATASET='.' csCreatePanopticImgs --use-train-id

  # [Optional] Generate panoptic groundtruth with EvalId to match evaluation
  # on the server. This step is not required for generating TFRecords.
  CITYSCAPES_DATASET='.' csCreatePanopticImgs
```

After running above commandlines, the expected directory structure should be as
follows:

```
cityscapes
+-- gtFine
|   |
|   +-- train
|   |   |
|   |   +-- aachen
|   |       |
|   |       +-- *_color.png
|   |       +-- *_instanceIds.png
|   |       +-- *_labelIds.png
|   |       +-- *_polygons.json
|   |       +-- *_labelTrainIds.png
|   |   ...
|   +-- val
|   +-- test
|   +-- cityscapes_panoptic_{train|val|test}_trainId.json
|   +-- cityscapes_panoptic_{train|val|test}_trainId
|   |   |
|   |   +-- *_panoptic.png
|   +-- cityscapes_panoptic_{train|val|test}.json
|   +-- cityscapes_panoptic_{train|val|test}
|       |
|       +-- *_panoptic.png
|
+-- leftImg8bit
     |
     +-- train
     +-- val
     +-- test
```

## Convert prepared dataset to TFRecord

Note: the rest of this doc and released DeepLab2 models use `TrainId` instead of
`EvalId` (which is used on the evaluation server). For evaluation on the server,
you would need to convert the predicted labels to `EvalId` .

Use the following commandline to generate cityscapes TFRecords:

```bash
# Assuming we are under the folder where deeplab2 is cloned to:

# For generating data for semantic segmentation task only
python deeplab2/data/build_cityscapes_data.py \
  --cityscapes_root=${PATH_TO_CITYSCAPES_ROOT} \
  --output_dir=${OUTPUT_PATH_FOR_SEMANTIC} \
  --create_panoptic_data=false

# For generating data for panoptic segmentation task
python deeplab2/data/build_cityscapes_data.py \
  --cityscapes_root=${PATH_TO_CITYSCAPES_ROOT} \
  --output_dir=${OUTPUT_PATH_FOR_PANOPTIC}
```

Commandline above will output three sharded tfrecord files:
`{train|val|test}@10.tfrecord`. In the tfrecords, for `train` and `val` set, it
contains the RGB image pixels as well as corresponding annotations. For `test`
set, it contains RGB images only. These files will be used as the input for the
model training and evaluation.

### TFExample proto format for cityscapes

The Example proto contains the following fields:

*   `image/encoded`: encoded image content.
*   `image/filename`: image filename.
*   `image/format`: image file format.
*   `image/height`: image height.
*   `image/width`: image width.
*   `image/channels`: image channels.
*   `image/segmentation/class/encoded`: encoded segmentation content.
*   `image/segmentation/class/format`: segmentation encoding format.

For semantic segmentation (`--create_panoptic_data=false`), the encoded
segmentation map will be the same as PNG file created by
`createTrainIdLabelImgs.py`.

For panoptic segmentation, the encoded segmentation map will be the raw bytes of
a int32 panoptic map, where each pixel is assigned to a panoptic ID. Unlike the
ID used in Cityscapes script (`json2instanceImg.py`), this panoptic ID is
computed by:

```
  panoptic ID = semantic ID * label divisor + instance ID
```

where semantic ID will be:

*   ignore label (255) for pixels not belonging to any segment
*   for segments associated with `iscrowd` label:
    *   (default): ignore label (255)
    *   (if set `--treat_crowd_as_ignore=false` while running
        `build_cityscapes_data.py`): `category_id` (use TrainId)
*   `category_id` (use TrainId) for other segments

The instance ID will be 0 for pixels belonging to

*   `stuff` class
*   `thing` class with `iscrowd` label
*   pixels with ignore label

and `[1, label divisor)` otherwise.