# High Quality Entity Segmentation (ICCV 2023 Oral)
Lu Qi, Jason Kuen, Tiancheng Shen, Jiuxiang Gu, Weidong Guo, Jiaya Jia, Zhe Lin, Ming-Hsuan Yang
This project offers an implementation of the paper, "[High-Quality Entity Segmentation](https://arxiv.org/abs/2211.05776)". This repository serves as an unofficial extension to the [Adobe EntitySeg Github](https://github.com/adobe-research/EntitySeg-Dataset), where you can directly download the EntitySeg Dataset and the source code of our proposed CropFormer. For a more comprehensive view of our results and visualizations, we invite you to explore our [project website](http://luqi.info/entityv2.github.io/).
## News
2023-09-07 The dataset, code and pretrained models are released.
2023-08-01 Our paper is accepted as ICCV2023 oral.
## Data
Please refer to the official repo [EntitySeg-Dataset](https://github.com/adobe-research/EntitySeg-Dataset) for annotation files and image URLs.
For convenience, we provide the images in several links including [Google Drive](https://drive.google.com/drive/folders/1yX2rhOroyhUCGCrmzSm7DL4BfQWvcG0v?usp=drive_link) and [Hugging Face](https://huggingface.co/datasets/qqlu1992/Adobe_EntitySeg), but we do not own the copyright of the images. It is solely your responsibility to check the original licenses of the images before using them. Any use of the images are at your own discretion and risk.
Furthermore, please refer to [the dataset description](DATA.md) on how to set up the dataset before running our code.
## Code
We offer the instructions on installation, train, evaluation and visualization for the proposed CropFormer in [the code description](CODE.md).
## Model Zoo
Overall, we also provide the pretrained segmentation models in two links including [Google Drive](https://drive.google.com/drive/folders/1uCas3I3xxqZReiXtw2GyNeLgXwkmlsQ_?usp=sharing) or [Hugging Face](https://huggingface.co/datasets/qqlu1992/Adobe_EntitySeg). We illustrate them in the following. For the class-aware segmentation tasks, we directly use the COCO-pretrained Mask2Former Model as our pretrain weights.
### Entity Segmentation
We provide several entity segmentation models. For all the training, we use the COCO-Entity pretrained models as our initialization that are provided in [Google Drive](https://drive.google.com/drive/folders/1dFEE7pefbzvxj1M4-pBOOGEwDalVagJx?usp=drive_link) or [Hugging Face](https://huggingface.co/datasets/qqlu1992/Adobe_EntitySeg/tree/main/CropFormer_model/Entity_Segmentation/COCO_Pretrained_Weights). We evaluate the model on both the overlapped-free $AP$ and no-overlapped $AP^e$ in low-resolution (L) and high-resolution images (H).
| Method |
Backbone |
Sched |
AP_L |
AP_L^e |
AP_H |
AP_H^e |
download |
| Mask2Former |
Swin-T |
3x |
- |
38.8 |
- |
40.7 |
model |
| CropFormer |
Swin-T |
3x |
- |
40.6 |
- |
43.0 |
model |
| Mask2Former |
Swin-L |
3x |
- |
44.4 |
- |
46.2 |
model |
| CropFormer |
Swin-L |
3x |
- |
45.8 |
- |
48.2 |
model |
| Mask2Former |
Hornet-L |
3x |
51.0 |
47.1 |
53.6 |
49.2 |
model |
| CropFormer |
Hornet-L |
3x |
- |
49.1 |
- |
51.5 |
model |
### Instance Segmentation
| Method |
Backbone |
Sched |
AP_H |
download |
| Mask2Former |
Swin-T |
3x |
22.7 |
model |
| Mask2Former |
Swin-L |
3x |
30.3 |
model |
### Semantic Segmentation
| Method |
Backbone |
Sched |
mIoU_H |
download |
| Mask2Former |
Swin-T |
3x |
45.2 |
model |
| Mask2Former |
Swin-L |
3x |
51.1 |
model |
### Panoptic Segmentation
| Method |
Backbone |
Sched |
PQ_H |
download |
| Mask2Former |
Swin-T |
3x |
9.8 |
model |
| Mask2Former |
Swin-L |
3x |
13.5 |
model |
## Citing Ours
Consider to cite **High Quality Entity Segmentation** if it helps your research.
```
@inproceedings{qi2022high,
title={High Quality Entity Segmentation},
author={Qi, Lu and Kuen, Jason and Shen, Tiancheng and Gu, Jiuxiang and Guo, Weidong and Jia, Jiaya and Lin, Zhe and Yang, Ming-Hsuan},
booktitle={ICCV},
year={2023}
}
```
## License
The code and models are released under the [CC BY-NC 4.0 license](https://creativecommons.org/licenses/by-nc/4.0/).
