Hugging Face's logo

KieDani
/
SegformerPlusPlus

Image Segmentation
English
Model card Files
xet
Community
KieDani commited on
Commit
e879b89
·
verified ·
1 Parent(s): 60465e5

Added some metadata

Browse files
Files changed (1) hide show
  1. README.md +99 -93
README.md CHANGED
@@ -1,93 +1,99 @@
1
- # SegFormer++
2
-
3
- Paper: [Segformer++: Efficient Token-Merging Strategies for High-Resolution Semantic Segmentation](https://arxiv.org/abs/2405.14467)
4
-
5
- ![image](docs/figures/segmentation.png)
6
-
7
- ![image](docs/figures/pose.png)
8
-
9
- ## Abstract
10
-
11
- Utilizing transformer architectures for semantic segmentation of high-resolution images is hindered by the attention's quadratic computational complexity in the number of tokens. A solution to this challenge involves decreasing the number of tokens through token merging, which has exhibited remarkable enhancements in inference speed, training efficiency, and memory utilization for image classification tasks. In this paper, we explore various token merging strategies within the framework of the SegFormer architecture and perform experiments on multiple semantic segmentation and human pose estimation datasets. Notably, without model re-training, we, for example, achieve an inference acceleration of 61% on the Cityscapes dataset while maintaining the mIoU performance. Consequently, this paper facilitates the deployment of transformer-based architectures on resource-constrained devices and in real-time applications.
12
-
13
- **Update:** It is now possible to load the model via torch.hub. See [here](docs/setup/torchhub_setup.md).
14
-
15
- **Update:** It is now possible to run the model **without OpenMMLab dependencies**, enabling users to utilize the SegFormerPlusPlus architecture without installing the full OpenMMLab framework.
16
-
17
- ## Results and Models
18
-
19
- Memory refers to the VRAM requirements during the training process.
20
-
21
- ### Inference on Cityscapes (MiT-B5)
22
-
23
- The weights of the Segformer (Original) model were used to get the inference results.
24
-
25
- | Method | mIoU | Speed-Up | config | download |
26
- |-----------------------------------|------:|---------:|--------------------------------------------------------------------------------------------|----------------------------------------------------------------|
27
- | Segformer (Original) | 82.39 | - | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-default.py) | [model](https://mediastore.rz.uni-augsburg.de/get/yzE65lzm6N/) |
28
- | Segformer++<sub>HQ</sub> (ours) | 82.31 | 1.61 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-bsm-hq.py) | [model](https://mediastore.rz.uni-augsburg.de/get/yzE65lzm6N/) |
29
- | Segformer++<sub>fast</sub> (ours) | 82.04 | 1.94 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-bsm-fast.py) | [model](https://mediastore.rz.uni-augsburg.de/get/yzE65lzm6N/) |
30
- | Segformer++<sub>2x2</sub> (ours) | 81.96 | 1.90 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-n2d-2x2.py) | [model](https://mediastore.rz.uni-augsburg.de/get/yzE65lzm6N/) |
31
- | Segformer (Downsampling) | 77.31 | 6.51 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-downsample.py) | [model](https://mediastore.rz.uni-augsburg.de/get/yzE65lzm6N/) |
32
-
33
- ### Training on Cityscapes (MiT-B5)
34
-
35
- | Method | mIoU | Speed-Up | Memory (GB) | config | download |
36
- |-----------------------------------|------:|---------:|-------------|--------------------------------------------------------------------------------------------|-----------------------------------------------------------------|
37
- | Segformer (Original) | 82.39 | - | 48.3 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-default.py) | [model](https://mediastore.rz.uni-augsburg.de/get/yzE65lzm6N/) |
38
- | Segformer++<sub>HQ</sub> (ours) | 82.19 | 1.40 | 34.0 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-bsm-hq.py) | [model](https://mediastore.rz.uni-augsburg.de/get/i8fY8uXJrV/ ) |
39
- | Segformer++<sub>fast</sub> (ours) | 81.77 | 1.55 | 30.5 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-bsm-fast.py) | [model](https://mediastore.rz.uni-augsburg.de/get/cmG974iAxt/ ) |
40
- | Segformer++<sub>2x2</sub> (ours) | 82.38 | 1.63 | 31.1 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-n2d-2x2.py) | [model](https://mediastore.rz.uni-augsburg.de/get/p0uMKbw531/) |
41
- | Segformer (Downsampling) | 79.24 | 2.95 | 10.0 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-downsample.py) | [model](https://mediastore.rz.uni-augsburg.de/get/73zkKSO21t/) |
42
-
43
- ### Training on ADE20K (640x640) (MiT-B5)
44
-
45
- | Method | mIoU | Speed-Up | Memory (GB) | config | download |
46
- |-----------------------------------|------:|---------:|------------:|---------------------------------------------------------------------------------------|----------------------------------------------------------------|
47
- | Segformer (Original) | 49.72 | - | 33.7 | [config](mmsegmentation/local_configs/ade20k/B5/segformer-ade20k640-b5-default.py) | [model](https://mediastore.rz.uni-augsburg.de/get/nKEjUHNAfK/) |
48
- | Segformer++<sub>HQ</sub> (ours) | 49.77 | 1.15 | 29.2 | [config](mmsegmentation/local_configs/ade20k/B5/segformer-ade20k640-b5-bsm-hq.py) | [model](https://mediastore.rz.uni-augsburg.de/get/Odyie8usgj/) |
49
- | Segformer++<sub>fast</sub> (ours) | 49.10 | 1.20 | 28.0 | [config](mmsegmentation/local_configs/ade20k/B5/segformer-ade20k640-b5-bsm-fast.py) | [model](https://mediastore.rz.uni-augsburg.de/get/K0IGkx4O2s/) |
50
- | Segformer++<sub>2x2</sub> (ours) | 49.35 | 1.26 | 27.2 | [config](mmsegmentation/local_configs/ade20k/B5/segformer-ade20k640-b5-n2d-2x2.py) | [model](https://mediastore.rz.uni-augsburg.de/get/w5_Pxx4Q5C/) |
51
- | Segformer (Downsampling) | 46.71 | 1.89 | 12.4 | [config](mmsegmentation/local_configs/ade20k/B5/segformer-ade20k640-b5-downsample.py) | [model](https://mediastore.rz.uni-augsburg.de/get/dFVvZQL6iL/) |
52
-
53
- ### Training on JBD
54
-
55
- | Method | PCK@0.1 | PCK@0.05 | Speed-Up | Memory (GB) | config | download |
56
- |-----------------------------------|--------:|---------:|---------:|------------:|---------------------------------------------------------------------|----------------------------------------------------------------|
57
- | Segformer (Original) | 95.20 | 90.65 | - | 40.0 | [config](mmpose/local_configs/jbd/B5/segformer-jump-b5-default.py) | [model](https://mediastore.rz.uni-augsburg.de/get/psolrWXLLp/) |
58
- | Segformer++<sub>HQ</sub> (ours) | 95.18 | 90.51 | 1.19 | 36.0 | [config](mmpose/local_configs/jbd/B5/segformer-jump-b5-bsm-hq.py) | [model](https://mediastore.rz.uni-augsburg.de/get/jx1eyecMLF/) |
59
- | Segformer++<sub>fast</sub> (ours) | 94.58 | 89.87 | 1.25 | 34.6 | [config](mmpose/local_configs/jbd/B5/segformer-jump-b5-bsm-fast.py) | [model](https://mediastore.rz.uni-augsburg.de/get/K0IGkx4O2s/) |
60
- | Segformer++<sub>2x2</sub> (ours) | 95.17 | 90.16 | 1.27 | 33.4 | [config](mmpose/local_configs/jbd/B5/segformer-jump-b5-n2d-2x2.py) | [model](https://mediastore.rz.uni-augsburg.de/get/HumKbSB1vI/) |
61
-
62
- ### Training on MS COCO
63
-
64
- | Method | PCK@0.1 | PCK@0.05 | Speed-Up | Memory (GB) | config | download |
65
- |-----------------------------------|--------:|---------:|---------:|------------:|----------------------------------------------------------------------|----------------------------------------------------------------|
66
- | Segformer (Original) | 95.16 | 87.61 | - | 13.5 | [config](mmpose/local_configs/coco/B5/segformer-coco-b5-default.py) | [model](https://mediastore.rz.uni-augsburg.de/get/ZOgj2NmQLy/) |
67
- | Segformer++<sub>HQ</sub> (ours) | 94.97 | 87.35 | 0.97 | 13.1 | [config](mmpose/local_configs/coco/B5/segformer-coco-b5-bsm-hq.py) | [model](https://mediastore.rz.uni-augsburg.de/get/oAH5IlPxG8/) |
68
- | Segformer++<sub>fast</sub> (ours) | 95.02 | 87.37 | 0.99 | 12.9 | [config](mmpose/local_configs/coco/B5/segformer-coco-b5-bsm-fast.py) | [model](https://mediastore.rz.uni-augsburg.de/get/3E2mMNLAAn/) |
69
- | Segformer++<sub>2x2</sub> (ours) | 94.98 | 87.36 | 1.24 | 12.3 | [config](mmpose/local_configs/coco/B5/segformer-coco-b5-n2d-2x2.py) | [model](https://mediastore.rz.uni-augsburg.de/get/rzlgKC5XLc/) |
70
-
71
- ## Usage
72
-
73
- Easy usage:
74
- - [Use the SegFormer++ via TorchHub](docs/setup/torchhub_setup.md)
75
- - [Use the SegFormer++ without OpenMMLab](docs/setup/noopenmmlab_setup.md)
76
-
77
- Legacy Variants. To use our models for semantic segmentation or 2D human pose estimation, please follow the installation instructions for MMSegmentation and MMPose respectively, which can be found in the documentation of the respective repositories.
78
- - [Use the SegFormer++ with MMSegmentation](docs/setup/mmseg_setup.md)
79
- - [Use the SegFormer++ with MMPose](docs/setup/mmpose_setup.md)
80
- - [Use the SegFormer++ without MMSegmentation/MMPose](docs/setup/mmeng_setup.md)
81
-
82
- Explanation of the different token merging strategies:
83
- - [Token Merging Settings](docs/run/token_merging.md)
84
-
85
- ## Citation
86
- ```bibtex
87
- @article{kienzle2024segformer++,
88
- title={Segformer++: Efficient Token-Merging Strategies for High-Resolution Semantic Segmentation},
89
- author={Kienzle, Daniel and Kantonis, Marco and Sch{\"o}n, Robin and Lienhart, Rainer},
90
- journal={IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR)},
91
- year={2024}
92
- }
93
- ```
 
 
 
 
 
 
 
1
+ ---
2
+ license: gpl-3.0
3
+ language:
4
+ - en
5
+ pipeline_tag: image-segmentation
6
+ ---
7
+ # SegFormer++
8
+
9
+ Paper: [Segformer++: Efficient Token-Merging Strategies for High-Resolution Semantic Segmentation](https://arxiv.org/abs/2405.14467)
10
+
11
+ ![image](docs/figures/segmentation.png)
12
+
13
+ ![image](docs/figures/pose.png)
14
+
15
+ ## Abstract
16
+
17
+ Utilizing transformer architectures for semantic segmentation of high-resolution images is hindered by the attention's quadratic computational complexity in the number of tokens. A solution to this challenge involves decreasing the number of tokens through token merging, which has exhibited remarkable enhancements in inference speed, training efficiency, and memory utilization for image classification tasks. In this paper, we explore various token merging strategies within the framework of the SegFormer architecture and perform experiments on multiple semantic segmentation and human pose estimation datasets. Notably, without model re-training, we, for example, achieve an inference acceleration of 61% on the Cityscapes dataset while maintaining the mIoU performance. Consequently, this paper facilitates the deployment of transformer-based architectures on resource-constrained devices and in real-time applications.
18
+
19
+ **Update:** It is now possible to load the model via torch.hub. See [here](docs/setup/torchhub_setup.md).
20
+
21
+ **Update:** It is now possible to run the model **without OpenMMLab dependencies**, enabling users to utilize the SegFormerPlusPlus architecture without installing the full OpenMMLab framework.
22
+
23
+ ## Results and Models
24
+
25
+ Memory refers to the VRAM requirements during the training process.
26
+
27
+ ### Inference on Cityscapes (MiT-B5)
28
+
29
+ The weights of the Segformer (Original) model were used to get the inference results.
30
+
31
+ | Method | mIoU | Speed-Up | config | download |
32
+ |-----------------------------------|------:|---------:|--------------------------------------------------------------------------------------------|----------------------------------------------------------------|
33
+ | Segformer (Original) | 82.39 | - | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-default.py) | [model](https://mediastore.rz.uni-augsburg.de/get/yzE65lzm6N/) |
34
+ | Segformer++<sub>HQ</sub> (ours) | 82.31 | 1.61 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-bsm-hq.py) | [model](https://mediastore.rz.uni-augsburg.de/get/yzE65lzm6N/) |
35
+ | Segformer++<sub>fast</sub> (ours) | 82.04 | 1.94 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-bsm-fast.py) | [model](https://mediastore.rz.uni-augsburg.de/get/yzE65lzm6N/) |
36
+ | Segformer++<sub>2x2</sub> (ours) | 81.96 | 1.90 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-n2d-2x2.py) | [model](https://mediastore.rz.uni-augsburg.de/get/yzE65lzm6N/) |
37
+ | Segformer (Downsampling) | 77.31 | 6.51 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-downsample.py) | [model](https://mediastore.rz.uni-augsburg.de/get/yzE65lzm6N/) |
38
+
39
+ ### Training on Cityscapes (MiT-B5)
40
+
41
+ | Method | mIoU | Speed-Up | Memory (GB) | config | download |
42
+ |-----------------------------------|------:|---------:|-------------|--------------------------------------------------------------------------------------------|-----------------------------------------------------------------|
43
+ | Segformer (Original) | 82.39 | - | 48.3 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-default.py) | [model](https://mediastore.rz.uni-augsburg.de/get/yzE65lzm6N/) |
44
+ | Segformer++<sub>HQ</sub> (ours) | 82.19 | 1.40 | 34.0 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-bsm-hq.py) | [model](https://mediastore.rz.uni-augsburg.de/get/i8fY8uXJrV/ ) |
45
+ | Segformer++<sub>fast</sub> (ours) | 81.77 | 1.55 | 30.5 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-bsm-fast.py) | [model](https://mediastore.rz.uni-augsburg.de/get/cmG974iAxt/ ) |
46
+ | Segformer++<sub>2x2</sub> (ours) | 82.38 | 1.63 | 31.1 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-n2d-2x2.py) | [model](https://mediastore.rz.uni-augsburg.de/get/p0uMKbw531/) |
47
+ | Segformer (Downsampling) | 79.24 | 2.95 | 10.0 | [config](mmsegmentation/local_configs/cityscapes/B5/segformer-cityscapes-b5-downsample.py) | [model](https://mediastore.rz.uni-augsburg.de/get/73zkKSO21t/) |
48
+
49
+ ### Training on ADE20K (640x640) (MiT-B5)
50
+
51
+ | Method | mIoU | Speed-Up | Memory (GB) | config | download |
52
+ |-----------------------------------|------:|---------:|------------:|---------------------------------------------------------------------------------------|----------------------------------------------------------------|
53
+ | Segformer (Original) | 49.72 | - | 33.7 | [config](mmsegmentation/local_configs/ade20k/B5/segformer-ade20k640-b5-default.py) | [model](https://mediastore.rz.uni-augsburg.de/get/nKEjUHNAfK/) |
54
+ | Segformer++<sub>HQ</sub> (ours) | 49.77 | 1.15 | 29.2 | [config](mmsegmentation/local_configs/ade20k/B5/segformer-ade20k640-b5-bsm-hq.py) | [model](https://mediastore.rz.uni-augsburg.de/get/Odyie8usgj/) |
55
+ | Segformer++<sub>fast</sub> (ours) | 49.10 | 1.20 | 28.0 | [config](mmsegmentation/local_configs/ade20k/B5/segformer-ade20k640-b5-bsm-fast.py) | [model](https://mediastore.rz.uni-augsburg.de/get/K0IGkx4O2s/) |
56
+ | Segformer++<sub>2x2</sub> (ours) | 49.35 | 1.26 | 27.2 | [config](mmsegmentation/local_configs/ade20k/B5/segformer-ade20k640-b5-n2d-2x2.py) | [model](https://mediastore.rz.uni-augsburg.de/get/w5_Pxx4Q5C/) |
57
+ | Segformer (Downsampling) | 46.71 | 1.89 | 12.4 | [config](mmsegmentation/local_configs/ade20k/B5/segformer-ade20k640-b5-downsample.py) | [model](https://mediastore.rz.uni-augsburg.de/get/dFVvZQL6iL/) |
58
+
59
+ ### Training on JBD
60
+
61
+ | Method | PCK@0.1 | PCK@0.05 | Speed-Up | Memory (GB) | config | download |
62
+ |-----------------------------------|--------:|---------:|---------:|------------:|---------------------------------------------------------------------|----------------------------------------------------------------|
63
+ | Segformer (Original) | 95.20 | 90.65 | - | 40.0 | [config](mmpose/local_configs/jbd/B5/segformer-jump-b5-default.py) | [model](https://mediastore.rz.uni-augsburg.de/get/psolrWXLLp/) |
64
+ | Segformer++<sub>HQ</sub> (ours) | 95.18 | 90.51 | 1.19 | 36.0 | [config](mmpose/local_configs/jbd/B5/segformer-jump-b5-bsm-hq.py) | [model](https://mediastore.rz.uni-augsburg.de/get/jx1eyecMLF/) |
65
+ | Segformer++<sub>fast</sub> (ours) | 94.58 | 89.87 | 1.25 | 34.6 | [config](mmpose/local_configs/jbd/B5/segformer-jump-b5-bsm-fast.py) | [model](https://mediastore.rz.uni-augsburg.de/get/K0IGkx4O2s/) |
66
+ | Segformer++<sub>2x2</sub> (ours) | 95.17 | 90.16 | 1.27 | 33.4 | [config](mmpose/local_configs/jbd/B5/segformer-jump-b5-n2d-2x2.py) | [model](https://mediastore.rz.uni-augsburg.de/get/HumKbSB1vI/) |
67
+
68
+ ### Training on MS COCO
69
+
70
+ | Method | PCK@0.1 | PCK@0.05 | Speed-Up | Memory (GB) | config | download |
71
+ |-----------------------------------|--------:|---------:|---------:|------------:|----------------------------------------------------------------------|----------------------------------------------------------------|
72
+ | Segformer (Original) | 95.16 | 87.61 | - | 13.5 | [config](mmpose/local_configs/coco/B5/segformer-coco-b5-default.py) | [model](https://mediastore.rz.uni-augsburg.de/get/ZOgj2NmQLy/) |
73
+ | Segformer++<sub>HQ</sub> (ours) | 94.97 | 87.35 | 0.97 | 13.1 | [config](mmpose/local_configs/coco/B5/segformer-coco-b5-bsm-hq.py) | [model](https://mediastore.rz.uni-augsburg.de/get/oAH5IlPxG8/) |
74
+ | Segformer++<sub>fast</sub> (ours) | 95.02 | 87.37 | 0.99 | 12.9 | [config](mmpose/local_configs/coco/B5/segformer-coco-b5-bsm-fast.py) | [model](https://mediastore.rz.uni-augsburg.de/get/3E2mMNLAAn/) |
75
+ | Segformer++<sub>2x2</sub> (ours) | 94.98 | 87.36 | 1.24 | 12.3 | [config](mmpose/local_configs/coco/B5/segformer-coco-b5-n2d-2x2.py) | [model](https://mediastore.rz.uni-augsburg.de/get/rzlgKC5XLc/) |
76
+
77
+ ## Usage
78
+
79
+ Easy usage:
80
+ - [Use the SegFormer++ via TorchHub](docs/setup/torchhub_setup.md)
81
+ - [Use the SegFormer++ without OpenMMLab](docs/setup/noopenmmlab_setup.md)
82
+
83
+ Legacy Variants. To use our models for semantic segmentation or 2D human pose estimation, please follow the installation instructions for MMSegmentation and MMPose respectively, which can be found in the documentation of the respective repositories.
84
+ - [Use the SegFormer++ with MMSegmentation](docs/setup/mmseg_setup.md)
85
+ - [Use the SegFormer++ with MMPose](docs/setup/mmpose_setup.md)
86
+ - [Use the SegFormer++ without MMSegmentation/MMPose](docs/setup/mmeng_setup.md)
87
+
88
+ Explanation of the different token merging strategies:
89
+ - [Token Merging Settings](docs/run/token_merging.md)
90
+
91
+ ## Citation
92
+ ```bibtex
93
+ @article{kienzle2024segformer++,
94
+ title={Segformer++: Efficient Token-Merging Strategies for High-Resolution Semantic Segmentation},
95
+ author={Kienzle, Daniel and Kantonis, Marco and Sch{\"o}n, Robin and Lienhart, Rainer},
96
+ journal={IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR)},
97
+ year={2024}
98
+ }
99
+ ```