yujielu commited on
Commit
9f2a548
·
verified ·
1 Parent(s): 260caba

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -114
README.md CHANGED
@@ -1,6 +1,6 @@
1
  <div align="center">
2
 
3
- # SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation
4
 
5
  **Yujie Lu<sup>1*</sup>, Jingwen Li<sup>2*</sup>, Sibo Ju<sup>3</sup>, Yanzhou Su<sup>4</sup>, He Yao<sup>1</sup>, Yisong Liu<sup>1</sup>, Min Zhu<sup>1&dagger;</sup>, Junlong Cheng<sup>1&dagger;</sup>
6
 
@@ -32,119 +32,6 @@ SegMoTE aims to retain the flexible prompt interface and generalization ability
32
  of SAM while providing lightweight adaptation for multimodal medical image
33
  segmentation.
34
 
35
- <p align="center">
36
- <img src="figures/Introduction.png" width="98%" alt="SegMoTE motivation and comparison with previous SAM adaptation methods">
37
- </p>
38
-
39
- ## Architecture
40
-
41
- <p align="center">
42
- <img src="figures/SegMoTE.png" width="100%" alt="Overall architecture of SegMoTE">
43
- </p>
44
-
45
- SegMoTE extends SAM with two components. First, **MoTE** injects learnable
46
- expert tokens into the mask decoder and uses token-level routing to select
47
- specialized experts for different modalities and tasks. A load-balancing
48
- objective encourages effective expert utilization. Second, **PPT** pools image
49
- features into adaptive prompt tokens for selected few-class segmentation
50
- settings, reducing dependence on manual prompts. The framework supports point,
51
- bounding-box, and text prompts while retaining an efficient inference path.
52
-
53
- ## Highlights
54
-
55
- - **Token-level expert routing:** SegMoTE activates expert tokens
56
- for modality- and task-adaptive segmentation.
57
- - **Progressive prompt tokenization:** Feature-conditioned prompt tokens
58
- support automatic segmentation for suitable binary foreground-background
59
- tasks.
60
- - **Multimodal medical segmentation:** The framework is designed for medical
61
- datasets spanning CT, MRI, dermoscopy, X-ray, and other modalities.
62
- - **SAM-compatible interaction:** Point, bounding-box, and text prompts are
63
- supported in the released implementation.
64
-
65
- ## Updates
66
-
67
- - **May 2026:** Code release preparation and inference checkpoint packaging.
68
- - **May 2026:** Deterministic MoTE routing enabled during evaluation.
69
-
70
- ## Installation
71
-
72
- Create an environment with a CUDA-enabled PyTorch installation appropriate for
73
- your hardware, then install the remaining dependencies:
74
-
75
- ```bash
76
- git clone <repository-url>
77
- cd SegMoTE
78
- pip install -r requirements.txt
79
- ```
80
-
81
- The implementation uses PyTorch, TIMM, Transformers, MONAI, OpenCV, and common
82
- scientific Python packages.
83
-
84
-
85
- ## Checkpoint
86
-
87
- Two checkpoint files are required for evaluation:
88
-
89
- ```text
90
- checkpoints/sam_b.pth
91
- checkpoints/segmote.pth
92
- ```
93
-
94
- | Checkpoint | Usage |
95
- | --- | --- |
96
- | `sam_b.pth` | Base initialization checkpoint loaded with `--sam_checkpoint` before loading SegMoTE weights. |
97
- | `segmote.pth` | SegMoTE inference checkpoint loaded with `--pretrain_path`. |
98
-
99
- Download the checkpoints from Baidu Netdisk:
100
-
101
- ```text
102
- sam_b: https://pan.baidu.com/s/1HcmqPiwpWgnYr4CMf6Y9Pg Password:eank
103
- segmote: https://pan.baidu.com/s/1tzlOv3YSU-9s6Gaw4pCF6g Password:wja3
104
- ```
105
-
106
- After downloading, place both checkpoint files in the `checkpoints/`
107
- directory.
108
-
109
- ## Evaluation
110
-
111
- Evaluate the released checkpoint on a dataset with bounding-box prompts:
112
-
113
- ```bash
114
- python test.py \
115
- --data_dir dataset \
116
- --dataset_list BTCV \
117
- --sam_checkpoint checkpoints/sam_b.pth \
118
- --pretrain_path checkpoints/segmote.pth \
119
- --prompt_mode bboxes \
120
- --output_dir outputs/BTCV
121
- ```
122
-
123
- ## Training
124
-
125
- Train SegMoTE from the base initialization checkpoint:
126
-
127
- ```bash
128
- python train.py \
129
- --data_dir dataset \
130
- --dataset_list BTCV \
131
- --sam_checkpoint checkpoints/sam_b.pth \
132
- --task_name segmote_train
133
- ```
134
-
135
- For distributed training on multiple GPUs:
136
-
137
- ```bash
138
- python train.py \
139
- --data_dir dataset \
140
- --dataset_list BTCV \
141
- --sam_checkpoint checkpoints/sam_b.pth \
142
- --task_name segmote_train \
143
- --dist \
144
- --multi_gpu \
145
- --gpu_ids 0 1 2 3 4 5 6 7
146
- ```
147
-
148
 
149
  ## Citation
150
 
 
1
  <div align="center">
2
 
3
+ <h1 align="center">SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation</h1>
4
 
5
  **Yujie Lu<sup>1*</sup>, Jingwen Li<sup>2*</sup>, Sibo Ju<sup>3</sup>, Yanzhou Su<sup>4</sup>, He Yao<sup>1</sup>, Yisong Liu<sup>1</sup>, Min Zhu<sup>1&dagger;</sup>, Junlong Cheng<sup>1&dagger;</sup>
6
 
 
32
  of SAM while providing lightweight adaptation for multimodal medical image
33
  segmentation.
34
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
35
 
36
  ## Citation
37