Update README.md
Browse files
README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
<div align="center">
|
| 2 |
|
| 3 |
-
|
| 4 |
|
| 5 |
**Yujie Lu<sup>1*</sup>, Jingwen Li<sup>2*</sup>, Sibo Ju<sup>3</sup>, Yanzhou Su<sup>4</sup>, He Yao<sup>1</sup>, Yisong Liu<sup>1</sup>, Min Zhu<sup>1†</sup>, Junlong Cheng<sup>1†</sup>
|
| 6 |
|
|
@@ -32,119 +32,6 @@ SegMoTE aims to retain the flexible prompt interface and generalization ability
|
|
| 32 |
of SAM while providing lightweight adaptation for multimodal medical image
|
| 33 |
segmentation.
|
| 34 |
|
| 35 |
-
<p align="center">
|
| 36 |
-
<img src="figures/Introduction.png" width="98%" alt="SegMoTE motivation and comparison with previous SAM adaptation methods">
|
| 37 |
-
</p>
|
| 38 |
-
|
| 39 |
-
## Architecture
|
| 40 |
-
|
| 41 |
-
<p align="center">
|
| 42 |
-
<img src="figures/SegMoTE.png" width="100%" alt="Overall architecture of SegMoTE">
|
| 43 |
-
</p>
|
| 44 |
-
|
| 45 |
-
SegMoTE extends SAM with two components. First, **MoTE** injects learnable
|
| 46 |
-
expert tokens into the mask decoder and uses token-level routing to select
|
| 47 |
-
specialized experts for different modalities and tasks. A load-balancing
|
| 48 |
-
objective encourages effective expert utilization. Second, **PPT** pools image
|
| 49 |
-
features into adaptive prompt tokens for selected few-class segmentation
|
| 50 |
-
settings, reducing dependence on manual prompts. The framework supports point,
|
| 51 |
-
bounding-box, and text prompts while retaining an efficient inference path.
|
| 52 |
-
|
| 53 |
-
## Highlights
|
| 54 |
-
|
| 55 |
-
- **Token-level expert routing:** SegMoTE activates expert tokens
|
| 56 |
-
for modality- and task-adaptive segmentation.
|
| 57 |
-
- **Progressive prompt tokenization:** Feature-conditioned prompt tokens
|
| 58 |
-
support automatic segmentation for suitable binary foreground-background
|
| 59 |
-
tasks.
|
| 60 |
-
- **Multimodal medical segmentation:** The framework is designed for medical
|
| 61 |
-
datasets spanning CT, MRI, dermoscopy, X-ray, and other modalities.
|
| 62 |
-
- **SAM-compatible interaction:** Point, bounding-box, and text prompts are
|
| 63 |
-
supported in the released implementation.
|
| 64 |
-
|
| 65 |
-
## Updates
|
| 66 |
-
|
| 67 |
-
- **May 2026:** Code release preparation and inference checkpoint packaging.
|
| 68 |
-
- **May 2026:** Deterministic MoTE routing enabled during evaluation.
|
| 69 |
-
|
| 70 |
-
## Installation
|
| 71 |
-
|
| 72 |
-
Create an environment with a CUDA-enabled PyTorch installation appropriate for
|
| 73 |
-
your hardware, then install the remaining dependencies:
|
| 74 |
-
|
| 75 |
-
```bash
|
| 76 |
-
git clone <repository-url>
|
| 77 |
-
cd SegMoTE
|
| 78 |
-
pip install -r requirements.txt
|
| 79 |
-
```
|
| 80 |
-
|
| 81 |
-
The implementation uses PyTorch, TIMM, Transformers, MONAI, OpenCV, and common
|
| 82 |
-
scientific Python packages.
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
## Checkpoint
|
| 86 |
-
|
| 87 |
-
Two checkpoint files are required for evaluation:
|
| 88 |
-
|
| 89 |
-
```text
|
| 90 |
-
checkpoints/sam_b.pth
|
| 91 |
-
checkpoints/segmote.pth
|
| 92 |
-
```
|
| 93 |
-
|
| 94 |
-
| Checkpoint | Usage |
|
| 95 |
-
| --- | --- |
|
| 96 |
-
| `sam_b.pth` | Base initialization checkpoint loaded with `--sam_checkpoint` before loading SegMoTE weights. |
|
| 97 |
-
| `segmote.pth` | SegMoTE inference checkpoint loaded with `--pretrain_path`. |
|
| 98 |
-
|
| 99 |
-
Download the checkpoints from Baidu Netdisk:
|
| 100 |
-
|
| 101 |
-
```text
|
| 102 |
-
sam_b: https://pan.baidu.com/s/1HcmqPiwpWgnYr4CMf6Y9Pg Password:eank
|
| 103 |
-
segmote: https://pan.baidu.com/s/1tzlOv3YSU-9s6Gaw4pCF6g Password:wja3
|
| 104 |
-
```
|
| 105 |
-
|
| 106 |
-
After downloading, place both checkpoint files in the `checkpoints/`
|
| 107 |
-
directory.
|
| 108 |
-
|
| 109 |
-
## Evaluation
|
| 110 |
-
|
| 111 |
-
Evaluate the released checkpoint on a dataset with bounding-box prompts:
|
| 112 |
-
|
| 113 |
-
```bash
|
| 114 |
-
python test.py \
|
| 115 |
-
--data_dir dataset \
|
| 116 |
-
--dataset_list BTCV \
|
| 117 |
-
--sam_checkpoint checkpoints/sam_b.pth \
|
| 118 |
-
--pretrain_path checkpoints/segmote.pth \
|
| 119 |
-
--prompt_mode bboxes \
|
| 120 |
-
--output_dir outputs/BTCV
|
| 121 |
-
```
|
| 122 |
-
|
| 123 |
-
## Training
|
| 124 |
-
|
| 125 |
-
Train SegMoTE from the base initialization checkpoint:
|
| 126 |
-
|
| 127 |
-
```bash
|
| 128 |
-
python train.py \
|
| 129 |
-
--data_dir dataset \
|
| 130 |
-
--dataset_list BTCV \
|
| 131 |
-
--sam_checkpoint checkpoints/sam_b.pth \
|
| 132 |
-
--task_name segmote_train
|
| 133 |
-
```
|
| 134 |
-
|
| 135 |
-
For distributed training on multiple GPUs:
|
| 136 |
-
|
| 137 |
-
```bash
|
| 138 |
-
python train.py \
|
| 139 |
-
--data_dir dataset \
|
| 140 |
-
--dataset_list BTCV \
|
| 141 |
-
--sam_checkpoint checkpoints/sam_b.pth \
|
| 142 |
-
--task_name segmote_train \
|
| 143 |
-
--dist \
|
| 144 |
-
--multi_gpu \
|
| 145 |
-
--gpu_ids 0 1 2 3 4 5 6 7
|
| 146 |
-
```
|
| 147 |
-
|
| 148 |
|
| 149 |
## Citation
|
| 150 |
|
|
|
|
| 1 |
<div align="center">
|
| 2 |
|
| 3 |
+
<h1 align="center">SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation</h1>
|
| 4 |
|
| 5 |
**Yujie Lu<sup>1*</sup>, Jingwen Li<sup>2*</sup>, Sibo Ju<sup>3</sup>, Yanzhou Su<sup>4</sup>, He Yao<sup>1</sup>, Yisong Liu<sup>1</sup>, Min Zhu<sup>1†</sup>, Junlong Cheng<sup>1†</sup>
|
| 6 |
|
|
|
|
| 32 |
of SAM while providing lightweight adaptation for multimodal medical image
|
| 33 |
segmentation.
|
| 34 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
|
| 36 |
## Citation
|
| 37 |
|