File size: 11,014 Bytes
86285ed dec55d5 86285ed | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 | # CD-DCU-Net
PyTorch code and checkpoints for the CD-DCU-Net experiments. The repository contains two parts:
- CD-DCU-Net with Concrete Dropout for uncertainty estimation;
- CD-DCU-Net and four backbone comparison networks without Concrete Dropout.
## Contents
The public repository is arranged as follows:
```text
.
|-- CD-DCU-Net(Incorporate CD-module)/
| |-- CD-DCU-Net.py
| |-- dataloader.py
| |-- train_with_α.py
| |-- train_without_α.py
| |-- predict.py
| |-- porposed.pth
| `-- proposed(without_α).pth
|
|-- dataset_upload/
| |-- input/
| | |-- 1/
| | |-- 2/
| | `-- ...
| `-- output/
| |-- 1/
| |-- 2/
| `-- ...
|
`-- Different networks compare(without CD-model)/
|-- dataloader.py
|-- train.py
|-- predict.py
|-- CD-DCU-Net/
| |-- CD-DCU-Net.py
| |-- proposed_initial.pth
| `-- kaiming_initial.pth
|-- DFPP-Net/
| |-- DFPP-Net.py
| `-- DFPP-Net.pth
|-- DR-U-Net/
| |-- DR-U-Net.py
| `-- DR-U-Net.pth
|-- ISP-Net/
| |-- ISP-Net.py
| `-- ISP-Net.pth
`-- TS-U-Net/
|-- TS-U-Net.py
`-- TS-U-Net.pth
```
## Concrete Dropout Experiments
`CD-DCU-Net(Incorporate CD-module)/CD-DCU-Net.py` adds Concrete Dropout after each encoder and decoder stage and at the bottleneck. Its output is
```python
{
"M_mean": M_mean,
"M_var": M_var,
"D_mean": D_mean,
"D_var": D_var,
}
```
The two training scripts differ in whether the temperature constant is applied to the log-variance term of the NLL loss:
```text
train_with_α.py
10e-4 * log(var + eps) + (target - mean)^2 / (var + eps)
train_without_α.py
log(var + eps) + (target - mean)^2 / (var + eps)
```
The temperature constant is `10e-4` and is applied to `log(var + eps)`. It is not the `BNNLoss(alpha=...)` argument, which is present in both scripts and scales the full prediction loss.
`10e-4` is the literal used by the code and evaluates to `1e-3` in Python. It is left unchanged in this repository.
The corresponding checkpoints are:
| Training setting | Checkpoint |
|---|---|
| With the temperature constant in the log-variance term | `porposed.pth` |
| Without the temperature constant | `proposed(without_α).pth` |
The spelling of `porposed.pth` matches the provided file.
## Backbone Comparisons
The comparison directory uses one shared data loader, training script, and prediction script. The available models are:
| Model | Model file | Checkpoint |
|---|---|---|
| CD-DCU-Net, proposed initialization | `CD-DCU-Net/CD-DCU-Net.py` | `CD-DCU-Net/proposed_initial.pth` |
| CD-DCU-Net, Kaiming initialization | `CD-DCU-Net/CD-DCU-Net.py` | `CD-DCU-Net/kaiming_initial.pth` |
| DFPP-Net | `DFPP-Net/DFPP-Net.py` | `DFPP-Net/DFPP-Net.pth` |
| DR-U-Net | `DR-U-Net/DR-U-Net.py` | `DR-U-Net/DR-U-Net.pth` |
| ISP-Net | `ISP-Net/ISP-Net.py` | `ISP-Net/ISP-Net.pth` |
| TS-U-Net | `TS-U-Net/TS-U-Net.py` | `TS-U-Net/TS-U-Net.pth` |
CD-DCU-Net is the proposed backbone. `kaiming_initial.pth` is the Kaiming-initialized result. Checkpoints without an explicit Kaiming label use the proposed initialization setting.
All comparison models return the same interface:
```python
{"M_mean": M_mean, "D_mean": D_mean}
```
The shared `train.py` and `predict.py` are currently configured for DFPP-Net. See [Selecting a model](#selecting-a-model) before running another backbone.
## Environment
Required packages:
```text
torch
torchvision
numpy
scipy
pillow
matplotlib
tqdm
opencv-python
torchsummary
```
Install PyTorch for the required CUDA version, then install the remaining packages:
```bash
pip install numpy scipy pillow matplotlib tqdm opencv-python torchsummary
```
Multi-GPU training uses the NCCL backend.
## Data Format
In the public repository layout, both experiment directories use one shared `dataset/` folder at the repository root. The dataset is not duplicated inside either experiment directory. `input/` contains the three input images and `output/` contains the corresponding labels. Sample folder names are numeric.
```text
dataset/
|-- input/
| |-- 1/
| | |-- image1.png
| | |-- image2.png
| | `-- image3.png
| |-- 2/
| | |-- image1.png
| | |-- image2.png
| | `-- image3.png
| `-- ...
`-- output/
|-- 1/
| `-- label.mat
|-- 2/
| `-- label.mat
`-- ...
```
Each `label.mat` must contain two arrays named `M` and `D`. For every sample:
- the three PNG files must be single-channel images of the same size;
- `M` and `D` must match the image height and width;
- the numeric folder name must be the same under `input/` and `output/`.
The data loader divides input images and labels by `1000.0`. It applies synchronized geometric augmentation only to the training split.
### Train/validation split
The training and validation sets are generated from the sample folders rather than stored in separate directories. All current training scripts use an 80/20 split:
```python
train_ratio = 0.8
seed = 42
```
The split is produced in the following order:
1. Read and sort the folder names under `dataset/input/` with Python `sorted()`.
2. Create one index for each sorted folder name.
3. Shuffle the indices with `np.random.RandomState(seed)`.
4. Use the first `int(0.8 * number_of_samples)` indices for training.
5. Use all remaining indices, approximately 20% of the samples, for validation.
The following script prints the exact sample folders assigned to each set:
```python
import os
import numpy as np
data_dir = "dataset"
train_ratio = 0.8
seed = 42
samples = sorted(os.listdir(os.path.join(data_dir, "input")))
indices = list(range(len(samples)))
rng = np.random.RandomState(seed)
rng.shuffle(indices)
train_size = int(train_ratio * len(samples))
train_samples = [samples[i] for i in indices[:train_size]]
val_samples = [samples[i] for i in indices[train_size:]]
print("Training samples:", train_samples)
print("Validation samples:", val_samples)
```
Run this script from the repository root. It reproduces the membership generated by `create_data_loaders()`. The distributed sampler may change the order in which training samples are read during each epoch, but it does not change which samples belong to the training or validation set.
Folder names are sorted as strings because the data loader uses `sorted(os.listdir(...))`. For example, non-padded names may be ordered as `1`, `10`, `11`, `2`. The script above deliberately uses the same rule so that its result matches the training code exactly.
## Selecting a Model
The architecture filenames contain hyphens, which cannot be used in a normal Python import. Copy or rename the selected file to an import-safe name before running the scripts.
### Concrete Dropout model
Inside `CD-DCU-Net(Incorporate CD-module)/`, copy or rename `CD-DCU-Net.py` to `cd_dcu_net.py`. Then use the same import in both training scripts and in `predict.py`:
```python
from cd_dcu_net import UNet
```
The constructor is:
```python
UNet(n_channels=3, bilinear=False)
```
This replaces `from mcunet1 import UNet` in the training scripts and `from proposed_uncertainty import UNet` in the prediction script.
### Comparison models
Copy the selected model file into `Different networks compare(without CD-model)/`, give it the module name shown below, and update both `train.py` and `predict.py`.
| Model | Module and import | Constructor |
|---|---|---|
| CD-DCU-Net | `from cd_dcu_net import UNet` | `UNet(n_channels=3, bilinear=False)` |
| DFPP-Net | `from dfppnet import DFPPNet3FrameMD` | `DFPPNet3FrameMD(n_feat=4, stage0_align=True, stage1_align=False, stage2_align=False, return_aux=False)` |
| DR-U-Net | `from dr_unet import UNet` | `UNet(n_channels=3, out_channels=2, bilinear=False)` |
| ISP-Net | `from isp_net import UNet` | `UNet(n_channels=3, bilinear=False)` |
| TS-U-Net | `from ts_unet import UNet` | `UNet(n_channels=3, bilinear=False)` |
Change the constructor in two places:
- the main block of `train.py`;
- `Predictor._load_model()` in `predict.py`.
The architecture and constructor must match the checkpoint.
## Training
Edit the configuration block at the bottom of the selected training script:
```python
data_dir = "../dataset"
batch_size = 4
train_ratio = 0.8
crop_size = (1360, 672)
num_workers = 16
num_epochs = 300
learning_rate = 0.0001
pretrained_path = "path/to/initialization_checkpoint.pth"
```
Also set `save_dir` in the `Trainer(...)` call. `crop_size` is ordered as `(height, width)` and must not exceed the image size.
Training is initialized from a checkpoint:
```python
checkpoint = torch.load(pretrained_path, map_location="cpu")
model.load_state_dict(checkpoint["model_state_dict"], strict=True)
```
The checkpoint must contain `model_state_dict` and must match the selected architecture.
Run one of the Concrete Dropout experiments from its directory:
```bash
python "train_with_α.py"
python "train_without_α.py"
```
Run a backbone comparison after updating the shared model import and constructor:
```bash
cd "Different networks compare(without CD-model)"
python train.py
```
For multi-GPU training:
```bash
torchrun --standalone --nproc_per_node=2 "train_with_α.py"
```
The scripts save epoch checkpoints, `best_model.pth`, training curves, and the five most recent rolling checkpoints. Concrete Dropout training also saves the learned dropout-rate curves.
## Prediction
Set the checkpoint, crop size, device, and three image paths at the bottom of `predict.py`:
```python
predictor = Predictor(
model_path="path/to/model_checkpoint.pth",
crop_size=(1360, 672),
device="cuda",
)
img1_path = "../dataset/input/1/image1.png"
img2_path = "../dataset/input/1/image2.png"
img3_path = "../dataset/input/1/image3.png"
```
Use the same crop size as training. Set `device="cpu"` for CPU inference.
Run:
```bash
python predict.py
```
### Single prediction
```python
result = predictor.predict_single(img1_path, img2_path, img3_path)
predictor.save_prediction(result, "predictions/single_result.mat", mode="single")
```
Single prediction returns `M` and `D`.
### Uncertainty prediction
Monte Carlo prediction is used for the Concrete Dropout model:
```python
results = predictor.predict_with_uncertainty(
img1_path,
img2_path,
img3_path,
n_samples=50,
)
predictor.save_prediction(results, "predictions/mc_result.mat", mode="mc")
```
During Monte Carlo prediction, `predict.py` enables Concrete Dropout, runs `n_samples` stochastic forward passes, and restores evaluation mode afterward. The output contains mean predictions and model, data, and total uncertainty maps for both `M` and `D`.
The comparison backbones return only `M_mean` and `D_mean`; use `predict_single()` for those checkpoints.
## Hugging Face Upload
The checkpoint files are large. Track them with Git LFS before pushing the repository:
```bash
git lfs install
git lfs track "*.pth"
git add .gitattributes
git add .
git commit -m "Add code and checkpoints"
```
|