Image-to-Image
Diffusers
Safetensors
sar-to-eo
remote-sensing
flow-matching
synthetic-aperture-radar
Instructions to use JeonghyeokDo/ReFlowSET with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use JeonghyeokDo/ReFlowSET with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import load_image # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("JeonghyeokDo/ReFlowSET", dtype=torch.bfloat16, device_map="cuda") prompt = "Turn this cat into a dog" input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png") image = pipe(image=input_image, prompt=prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Upload folder using huggingface_hub
Browse files- README.md +24 -4
- baselines/README.md +313 -0
- baselines/licenses/LICENSE-SoftPool-MIT.txt +21 -0
- baselines/licenses/LICENSE-bbdm.txt +21 -0
- baselines/licenses/LICENSE-cbbdm.txt +21 -0
- baselines/licenses/LICENSE-cdiffset.txt +21 -0
- baselines/licenses/LICENSE-cyclegan.txt +58 -0
- baselines/licenses/LICENSE-hidiff.txt +201 -0
- baselines/licenses/LICENSE-p2phd.txt +45 -0
- baselines/licenses/LICENSE-pix2pix.txt +58 -0
- baselines/licenses/LICENSE-resshift.txt +35 -0
- baselines/licenses/LICENSE-sd21ft.txt +21 -0
- baselines/licenses/LICENSE-spade.txt +177 -0
- baselines/licenses/NO-UPSTREAM-LICENSE.md +189 -0
- baselines/qxs-saropt/bbdm/README.md +136 -0
- baselines/qxs-saropt/cbbdm/README.md +153 -0
- baselines/qxs-saropt/cdiffset/README.md +164 -0
- baselines/qxs-saropt/cdiffset/config.json +45 -0
- baselines/qxs-saropt/conddiff/README.md +173 -0
- baselines/qxs-saropt/controlnet/README.md +127 -0
- baselines/qxs-saropt/controlnet/config.json +56 -0
- baselines/qxs-saropt/cyclegan/README.md +169 -0
- baselines/qxs-saropt/ddpm/README.md +174 -0
- baselines/qxs-saropt/e3diff/README.md +177 -0
- baselines/qxs-saropt/hidiff/README.md +162 -0
- baselines/qxs-saropt/p2phd/README.md +142 -0
- baselines/qxs-saropt/pix2pix/README.md +141 -0
- baselines/qxs-saropt/resshift/README.md +144 -0
- baselines/qxs-saropt/sd21ft/README.md +156 -0
- baselines/qxs-saropt/sd21ft/config.json +45 -0
- baselines/qxs-saropt/spade/README.md +176 -0
- baselines/qxs-saropt/stegogan/README.md +200 -0
- baselines/sar2opt/bbdm/README.md +138 -0
- baselines/sar2opt/cbbdm/README.md +155 -0
- baselines/sar2opt/cdiffset/README.md +166 -0
- baselines/sar2opt/cdiffset/config.json +45 -0
- baselines/sar2opt/conddiff/README.md +175 -0
- baselines/sar2opt/controlnet/README.md +128 -0
- baselines/sar2opt/controlnet/config.json +56 -0
- baselines/sar2opt/cyclegan/README.md +172 -0
- baselines/sar2opt/ddpm/README.md +176 -0
- baselines/sar2opt/e3diff/README.md +179 -0
- baselines/sar2opt/hidiff/README.md +164 -0
- baselines/sar2opt/p2phd/README.md +145 -0
- baselines/sar2opt/pix2pix/README.md +144 -0
- baselines/sar2opt/resshift/README.md +146 -0
- baselines/sar2opt/sd21ft/README.md +158 -0
- baselines/sar2opt/sd21ft/config.json +45 -0
- baselines/sar2opt/spade/README.md +179 -0
- baselines/sar2opt/stegogan/README.md +205 -0
README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
| 3 |
-
license_name:
|
| 4 |
pipeline_tag: image-to-image
|
| 5 |
library_name: diffusers
|
| 6 |
tags:
|
|
@@ -20,7 +20,7 @@ from scratch inside a frozen high-fidelity autoencoder latent space.
|
|
| 20 |
- **Paper:** https://arxiv.org/abs/{{ARXIV_ID}}
|
| 21 |
- **Code:** https://github.com/KAIST-VICLab/ReFlowSET
|
| 22 |
- **Project page:** https://kaist-viclab.github.io/ReFlowSET_site/
|
| 23 |
-
- **Comparison-method weights:** [`
|
| 24 |
|
| 25 |
## Checkpoints
|
| 26 |
|
|
@@ -115,9 +115,29 @@ QXS-SAROPT and SAR2Opt only, one dataset per arm, trained from scratch. No
|
|
| 115 |
pretraining corpus. Neither dataset is redistributed. QXS-SAROPT requires citing
|
| 116 |
arXiv:2103.08259 for research use.
|
| 117 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
## Licence and provenance
|
| 119 |
|
| 120 |
-
**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
|
| 122 |
The frozen autoencoder bundled in `vae/` is the **Apache-2.0** autoencoder from
|
| 123 |
[`black-forest-labs/FLUX.2-klein-base-4B`](https://huggingface.co/black-forest-labs/FLUX.2-klein-base-4B),
|
|
|
|
| 1 |
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: mixed-see-model-card
|
| 4 |
pipeline_tag: image-to-image
|
| 5 |
library_name: diffusers
|
| 6 |
tags:
|
|
|
|
| 20 |
- **Paper:** https://arxiv.org/abs/{{ARXIV_ID}}
|
| 21 |
- **Code:** https://github.com/KAIST-VICLab/ReFlowSET
|
| 22 |
- **Project page:** https://kaist-viclab.github.io/ReFlowSET_site/
|
| 23 |
+
- **Comparison-method weights:** [`baselines/`](https://huggingface.co/JeonghyeokDo/ReFlowSET/tree/main/baselines) in this repository — all fifteen prior methods, both datasets
|
| 24 |
|
| 25 |
## Checkpoints
|
| 26 |
|
|
|
|
| 115 |
pretraining corpus. Neither dataset is redistributed. QXS-SAROPT requires citing
|
| 116 |
arXiv:2103.08259 for research use.
|
| 117 |
|
| 118 |
+
## Comparison methods, in this same repository
|
| 119 |
+
|
| 120 |
+
`baselines/` holds the **fifteen prior methods of the paper's main table**,
|
| 121 |
+
retrained by us on the same splits and scored by the same evaluator — thirty
|
| 122 |
+
checkpoints, both datasets, each with its own card, its measured row, its
|
| 123 |
+
upstream repository and its licence. Start at
|
| 124 |
+
[`baselines/README.md`](https://huggingface.co/JeonghyeokDo/ReFlowSET/blob/main/baselines/README.md).
|
| 125 |
+
|
| 126 |
+
```python
|
| 127 |
+
from diffusers import UNet2DConditionModel
|
| 128 |
+
unet = UNet2DConditionModel.from_pretrained(
|
| 129 |
+
"JeonghyeokDo/ReFlowSET", subfolder="baselines/qxs-saropt/cdiffset")
|
| 130 |
+
```
|
| 131 |
+
|
| 132 |
## Licence and provenance
|
| 133 |
|
| 134 |
+
**This repository is mixed-licence, so the Hub tag is `other`.** ReFlowSET's own
|
| 135 |
+
weights are **CC BY-NC 4.0** and the code is **Apache-2.0**. Everything under
|
| 136 |
+
`baselines/` is a third-party method carrying its own terms — MIT, BSD-3-Clause,
|
| 137 |
+
Apache-2.0, CreativeML-OpenRAIL-M, CC BY-NC-SA 4.0, S-Lab 1.0, and four with no
|
| 138 |
+
upstream licence file at all. The per-method cards state each one, and the texts
|
| 139 |
+
that must travel are in `baselines/licenses/`. Check the method you intend to
|
| 140 |
+
use; the repo-level tag is not a substitute.
|
| 141 |
|
| 142 |
The frozen autoencoder bundled in `vae/` is the **Apache-2.0** autoencoder from
|
| 143 |
[`black-forest-labs/FLUX.2-klein-base-4B`](https://huggingface.co/black-forest-labs/FLUX.2-klein-base-4B),
|
baselines/README.md
ADDED
|
@@ -0,0 +1,313 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: per-method-see-below
|
| 4 |
+
library_name: pytorch
|
| 5 |
+
pipeline_tag: image-to-image
|
| 6 |
+
tags:
|
| 7 |
+
- sar-to-eo
|
| 8 |
+
- image-translation
|
| 9 |
+
- remote-sensing
|
| 10 |
+
- baselines
|
| 11 |
+
- benchmark
|
| 12 |
+
- reflowset
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# ReFlowSET — retrained comparison-method checkpoints
|
| 16 |
+
|
| 17 |
+
The fifteen comparison methods of the **ReFlowSET** SAR-to-EO benchmark, on both
|
| 18 |
+
of its datasets: **QXS-SAROPT** (256 px) and **SAR2Opt** (512 px). Thirty
|
| 19 |
+
checkpoint cells in total.
|
| 20 |
+
|
| 21 |
+
**Every checkpoint here was retrained by us.** None is an original author's
|
| 22 |
+
released checkpoint. Each method was trained on the same official split, at the
|
| 23 |
+
same resolution, under a budget stated in optimizer updates, and every cell was
|
| 24 |
+
then scored through one evaluation pipeline with an identity-collapse and
|
| 25 |
+
ground-truth-leak audit. That is what makes the rows below comparable **to each
|
| 26 |
+
other** — and, for exactly the same reason, not comparable with the numbers in
|
| 27 |
+
the methods' own papers.
|
| 28 |
+
|
| 29 |
+
ReFlowSET itself is not in this repository. It lives at
|
| 30 |
+
[JeonghyeokDo/ReFlowSET](https://huggingface.co/JeonghyeokDo/ReFlowSET); code at https://github.com/KAIST-VICLab/ReFlowSET; project page
|
| 31 |
+
at https://kaist-viclab.github.io/ReFlowSET_site/.
|
| 32 |
+
|
| 33 |
+
## Results — all sixteen rows, both datasets
|
| 34 |
+
|
| 35 |
+
Our own measurements, on the images each checkpoint produced. No number below
|
| 36 |
+
is copied from any paper.
|
| 37 |
+
|
| 38 |
+
### QXS-SAROPT — n = 3,999, 256 px
|
| 39 |
+
|
| 40 |
+
| Method | Venue | FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 41 |
+
|---|---|---|---|---|---|---|
|
| 42 |
+
| *General image-to-image* | | | | | | |
|
| 43 |
+
| pix2pix | CVPR'17 | 174.6 | 0.373 | 0.665 | 0.203 | 12.33 |
|
| 44 |
+
| CycleGAN ⚠ | ICCV'17 | 104.4 | 0.376 | 0.653 | 0.262 | 12.92 |
|
| 45 |
+
| pix2pixHD | CVPR'18 | 85.7 | 0.298 | 0.573 | 0.358 | 16.13 |
|
| 46 |
+
| SPADE | CVPR'19 | 90.7 | 0.292 | 0.599 | 0.320 | 14.53 |
|
| 47 |
+
| DDPM (SR3-class) | TPAMI'22 | 43.8 | 0.311 | 0.620 | 0.359 | 14.04 |
|
| 48 |
+
| SD2.1 fine-tune only | CVPR'22 | 19.1 | 0.257 | 0.561 | 0.348 | 15.40 |
|
| 49 |
+
| BBDM | CVPR'23 | 76.6 | 0.270 | 0.568 | 0.352 | 15.34 |
|
| 50 |
+
| ControlNet | ICCV'23 | 50.4 | 0.307 | 0.604 | 0.297 | 13.42 |
|
| 51 |
+
| HI-Diff | NeurIPS'23 | 324.3 | 0.539 | 0.692 | 0.457 | 17.10 |
|
| 52 |
+
| ResShift | NeurIPS'23 | 140.2 | 0.334 | 0.607 | 0.217 | 14.20 |
|
| 53 |
+
| StegoGAN ⚠ | CVPR'24 | 106.8 | 0.384 | 0.658 | 0.254 | 12.96 |
|
| 54 |
+
| *SAR-to-EO* | | | | | | |
|
| 55 |
+
| Conditional Diffusion | GRSL'24 | 88.6 | 0.355 | 0.730 | 0.213 | 11.55 |
|
| 56 |
+
| cBBDM | GRSL'25 | 50.6 | 0.246 | 0.539 | 0.372 | 16.02 |
|
| 57 |
+
| E3Diff | GRSL'25 | 47.8 | 0.278 | 0.530 | 0.302 | 16.44 |
|
| 58 |
+
| C-DiffSET | TCSVT'26 | 19.9 | 0.233 | 0.526 | 0.380 | 16.92 |
|
| 59 |
+
| *Ours* | | | | | | |
|
| 60 |
+
| **ReFlowSET (ours)** | — | 19.1 | 0.231 | 0.534 | 0.355 | 16.09 |
|
| 61 |
+
|
| 62 |
+
### SAR2Opt — n = 627, 512 px (centre crops of the 600 px tiles)
|
| 63 |
+
|
| 64 |
+
| Method | Venue | FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 65 |
+
|---|---|---|---|---|---|---|
|
| 66 |
+
| *General image-to-image* | | | | | | |
|
| 67 |
+
| pix2pix | CVPR'17 | 261.9 | 0.347 | 0.657 | 0.199 | 13.39 |
|
| 68 |
+
| CycleGAN ⚠ | ICCV'17 | 143.5 | 0.330 | 0.650 | 0.178 | 12.90 |
|
| 69 |
+
| pix2pixHD | CVPR'18 | 146.3 | 0.283 | 0.567 | 0.268 | 15.95 |
|
| 70 |
+
| SPADE | CVPR'19 | 142.5 | 0.265 | 0.597 | 0.234 | 14.47 |
|
| 71 |
+
| DDPM (SR3-class) | TPAMI'22 | 122.5 | 0.295 | 0.610 | 0.313 | 13.65 |
|
| 72 |
+
| SD2.1 fine-tune only | CVPR'22 | 71.8 | 0.211 † | 0.541 | 0.293 | 16.24 |
|
| 73 |
+
| BBDM | CVPR'23 | 143.1 | 0.290 | 0.590 | 0.276 | 15.29 |
|
| 74 |
+
| ControlNet | ICCV'23 | 140.5 | 0.350 | 0.643 | 0.217 | 11.73 |
|
| 75 |
+
| HI-Diff | NeurIPS'23 | 319.8 | 0.473 | 0.692 | 0.384 | 17.36 |
|
| 76 |
+
| ResShift | NeurIPS'23 | 141.7 | 0.304 | 0.597 | 0.177 | 14.31 |
|
| 77 |
+
| StegoGAN ⚠ | CVPR'24 | 150.1 | 0.347 | 0.655 | 0.158 | 12.47 |
|
| 78 |
+
| *SAR-to-EO* | | | | | | |
|
| 79 |
+
| Conditional Diffusion | GRSL'24 | 211.8 | 0.415 | 0.686 | 0.248 | 12.48 |
|
| 80 |
+
| cBBDM | GRSL'25 | 222.3 | 0.377 | 0.571 | 0.361 | 17.05 |
|
| 81 |
+
| E3Diff | GRSL'25 | 104.7 | 0.232 | 0.529 | 0.249 | 16.09 |
|
| 82 |
+
| C-DiffSET | TCSVT'26 | 78.1 | 0.214 † | 0.529 | 0.314 | 16.81 |
|
| 83 |
+
| *Ours* | | | | | | |
|
| 84 |
+
| **ReFlowSET (ours)** | — | 66.3 | 0.185 | 0.522 | 0.287 | 16.06 |
|
| 85 |
+
|
| 86 |
+
⚠ **fails the input-copy audit** — see *Identity collapse* below.
|
| 87 |
+
† **these two DISTS values were re-measured.** The image dumps for these two
|
| 88 |
+
SAR2Opt cells were regenerated after their last extended-metrics pass, and that
|
| 89 |
+
pass's resume check keys on the number of images rather than on their content, so
|
| 90 |
+
it skipped them and their stored DISTS went stale. Both were re-measured on the
|
| 91 |
+
current dumps at full n=627 and the table prints the re-measurement (SD2.1-FT
|
| 92 |
+
0.212 -> **0.211**; C-DiffSET 0.2141 -> **0.2136**, unchanged at three decimals).
|
| 93 |
+
The harness was validated on a cell that is *not* stale, whose DISTS it
|
| 94 |
+
reproduces to 6e-6. FID, LPIPS, SSIM and PSNR come from a pass that rescores
|
| 95 |
+
every cell on every run and were never affected.
|
| 96 |
+
|
| 97 |
+
## Read before you compare these numbers with a paper
|
| 98 |
+
|
| 99 |
+
**These numbers are not comparable with the source papers, and are not meant to
|
| 100 |
+
be.** Three reasons, all deliberate:
|
| 101 |
+
|
| 102 |
+
1. **Budgets are matched across the benchmark, not to each method's own
|
| 103 |
+
recipe.** Several methods are trained here at a fraction of their published
|
| 104 |
+
schedule — HI-Diff, for instance, is trained for 50,000 updates against an
|
| 105 |
+
upstream recipe of 600,000, with progressive resizing disabled. Where we
|
| 106 |
+
departed from a published recipe, the method's own card says so and says by
|
| 107 |
+
how much.
|
| 108 |
+
2. **Budgets are quoted in optimizer updates, never epochs.** The two datasets
|
| 109 |
+
differ by more than tenfold in size (16,001 versus 1,450 training pairs), so
|
| 110 |
+
an epoch count means something completely different on each. Comparing
|
| 111 |
+
unpaired methods by epochs produced two successive wrong conclusions in our
|
| 112 |
+
own work before we changed the unit.
|
| 113 |
+
3. **One evaluation protocol for everyone.** Same test items, same resolution,
|
| 114 |
+
same metric implementations, same ground truth. SAR2Opt is **centre-cropped**
|
| 115 |
+
to 512 px and never resized, in training and in evaluation alike.
|
| 116 |
+
|
| 117 |
+
### The LPIPS convention
|
| 118 |
+
|
| 119 |
+
LPIPS here is **LPIPS-VGG on inputs mapped to `[-1, 1]`** before the network —
|
| 120 |
+
`lpips_vgg(x * 2 - 1, y * 2 - 1)` with the library's default `normalize=False`.
|
| 121 |
+
The other convention, which several released evaluators use, hands the network
|
| 122 |
+
`[0, 1]` inputs *unscaled* (still `normalize=False`) and produces a
|
| 123 |
+
**systematically lower number** — about 0.05 lower — on the same images. Note
|
| 124 |
+
that `[0, 1]` inputs with `normalize=True` is the *same* thing as ours, not the
|
| 125 |
+
other one; only the flag differs. Both are called "LPIPS" in the literature. If you compare a number from
|
| 126 |
+
this table against one from a paper, check which convention that paper used
|
| 127 |
+
first; otherwise the comparison is meaningless. The same care applies in reverse
|
| 128 |
+
to anyone quoting these numbers.
|
| 129 |
+
|
| 130 |
+
PSNR and SSIM are per-image torchmetrics with `data_range=1`. FID is
|
| 131 |
+
`pytorch-fid` against the size-matched ground-truth test set. DISTS is the
|
| 132 |
+
standard implementation on the same pairs. Where the generated size and the
|
| 133 |
+
ground-truth size differ, the ground truth is centre-cropped to match.
|
| 134 |
+
|
| 135 |
+
### Identity collapse
|
| 136 |
+
|
| 137 |
+
**Four of the thirty cells fail an input-copy audit**: CycleGAN and StegoGAN, on
|
| 138 |
+
both datasets. Their outputs are closer to the SAR image they were given than to
|
| 139 |
+
the EO image they were asked for — mean |output − SAR| < mean |output − ground
|
| 140 |
+
truth| — which is the signature of a model that has learned to pass its input
|
| 141 |
+
through rather than translate it.
|
| 142 |
+
|
| 143 |
+
| cell | ratio (SAR distance ÷ GT distance) | verdict |
|
| 144 |
+
|---|---|---|
|
| 145 |
+
| QXS-SAROPT CycleGAN | 0.847 | COLLAPSED |
|
| 146 |
+
| QXS-SAROPT StegoGAN | 0.919 | COLLAPSED |
|
| 147 |
+
| SAR2Opt CycleGAN | 0.795 | COLLAPSED |
|
| 148 |
+
| SAR2Opt StegoGAN | 0.758 | COLLAPSED |
|
| 149 |
+
|
| 150 |
+
**Their metric values are reported unchanged and marked in place.** They are what
|
| 151 |
+
the released implementation does at its own published protocol, and a benchmark
|
| 152 |
+
that quietly substitutes a better-behaved variant is not reporting the method.
|
| 153 |
+
But they should not be read as translation quality. No other cell on either
|
| 154 |
+
dataset collapses — in particular DDPM, Conditional Diffusion and BBDM do **not**
|
| 155 |
+
collapse here, whatever they may do elsewhere.
|
| 156 |
+
|
| 157 |
+
The companion leak audit is clean: no cell on either dataset shows a generation
|
| 158 |
+
that matches its own ground truth anomalously better than it matches other
|
| 159 |
+
images' ground truth. Per-cell numbers are on each method's card.
|
| 160 |
+
|
| 161 |
+
**One oracle you must not use.** StegoGAN's `fake_B` output is computed as
|
| 162 |
+
`netG_A(real_A, latent(real_B))` — conditioned on the ground truth, and
|
| 163 |
+
impossible to produce at deployment. Only `fake_B_clean = netG_A(real_A)` is a
|
| 164 |
+
prediction, and it is what the StegoGAN rows above report. Any published StegoGAN
|
| 165 |
+
number should say which of the two it is.
|
| 166 |
+
|
| 167 |
+
## Layout
|
| 168 |
+
|
| 169 |
+
```
|
| 170 |
+
<dataset>/<method>/<weight files> + README.md
|
| 171 |
+
dataset ∈ { qxs-saropt, sar2opt }
|
| 172 |
+
licenses/ third-party licence texts, one per method
|
| 173 |
+
```
|
| 174 |
+
|
| 175 |
+
Every one of the thirty folders carries its own `README.md` with that cell's
|
| 176 |
+
architecture, the file-by-file contents, the training budget in optimizer
|
| 177 |
+
updates, the measured metrics, a copy-pasteable load-and-translate snippet, the
|
| 178 |
+
traps specific to that method, and its licence stated factually.
|
| 179 |
+
|
| 180 |
+
| method | folder | files | upstream code |
|
| 181 |
+
|---|---|---|---|
|
| 182 |
+
| pix2pix | `pix2pix/` | `net_G.pth` | `junyanz/pytorch-CycleGAN-and-pix2pix` |
|
| 183 |
+
| CycleGAN | `cyclegan/` | `net_G_A.pth` (**SAR→EO**), `net_G_B.pth` | `junyanz/pytorch-CycleGAN-and-pix2pix` |
|
| 184 |
+
| pix2pixHD | `p2phd/` | `net_G.pth` | `NVIDIA/pix2pixHD` |
|
| 185 |
+
| SPADE | `spade/` | `net_G.pth` | `NVlabs/SPADE` |
|
| 186 |
+
| DDPM (SR3-class) | `ddpm/` | `gen.pth` | `DeepSARRS/E3Diff`, stage 1 |
|
| 187 |
+
| SD2.1 fine-tune only | `sd21ft/` | `diffusion_pytorch_model.safetensors`, `config.json` | `KAIST-VICLab/C-DiffSET`, stage 1, on SD 2.1-base |
|
| 188 |
+
| BBDM | `bbdm/` | `last_model.pth` | `xuekt98/BBDM` |
|
| 189 |
+
| ControlNet | `controlnet/` | `diffusion_pytorch_model.safetensors`, `config.json` | `huggingface/diffusers` `train_controlnet.py`, on SD 2.1-base |
|
| 190 |
+
| HI-Diff | `hidiff/` | `S1_*.pth` ×2, `S2_*.pth` ×3 (**only the three `S2_` files are needed at inference**) | `zhengchen1999/HI-Diff` |
|
| 191 |
+
| ResShift | `resshift/` | `ema_model.pth` | `zsyOAOA/ResShift` |
|
| 192 |
+
| StegoGAN | `stegogan/` | `net_G_A.pth` (**SAR→EO**), `net_G_B.pth` | `sian-wusidi/StegoGAN` |
|
| 193 |
+
| Conditional Diffusion | `conddiff/` | `ema_final.pt` | `Coordi777/Conditional-Diffusion-for-SAR-to-Optical-Image-Translation` |
|
| 194 |
+
| cBBDM | `cbbdm/` | `last_model.pth` | `egshkim/ConditionalBBDM-for-VHR-SAR-to-Optical` |
|
| 195 |
+
| E3Diff | `e3diff/` | `gen.pth` | `DeepSARRS/E3Diff`, stage 2 |
|
| 196 |
+
| C-DiffSET | `cdiffset/` | `diffusion_pytorch_model.safetensors`, `config.json` | `KAIST-VICLab/C-DiffSET`, stage 2 |
|
| 197 |
+
|
| 198 |
+
### Three checkpoints these weights need and this repository does not carry
|
| 199 |
+
|
| 200 |
+
* **BBDM and cBBDM** need the CompVis latent-diffusion **vq-f4 VQGAN**
|
| 201 |
+
(`model.ckpt`, 756,175,527 bytes) at the path named in their config — at
|
| 202 |
+
construction time, even though the same weights are also inside
|
| 203 |
+
`last_model.pth`. MIT.
|
| 204 |
+
* **ResShift** needs the CompVis **vq-f4 autoencoder** (221,364,711 bytes). MIT.
|
| 205 |
+
* **SD2.1-FT, ControlNet and C-DiffSET** need Stable Diffusion 2.1-base for the
|
| 206 |
+
frozen VAE, text encoder, tokenizer and scheduler. Their cards name the exact
|
| 207 |
+
repository we loaded.
|
| 208 |
+
* **DDPM and E3Diff** need no extra checkpoint, but do need **PPB-despeckled and
|
| 209 |
+
Canny condition images** computed from each SAR input; neither can be run from
|
| 210 |
+
a SAR file alone.
|
| 211 |
+
|
| 212 |
+
## Licences
|
| 213 |
+
|
| 214 |
+
**A single repository-wide licence tag would be wrong for this collection.** The
|
| 215 |
+
fifteen methods fall into three tiers, and the restrictive terms in tiers B and C
|
| 216 |
+
travel with the individual cells. Each method's card states its position in full;
|
| 217 |
+
the licence texts are in `licenses/`.
|
| 218 |
+
|
| 219 |
+
| method | upstream licence | redistribution | what propagates |
|
| 220 |
+
|---|---|---|---|
|
| 221 |
+
| pix2pix | multi-part BSD (CycleGAN / pix2pix / DCGAN notices) | ✅ with the notice | no-endorsement (DCGAN part) |
|
| 222 |
+
| CycleGAN | same file as pix2pix | ✅ with the notice | same |
|
| 223 |
+
| pix2pixHD | BSD (NVIDIA 2019) + bundled CycleGAN notice | ✅ with both notices | none |
|
| 224 |
+
| **SPADE** | **CC BY-NC-SA 4.0** | ⚠️ **non-commercial only** | **NonCommercial + ShareAlike** — this cell is itself distributed under CC BY-NC-SA 4.0 |
|
| 225 |
+
| **DDPM (SR3-class)** | **none declared** | ⚠️ **no express grant** — published by our decision | unknown; there is no licence to read |
|
| 226 |
+
| SD2.1 fine-tune only | MIT code (C-DiffSET) / **Open RAIL++-M** weights | ✅ under Open RAIL++-M | **Attachment A use restrictions** |
|
| 227 |
+
| BBDM | MIT | ✅ | none (needs the external vq-f4 VQGAN, MIT) |
|
| 228 |
+
| ControlNet | Apache-2.0 code (diffusers) / **Open RAIL++-M** weights | ✅ under Open RAIL++-M | **Attachment A use restrictions** |
|
| 229 |
+
| HI-Diff | Apache-2.0 | ✅ | ship the licence, keep notices, state your modifications |
|
| 230 |
+
| **ResShift** | **S-Lab License 1.0** | ⚠️ **non-commercial only** | **NonCommercial**; no-endorsement |
|
| 231 |
+
| **StegoGAN** | **none declared** | ⚠️ **no express grant** — published by our decision | unknown |
|
| 232 |
+
| **Conditional Diffusion** | **none declared** | ⚠️ **no express grant** — published by our decision | unknown (the guided-diffusion parts it forks are MIT) |
|
| 233 |
+
| cBBDM | MIT | ✅ | none (needs the external vq-f4 VQGAN, MIT) |
|
| 234 |
+
| **E3Diff** | **none declared** | ⚠️ **no express grant** — published by our decision | unknown |
|
| 235 |
+
| C-DiffSET | MIT code / **Open RAIL++-M** weights | ✅ under Open RAIL++-M | **Attachment A use restrictions** |
|
| 236 |
+
|
| 237 |
+
### The four with no upstream licence
|
| 238 |
+
|
| 239 |
+
**StegoGAN, E3Diff, DDPM (SR3-class) and Conditional Diffusion are derived from
|
| 240 |
+
upstream code that publishes no licence at all** — no LICENSE, LICENCE, COPYING
|
| 241 |
+
or NOTICE file, no licence section in the README, and no licence declared to
|
| 242 |
+
GitHub. Under default copyright, all rights remain reserved by those authors and
|
| 243 |
+
**no express permission to redistribute derived work has been granted** to us or
|
| 244 |
+
to you.
|
| 245 |
+
|
| 246 |
+
We publish these four anyway, so that the benchmark is reproducible end to end,
|
| 247 |
+
and we say so plainly rather than implying a permission that does not exist. If
|
| 248 |
+
you intend to redistribute them or build on them, assess that for yourself, and
|
| 249 |
+
consider asking the upstream authors directly.
|
| 250 |
+
`licenses/NO-UPSTREAM-LICENSE.md` records, per method, exactly what we checked
|
| 251 |
+
and exactly what we found.
|
| 252 |
+
|
| 253 |
+
### Files in `licenses/`
|
| 254 |
+
|
| 255 |
+
`LICENSE-pix2pix.txt`, `LICENSE-cyclegan.txt`, `LICENSE-p2phd.txt`,
|
| 256 |
+
`LICENSE-spade.txt`, `LICENSE-bbdm.txt`, `LICENSE-cbbdm.txt`,
|
| 257 |
+
`LICENSE-hidiff.txt`, `LICENSE-resshift.txt`, `LICENSE-cdiffset.txt`,
|
| 258 |
+
`LICENSE-sd21ft.txt` — one per method, copied verbatim from the upstream
|
| 259 |
+
repository we trained with. `LICENSE-SoftPool-MIT.txt` is a vendored dependency
|
| 260 |
+
of the E3Diff code base and is included because it is genuinely part of what
|
| 261 |
+
those two cells were trained with; it is **not** a licence for E3Diff.
|
| 262 |
+
`NO-UPSTREAM-LICENSE.md` covers the four methods with no grant.
|
| 263 |
+
`LICENSE-WEIGHTS.md` at the repository root is the CreativeML Open RAIL++-M
|
| 264 |
+
text, including Attachment A, which governs the three Stable-Diffusion-derived
|
| 265 |
+
cells.
|
| 266 |
+
|
| 267 |
+
## Datasets
|
| 268 |
+
|
| 269 |
+
Neither corpus is redistributed here, in any form, and neither may be
|
| 270 |
+
redistributed from here.
|
| 271 |
+
|
| 272 |
+
* **QXS-SAROPT** publishes no licence file; its one stated term is a citation
|
| 273 |
+
requirement — *"This paper must be cited when the dataset is used for research
|
| 274 |
+
purposes"* (Huang et al., [arXiv:2103.08259](https://arxiv.org/abs/2103.08259)).
|
| 275 |
+
Its distribution is request-gated by the authors. Obtain it from them.
|
| 276 |
+
* **SAR2Opt** ships an MIT LICENSE, `Copyright (c) 2021 MarsZhaoYT`. The MIT
|
| 277 |
+
text grants rights in "the Software"; the repository does not name the imagery
|
| 278 |
+
sources, so that file is best read as covering the authors' packaging and code.
|
| 279 |
+
Citation requested: Zhao, Celik, Liu and Li, *A Comparative Analysis of
|
| 280 |
+
GAN-based Methods for SAR-to-Optical Image Translation*, IEEE GRSL, 2022.
|
| 281 |
+
* **The optical side of both datasets is Google-Earth-derived.** Copyright in
|
| 282 |
+
that imagery is held by third parties, not by the dataset authors and not by
|
| 283 |
+
us. No such imagery appears in this repository.
|
| 284 |
+
|
| 285 |
+
## A note on two venue labels
|
| 286 |
+
|
| 287 |
+
The table above prints the venue strings as they appear in the ReFlowSET paper.
|
| 288 |
+
Two of them do not match the year in the upstream repositories' own citation
|
| 289 |
+
blocks, and both refer to the same paper under an early-access versus issue year:
|
| 290 |
+
|
| 291 |
+
* **Conditional Diffusion** — the authors' bibtex says **2023**,
|
| 292 |
+
[doi:10.1109/LGRS.2023.3337143](https://doi.org/10.1109/LGRS.2023.3337143).
|
| 293 |
+
* **E3Diff** — the authors' bibtex says **2024**,
|
| 294 |
+
[doi:10.1109/LGRS.2024.3506566](https://doi.org/10.1109/LGRS.2024.3506566).
|
| 295 |
+
|
| 296 |
+
Cite by DOI.
|
| 297 |
+
|
| 298 |
+
## Citation
|
| 299 |
+
|
| 300 |
+
If you use this benchmark or any of these weights, please cite ReFlowSET:
|
| 301 |
+
|
| 302 |
+
```bibtex
|
| 303 |
+
@article{do2026reflowset,
|
| 304 |
+
title = {ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation},
|
| 305 |
+
author = {Jeonghyeok Do<sup>1</sup>, Seungchul Lee<sup>2</sup>, Munchurl Kim<sup>1*</sup>},
|
| 306 |
+
journal = {arXiv preprint arXiv:{{ARXIV_ID}}},
|
| 307 |
+
year = {2026}
|
| 308 |
+
}
|
| 309 |
+
```
|
| 310 |
+
|
| 311 |
+
**And please cite each method's original paper** — every model card carries the
|
| 312 |
+
reference — **and both dataset papers.** The QXS-SAROPT citation is required by
|
| 313 |
+
that dataset's own terms.
|
baselines/licenses/LICENSE-SoftPool-MIT.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2020 Alexandros Stergiou
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
baselines/licenses/LICENSE-bbdm.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2023 xuekt98
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
baselines/licenses/LICENSE-cbbdm.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2025 egshkim
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
baselines/licenses/LICENSE-cdiffset.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 KAIST VICLab
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
baselines/licenses/LICENSE-cyclegan.txt
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Copyright (c) 2017, Jun-Yan Zhu and Taesung Park
|
| 2 |
+
All rights reserved.
|
| 3 |
+
|
| 4 |
+
Redistribution and use in source and binary forms, with or without
|
| 5 |
+
modification, are permitted provided that the following conditions are met:
|
| 6 |
+
|
| 7 |
+
* Redistributions of source code must retain the above copyright notice, this
|
| 8 |
+
list of conditions and the following disclaimer.
|
| 9 |
+
|
| 10 |
+
* Redistributions in binary form must reproduce the above copyright notice,
|
| 11 |
+
this list of conditions and the following disclaimer in the documentation
|
| 12 |
+
and/or other materials provided with the distribution.
|
| 13 |
+
|
| 14 |
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
| 15 |
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
| 16 |
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
| 17 |
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
| 18 |
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
| 19 |
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
| 20 |
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
| 21 |
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
| 22 |
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
| 23 |
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
--------------------------- LICENSE FOR pix2pix --------------------------------
|
| 27 |
+
BSD License
|
| 28 |
+
|
| 29 |
+
For pix2pix software
|
| 30 |
+
Copyright (c) 2016, Phillip Isola and Jun-Yan Zhu
|
| 31 |
+
All rights reserved.
|
| 32 |
+
|
| 33 |
+
Redistribution and use in source and binary forms, with or without
|
| 34 |
+
modification, are permitted provided that the following conditions are met:
|
| 35 |
+
|
| 36 |
+
* Redistributions of source code must retain the above copyright notice, this
|
| 37 |
+
list of conditions and the following disclaimer.
|
| 38 |
+
|
| 39 |
+
* Redistributions in binary form must reproduce the above copyright notice,
|
| 40 |
+
this list of conditions and the following disclaimer in the documentation
|
| 41 |
+
and/or other materials provided with the distribution.
|
| 42 |
+
|
| 43 |
+
----------------------------- LICENSE FOR DCGAN --------------------------------
|
| 44 |
+
BSD License
|
| 45 |
+
|
| 46 |
+
For dcgan.torch software
|
| 47 |
+
|
| 48 |
+
Copyright (c) 2015, Facebook, Inc. All rights reserved.
|
| 49 |
+
|
| 50 |
+
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
|
| 51 |
+
|
| 52 |
+
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
|
| 53 |
+
|
| 54 |
+
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
|
| 55 |
+
|
| 56 |
+
Neither the name Facebook nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
|
| 57 |
+
|
| 58 |
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
baselines/licenses/LICENSE-hidiff.txt
ADDED
|
@@ -0,0 +1,201 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Apache License
|
| 2 |
+
Version 2.0, January 2004
|
| 3 |
+
http://www.apache.org/licenses/
|
| 4 |
+
|
| 5 |
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
| 6 |
+
|
| 7 |
+
1. Definitions.
|
| 8 |
+
|
| 9 |
+
"License" shall mean the terms and conditions for use, reproduction,
|
| 10 |
+
and distribution as defined by Sections 1 through 9 of this document.
|
| 11 |
+
|
| 12 |
+
"Licensor" shall mean the copyright owner or entity authorized by
|
| 13 |
+
the copyright owner that is granting the License.
|
| 14 |
+
|
| 15 |
+
"Legal Entity" shall mean the union of the acting entity and all
|
| 16 |
+
other entities that control, are controlled by, or are under common
|
| 17 |
+
control with that entity. For the purposes of this definition,
|
| 18 |
+
"control" means (i) the power, direct or indirect, to cause the
|
| 19 |
+
direction or management of such entity, whether by contract or
|
| 20 |
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
| 21 |
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
| 22 |
+
|
| 23 |
+
"You" (or "Your") shall mean an individual or Legal Entity
|
| 24 |
+
exercising permissions granted by this License.
|
| 25 |
+
|
| 26 |
+
"Source" form shall mean the preferred form for making modifications,
|
| 27 |
+
including but not limited to software source code, documentation
|
| 28 |
+
source, and configuration files.
|
| 29 |
+
|
| 30 |
+
"Object" form shall mean any form resulting from mechanical
|
| 31 |
+
transformation or translation of a Source form, including but
|
| 32 |
+
not limited to compiled object code, generated documentation,
|
| 33 |
+
and conversions to other media types.
|
| 34 |
+
|
| 35 |
+
"Work" shall mean the work of authorship, whether in Source or
|
| 36 |
+
Object form, made available under the License, as indicated by a
|
| 37 |
+
copyright notice that is included in or attached to the work
|
| 38 |
+
(an example is provided in the Appendix below).
|
| 39 |
+
|
| 40 |
+
"Derivative Works" shall mean any work, whether in Source or Object
|
| 41 |
+
form, that is based on (or derived from) the Work and for which the
|
| 42 |
+
editorial revisions, annotations, elaborations, or other modifications
|
| 43 |
+
represent, as a whole, an original work of authorship. For the purposes
|
| 44 |
+
of this License, Derivative Works shall not include works that remain
|
| 45 |
+
separable from, or merely link (or bind by name) to the interfaces of,
|
| 46 |
+
the Work and Derivative Works thereof.
|
| 47 |
+
|
| 48 |
+
"Contribution" shall mean any work of authorship, including
|
| 49 |
+
the original version of the Work and any modifications or additions
|
| 50 |
+
to that Work or Derivative Works thereof, that is intentionally
|
| 51 |
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
| 52 |
+
or by an individual or Legal Entity authorized to submit on behalf of
|
| 53 |
+
the copyright owner. For the purposes of this definition, "submitted"
|
| 54 |
+
means any form of electronic, verbal, or written communication sent
|
| 55 |
+
to the Licensor or its representatives, including but not limited to
|
| 56 |
+
communication on electronic mailing lists, source code control systems,
|
| 57 |
+
and issue tracking systems that are managed by, or on behalf of, the
|
| 58 |
+
Licensor for the purpose of discussing and improving the Work, but
|
| 59 |
+
excluding communication that is conspicuously marked or otherwise
|
| 60 |
+
designated in writing by the copyright owner as "Not a Contribution."
|
| 61 |
+
|
| 62 |
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
| 63 |
+
on behalf of whom a Contribution has been received by Licensor and
|
| 64 |
+
subsequently incorporated within the Work.
|
| 65 |
+
|
| 66 |
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
| 67 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 68 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 69 |
+
copyright license to reproduce, prepare Derivative Works of,
|
| 70 |
+
publicly display, publicly perform, sublicense, and distribute the
|
| 71 |
+
Work and such Derivative Works in Source or Object form.
|
| 72 |
+
|
| 73 |
+
3. Grant of Patent License. Subject to the terms and conditions of
|
| 74 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 75 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 76 |
+
(except as stated in this section) patent license to make, have made,
|
| 77 |
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
| 78 |
+
where such license applies only to those patent claims licensable
|
| 79 |
+
by such Contributor that are necessarily infringed by their
|
| 80 |
+
Contribution(s) alone or by combination of their Contribution(s)
|
| 81 |
+
with the Work to which such Contribution(s) was submitted. If You
|
| 82 |
+
institute patent litigation against any entity (including a
|
| 83 |
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
| 84 |
+
or a Contribution incorporated within the Work constitutes direct
|
| 85 |
+
or contributory patent infringement, then any patent licenses
|
| 86 |
+
granted to You under this License for that Work shall terminate
|
| 87 |
+
as of the date such litigation is filed.
|
| 88 |
+
|
| 89 |
+
4. Redistribution. You may reproduce and distribute copies of the
|
| 90 |
+
Work or Derivative Works thereof in any medium, with or without
|
| 91 |
+
modifications, and in Source or Object form, provided that You
|
| 92 |
+
meet the following conditions:
|
| 93 |
+
|
| 94 |
+
(a) You must give any other recipients of the Work or
|
| 95 |
+
Derivative Works a copy of this License; and
|
| 96 |
+
|
| 97 |
+
(b) You must cause any modified files to carry prominent notices
|
| 98 |
+
stating that You changed the files; and
|
| 99 |
+
|
| 100 |
+
(c) You must retain, in the Source form of any Derivative Works
|
| 101 |
+
that You distribute, all copyright, patent, trademark, and
|
| 102 |
+
attribution notices from the Source form of the Work,
|
| 103 |
+
excluding those notices that do not pertain to any part of
|
| 104 |
+
the Derivative Works; and
|
| 105 |
+
|
| 106 |
+
(d) If the Work includes a "NOTICE" text file as part of its
|
| 107 |
+
distribution, then any Derivative Works that You distribute must
|
| 108 |
+
include a readable copy of the attribution notices contained
|
| 109 |
+
within such NOTICE file, excluding those notices that do not
|
| 110 |
+
pertain to any part of the Derivative Works, in at least one
|
| 111 |
+
of the following places: within a NOTICE text file distributed
|
| 112 |
+
as part of the Derivative Works; within the Source form or
|
| 113 |
+
documentation, if provided along with the Derivative Works; or,
|
| 114 |
+
within a display generated by the Derivative Works, if and
|
| 115 |
+
wherever such third-party notices normally appear. The contents
|
| 116 |
+
of the NOTICE file are for informational purposes only and
|
| 117 |
+
do not modify the License. You may add Your own attribution
|
| 118 |
+
notices within Derivative Works that You distribute, alongside
|
| 119 |
+
or as an addendum to the NOTICE text from the Work, provided
|
| 120 |
+
that such additional attribution notices cannot be construed
|
| 121 |
+
as modifying the License.
|
| 122 |
+
|
| 123 |
+
You may add Your own copyright statement to Your modifications and
|
| 124 |
+
may provide additional or different license terms and conditions
|
| 125 |
+
for use, reproduction, or distribution of Your modifications, or
|
| 126 |
+
for any such Derivative Works as a whole, provided Your use,
|
| 127 |
+
reproduction, and distribution of the Work otherwise complies with
|
| 128 |
+
the conditions stated in this License.
|
| 129 |
+
|
| 130 |
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
| 131 |
+
any Contribution intentionally submitted for inclusion in the Work
|
| 132 |
+
by You to the Licensor shall be under the terms and conditions of
|
| 133 |
+
this License, without any additional terms or conditions.
|
| 134 |
+
Notwithstanding the above, nothing herein shall supersede or modify
|
| 135 |
+
the terms of any separate license agreement you may have executed
|
| 136 |
+
with Licensor regarding such Contributions.
|
| 137 |
+
|
| 138 |
+
6. Trademarks. This License does not grant permission to use the trade
|
| 139 |
+
names, trademarks, service marks, or product names of the Licensor,
|
| 140 |
+
except as required for reasonable and customary use in describing the
|
| 141 |
+
origin of the Work and reproducing the content of the NOTICE file.
|
| 142 |
+
|
| 143 |
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
| 144 |
+
agreed to in writing, Licensor provides the Work (and each
|
| 145 |
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
| 146 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
| 147 |
+
implied, including, without limitation, any warranties or conditions
|
| 148 |
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
| 149 |
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
| 150 |
+
appropriateness of using or redistributing the Work and assume any
|
| 151 |
+
risks associated with Your exercise of permissions under this License.
|
| 152 |
+
|
| 153 |
+
8. Limitation of Liability. In no event and under no legal theory,
|
| 154 |
+
whether in tort (including negligence), contract, or otherwise,
|
| 155 |
+
unless required by applicable law (such as deliberate and grossly
|
| 156 |
+
negligent acts) or agreed to in writing, shall any Contributor be
|
| 157 |
+
liable to You for damages, including any direct, indirect, special,
|
| 158 |
+
incidental, or consequential damages of any character arising as a
|
| 159 |
+
result of this License or out of the use or inability to use the
|
| 160 |
+
Work (including but not limited to damages for loss of goodwill,
|
| 161 |
+
work stoppage, computer failure or malfunction, or any and all
|
| 162 |
+
other commercial damages or losses), even if such Contributor
|
| 163 |
+
has been advised of the possibility of such damages.
|
| 164 |
+
|
| 165 |
+
9. Accepting Warranty or Additional Liability. While redistributing
|
| 166 |
+
the Work or Derivative Works thereof, You may choose to offer,
|
| 167 |
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
| 168 |
+
or other liability obligations and/or rights consistent with this
|
| 169 |
+
License. However, in accepting such obligations, You may act only
|
| 170 |
+
on Your own behalf and on Your sole responsibility, not on behalf
|
| 171 |
+
of any other Contributor, and only if You agree to indemnify,
|
| 172 |
+
defend, and hold each Contributor harmless for any liability
|
| 173 |
+
incurred by, or claims asserted against, such Contributor by reason
|
| 174 |
+
of your accepting any such warranty or additional liability.
|
| 175 |
+
|
| 176 |
+
END OF TERMS AND CONDITIONS
|
| 177 |
+
|
| 178 |
+
APPENDIX: How to apply the Apache License to your work.
|
| 179 |
+
|
| 180 |
+
To apply the Apache License to your work, attach the following
|
| 181 |
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
| 182 |
+
replaced with your own identifying information. (Don't include
|
| 183 |
+
the brackets!) The text should be enclosed in the appropriate
|
| 184 |
+
comment syntax for the file format. We also recommend that a
|
| 185 |
+
file or class name and description of purpose be included on the
|
| 186 |
+
same "printed page" as the copyright notice for easier
|
| 187 |
+
identification within third-party archives.
|
| 188 |
+
|
| 189 |
+
Copyright 2023 HI-Diff Authors
|
| 190 |
+
|
| 191 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
| 192 |
+
you may not use this file except in compliance with the License.
|
| 193 |
+
You may obtain a copy of the License at
|
| 194 |
+
|
| 195 |
+
http://www.apache.org/licenses/LICENSE-2.0
|
| 196 |
+
|
| 197 |
+
Unless required by applicable law or agreed to in writing, software
|
| 198 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
| 199 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 200 |
+
See the License for the specific language governing permissions and
|
| 201 |
+
limitations under the License.
|
baselines/licenses/LICENSE-p2phd.txt
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Copyright (C) 2019 NVIDIA Corporation. Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu.
|
| 2 |
+
BSD License. All rights reserved.
|
| 3 |
+
|
| 4 |
+
Redistribution and use in source and binary forms, with or without
|
| 5 |
+
modification, are permitted provided that the following conditions are met:
|
| 6 |
+
|
| 7 |
+
* Redistributions of source code must retain the above copyright notice, this
|
| 8 |
+
list of conditions and the following disclaimer.
|
| 9 |
+
|
| 10 |
+
* Redistributions in binary form must reproduce the above copyright notice,
|
| 11 |
+
this list of conditions and the following disclaimer in the documentation
|
| 12 |
+
and/or other materials provided with the distribution.
|
| 13 |
+
|
| 14 |
+
THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL
|
| 15 |
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR ANY PARTICULAR PURPOSE.
|
| 16 |
+
IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL
|
| 17 |
+
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
|
| 18 |
+
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING
|
| 19 |
+
OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
--------------------------- LICENSE FOR pytorch-CycleGAN-and-pix2pix ----------------
|
| 23 |
+
Copyright (c) 2017, Jun-Yan Zhu and Taesung Park
|
| 24 |
+
All rights reserved.
|
| 25 |
+
|
| 26 |
+
Redistribution and use in source and binary forms, with or without
|
| 27 |
+
modification, are permitted provided that the following conditions are met:
|
| 28 |
+
|
| 29 |
+
* Redistributions of source code must retain the above copyright notice, this
|
| 30 |
+
list of conditions and the following disclaimer.
|
| 31 |
+
|
| 32 |
+
* Redistributions in binary form must reproduce the above copyright notice,
|
| 33 |
+
this list of conditions and the following disclaimer in the documentation
|
| 34 |
+
and/or other materials provided with the distribution.
|
| 35 |
+
|
| 36 |
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
| 37 |
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
| 38 |
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
| 39 |
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
| 40 |
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
| 41 |
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
| 42 |
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
| 43 |
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
| 44 |
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
| 45 |
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
baselines/licenses/LICENSE-pix2pix.txt
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Copyright (c) 2017, Jun-Yan Zhu and Taesung Park
|
| 2 |
+
All rights reserved.
|
| 3 |
+
|
| 4 |
+
Redistribution and use in source and binary forms, with or without
|
| 5 |
+
modification, are permitted provided that the following conditions are met:
|
| 6 |
+
|
| 7 |
+
* Redistributions of source code must retain the above copyright notice, this
|
| 8 |
+
list of conditions and the following disclaimer.
|
| 9 |
+
|
| 10 |
+
* Redistributions in binary form must reproduce the above copyright notice,
|
| 11 |
+
this list of conditions and the following disclaimer in the documentation
|
| 12 |
+
and/or other materials provided with the distribution.
|
| 13 |
+
|
| 14 |
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
| 15 |
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
| 16 |
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
| 17 |
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
| 18 |
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
| 19 |
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
| 20 |
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
| 21 |
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
| 22 |
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
| 23 |
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
--------------------------- LICENSE FOR pix2pix --------------------------------
|
| 27 |
+
BSD License
|
| 28 |
+
|
| 29 |
+
For pix2pix software
|
| 30 |
+
Copyright (c) 2016, Phillip Isola and Jun-Yan Zhu
|
| 31 |
+
All rights reserved.
|
| 32 |
+
|
| 33 |
+
Redistribution and use in source and binary forms, with or without
|
| 34 |
+
modification, are permitted provided that the following conditions are met:
|
| 35 |
+
|
| 36 |
+
* Redistributions of source code must retain the above copyright notice, this
|
| 37 |
+
list of conditions and the following disclaimer.
|
| 38 |
+
|
| 39 |
+
* Redistributions in binary form must reproduce the above copyright notice,
|
| 40 |
+
this list of conditions and the following disclaimer in the documentation
|
| 41 |
+
and/or other materials provided with the distribution.
|
| 42 |
+
|
| 43 |
+
----------------------------- LICENSE FOR DCGAN --------------------------------
|
| 44 |
+
BSD License
|
| 45 |
+
|
| 46 |
+
For dcgan.torch software
|
| 47 |
+
|
| 48 |
+
Copyright (c) 2015, Facebook, Inc. All rights reserved.
|
| 49 |
+
|
| 50 |
+
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
|
| 51 |
+
|
| 52 |
+
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
|
| 53 |
+
|
| 54 |
+
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
|
| 55 |
+
|
| 56 |
+
Neither the name Facebook nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
|
| 57 |
+
|
| 58 |
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
baselines/licenses/LICENSE-resshift.txt
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
S-Lab License 1.0
|
| 2 |
+
|
| 3 |
+
Copyright 2022 S-Lab
|
| 4 |
+
|
| 5 |
+
Redistribution and use for non-commercial purpose in source and
|
| 6 |
+
binary forms, with or without modification, are permitted provided
|
| 7 |
+
that the following conditions are met:
|
| 8 |
+
|
| 9 |
+
1. Redistributions of source code must retain the above copyright
|
| 10 |
+
notice, this list of conditions and the following disclaimer.
|
| 11 |
+
|
| 12 |
+
2. Redistributions in binary form must reproduce the above copyright
|
| 13 |
+
notice, this list of conditions and the following disclaimer in
|
| 14 |
+
the documentation and/or other materials provided with the
|
| 15 |
+
distribution.
|
| 16 |
+
|
| 17 |
+
3. Neither the name of the copyright holder nor the names of its
|
| 18 |
+
contributors may be used to endorse or promote products derived
|
| 19 |
+
from this software without specific prior written permission.
|
| 20 |
+
|
| 21 |
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
| 22 |
+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
| 23 |
+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
| 24 |
+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
| 25 |
+
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
| 26 |
+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
| 27 |
+
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
| 28 |
+
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
| 29 |
+
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
| 30 |
+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
| 31 |
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
| 32 |
+
|
| 33 |
+
In the event that redistribution and/or use for commercial purpose in
|
| 34 |
+
source or binary forms, with or without modification is required,
|
| 35 |
+
please contact the contributor(s) of the work.
|
baselines/licenses/LICENSE-sd21ft.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 KAIST VICLab
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
baselines/licenses/LICENSE-spade.txt
ADDED
|
@@ -0,0 +1,177 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
## creative commons
|
| 2 |
+
|
| 3 |
+
# Attribution-NonCommercial-ShareAlike 4.0 International
|
| 4 |
+
|
| 5 |
+
Creative Commons Corporation (“Creative Commons”) is not a law firm and does not provide legal services or legal advice. Distribution of Creative Commons public licenses does not create a lawyer-client or other relationship. Creative Commons makes its licenses and related information available on an “as-is” basis. Creative Commons gives no warranties regarding its licenses, any material licensed under their terms and conditions, or any related information. Creative Commons disclaims all liability for damages resulting from their use to the fullest extent possible.
|
| 6 |
+
|
| 7 |
+
### Using Creative Commons Public Licenses
|
| 8 |
+
|
| 9 |
+
Creative Commons public licenses provide a standard set of terms and conditions that creators and other rights holders may use to share original works of authorship and other material subject to copyright and certain other rights specified in the public license below. The following considerations are for informational purposes only, are not exhaustive, and do not form part of our licenses.
|
| 10 |
+
|
| 11 |
+
* __Considerations for licensors:__ Our public licenses are intended for use by those authorized to give the public permission to use material in ways otherwise restricted by copyright and certain other rights. Our licenses are irrevocable. Licensors should read and understand the terms and conditions of the license they choose before applying it. Licensors should also secure all rights necessary before applying our licenses so that the public can reuse the material as expected. Licensors should clearly mark any material not subject to the license. This includes other CC-licensed material, or material used under an exception or limitation to copyright. [More considerations for licensors](http://wiki.creativecommons.org/Considerations_for_licensors_and_licensees#Considerations_for_licensors).
|
| 12 |
+
|
| 13 |
+
* __Considerations for the public:__ By using one of our public licenses, a licensor grants the public permission to use the licensed material under specified terms and conditions. If the licensor’s permission is not necessary for any reason–for example, because of any applicable exception or limitation to copyright–then that use is not regulated by the license. Our licenses grant only permissions under copyright and certain other rights that a licensor has authority to grant. Use of the licensed material may still be restricted for other reasons, including because others have copyright or other rights in the material. A licensor may make special requests, such as asking that all changes be marked or described. Although not required by our licenses, you are encouraged to respect those requests where reasonable. [More considerations for the public](http://wiki.creativecommons.org/Considerations_for_licensors_and_licensees#Considerations_for_licensees).
|
| 14 |
+
|
| 15 |
+
## Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License
|
| 16 |
+
|
| 17 |
+
By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions.
|
| 18 |
+
|
| 19 |
+
### Section 1 – Definitions.
|
| 20 |
+
|
| 21 |
+
a. __Adapted Material__ means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image.
|
| 22 |
+
|
| 23 |
+
b. __Adapter's License__ means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License.
|
| 24 |
+
|
| 25 |
+
c. __BY-NC-SA Compatible License__ means a license listed at [creativecommons.org/compatiblelicenses](http://creativecommons.org/compatiblelicenses), approved by Creative Commons as essentially the equivalent of this Public License.
|
| 26 |
+
|
| 27 |
+
d. __Copyright and Similar Rights__ means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights.
|
| 28 |
+
|
| 29 |
+
e. __Effective Technological Measures__ means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements.
|
| 30 |
+
|
| 31 |
+
f. __Exceptions and Limitations__ means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material.
|
| 32 |
+
|
| 33 |
+
g. __License Elements__ means the license attributes listed in the name of a Creative Commons Public License. The License Elements of this Public License are Attribution, NonCommercial, and ShareAlike.
|
| 34 |
+
|
| 35 |
+
h. __Licensed Material__ means the artistic or literary work, database, or other material to which the Licensor applied this Public License.
|
| 36 |
+
|
| 37 |
+
i. __Licensed Rights__ means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license.
|
| 38 |
+
|
| 39 |
+
h. __Licensor__ means the individual(s) or entity(ies) granting rights under this Public License.
|
| 40 |
+
|
| 41 |
+
i. __NonCommercial__ means not primarily intended for or directed towards commercial advantage or monetary compensation. For purposes of this Public License, the exchange of the Licensed Material for other material subject to Copyright and Similar Rights by digital file-sharing or similar means is NonCommercial provided there is no payment of monetary compensation in connection with the exchange.
|
| 42 |
+
|
| 43 |
+
j. __Share__ means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them.
|
| 44 |
+
|
| 45 |
+
k. __Sui Generis Database Rights__ means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world.
|
| 46 |
+
|
| 47 |
+
l. __You__ means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning.
|
| 48 |
+
|
| 49 |
+
### Section 2 – Scope.
|
| 50 |
+
|
| 51 |
+
a. ___License grant.___
|
| 52 |
+
|
| 53 |
+
1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to:
|
| 54 |
+
|
| 55 |
+
A. reproduce and Share the Licensed Material, in whole or in part, for NonCommercial purposes only; and
|
| 56 |
+
|
| 57 |
+
B. produce, reproduce, and Share Adapted Material for NonCommercial purposes only.
|
| 58 |
+
|
| 59 |
+
2. __Exceptions and Limitations.__ For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions.
|
| 60 |
+
|
| 61 |
+
3. __Term.__ The term of this Public License is specified in Section 6(a).
|
| 62 |
+
|
| 63 |
+
4. __Media and formats; technical modifications allowed.__ The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a)(4) never produces Adapted Material.
|
| 64 |
+
|
| 65 |
+
5. __Downstream recipients.__
|
| 66 |
+
|
| 67 |
+
A. __Offer from the Licensor – Licensed Material.__ Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License.
|
| 68 |
+
|
| 69 |
+
B. __Additional offer from the Licensor – Adapted Material.__ Every recipient of Adapted Material from You automatically receives an offer from the Licensor to exercise the Licensed Rights in the Adapted Material under the conditions of the Adapter’s License You apply.
|
| 70 |
+
|
| 71 |
+
C. __No downstream restrictions.__ You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material.
|
| 72 |
+
|
| 73 |
+
6. __No endorsement.__ Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i).
|
| 74 |
+
|
| 75 |
+
b. ___Other rights.___
|
| 76 |
+
|
| 77 |
+
1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise.
|
| 78 |
+
|
| 79 |
+
2. Patent and trademark rights are not licensed under this Public License.
|
| 80 |
+
|
| 81 |
+
3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties, including when the Licensed Material is used other than for NonCommercial purposes.
|
| 82 |
+
|
| 83 |
+
### Section 3 – License Conditions.
|
| 84 |
+
|
| 85 |
+
Your exercise of the Licensed Rights is expressly made subject to the following conditions.
|
| 86 |
+
|
| 87 |
+
a. ___Attribution.___
|
| 88 |
+
|
| 89 |
+
1. If You Share the Licensed Material (including in modified form), You must:
|
| 90 |
+
|
| 91 |
+
A. retain the following if it is supplied by the Licensor with the Licensed Material:
|
| 92 |
+
|
| 93 |
+
i. identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated);
|
| 94 |
+
|
| 95 |
+
ii. a copyright notice;
|
| 96 |
+
|
| 97 |
+
iii. a notice that refers to this Public License;
|
| 98 |
+
|
| 99 |
+
iv. a notice that refers to the disclaimer of warranties;
|
| 100 |
+
|
| 101 |
+
v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable;
|
| 102 |
+
|
| 103 |
+
B. indicate if You modified the Licensed Material and retain an indication of any previous modifications; and
|
| 104 |
+
|
| 105 |
+
C. indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License.
|
| 106 |
+
|
| 107 |
+
2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information.
|
| 108 |
+
|
| 109 |
+
3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable.
|
| 110 |
+
|
| 111 |
+
b. ___ShareAlike.___
|
| 112 |
+
|
| 113 |
+
In addition to the conditions in Section 3(a), if You Share Adapted Material You produce, the following conditions also apply.
|
| 114 |
+
|
| 115 |
+
1. The Adapter’s License You apply must be a Creative Commons license with the same License Elements, this version or later, or a BY-NC-SA Compatible License.
|
| 116 |
+
|
| 117 |
+
2. You must include the text of, or the URI or hyperlink to, the Adapter's License You apply. You may satisfy this condition in any reasonable manner based on the medium, means, and context in which You Share Adapted Material.
|
| 118 |
+
|
| 119 |
+
3. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, Adapted Material that restrict exercise of the rights granted under the Adapter's License You apply.
|
| 120 |
+
|
| 121 |
+
### Section 4 – Sui Generis Database Rights.
|
| 122 |
+
|
| 123 |
+
Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material:
|
| 124 |
+
|
| 125 |
+
a. for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database for NonCommercial purposes only;
|
| 126 |
+
|
| 127 |
+
b. if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material, including for purposes of Section 3(b); and
|
| 128 |
+
|
| 129 |
+
c. You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database.
|
| 130 |
+
|
| 131 |
+
For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights.
|
| 132 |
+
|
| 133 |
+
### Section 5 – Disclaimer of Warranties and Limitation of Liability.
|
| 134 |
+
|
| 135 |
+
a. __Unless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.__
|
| 136 |
+
|
| 137 |
+
b. __To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.__
|
| 138 |
+
|
| 139 |
+
c. The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.
|
| 140 |
+
|
| 141 |
+
### Section 6 – Term and Termination.
|
| 142 |
+
|
| 143 |
+
a. This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically.
|
| 144 |
+
|
| 145 |
+
b. Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates:
|
| 146 |
+
|
| 147 |
+
1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or
|
| 148 |
+
|
| 149 |
+
2. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or
|
| 150 |
+
|
| 151 |
+
For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License.
|
| 152 |
+
|
| 153 |
+
c. For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License.
|
| 154 |
+
|
| 155 |
+
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public License.
|
| 156 |
+
|
| 157 |
+
### Section 7 – Other Terms and Conditions.
|
| 158 |
+
|
| 159 |
+
a. The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed.
|
| 160 |
+
|
| 161 |
+
b. Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License.
|
| 162 |
+
|
| 163 |
+
### Section 8 – Interpretation.
|
| 164 |
+
|
| 165 |
+
a. For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License.
|
| 166 |
+
|
| 167 |
+
b. To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions.
|
| 168 |
+
|
| 169 |
+
c. No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor.
|
| 170 |
+
|
| 171 |
+
d. Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority.
|
| 172 |
+
|
| 173 |
+
```
|
| 174 |
+
Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances will be considered the “Licensor.” Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as otherwise permitted by the Creative Commons policies published at [creativecommons.org/policies](http://creativecommons.org/policies), Creative Commons does not authorize the use of the trademark “Creative Commons” or any other trademark or logo of Creative Commons without its prior written consent including, without limitation, in connection with any unauthorized modifications to any of its public licenses or any other arrangements, understandings, or agreements concerning use of licensed material. For the avoidance of doubt, this paragraph does not form part of the public licenses.
|
| 175 |
+
|
| 176 |
+
Creative Commons may be contacted at [creativecommons.org](http://creativecommons.org/).
|
| 177 |
+
```
|
baselines/licenses/NO-UPSTREAM-LICENSE.md
ADDED
|
@@ -0,0 +1,189 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Four upstream code bases publish no licence
|
| 2 |
+
|
| 3 |
+
This file records, per method, **exactly what we checked and exactly what we
|
| 4 |
+
found**. It exists because four of the fifteen comparison methods in this
|
| 5 |
+
repository are derived from upstream code that grants no licence at all, and a
|
| 6 |
+
reader of these weights is entitled to that fact stated plainly rather than
|
| 7 |
+
inferred from a missing file.
|
| 8 |
+
|
| 9 |
+
The four are **StegoGAN**, **E3Diff**, **DDPM (SR3-class)** — which is the E3Diff
|
| 10 |
+
code base run in its stage-1 configuration — and **Conditional Diffusion
|
| 11 |
+
(CondDiff)**.
|
| 12 |
+
|
| 13 |
+
The checkpoints derived from them **are published here anyway**, as a deliberate
|
| 14 |
+
decision, so that the benchmark is reproducible end to end. We are not asserting
|
| 15 |
+
that a licence exists, and we are not granting one. Under default copyright, all
|
| 16 |
+
rights in the upstream code remain reserved by its authors, and **no express
|
| 17 |
+
permission to redistribute derived work has been granted to us or to you**. If
|
| 18 |
+
you intend to redistribute these four checkpoints or build on them, assess that
|
| 19 |
+
for yourself, and consider asking the upstream authors directly.
|
| 20 |
+
|
| 21 |
+
Everything below was checked on **2026-08-28**.
|
| 22 |
+
|
| 23 |
+
---
|
| 24 |
+
|
| 25 |
+
## What "checked" means
|
| 26 |
+
|
| 27 |
+
For each upstream we ran three tests. Where a test does not apply (for example,
|
| 28 |
+
a vendored tree that is not a git clone has no tracked-file list), that is stated
|
| 29 |
+
rather than skipped silently.
|
| 30 |
+
|
| 31 |
+
1. **Filesystem.** `find <repo> -maxdepth 3 \( -iname '*licen*' -o -iname
|
| 32 |
+
'COPYING*' -o -iname 'NOTICE*' \)` over the vendored working tree.
|
| 33 |
+
2. **Version control.** `git ls-files | grep -iE 'licen|copying|notice'` at the
|
| 34 |
+
pinned commit, which catches a licence file that exists upstream but was
|
| 35 |
+
deleted locally.
|
| 36 |
+
3. **Upstream host.** The GitHub REST API for the repository
|
| 37 |
+
(`license` field) and its `/license` endpoint.
|
| 38 |
+
4. **README.** `grep -inE 'licen|copyright|all rights reserved|commercial'`
|
| 39 |
+
over the repository README, because some projects state terms in prose
|
| 40 |
+
instead of in a file.
|
| 41 |
+
|
| 42 |
+
---
|
| 43 |
+
|
| 44 |
+
## StegoGAN
|
| 45 |
+
|
| 46 |
+
*StegoGAN: Leveraging Steganography for Non-Bijective Image-to-Image
|
| 47 |
+
Translation*, Wu, Chen, Mermet, Hurni, Schindler, Gonthier and Landrieu,
|
| 48 |
+
CVPR 2024.
|
| 49 |
+
|
| 50 |
+
| test | result |
|
| 51 |
+
|---|---|
|
| 52 |
+
| upstream | `https://github.com/sian-wusidi/StegoGAN` (the official CVPR 2024 code) |
|
| 53 |
+
| pinned commit | `cad61997c0f82793444f60f81298142b80cdf3c1` |
|
| 54 |
+
| filesystem, maxdepth 3 | **no match** — no LICENSE, LICENCE, COPYING or NOTICE at any depth |
|
| 55 |
+
| tracked files at that commit | **0 of 52** tracked paths are licence-shaped |
|
| 56 |
+
| GitHub API | repository exists; `"license": null`; the `/license` endpoint returns **HTTP 404** |
|
| 57 |
+
| README | **no match** for `licen`, `copyright`, `all rights reserved` or `commercial` — the README has no licence section |
|
| 58 |
+
|
| 59 |
+
**Lineage.** StegoGAN's README states it is built on
|
| 60 |
+
`junyanz/pytorch-CycleGAN-and-pix2pix`. The parts of the tree that are
|
| 61 |
+
unmodified CycleGAN carry that project's BSD notice, reproduced here as
|
| 62 |
+
`LICENSE-cyclegan.txt`. **That notice does not extend to StegoGAN's own
|
| 63 |
+
contributions**, which are the mismatch-mask generator (`resnet_9blocks_maskv1` /
|
| 64 |
+
`maskv3`), the fusion block and the consistency/regularisation losses — i.e.
|
| 65 |
+
precisely the parts that make the released `net_G_A.pth` what it is.
|
| 66 |
+
|
| 67 |
+
**Conclusion: no licence. Weights published by decision, with this disclosure.**
|
| 68 |
+
|
| 69 |
+
---
|
| 70 |
+
|
| 71 |
+
## E3Diff
|
| 72 |
+
|
| 73 |
+
*Efficient End-to-End Diffusion Model for One-Step SAR-to-Optical Translation*,
|
| 74 |
+
Qin, Zou, Li and Zhang, IEEE GRSL, doi:10.1109/LGRS.2024.3506566.
|
| 75 |
+
|
| 76 |
+
| test | result |
|
| 77 |
+
|---|---|
|
| 78 |
+
| upstream | `https://github.com/DeepSARRS/E3Diff` |
|
| 79 |
+
| pinned commit | `38601093ab8f8e4b478144621f20890b100a3b74` |
|
| 80 |
+
| filesystem, maxdepth 3 | **one match, and it is not a grant for E3Diff**: `SoftPool/LICENSE.txt` |
|
| 81 |
+
| tracked files at that commit | **1 of 58** — the same `SoftPool/LICENSE.txt` |
|
| 82 |
+
| GitHub API | repository exists; `"license": null`; `/license` returns **HTTP 404** |
|
| 83 |
+
| README | **no match** — no licence section |
|
| 84 |
+
|
| 85 |
+
**About the one licence file in the tree.** `SoftPool/LICENSE.txt` is the MIT
|
| 86 |
+
licence of a **vendored third-party dependency**, `alexandrosstergiou/SoftPool`,
|
| 87 |
+
`Copyright (c) 2020 Alexandros Stergiou`. It is reproduced here as
|
| 88 |
+
`LICENSE-SoftPool-MIT.txt` because that dependency's code is genuinely part of
|
| 89 |
+
what the checkpoint was trained with (E3Diff's CPEN module imports SoftPool
|
| 90 |
+
unconditionally). It is **not** a licence for E3Diff, and it must not be read
|
| 91 |
+
as one.
|
| 92 |
+
|
| 93 |
+
**Lineage.** E3Diff's README credits SR3
|
| 94 |
+
(`Janspiry/Image-Super-Resolution-via-Iterative-Refinement`, Apache-2.0),
|
| 95 |
+
`GaParmar/img2img-turbo` (MIT) and `alexandrosstergiou/SoftPool` (MIT). Those
|
| 96 |
+
licences cover the borrowed parts only, not E3Diff's own two-stage contribution.
|
| 97 |
+
We do not ship the SR3 or img2img-turbo licence texts here because neither file
|
| 98 |
+
is vendored in the tree we trained from; fetch them from those repositories if
|
| 99 |
+
you need them.
|
| 100 |
+
|
| 101 |
+
**These are our own retrained weights**, never the authors' released checkpoint.
|
| 102 |
+
|
| 103 |
+
**Conclusion: no licence. Weights published by decision, with this disclosure.**
|
| 104 |
+
|
| 105 |
+
---
|
| 106 |
+
|
| 107 |
+
## DDPM (SR3-class)
|
| 108 |
+
|
| 109 |
+
| test | result |
|
| 110 |
+
|---|---|
|
| 111 |
+
| upstream | **the same code base as E3Diff**, `https://github.com/DeepSARRS/E3Diff`, run in its stage-1 configuration (`"stage": 1`) |
|
| 112 |
+
| pinned commit | `38601093ab8f8e4b478144621f20890b100a3b74` |
|
| 113 |
+
| all four tests | **identical results to the E3Diff row above** |
|
| 114 |
+
|
| 115 |
+
This row reproduces the **SR3 method class** — an eps-prediction conditional
|
| 116 |
+
DDPM on the source image — using E3Diff's stage-1 code. It is **not** the SR3
|
| 117 |
+
authors' implementation and no code from the SR3 release was used. Its licence
|
| 118 |
+
position is therefore exactly E3Diff's: none.
|
| 119 |
+
|
| 120 |
+
**Conclusion: no licence. Weights published by decision, with this disclosure.**
|
| 121 |
+
|
| 122 |
+
---
|
| 123 |
+
|
| 124 |
+
## Conditional Diffusion (CondDiff)
|
| 125 |
+
|
| 126 |
+
*Conditional Diffusion for SAR to Optical Image Translation*, Bai, Pu and Xu,
|
| 127 |
+
IEEE GRSL, doi:10.1109/LGRS.2023.3337143.
|
| 128 |
+
|
| 129 |
+
| test | result |
|
| 130 |
+
|---|---|
|
| 131 |
+
| upstream | `https://github.com/Coordi777/Conditional-Diffusion-for-SAR-to-Optical-Image-Translation` |
|
| 132 |
+
| pinned commit | **not recoverable — see below** |
|
| 133 |
+
| filesystem, maxdepth 3 | **no match** — no LICENSE, LICENCE, COPYING or NOTICE anywhere in the vendored tree |
|
| 134 |
+
| tracked files | **test does not apply**: the vendored copy is not a git clone (no `.git`, no `.gitmodules`, no VERSION file) |
|
| 135 |
+
| GitHub API | repository exists; `"license": null`; `/license` returns **HTTP 404** |
|
| 136 |
+
| README | **no match** — no licence section |
|
| 137 |
+
|
| 138 |
+
**A second, separate reproducibility gap.** Because the vendored copy carries no
|
| 139 |
+
version control and records no upstream URL inside the tree, **the exact upstream
|
| 140 |
+
commit this checkpoint was trained from cannot be recovered.** The repository
|
| 141 |
+
URL above is recorded in our own collection script, not in the vendored code. A
|
| 142 |
+
"clone upstream, then apply our patch" recipe is therefore not possible for this
|
| 143 |
+
row; the vendored tree itself is the only source of truth.
|
| 144 |
+
|
| 145 |
+
**Lineage.** CondDiff's README states the repository is based on
|
| 146 |
+
`openai/guided-diffusion` with modifications, and the vendored tree is visibly
|
| 147 |
+
that code base — OpenAI provenance comments survive in
|
| 148 |
+
`guided_diffusion/logger.py` and `guided_diffusion/unet.py`.
|
| 149 |
+
`openai/guided-diffusion` is **MIT**. The unmodified guided-diffusion parts carry
|
| 150 |
+
that MIT licence; it does not extend to the authors' modifications. We do not
|
| 151 |
+
ship the guided-diffusion MIT text here because it is not vendored in the tree we
|
| 152 |
+
trained from — fetch it from that repository if you need it.
|
| 153 |
+
|
| 154 |
+
**We modified the code further**, and say so as a matter of discipline rather
|
| 155 |
+
than because any licence compels it:
|
| 156 |
+
|
| 157 |
+
* the distributed-init helper no longer overwrites `CUDA_VISIBLE_DEVICES`;
|
| 158 |
+
* **the sampler sorts its directory listings.** Upstream paired SAR with EO by
|
| 159 |
+
*unsorted* `os.listdir` position, i.e. by filesystem order, so any number
|
| 160 |
+
produced with the unpatched sampler is measured against effectively arbitrary
|
| 161 |
+
ground truth. This is a correctness fix, not a convenience;
|
| 162 |
+
* the noise tensor for a partial last batch is shaped from the batch, not from
|
| 163 |
+
the `--batch_size` flag;
|
| 164 |
+
* the sampler takes `--test_dir` / `--out_dir` and builds its resize transform at
|
| 165 |
+
run time, so the 512 px cell is not silently resized to 256.
|
| 166 |
+
|
| 167 |
+
All four changes are commented in place in the vendored source. Sampling uses
|
| 168 |
+
respaced DDPM with 250 steps, which is the authors' own `sample.sh` protocol;
|
| 169 |
+
the released DDIM path is broken upstream (`ddim_sample_loop` has no `condition`
|
| 170 |
+
parameter, so passing `--use_ddim True` raises `TypeError`).
|
| 171 |
+
|
| 172 |
+
**Conclusion: no licence. Weights published by decision, with this disclosure.**
|
| 173 |
+
|
| 174 |
+
---
|
| 175 |
+
|
| 176 |
+
## Summary
|
| 177 |
+
|
| 178 |
+
| method | upstream | licence file | GitHub `license` | verdict |
|
| 179 |
+
|---|---|---|---|---|
|
| 180 |
+
| StegoGAN | `sian-wusidi/StegoGAN` | none (0 of 52 tracked) | `null`, `/license` 404 | **no grant** |
|
| 181 |
+
| E3Diff | `DeepSARRS/E3Diff` | only a dependency's (`SoftPool/LICENSE.txt`, MIT) | `null`, `/license` 404 | **no grant** |
|
| 182 |
+
| DDPM (SR3-class) | same repo, stage 1 | as above | `null`, `/license` 404 | **no grant** |
|
| 183 |
+
| CondDiff | `Coordi777/Conditional-Diffusion-…` | none | `null`, `/license` 404 | **no grant** |
|
| 184 |
+
|
| 185 |
+
Every other upstream that **does** publish a licence file has its text in this
|
| 186 |
+
same directory, one file per method. The one remaining method, ControlNet, is
|
| 187 |
+
trained with `huggingface/diffusers`, whose grant is an Apache-2.0 header in the
|
| 188 |
+
trainer source rather than a repository LICENSE file, so there is no file to
|
| 189 |
+
copy; that row's weight terms are the Open RAIL++-M text at the repository root.
|
baselines/qxs-saropt/bbdm/README.md
ADDED
|
@@ -0,0 +1,136 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# BBDM — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**BBDM: Image-to-Image Translation with Brownian Bridge Diffusion Models**, Li,
|
| 4 |
+
Xue, Liu and Lai, *CVPR 2023*
|
| 5 |
+
([arXiv:2205.07680](https://arxiv.org/abs/2205.07680)). Upstream code:
|
| 6 |
+
[`xuekt98/BBDM`](https://github.com/xuekt98/BBDM) at commit
|
| 7 |
+
`02c3b13c9f9dfab0853e32123100680a0640c4ed`.
|
| 8 |
+
|
| 9 |
+
This is the latent Brownian-bridge diffusion model we retrained ourselves on
|
| 10 |
+
QXS-SAROPT, and it is the checkpoint behind the BBDM row of the ReFlowSET
|
| 11 |
+
comparison table.
|
| 12 |
+
|
| 13 |
+
## What is in this folder
|
| 14 |
+
|
| 15 |
+
| file | bytes | what it is |
|
| 16 |
+
|---|---|---|
|
| 17 |
+
| `last_model.pth` | 2,118,317,140 | the whole trained system — see below |
|
| 18 |
+
|
| 19 |
+
`last_model.pth` is a pickled dictionary, not a bare state dict. It holds:
|
| 20 |
+
|
| 21 |
+
* `['model']` — the full `LatentBrownianBridgeModel` state dict: the denoising
|
| 22 |
+
UNet **and**, embedded inside it, the frozen vq-f4 VQGAN submodule;
|
| 23 |
+
* `['ema']` — the EMA shadow weights. **Sampling uses these**, not `['model']`;
|
| 24 |
+
the runner calls `apply_ema()` before generating;
|
| 25 |
+
* `['step']` and `['epoch']`.
|
| 26 |
+
|
| 27 |
+
UNet: latent resolution 64×64, `model_channels` 128, `num_res_blocks` 2,
|
| 28 |
+
attention at resolutions 32/16/8, `channel_mult` (1, 4, 8), 8 heads,
|
| 29 |
+
64 channels per head, scale-shift normalisation, resblock up/downsampling.
|
| 30 |
+
|
| 31 |
+
The denoising UNet has `condition_key: nocond` — there is no separate
|
| 32 |
+
conditioning branch, because in BBDM the SAR image *is* one endpoint of the
|
| 33 |
+
Brownian bridge and the EO image is the other. The translation is the bridge,
|
| 34 |
+
not a conditioner.
|
| 35 |
+
|
| 36 |
+
## Training budget we used
|
| 37 |
+
|
| 38 |
+
| | |
|
| 39 |
+
|---|---|
|
| 40 |
+
| **generator updates released** | **50,000** |
|
| 41 |
+
| batch size / resolution | 32 @ 256 px |
|
| 42 |
+
| iterations per epoch | 500 |
|
| 43 |
+
| epochs | 100 |
|
| 44 |
+
| optimizer | Adam, lr 1e-4, β₁ 0.9, no weight decay; `ReduceLROnPlateau` (factor 0.5, patience 3,000, min lr 5e-7) |
|
| 45 |
+
| EMA | decay 0.995, started at step 30,000, updated every 8 steps |
|
| 46 |
+
| bridge | `mt_type` linear, objective `grad`, L1 loss, T = 1,000, `max_var` 1.0 |
|
| 47 |
+
| sampler (test) | 200 skip-sampled steps, η = 1.0, `clip_denoised` off |
|
| 48 |
+
| latent space | CompVis **vq-f4** VQGAN, frozen (`embed_dim` 3, `n_embed` 8192) |
|
| 49 |
+
| horizontal flip | disabled |
|
| 50 |
+
|
| 51 |
+
The step cap and the epoch count coincide exactly here.
|
| 52 |
+
|
| 53 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 54 |
+
|
| 55 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 56 |
+
|---|---|---|---|---|
|
| 57 |
+
| 76.6 | 0.270 | 0.568 | 0.352 | 15.34 |
|
| 58 |
+
|
| 59 |
+
Evaluated on the official split's 3,999 test chips at their native 256 px.
|
| 60 |
+
No subsampling: every metric on this page is measured over all 3,999 pairs.
|
| 61 |
+
|
| 62 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 63 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 64 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 65 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 66 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 67 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 68 |
+
convention gives a systematically different number and the two must never be
|
| 69 |
+
mixed, or compared against a paper that used the other one.
|
| 70 |
+
|
| 71 |
+
## Load it and translate one SAR image
|
| 72 |
+
|
| 73 |
+
There is no smaller entry point than the repository's own runner, because the
|
| 74 |
+
model is constructed with a VQGAN whose weights are loaded from a path in the
|
| 75 |
+
config **even though the same weights are already inside `last_model.pth`**.
|
| 76 |
+
You must have the CompVis latent-diffusion **vq-f4** checkpoint
|
| 77 |
+
(`model.ckpt`, 756,175,527 bytes) on disk and pointed at by
|
| 78 |
+
`model.VQGAN.params.ckpt_path`, or construction fails before any of this runs.
|
| 79 |
+
|
| 80 |
+
```bash
|
| 81 |
+
# from the BBDM repository root, with a config whose data.dataset_config
|
| 82 |
+
# points at your SAR / EO directories and whose VQGAN ckpt_path exists:
|
| 83 |
+
python main.py -c configs/<your>-LBBDM-f4.yaml --gpu_ids 0 \
|
| 84 |
+
-r <output dir> --sample_to_eval --resume_model last_model.pth
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
`--resume_model` sets `model.model_load_path`; the runner loads it with
|
| 88 |
+
`torch.load(..., weights_only=False)`, restores the EMA shadow and applies it
|
| 89 |
+
before sampling.
|
| 90 |
+
|
| 91 |
+
Outputs land in
|
| 92 |
+
`<output dir>/<dataset_name>/LBBDM-f4/sample_to_eval/`,
|
| 93 |
+
in three sibling directories: **`200/` holds the generated EO** (the name is the
|
| 94 |
+
`sample_step` count), `condition/` holds the SAR inputs and `ground_truth/` holds
|
| 95 |
+
the targets. Score `200/`.
|
| 96 |
+
|
| 97 |
+
## Read before using this checkpoint
|
| 98 |
+
|
| 99 |
+
* **Sampling must use the EMA weights.** `['model']` and `['ema']` are both in
|
| 100 |
+
the file and they are different networks. The reported numbers are the EMA.
|
| 101 |
+
* **You need the external vq-f4 VQGAN.** It is not redistributed here — it is
|
| 102 |
+
CompVis latent-diffusion material with its own (MIT) licence, and the config
|
| 103 |
+
requires it at construction time regardless.
|
| 104 |
+
* **`torch.load` needs `weights_only=False`.** PyTorch ≥ 2.6 flipped that
|
| 105 |
+
default; these are pickled Lightning-style checkpoints. We made that change in
|
| 106 |
+
four places in the repository (`model/VQGAN/vqgan.py`, `runners/BaseRunner.py`
|
| 107 |
+
twice, `runners/.../BBDMRunner.py`) and also removed a `verbose=True` kwarg
|
| 108 |
+
from `ReduceLROnPlateau` that PyTorch 2.x no longer accepts. All four are
|
| 109 |
+
compatibility fixes; none changes the maths.
|
| 110 |
+
* Passes the identity-collapse audit on both datasets.
|
| 111 |
+
|
| 112 |
+
## Licence — stated factually
|
| 113 |
+
|
| 114 |
+
Upstream `xuekt98/BBDM` ships a LICENSE file, reproduced verbatim here as
|
| 115 |
+
`licenses/LICENSE-bbdm.txt`: the **MIT licence**, `Copyright (c) 2023 xuekt98`.
|
| 116 |
+
|
| 117 |
+
**Redistribution of these weights is permitted.** No non-commercial clause, no
|
| 118 |
+
share-alike clause.
|
| 119 |
+
|
| 120 |
+
One dependency note that is attribution, not restriction: this checkpoint cannot
|
| 121 |
+
be run without the CompVis latent-diffusion **vq-f4** VQGAN, which is MIT and is
|
| 122 |
+
**not** redistributed here.
|
| 123 |
+
|
| 124 |
+
Attribution: *trained with `xuekt98/BBDM` (MIT, Copyright (c) 2023 xuekt98).
|
| 125 |
+
Requires the CompVis latent-diffusion vq-f4 VQGAN (MIT,
|
| 126 |
+
https://github.com/CompVis/latent-diffusion), which is not redistributed here.
|
| 127 |
+
Full text in `licenses/LICENSE-bbdm.txt`.*
|
| 128 |
+
|
| 129 |
+
---
|
| 130 |
+
|
| 131 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 132 |
+
checkpoint that we retrained ourselves on QXS-SAROPT; it is not ReFlowSET
|
| 133 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 134 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 135 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 136 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/qxs-saropt/cbbdm/README.md
ADDED
|
@@ -0,0 +1,153 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# cBBDM — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**Conditional Brownian Bridge Diffusion Model for VHR SAR to Optical Image
|
| 4 |
+
Translation**, Kim and Chung, *IEEE Geoscience and Remote Sensing Letters*, 2025
|
| 5 |
+
([arXiv:2408.07947](https://arxiv.org/abs/2408.07947); the venue and year are
|
| 6 |
+
taken from the authors' own citation block — the repository records no DOI).
|
| 7 |
+
Upstream code:
|
| 8 |
+
[`egshkim/ConditionalBBDM-for-VHR-SAR-to-Optical`](https://github.com/egshkim/ConditionalBBDM-for-VHR-SAR-to-Optical)
|
| 9 |
+
at commit `8ce15934f4d4e3f01efe70d11e2d9b9e0859210c`, a fork of
|
| 10 |
+
[`xuekt98/BBDM`](https://github.com/xuekt98/BBDM).
|
| 11 |
+
|
| 12 |
+
This is the conditional Brownian-bridge latent diffusion model we retrained
|
| 13 |
+
ourselves on QXS-SAROPT, and it is the checkpoint behind the cBBDM row of the
|
| 14 |
+
ReFlowSET comparison table.
|
| 15 |
+
|
| 16 |
+
## What is in this folder
|
| 17 |
+
|
| 18 |
+
| file | bytes | what it is |
|
| 19 |
+
|---|---|---|
|
| 20 |
+
| `last_model.pth` | 2,118,345,622 | the whole trained system — see below |
|
| 21 |
+
|
| 22 |
+
`last_model.pth` is a pickled dictionary, not a bare state dict. It holds:
|
| 23 |
+
|
| 24 |
+
* `['model']` — the full `LatentBrownianBridgeModel` state dict: the denoising
|
| 25 |
+
UNet **and**, embedded inside it, the frozen vq-f4 VQGAN submodule;
|
| 26 |
+
* `['ema']` — the EMA shadow weights. **Sampling uses these**, not `['model']`;
|
| 27 |
+
the runner calls `apply_ema()` before generating;
|
| 28 |
+
* `['step']` and `['epoch']`.
|
| 29 |
+
|
| 30 |
+
UNet: latent resolution 64×64, `model_channels` 128, `num_res_blocks` 2,
|
| 31 |
+
attention at resolutions 32/16/8, `channel_mult` (1, 4, 8), 8 heads,
|
| 32 |
+
64 channels per head, scale-shift normalisation, resblock up/downsampling.
|
| 33 |
+
|
| 34 |
+
The denoising UNet differs from plain BBDM in exactly two ways, and they are
|
| 35 |
+
the method: **`in_channels` is 6, not 3**, and **`condition_key` is
|
| 36 |
+
`SpatialRescaler`** — a small trained module that rescales the SAR latent and
|
| 37 |
+
concatenates it to the bridge state at every step, so the SAR conditioning is
|
| 38 |
+
explicit rather than implicit in the bridge endpoints. The SpatialRescaler is
|
| 39 |
+
trained (it is in the model's parameter list) and is inside this file.
|
| 40 |
+
|
| 41 |
+
## Training budget we used
|
| 42 |
+
|
| 43 |
+
| | |
|
| 44 |
+
|---|---|
|
| 45 |
+
| **generator updates released** | **50,000** |
|
| 46 |
+
| batch size / resolution | 32 @ 256 px |
|
| 47 |
+
| iterations per epoch | 500 |
|
| 48 |
+
| epochs | 100 |
|
| 49 |
+
| optimizer | Adam, lr 1e-4, β₁ 0.9, no weight decay; `ReduceLROnPlateau` (factor 0.5, patience 3,000, min lr 5e-7) |
|
| 50 |
+
| EMA | decay 0.995, started at step 30,000, updated every 8 steps |
|
| 51 |
+
| bridge | `mt_type` linear, objective `grad`, L1 loss, T = 1,000, `max_var` 1.0 |
|
| 52 |
+
| sampler (test) | 200 skip-sampled steps, η = 1.0, `clip_denoised` off |
|
| 53 |
+
| latent space | CompVis **vq-f4** VQGAN, frozen (`embed_dim` 3, `n_embed` 8192) |
|
| 54 |
+
| horizontal flip | disabled |
|
| 55 |
+
|
| 56 |
+
The step cap and the epoch count coincide exactly here.
|
| 57 |
+
|
| 58 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 59 |
+
|
| 60 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 61 |
+
|---|---|---|---|---|
|
| 62 |
+
| 50.6 | 0.246 | 0.539 | 0.372 | 16.02 |
|
| 63 |
+
|
| 64 |
+
Evaluated on the official split's 3,999 test chips at their native 256 px.
|
| 65 |
+
No subsampling: every metric on this page is measured over all 3,999 pairs.
|
| 66 |
+
|
| 67 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 68 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 69 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 70 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 71 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 72 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 73 |
+
convention gives a systematically different number and the two must never be
|
| 74 |
+
mixed, or compared against a paper that used the other one.
|
| 75 |
+
|
| 76 |
+
## Load it and translate one SAR image
|
| 77 |
+
|
| 78 |
+
There is no smaller entry point than the repository's own runner, because the
|
| 79 |
+
model is constructed with a VQGAN whose weights are loaded from a path in the
|
| 80 |
+
config **even though the same weights are already inside `last_model.pth`**.
|
| 81 |
+
You must have the CompVis latent-diffusion **vq-f4** checkpoint
|
| 82 |
+
(`model.ckpt`, 756,175,527 bytes) on disk and pointed at by
|
| 83 |
+
`model.VQGAN.params.ckpt_path`, or construction fails before any of this runs.
|
| 84 |
+
|
| 85 |
+
```bash
|
| 86 |
+
# from the cBBDM repository root, with a config whose data.dataset_config
|
| 87 |
+
# points at your SAR / EO directories and whose VQGAN ckpt_path exists:
|
| 88 |
+
python main.py -c configs/<your>-cBBDM-f4.yaml --gpu_ids 0 \
|
| 89 |
+
-r <output dir> --sample_to_eval --resume_model last_model.pth
|
| 90 |
+
```
|
| 91 |
+
|
| 92 |
+
`--resume_model` sets `model.model_load_path`; the runner loads it with
|
| 93 |
+
`torch.load(..., weights_only=False)`, restores the EMA shadow and applies it
|
| 94 |
+
before sampling.
|
| 95 |
+
|
| 96 |
+
Outputs land in
|
| 97 |
+
`<output dir>/<dataset_name>/cBBDM-f4/sample_to_eval/`,
|
| 98 |
+
in three sibling directories: **`200/` holds the generated EO** (the name is the
|
| 99 |
+
`sample_step` count), `condition/` holds the SAR inputs and `ground_truth/` holds
|
| 100 |
+
the targets. Score `200/`.
|
| 101 |
+
|
| 102 |
+
## Read before using this checkpoint
|
| 103 |
+
|
| 104 |
+
* **Sampling must use the EMA weights.** `['model']` and `['ema']` are both in
|
| 105 |
+
the file and they are different networks. The reported numbers are the EMA.
|
| 106 |
+
* **You need the external vq-f4 VQGAN.** It is not redistributed here — it is
|
| 107 |
+
CompVis latent-diffusion material with its own (MIT) licence, and the config
|
| 108 |
+
requires it at construction time regardless.
|
| 109 |
+
* **`torch.load` needs `weights_only=False`.** PyTorch ≥ 2.6 flipped that
|
| 110 |
+
default; these are pickled Lightning-style checkpoints. We made that change in
|
| 111 |
+
four places in the repository (`model/VQGAN/vqgan.py`, `runners/BaseRunner.py`
|
| 112 |
+
twice, `runners/.../BBDMRunner.py`) and also removed a `verbose=True` kwarg
|
| 113 |
+
from `ReduceLROnPlateau` that PyTorch 2.x no longer accepts. All four are
|
| 114 |
+
compatibility fixes; none changes the maths.
|
| 115 |
+
* **A performance patch we applied, and when.** This fork regressed the
|
| 116 |
+
DataLoader worker count from upstream BBDM's 8 to 0, which dominated wall-clock
|
| 117 |
+
time (one 32-pair batch: 6.5 s serial versus 1.1 s across 8 threads). We
|
| 118 |
+
restored `num_workers=8`. **Both cells published here were trained *before*
|
| 119 |
+
that change, i.e. with 0 workers** — same data, same order, same step count,
|
| 120 |
+
only slower. The numbers on this page are unaffected either way: worker count
|
| 121 |
+
cannot change which samples the model sees.
|
| 122 |
+
* Passes the identity-collapse audit on both datasets.
|
| 123 |
+
|
| 124 |
+
## Licence — stated factually
|
| 125 |
+
|
| 126 |
+
Upstream `egshkim/ConditionalBBDM-for-VHR-SAR-to-Optical` ships a LICENSE file,
|
| 127 |
+
reproduced verbatim here as `licenses/LICENSE-cbbdm.txt`: the **MIT licence**,
|
| 128 |
+
`Copyright (c) 2025 egshkim`. Its README adds: *"The original source code is
|
| 129 |
+
licensed under the MIT License, and this repository continues to comply with its
|
| 130 |
+
terms"* — the original being `xuekt98/BBDM`, also MIT
|
| 131 |
+
(`licenses/LICENSE-bbdm.txt`).
|
| 132 |
+
|
| 133 |
+
**Redistribution of these weights is permitted.** No non-commercial clause, no
|
| 134 |
+
share-alike clause.
|
| 135 |
+
|
| 136 |
+
One dependency note that is attribution, not restriction: this checkpoint cannot
|
| 137 |
+
be run without the CompVis latent-diffusion **vq-f4** VQGAN, which is MIT and is
|
| 138 |
+
**not** redistributed here.
|
| 139 |
+
|
| 140 |
+
Attribution: *trained with `egshkim/ConditionalBBDM-for-VHR-SAR-to-Optical`
|
| 141 |
+
(MIT, Copyright (c) 2025 egshkim), a fork of `xuekt98/BBDM` (MIT, Copyright (c)
|
| 142 |
+
2023 xuekt98). Requires the CompVis vq-f4 VQGAN (MIT,
|
| 143 |
+
https://github.com/CompVis/latent-diffusion), not redistributed here. Full texts
|
| 144 |
+
in `licenses/LICENSE-cbbdm.txt` and `licenses/LICENSE-bbdm.txt`.*
|
| 145 |
+
|
| 146 |
+
---
|
| 147 |
+
|
| 148 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 149 |
+
checkpoint that we retrained ourselves on QXS-SAROPT; it is not ReFlowSET
|
| 150 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 151 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 152 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 153 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/qxs-saropt/cdiffset/README.md
ADDED
|
@@ -0,0 +1,164 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# C-DiffSET — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**C-DiffSET: Leveraging Latent Diffusion for SAR-to-EO Image Translation with
|
| 4 |
+
Confidence-Guided Reliable Object Generation**, Do, Lee, Lee and Kim,
|
| 5 |
+
*IEEE Transactions on Circuits and Systems for Video Technology*, 2026
|
| 6 |
+
([doi:10.1109/TCSVT.2026.3701447](https://doi.org/10.1109/TCSVT.2026.3701447),
|
| 7 |
+
[arXiv:2411.10788](https://arxiv.org/abs/2411.10788)). Upstream code:
|
| 8 |
+
[`KAIST-VICLab/C-DiffSET`](https://github.com/KAIST-VICLab/C-DiffSET).
|
| 9 |
+
|
| 10 |
+
This is the **stage-2 (confidence-guided) UNet** we retrained ourselves on
|
| 11 |
+
QXS-SAROPT, and it is the checkpoint behind the C-DiffSET row of the ReFlowSET
|
| 12 |
+
comparison table.
|
| 13 |
+
|
| 14 |
+
## What is in this folder
|
| 15 |
+
|
| 16 |
+
A single `UNet2DConditionModel` (865,925,125 parameters, fp32, 686 tensors) with
|
| 17 |
+
two channel modifications relative to stock SD 2.1-base:
|
| 18 |
+
|
| 19 |
+
* **`conv_in` 4 -> 8 channels.** The UNet is conditioned by channel
|
| 20 |
+
concatenation in latent space, **SAR first**:
|
| 21 |
+
`torch.cat([sar_latent, noisy_eo_latent], dim=1)`. The extra half was
|
| 22 |
+
initialised by duplicating the pretrained kernel and halving it; in this
|
| 23 |
+
trained checkpoint the two halves have clearly diverged
|
| 24 |
+
(mean |w| 0.01424 on channels 0-3 vs 0.02810 on channels 4-7).
|
| 25 |
+
* **`conv_out` 4 -> 5 channels.** Channels 0-3 are the usual epsilon
|
| 26 |
+
prediction; channel 4 is a **raw spatial variance** used by the
|
| 27 |
+
confidence-guided loss. It was zero-initialised and is no longer zero
|
| 28 |
+
(mean |w| 0.02922), i.e. the head really trained. To turn it into a variance:
|
| 29 |
+
`var = clamp(softplus(raw + c) + 1e-6, 1e-6, 10)` where the offset
|
| 30 |
+
`c = log(exp(1 - 1e-6) - 1)` is the inverse softplus of the initial variance,
|
| 31 |
+
and confidence is `1 / var`. **The scheduler must only ever see channels 0-3.**
|
| 32 |
+
|
| 33 |
+
There is no VAE, no text encoder and no scheduler here — take those from the
|
| 34 |
+
base model (below).
|
| 35 |
+
|
| 36 |
+
## Training budget we used
|
| 37 |
+
|
| 38 |
+
| | |
|
| 39 |
+
|---|---|
|
| 40 |
+
| optimizer updates released | **40,000** (fixed-step snapshot `checkpoint-40000`) |
|
| 41 |
+
| updates the run was configured for | 50,000 |
|
| 42 |
+
| initialised from | our own stage-1 checkpoint (the `sd21ft` folder's run, at its final 50,000-update state) |
|
| 43 |
+
| batch size / resolution | 64 @ 256 px |
|
| 44 |
+
| epochs the 40,000 updates work out to | ~160 (16,001 training pairs, 250 updates/epoch) |
|
| 45 |
+
| optimizer | AdamW, lr 3e-5, weight decay 0.01, cosine schedule, 100-step warmup |
|
| 46 |
+
| precision | fp32 (no mixed precision) |
|
| 47 |
+
| seed | 2024 |
|
| 48 |
+
| prediction type | epsilon, with the confidence-guided (Gaussian-NLL) loss |
|
| 49 |
+
| prompt | the fixed string `"electro-optical image"` |
|
| 50 |
+
| augmentation | horizontal flip, vertical flip, 90-degree rotation |
|
| 51 |
+
| test-time sampler | **DDIM, 50 steps**, fp32, one image at a time |
|
| 52 |
+
|
| 53 |
+
The budget is quoted in **optimizer updates**, not epochs, because the two
|
| 54 |
+
datasets have very different training-set sizes. Two things a reader
|
| 55 |
+
reproducing this needs to know: the released snapshot is taken at update 40,000
|
| 56 |
+
of a cosine schedule that was laid out for 50,000, so the learning rate had
|
| 57 |
+
**not** finished decaying at that point; and we publish the fixed-step snapshot
|
| 58 |
+
rather than the repository's validation-selected `best/`, because every other
|
| 59 |
+
method in our comparison publishes its last checkpoint and selecting on
|
| 60 |
+
validation PSNR would have been an asymmetry in C-DiffSET's favour.
|
| 61 |
+
|
| 62 |
+
Data: 256 px, no crop and no resize (the chips are already 256 px). SAR chips come from the `sar_256_oc_0.2` directory of QXS-SAROPT; the EO target is the same stem under `opt_256_oc_0.2`.
|
| 63 |
+
|
| 64 |
+
## Base model
|
| 65 |
+
|
| 66 |
+
This checkpoint is a derivative of **Stable Diffusion 2.1-base**. Because the
|
| 67 |
+
original `stabilityai/stable-diffusion-2-1-base` repository is no longer
|
| 68 |
+
available on the Hub, our training and every snippet below load the community
|
| 69 |
+
mirror [`Manojb/stable-diffusion-2-1-base`](https://huggingface.co/Manojb/stable-diffusion-2-1-base).
|
| 70 |
+
That is what we actually trained against — verified in the training
|
| 71 |
+
configuration files, the launch scripts and the training logs, none of which
|
| 72 |
+
ever reference `stabilityai/...`. The frozen VAE, CLIP text encoder, tokenizer
|
| 73 |
+
and scheduler all come from that repo. A different SD 2.1-base mirror is
|
| 74 |
+
expected to work but has not been verified by us.
|
| 75 |
+
|
| 76 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 77 |
+
|
| 78 |
+
| FID↓ | KID↓ | DISTS↓ | LPIPS↓ | PSNR↑ | SSIM↑ |
|
| 79 |
+
|---|---|---|---|---|---|
|
| 80 |
+
| 19.9 | 0.0055 | 0.233 | 0.526 | 16.92 | 0.380 |
|
| 81 |
+
|
| 82 |
+
These are our own re-evaluation numbers, not numbers copied from any paper.
|
| 83 |
+
LPIPS is **LPIPS-VGG on inputs mapped to [-1, 1]** (the `normalize=False`
|
| 84 |
+
convention); the alternative [0, 1] convention gives a systematically different
|
| 85 |
+
number and the two must never be mixed. PSNR/SSIM are per-image torchmetrics
|
| 86 |
+
with `data_range=1`; FID is `pytorch-fid` against the size-matched ground truth.
|
| 87 |
+
|
| 88 |
+
## Load it and translate one SAR PNG
|
| 89 |
+
|
| 90 |
+
```python
|
| 91 |
+
import numpy as np
|
| 92 |
+
import torch
|
| 93 |
+
from PIL import Image
|
| 94 |
+
from diffusers import UNet2DConditionModel, AutoencoderKL, DDIMScheduler
|
| 95 |
+
from transformers import CLIPTextModel, CLIPTokenizer
|
| 96 |
+
|
| 97 |
+
REPO = "JeonghyeokDo/ReFlowSET" # comparison cells live under baselines/
|
| 98 |
+
BASE = "Manojb/stable-diffusion-2-1-base" # frozen VAE / text encoder / scheduler
|
| 99 |
+
DEV, DT = "cuda", torch.float32
|
| 100 |
+
|
| 101 |
+
unet = UNet2DConditionModel.from_pretrained(
|
| 102 |
+
REPO, subfolder="baselines/qxs-saropt/cdiffset", torch_dtype=DT).to(DEV).eval()
|
| 103 |
+
vae = AutoencoderKL.from_pretrained(BASE, subfolder="vae", torch_dtype=DT).to(DEV).eval()
|
| 104 |
+
tok = CLIPTokenizer.from_pretrained(BASE, subfolder="tokenizer")
|
| 105 |
+
txt = CLIPTextModel.from_pretrained(BASE, subfolder="text_encoder", torch_dtype=DT).to(DEV).eval()
|
| 106 |
+
sch = DDIMScheduler.from_pretrained(BASE, subfolder="scheduler")
|
| 107 |
+
sch.set_timesteps(50, device=DEV)
|
| 108 |
+
|
| 109 |
+
# The fixed training prompt, tokenized exactly as in training (no padding).
|
| 110 |
+
ids = tok("electro-optical image", padding="do_not_pad",
|
| 111 |
+
max_length=tok.model_max_length, truncation=True,
|
| 112 |
+
return_tensors="pt").input_ids.to(DEV)
|
| 113 |
+
with torch.no_grad():
|
| 114 |
+
embed = txt(ids)[0]
|
| 115 |
+
|
| 116 |
+
# SAR input: 256x256 PNG -> RGB -> [-1, 1]. Do not resize; crop if needed.
|
| 117 |
+
sar = Image.open("sar.png").convert("RGB")
|
| 118 |
+
x = torch.from_numpy(np.array(sar)).permute(2, 0, 1)[None]
|
| 119 |
+
x = (x.to(DEV, DT) / 255.0) * 2 - 1
|
| 120 |
+
|
| 121 |
+
with torch.no_grad():
|
| 122 |
+
sar_lat = vae.encode(x).latent_dist.mean * vae.config.scaling_factor # .mean, not .sample()
|
| 123 |
+
eo_lat = torch.randn_like(sar_lat)
|
| 124 |
+
for t in sch.timesteps:
|
| 125 |
+
out = unet(torch.cat([sar_lat, eo_lat], 1), # SAR first, then the noisy EO latent
|
| 126 |
+
t, encoder_hidden_states=embed).sample
|
| 127 |
+
# channels 0-3 are epsilon; channel 4 is the raw variance and never
|
| 128 |
+
# goes to the scheduler.
|
| 129 |
+
eo_lat = sch.step(out[:, :4], t, eo_lat).prev_sample
|
| 130 |
+
eo = vae.decode(eo_lat / vae.config.scaling_factor).sample
|
| 131 |
+
eo = (eo * 0.5 + 0.5).clamp(0, 1)[0].cpu()
|
| 132 |
+
|
| 133 |
+
Image.fromarray((eo.permute(1, 2, 0).numpy() * 255).round().astype(np.uint8)).save("eo.png")
|
| 134 |
+
```
|
| 135 |
+
|
| 136 |
+
The three things that silently produce a plausible-but-wrong image if you get
|
| 137 |
+
them backwards: the concatenation order is **SAR first**, only `out[:, :4]`
|
| 138 |
+
reaches the scheduler, and the VAE must be read as `latent_dist.mean`, not
|
| 139 |
+
`.sample()`.
|
| 140 |
+
|
| 141 |
+
## Licence status — stated factually
|
| 142 |
+
|
| 143 |
+
* **Training code:** MIT, `Copyright (c) 2026 KAIST VICLab`
|
| 144 |
+
([`KAIST-VICLab/C-DiffSET`](https://github.com/KAIST-VICLab/C-DiffSET) ships a
|
| 145 |
+
LICENSE file).
|
| 146 |
+
* **These weights:** they are a fine-tune of Stable Diffusion 2.1-base and are
|
| 147 |
+
therefore a *Derivative of the Model* under the **CreativeML Open RAIL++-M**
|
| 148 |
+
licence. The base repository we loaded declares `openrail++`.
|
| 149 |
+
The **Attachment A use restrictions travel with these weights** — they bind
|
| 150 |
+
you, and you must pass them on to anyone you redistribute to. The full text,
|
| 151 |
+
including Attachment A, ships in this repository as `LICENSE-WEIGHTS.md`.
|
| 152 |
+
|
| 153 |
+
Attribution: *fine-tuned from Stable Diffusion 2.1-base (obtained as
|
| 154 |
+
`Manojb/stable-diffusion-2-1-base`, a re-upload of
|
| 155 |
+
`stabilityai/stable-diffusion-2-1-base`) with the training code of
|
| 156 |
+
`KAIST-VICLab/C-DiffSET`. Do, Lee, Lee and Kim, "C-DiffSET", IEEE TCSVT 2026,
|
| 157 |
+
doi:10.1109/TCSVT.2026.3701447.*
|
| 158 |
+
|
| 159 |
+
---
|
| 160 |
+
|
| 161 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 162 |
+
checkpoint retrained by us on QXS-SAROPT; it is not ReFlowSET itself. Every
|
| 163 |
+
comparison method in the release was retrained by us on the same splits at the
|
| 164 |
+
same resolution, so the rows are directly comparable to each other.
|
baselines/qxs-saropt/cdiffset/config.json
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_class_name": "UNet2DConditionModel",
|
| 3 |
+
"_diffusers_version": "0.10.0.dev0",
|
| 4 |
+
"act_fn": "silu",
|
| 5 |
+
"attention_head_dim": [
|
| 6 |
+
5,
|
| 7 |
+
10,
|
| 8 |
+
20,
|
| 9 |
+
20
|
| 10 |
+
],
|
| 11 |
+
"block_out_channels": [
|
| 12 |
+
320,
|
| 13 |
+
640,
|
| 14 |
+
1280,
|
| 15 |
+
1280
|
| 16 |
+
],
|
| 17 |
+
"center_input_sample": false,
|
| 18 |
+
"cross_attention_dim": 1024,
|
| 19 |
+
"down_block_types": [
|
| 20 |
+
"CrossAttnDownBlock2D",
|
| 21 |
+
"CrossAttnDownBlock2D",
|
| 22 |
+
"CrossAttnDownBlock2D",
|
| 23 |
+
"DownBlock2D"
|
| 24 |
+
],
|
| 25 |
+
"downsample_padding": 1,
|
| 26 |
+
"dual_cross_attention": false,
|
| 27 |
+
"flip_sin_to_cos": true,
|
| 28 |
+
"freq_shift": 0,
|
| 29 |
+
"in_channels": 8,
|
| 30 |
+
"layers_per_block": 2,
|
| 31 |
+
"mid_block_scale_factor": 1,
|
| 32 |
+
"norm_eps": 1e-05,
|
| 33 |
+
"norm_num_groups": 32,
|
| 34 |
+
"num_class_embeds": null,
|
| 35 |
+
"only_cross_attention": false,
|
| 36 |
+
"out_channels": 5,
|
| 37 |
+
"sample_size": 64,
|
| 38 |
+
"up_block_types": [
|
| 39 |
+
"UpBlock2D",
|
| 40 |
+
"CrossAttnUpBlock2D",
|
| 41 |
+
"CrossAttnUpBlock2D",
|
| 42 |
+
"CrossAttnUpBlock2D"
|
| 43 |
+
],
|
| 44 |
+
"use_linear_projection": true
|
| 45 |
+
}
|
baselines/qxs-saropt/conddiff/README.md
ADDED
|
@@ -0,0 +1,173 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Conditional Diffusion — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**Conditional Diffusion for SAR to Optical Image Translation**, Bai, Pu and Xu,
|
| 4 |
+
*IEEE Geoscience and Remote Sensing Letters*, 2023
|
| 5 |
+
([doi:10.1109/LGRS.2023.3337143](https://doi.org/10.1109/LGRS.2023.3337143); the
|
| 6 |
+
year is the one in the authors' own citation block — a table that labels this
|
| 7 |
+
row with a later issue year is referring to the same paper).
|
| 8 |
+
|
| 9 |
+
Upstream code:
|
| 10 |
+
[`Coordi777/Conditional-Diffusion-for-SAR-to-Optical-Image-Translation`](https://github.com/Coordi777/Conditional-Diffusion-for-SAR-to-Optical-Image-Translation),
|
| 11 |
+
a modified copy of [`openai/guided-diffusion`](https://github.com/openai/guided-diffusion).
|
| 12 |
+
**The exact upstream commit is not recoverable** — see the licence section.
|
| 13 |
+
|
| 14 |
+
This is the pixel-space conditional diffusion model we retrained ourselves on
|
| 15 |
+
QXS-SAROPT, and it is the checkpoint behind the Conditional Diffusion row of the
|
| 16 |
+
ReFlowSET comparison table.
|
| 17 |
+
|
| 18 |
+
## What is in this folder
|
| 19 |
+
|
| 20 |
+
| file | bytes | what it is |
|
| 21 |
+
|---|---|---|
|
| 22 |
+
| `ema_final.pt` | 657,495,287 | EMA (decay 0.9999) of the UNet at update 50,000 — a **bare state dict**, no wrapper key |
|
| 23 |
+
|
| 24 |
+
`ema_final.pt` is the EMA of a **164.3 M-parameter guided-diffusion UNet**:
|
| 25 |
+
`num_channels` 128, `num_res_blocks` 3, `learn_sigma` **False**, attention at
|
| 26 |
+
resolutions 16 and 8. It is a plain `state_dict` — load it directly, with no
|
| 27 |
+
`['model']` or `['ema']` indirection.
|
| 28 |
+
|
| 29 |
+
The QXS-SAROPT and SAR2Opt files differ in size (657,495,287 vs 662,459,343
|
| 30 |
+
bytes) purely because of the 256 px versus 512 px positional and attention
|
| 31 |
+
buffers; the parameter count is otherwise the same.
|
| 32 |
+
|
| 33 |
+
**Conditioning.** The SAR image is concatenated to the noisy state **noise-free**
|
| 34 |
+
at every reverse step and at training time — that is the paper's claim, and the
|
| 35 |
+
code does exactly that.
|
| 36 |
+
|
| 37 |
+
## Training budget we used
|
| 38 |
+
|
| 39 |
+
| | |
|
| 40 |
+
|---|---|
|
| 41 |
+
| **generator updates released** | **50,000** |
|
| 42 |
+
| batch size / resolution | 24 @ 256 px |
|
| 43 |
+
| optimizer | Adam, **constant** lr 1e-4 with `--lr_anneal_steps 50000` (linear decay to zero, which is also the only stop mechanism in the released code) |
|
| 44 |
+
| EMA | decay 0.9999 |
|
| 45 |
+
| diffusion | T = 2,000, linear β schedule, eps-prediction, `learn_sigma` False |
|
| 46 |
+
| sampler (test) | **respaced DDPM, 250 steps**, `clip_denoised` on |
|
| 47 |
+
| augmentation | none — the released code has none |
|
| 48 |
+
| input | the chips are natively 256 px, used as they are |
|
| 49 |
+
|
| 50 |
+
**The paper and the released code disagree about the learning-rate schedule.**
|
| 51 |
+
The paper describes warmup plus cosine; the released code implements neither. We
|
| 52 |
+
ran the code.
|
| 53 |
+
|
| 54 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 55 |
+
|
| 56 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 57 |
+
|---|---|---|---|---|
|
| 58 |
+
| 88.6 | 0.355 | 0.730 | 0.213 | 11.55 |
|
| 59 |
+
|
| 60 |
+
Evaluated on the official split's 3,999 test chips at their native 256 px.
|
| 61 |
+
No subsampling: every metric on this page is measured over all 3,999 pairs.
|
| 62 |
+
|
| 63 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 64 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 65 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 66 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 67 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 68 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 69 |
+
convention gives a systematically different number and the two must never be
|
| 70 |
+
mixed, or compared against a paper that used the other one.
|
| 71 |
+
|
| 72 |
+
## Load it and translate one SAR image
|
| 73 |
+
|
| 74 |
+
```python
|
| 75 |
+
import torch
|
| 76 |
+
from guided_diffusion.script_util import (create_model_and_diffusion,
|
| 77 |
+
model_and_diffusion_defaults)
|
| 78 |
+
|
| 79 |
+
d = model_and_diffusion_defaults()
|
| 80 |
+
d.update(image_size=256, num_channels=128, num_res_blocks=3, learn_sigma=False,
|
| 81 |
+
diffusion_steps=2000, noise_schedule='linear', timestep_respacing='250')
|
| 82 |
+
model, diffusion = create_model_and_diffusion(**d)
|
| 83 |
+
model.load_state_dict(torch.load('ema_final.pt', map_location='cpu'))
|
| 84 |
+
model.cuda().eval()
|
| 85 |
+
|
| 86 |
+
# sar: (1, 3, 256, 256) float tensor in [-1, 1]
|
| 87 |
+
sample = diffusion.p_sample_loop(model, (1, 3, 256, 256),
|
| 88 |
+
clip_denoised=True, model_kwargs={},
|
| 89 |
+
noise=None, condition=sar)
|
| 90 |
+
```
|
| 91 |
+
|
| 92 |
+
The repository imports `blobfile` and `mpi4py` unconditionally; a single-process
|
| 93 |
+
run on a local filesystem needs either those packages or small local stand-ins on
|
| 94 |
+
`PYTHONPATH`.
|
| 95 |
+
|
| 96 |
+
**Never pass `use_ddim=True`.** See below.
|
| 97 |
+
|
| 98 |
+
## Read before using this checkpoint
|
| 99 |
+
|
| 100 |
+
* **DDIM is broken upstream, and that is not a choice we made.**
|
| 101 |
+
`p_sample_loop(..., condition=None, ...)` accepts and threads the SAR
|
| 102 |
+
condition; `ddim_sample_loop(...)` has **no `condition` parameter at all**, so
|
| 103 |
+
passing `--use_ddim True` raises `TypeError`. Sampling is respaced DDPM with
|
| 104 |
+
250 steps, which is also what the authors' own `sample.sh` uses.
|
| 105 |
+
* **A correctness bug in the released sampler, which we fixed.** Upstream paired
|
| 106 |
+
each SAR image with an EO image by **unsorted `os.listdir` position** — i.e. by
|
| 107 |
+
filesystem order. Any number produced with the unpatched sampler is measured
|
| 108 |
+
against effectively arbitrary ground truth. We sort both listings. If you
|
| 109 |
+
reproduce this row from a clean upstream checkout, apply that fix or your
|
| 110 |
+
metrics are meaningless.
|
| 111 |
+
* **Three further changes we made**, all commented in place: the
|
| 112 |
+
distributed-init helper no longer overwrites `CUDA_VISIBLE_DEVICES` (upstream
|
| 113 |
+
pins rank % 8, which on a shared machine hijacks another user's device); the
|
| 114 |
+
noise tensor for a partial last batch is shaped from the batch rather than from
|
| 115 |
+
the `--batch_size` flag; and the sampler takes explicit input/output directories
|
| 116 |
+
and builds its resize transform at run time, because the module-level transform
|
| 117 |
+
hard-codes 256 px and would silently downsize the 512 px cell.
|
| 118 |
+
* **The training loader requires integer filenames.** It sorts with
|
| 119 |
+
`int(stem)`, so any non-numeric stem raises `ValueError`. Feed it an
|
| 120 |
+
integer-named adapter directory and keep a manifest to map back to the real
|
| 121 |
+
stems.
|
| 122 |
+
* **Do not let the released 512 px path resize.** The fork's `center_crop_arr`
|
| 123 |
+
*resizes* 600 → 512. We wrote deterministic centre-512 crops instead, to keep a
|
| 124 |
+
crop-not-resize protocol across the whole benchmark.
|
| 125 |
+
* Passes the identity-collapse audit on both datasets.
|
| 126 |
+
|
| 127 |
+
## Licence — stated factually ⚠ no upstream licence exists
|
| 128 |
+
|
| 129 |
+
**The upstream code base publishes no licence.**
|
| 130 |
+
[`Coordi777/Conditional-Diffusion-for-SAR-to-Optical-Image-Translation`](https://github.com/Coordi777/Conditional-Diffusion-for-SAR-to-Optical-Image-Translation)
|
| 131 |
+
has no LICENSE, LICENCE, COPYING or NOTICE file anywhere in the tree we trained
|
| 132 |
+
from, no licence section in its README, and the GitHub API reports no declared
|
| 133 |
+
licence, with the `/license` endpoint returning 404. Checked 2026-08-28.
|
| 134 |
+
|
| 135 |
+
Under default copyright that means **all rights are reserved by the authors and
|
| 136 |
+
no express permission to redistribute derived work has been granted** to us or
|
| 137 |
+
to you. We publish this checkpoint anyway and state the position plainly. Assess
|
| 138 |
+
redistribution for yourself; consider asking the authors.
|
| 139 |
+
|
| 140 |
+
**A second gap, which is about reproducibility rather than licensing.** Our
|
| 141 |
+
vendored copy of this repository carries no version control and records no
|
| 142 |
+
upstream URL inside its tree, so **the exact commit these weights were trained
|
| 143 |
+
from cannot be recovered**. A "clone upstream, then apply our patch" recipe is
|
| 144 |
+
therefore not possible for this row.
|
| 145 |
+
|
| 146 |
+
**Lineage.** The README states the repository is based on
|
| 147 |
+
`openai/guided-diffusion` with modifications, and the tree is visibly that code
|
| 148 |
+
base — OpenAI provenance comments survive in `guided_diffusion/logger.py` and
|
| 149 |
+
`guided_diffusion/unet.py`. `openai/guided-diffusion` is **MIT**; the unmodified
|
| 150 |
+
guided-diffusion parts carry that licence, which does **not** extend to the
|
| 151 |
+
authors' modifications. We do not ship the guided-diffusion licence text here
|
| 152 |
+
because it is not vendored in the tree we trained from — take it from that
|
| 153 |
+
repository if you need it, and make sure the copyright line you carry is
|
| 154 |
+
OpenAI's rather than another project's MIT file.
|
| 155 |
+
|
| 156 |
+
**We modified the code further**, and say so as a matter of discipline rather
|
| 157 |
+
than because any licence compels it: the four changes listed in the section
|
| 158 |
+
above, of which the sorted-listing fix is a correctness fix.
|
| 159 |
+
|
| 160 |
+
Please cite: Bai, Pu and Xu, *Conditional Diffusion for SAR to Optical Image
|
| 161 |
+
Translation*, IEEE Geoscience and Remote Sensing Letters,
|
| 162 |
+
[doi:10.1109/LGRS.2023.3337143](https://doi.org/10.1109/LGRS.2023.3337143).
|
| 163 |
+
The full record of what we checked, per method, is in
|
| 164 |
+
`licenses/NO-UPSTREAM-LICENSE.md` at the root of this repository.
|
| 165 |
+
|
| 166 |
+
---
|
| 167 |
+
|
| 168 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 169 |
+
checkpoint that we retrained ourselves on QXS-SAROPT; it is not ReFlowSET
|
| 170 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 171 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 172 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 173 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/qxs-saropt/controlnet/README.md
ADDED
|
@@ -0,0 +1,127 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ControlNet — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**Adding Conditional Control to Text-to-Image Diffusion Models**, Zhang, Rao and
|
| 4 |
+
Agrawala, *ICCV* 2023. The SAR image is the control signal and the base
|
| 5 |
+
Stable Diffusion 2.1-base UNet stays frozen.
|
| 6 |
+
|
| 7 |
+
> **How our row was produced, precisely.** We trained with the
|
| 8 |
+
> `examples/controlnet/train_controlnet.py` script from
|
| 9 |
+
> [`huggingface/diffusers`](https://github.com/huggingface/diffusers)
|
| 10 |
+
> (Apache-2.0, Copyright 2025 The HuggingFace Inc. team), initialising the
|
| 11 |
+
> adapter with `ControlNetModel.from_unet` on SD 2.1-base. **No code from
|
| 12 |
+
> `lllyasviel/ControlNet` was used.** The method is theirs; the implementation
|
| 13 |
+
> is the diffusers one.
|
| 14 |
+
|
| 15 |
+
## What is in this folder
|
| 16 |
+
|
| 17 |
+
A complete diffusers `ControlNetModel` folder — `config.json` plus
|
| 18 |
+
`diffusion_pytorch_model.safetensors` (364,228,240 parameters, fp32, 340
|
| 19 |
+
tensors). It is the encoder half of the UNet plus the zero-convolutions and the
|
| 20 |
+
conditioning embedder; it has no `up_blocks` and no `conv_out`, so **it cannot
|
| 21 |
+
generate anything on its own** and must be paired with the base pipeline.
|
| 22 |
+
|
| 23 |
+
The conditioning input is 3-channel RGB (`conditioning_channels: 3`) with
|
| 24 |
+
embedder widths `[16, 32, 96, 256]`; the SAR PNG is opened as RGB, so the three
|
| 25 |
+
channels are identical.
|
| 26 |
+
|
| 27 |
+
## Training budget we used
|
| 28 |
+
|
| 29 |
+
| | |
|
| 30 |
+
|---|---|
|
| 31 |
+
| optimizer updates | **50,000** |
|
| 32 |
+
| batch size / resolution | 32 @ 256 px |
|
| 33 |
+
| epochs the 50,000 updates work out to | ~100 (16,001 training pairs, 500 updates/epoch) |
|
| 34 |
+
| optimizer | AdamW, lr 1e-5, **constant** schedule (the script's default) |
|
| 35 |
+
| precision | bf16 mixed precision — the saved tensors are the fp32 master weights |
|
| 36 |
+
| seed | 42 |
|
| 37 |
+
| trainable parameters | the adapter only; the SD 2.1-base UNet, VAE and text encoder stayed frozen |
|
| 38 |
+
| prompt | the fixed string `"electro-optical image"` for every pair |
|
| 39 |
+
| augmentation | **none** — the diffusers trainer applies only `Resize(res)` + `CenterCrop(res)`, both no-ops at our input size, and no flips or rotations |
|
| 40 |
+
| test-time sampler | **UniPCMultistepScheduler, 50 steps, guidance scale 7.5**, bf16 |
|
| 41 |
+
|
| 42 |
+
Unlike the two UNet folders in this repository, this is **not** an SD 2.1
|
| 43 |
+
fine-tune: the base weights were never updated, which is why only the 364 M
|
| 44 |
+
adapter ships.
|
| 45 |
+
|
| 46 |
+
Reproduction note: our inference script creates **one** `torch.Generator`
|
| 47 |
+
seeded 42 and reuses it across the whole test set in batches of 32,
|
| 48 |
+
so a single image regenerated on its own will not be bit-identical to our dump
|
| 49 |
+
even at the same seed. The conditioning image is
|
| 50 |
+
the SAR PNG opened as RGB and bilinearly resized to 256x256 (a no-op at our
|
| 51 |
+
evaluation resolution, since the SAR inputs are already 256 px).
|
| 52 |
+
|
| 53 |
+
Data: 256 px chips, used as-is (no crop, no resize). SAR chips come from the `sar_256_oc_0.2` directory of QXS-SAROPT; the EO target is the same stem under `opt_256_oc_0.2`.
|
| 54 |
+
|
| 55 |
+
## Base model
|
| 56 |
+
|
| 57 |
+
This checkpoint is a derivative of **Stable Diffusion 2.1-base**. Because the
|
| 58 |
+
original `stabilityai/stable-diffusion-2-1-base` repository is no longer
|
| 59 |
+
available on the Hub, our training and every snippet below load the community
|
| 60 |
+
mirror [`Manojb/stable-diffusion-2-1-base`](https://huggingface.co/Manojb/stable-diffusion-2-1-base).
|
| 61 |
+
That is what we actually trained against — verified in the training
|
| 62 |
+
configuration files, the launch scripts and the training logs, none of which
|
| 63 |
+
ever reference `stabilityai/...`. The frozen VAE, CLIP text encoder, tokenizer
|
| 64 |
+
and scheduler all come from that repo. A different SD 2.1-base mirror is
|
| 65 |
+
expected to work but has not been verified by us.
|
| 66 |
+
|
| 67 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 68 |
+
|
| 69 |
+
| FID↓ | KID↓ | DISTS↓ | LPIPS↓ | PSNR↑ | SSIM↑ |
|
| 70 |
+
|---|---|---|---|---|---|
|
| 71 |
+
| 50.4 | 0.0211 | 0.307 | 0.604 | 13.42 | 0.297 |
|
| 72 |
+
|
| 73 |
+
These are our own re-evaluation numbers, not numbers copied from any paper.
|
| 74 |
+
LPIPS is **LPIPS-VGG on inputs mapped to [-1, 1]** (the `normalize=False`
|
| 75 |
+
convention); the alternative [0, 1] convention gives a systematically different
|
| 76 |
+
number and the two must never be mixed. PSNR/SSIM are per-image torchmetrics
|
| 77 |
+
with `data_range=1`; FID is `pytorch-fid` against the size-matched ground truth.
|
| 78 |
+
|
| 79 |
+
## Load it and translate one SAR PNG
|
| 80 |
+
|
| 81 |
+
```python
|
| 82 |
+
import torch
|
| 83 |
+
from PIL import Image
|
| 84 |
+
from diffusers import (ControlNetModel, StableDiffusionControlNetPipeline,
|
| 85 |
+
UniPCMultistepScheduler)
|
| 86 |
+
|
| 87 |
+
REPO = "JeonghyeokDo/ReFlowSET" # comparison cells live under baselines/
|
| 88 |
+
BASE = "Manojb/stable-diffusion-2-1-base"
|
| 89 |
+
|
| 90 |
+
controlnet = ControlNetModel.from_pretrained(
|
| 91 |
+
REPO, subfolder="baselines/qxs-saropt/controlnet", torch_dtype=torch.bfloat16)
|
| 92 |
+
pipe = StableDiffusionControlNetPipeline.from_pretrained(
|
| 93 |
+
BASE, controlnet=controlnet, torch_dtype=torch.bfloat16,
|
| 94 |
+
safety_checker=None).to("cuda")
|
| 95 |
+
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
|
| 96 |
+
|
| 97 |
+
sar = Image.open("sar.png").convert("RGB").resize((256, 256), Image.BILINEAR)
|
| 98 |
+
eo = pipe("electro-optical image", image=sar, height=256, width=256,
|
| 99 |
+
num_inference_steps=50, guidance_scale=7.5,
|
| 100 |
+
generator=torch.Generator("cuda").manual_seed(42)).images[0]
|
| 101 |
+
eo.save("eo.png")
|
| 102 |
+
```
|
| 103 |
+
|
| 104 |
+
## Licence status — stated factually
|
| 105 |
+
|
| 106 |
+
* **Training code:** Apache-2.0. The trainer is
|
| 107 |
+
`examples/controlnet/train_controlnet.py` from
|
| 108 |
+
[`huggingface/diffusers`](https://github.com/huggingface/diffusers),
|
| 109 |
+
`Copyright 2025 The HuggingFace Inc. team`; the Apache-2.0 grant is in the
|
| 110 |
+
file header. Our copy of that script is four loose files rather than a full
|
| 111 |
+
clone, so it carries no separate repository LICENSE file — the header is the
|
| 112 |
+
grant.
|
| 113 |
+
* **Method credit:** ControlNet (Zhang et al., ICCV 2023),
|
| 114 |
+
[`lllyasviel/ControlNet`](https://github.com/lllyasviel/ControlNet),
|
| 115 |
+
Apache-2.0. **No code from that repository was used in this row.**
|
| 116 |
+
* **These weights:** initialised from Stable Diffusion 2.1-base via
|
| 117 |
+
`ControlNetModel.from_unet`, so they are an SD 2.1 derivative under the
|
| 118 |
+
**CreativeML Open RAIL++-M** licence (`openrail++`). The **Attachment A use
|
| 119 |
+
restrictions travel with these weights**. Full text, including Attachment A,
|
| 120 |
+
ships in this repository as `LICENSE-WEIGHTS.md`.
|
| 121 |
+
|
| 122 |
+
---
|
| 123 |
+
|
| 124 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 125 |
+
checkpoint retrained by us on QXS-SAROPT; it is not ReFlowSET itself. Every
|
| 126 |
+
comparison method in the release was retrained by us on the same splits at the
|
| 127 |
+
same resolution, so the rows are directly comparable to each other.
|
baselines/qxs-saropt/controlnet/config.json
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_class_name": "ControlNetModel",
|
| 3 |
+
"_diffusers_version": "0.37.1",
|
| 4 |
+
"act_fn": "silu",
|
| 5 |
+
"addition_embed_type": null,
|
| 6 |
+
"addition_embed_type_num_heads": 64,
|
| 7 |
+
"addition_time_embed_dim": null,
|
| 8 |
+
"attention_head_dim": [
|
| 9 |
+
5,
|
| 10 |
+
10,
|
| 11 |
+
20,
|
| 12 |
+
20
|
| 13 |
+
],
|
| 14 |
+
"block_out_channels": [
|
| 15 |
+
320,
|
| 16 |
+
640,
|
| 17 |
+
1280,
|
| 18 |
+
1280
|
| 19 |
+
],
|
| 20 |
+
"class_embed_type": null,
|
| 21 |
+
"conditioning_channels": 3,
|
| 22 |
+
"conditioning_embedding_out_channels": [
|
| 23 |
+
16,
|
| 24 |
+
32,
|
| 25 |
+
96,
|
| 26 |
+
256
|
| 27 |
+
],
|
| 28 |
+
"controlnet_conditioning_channel_order": "rgb",
|
| 29 |
+
"cross_attention_dim": 1024,
|
| 30 |
+
"down_block_types": [
|
| 31 |
+
"CrossAttnDownBlock2D",
|
| 32 |
+
"CrossAttnDownBlock2D",
|
| 33 |
+
"CrossAttnDownBlock2D",
|
| 34 |
+
"DownBlock2D"
|
| 35 |
+
],
|
| 36 |
+
"downsample_padding": 1,
|
| 37 |
+
"encoder_hid_dim": null,
|
| 38 |
+
"encoder_hid_dim_type": null,
|
| 39 |
+
"flip_sin_to_cos": true,
|
| 40 |
+
"freq_shift": 0,
|
| 41 |
+
"global_pool_conditions": false,
|
| 42 |
+
"in_channels": 4,
|
| 43 |
+
"layers_per_block": 2,
|
| 44 |
+
"mid_block_scale_factor": 1,
|
| 45 |
+
"mid_block_type": "UNetMidBlock2DCrossAttn",
|
| 46 |
+
"norm_eps": 1e-05,
|
| 47 |
+
"norm_num_groups": 32,
|
| 48 |
+
"num_attention_heads": null,
|
| 49 |
+
"num_class_embeds": null,
|
| 50 |
+
"only_cross_attention": false,
|
| 51 |
+
"projection_class_embeddings_input_dim": null,
|
| 52 |
+
"resnet_time_scale_shift": "default",
|
| 53 |
+
"transformer_layers_per_block": 1,
|
| 54 |
+
"upcast_attention": false,
|
| 55 |
+
"use_linear_projection": true
|
| 56 |
+
}
|
baselines/qxs-saropt/cyclegan/README.md
ADDED
|
@@ -0,0 +1,169 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# CycleGAN — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial
|
| 4 |
+
Networks**, Zhu, Park, Isola and Efros, *ICCV 2017*
|
| 5 |
+
([arXiv:1703.10593](https://arxiv.org/abs/1703.10593)). Upstream code:
|
| 6 |
+
[`junyanz/pytorch-CycleGAN-and-pix2pix`](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix)
|
| 7 |
+
at commit `2a7afba2895d52556dd5dfe07e8555ef657ced6f`.
|
| 8 |
+
|
| 9 |
+
This is the unpaired cycle-consistent GAN we retrained ourselves on QXS-SAROPT,
|
| 10 |
+
and it is the checkpoint behind the CycleGAN row of the ReFlowSET comparison
|
| 11 |
+
table. **It fails our input-copy audit — see the section below before you use
|
| 12 |
+
or quote it.**
|
| 13 |
+
|
| 14 |
+
## What is in this folder
|
| 15 |
+
|
| 16 |
+
| file | bytes | what it is |
|
| 17 |
+
|---|---|---|
|
| 18 |
+
| `net_G_A.pth` | 45,533,133 | **SAR → EO** — this is the translation direction the table reports |
|
| 19 |
+
| `net_G_B.pth` | 45,533,133 | EO → SAR, the reverse generator that closes the cycle; not used to produce any reported number |
|
| 20 |
+
|
| 21 |
+
Both are **`resnet_9blocks` generators** with instance normalisation,
|
| 22 |
+
`ngf = 64`, 3 in / 3 out, dropout disabled. The direction assignment is not a
|
| 23 |
+
convention we chose: upstream's unaligned loader binds `A_paths` to the
|
| 24 |
+
`trainA` / `testA` directory, which holds the SAR side, and the model computes
|
| 25 |
+
`fake_B = netG_A(real_A)`. So **`net_G_A` is the SAR → EO generator**.
|
| 26 |
+
|
| 27 |
+
`net_G_B` is shipped because it is half of the trained system and someone
|
| 28 |
+
reproducing the cycle needs it; it is not needed to reproduce our numbers. The
|
| 29 |
+
two discriminators are not released.
|
| 30 |
+
|
| 31 |
+
## Training budget we used
|
| 32 |
+
|
| 33 |
+
| | |
|
| 34 |
+
|---|---|
|
| 35 |
+
| **generator updates released** | **100,050** |
|
| 36 |
+
| batch size | 8 |
|
| 37 |
+
| epochs (constant LR + linear decay) | 25 + 25 |
|
| 38 |
+
| iterations per epoch | 2,001 (16,001 training images, the loader does not drop the last batch) |
|
| 39 |
+
| optimizer | Adam, lr 2e-4, β₁ 0.5, linear decay to 0 over the second half |
|
| 40 |
+
| adversarial loss | LSGAN; discriminator `basic` (70×70 PatchGAN) |
|
| 41 |
+
| cycle losses | λ_A 10, λ_B 10 |
|
| 42 |
+
| identity loss | λ_identity 0.5 |
|
| 43 |
+
| augmentation | horizontal flip |
|
| 44 |
+
| input | 256 px, no resize |
|
| 45 |
+
|
| 46 |
+
Budgets are quoted in **optimizer updates**, never epochs. This is the single
|
| 47 |
+
most important comparability rule for the unpaired methods in this benchmark:
|
| 48 |
+
"100 + 100 epochs" is 72,600 updates on 1,450 images and 400,000+ on
|
| 49 |
+
16,001, a difference that reads as budget-matched and is not.
|
| 50 |
+
|
| 51 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 52 |
+
|
| 53 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 54 |
+
|---|---|---|---|---|
|
| 55 |
+
| 104.4 | 0.376 | 0.653 | 0.262 | 12.92 |
|
| 56 |
+
|
| 57 |
+
Evaluated on the official split's 3,999 test chips at their native 256 px.
|
| 58 |
+
No subsampling: every metric on this page is measured over all 3,999 pairs.
|
| 59 |
+
|
| 60 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 61 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 62 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 63 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 64 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 65 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 66 |
+
convention gives a systematically different number and the two must never be
|
| 67 |
+
mixed, or compared against a paper that used the other one.
|
| 68 |
+
|
| 69 |
+
## Load it and translate one SAR image
|
| 70 |
+
|
| 71 |
+
Clone `junyanz/pytorch-CycleGAN-and-pix2pix` at the commit above and run this
|
| 72 |
+
from its root. The model code is stock — we applied no patch to it.
|
| 73 |
+
|
| 74 |
+
```python
|
| 75 |
+
import numpy as np, torch
|
| 76 |
+
from PIL import Image
|
| 77 |
+
from models.networks import define_G # from the junyanz repo
|
| 78 |
+
|
| 79 |
+
G_A = define_G(3, 3, 64, 'resnet_9blocks', 'instance', use_dropout=False,
|
| 80 |
+
init_type='normal', init_gain=0.02, gpu_ids=[])
|
| 81 |
+
sd = torch.load('net_G_A.pth', map_location='cpu')
|
| 82 |
+
if hasattr(sd, '_metadata'):
|
| 83 |
+
del sd._metadata
|
| 84 |
+
G_A.load_state_dict(sd)
|
| 85 |
+
G_A.eval()
|
| 86 |
+
|
| 87 |
+
sar = Image.open('sar.png').convert('RGB')
|
| 88 |
+
x = torch.from_numpy(np.asarray(sar, np.float32) / 127.5 - 1).permute(2, 0, 1)[None]
|
| 89 |
+
|
| 90 |
+
with torch.no_grad():
|
| 91 |
+
y = G_A(x) # this is fake_B
|
| 92 |
+
|
| 93 |
+
eo = ((y[0].permute(1, 2, 0).numpy() + 1) * 127.5).clip(0, 255).astype(np.uint8)
|
| 94 |
+
Image.fromarray(eo).save('eo.png')
|
| 95 |
+
```
|
| 96 |
+
|
| 97 |
+
Upstream also supports a SAR-only run without this snippet:
|
| 98 |
+
`test.py --model test --dataset_mode single --no_dropout --model_suffix _A`,
|
| 99 |
+
which loads `latest_net_G_A.pth` directly.
|
| 100 |
+
|
| 101 |
+
## ⚠ This cell fails the input-copy audit
|
| 102 |
+
|
| 103 |
+
We audit every cell for **identity collapse**: a model that has learned to pass
|
| 104 |
+
its input through rather than translate it. The test compares the mean absolute
|
| 105 |
+
distance from the generated image to the SAR **input** against the distance to
|
| 106 |
+
the EO **target**; a ratio below 1.0 means the output is closer to the input it
|
| 107 |
+
was given than to the answer it was asked for.
|
| 108 |
+
|
| 109 |
+
| quantity | value |
|
| 110 |
+
|---|---|
|
| 111 |
+
| mean \|generated − ground truth\| | 47.548 |
|
| 112 |
+
| mean \|generated − SAR input\| | 40.274 |
|
| 113 |
+
| **ratio** (SAR distance ÷ GT distance) | **0.847** — below 1.0 |
|
| 114 |
+
| verdict | **COLLAPSED** |
|
| 115 |
+
| cross-pair separation | 0.955 (cohort median 1.160) — no oracle behaviour |
|
| 116 |
+
|
| 117 |
+
**Read the metrics on this page with that in mind.** They are reported unchanged
|
| 118 |
+
because they are what the released implementation does at its own published
|
| 119 |
+
protocol, and a benchmark that quietly substitutes a better-behaved variant is
|
| 120 |
+
not reporting the method. But they should not be read as translation quality.
|
| 121 |
+
The separation figure is the companion check for the *opposite* failure: it is
|
| 122 |
+
the mean distance from a generated image to **other** images' ground truth
|
| 123 |
+
divided by the distance to its **own**, so a value well above the cohort median
|
| 124 |
+
would indicate that ground-truth information had leaked into the generation.
|
| 125 |
+
This cell sits at or below the median, so there is no leak — the problem here is
|
| 126 |
+
input-copying, and only input-copying.
|
| 127 |
+
|
| 128 |
+
On this dataset the collapse is at least partly a budget effect: a diagnostic control run of the same code at **400,000** updates (4× this one) does **not** collapse (ratio 1.127). That control is not published here and is not the table's CycleGAN row, which is the method at its own protocol's budget.
|
| 129 |
+
|
| 130 |
+
## Read before using this checkpoint
|
| 131 |
+
|
| 132 |
+
* **`net_G_A` is SAR → EO.** Loading `net_G_B` and expecting an EO image gives
|
| 133 |
+
you the reverse mapping and a confusing result.
|
| 134 |
+
* Build the network with `gpu_ids=[]` — the checkpoints have bare keys and a
|
| 135 |
+
non-empty `gpu_ids` introduces a `module.` prefix the load will reject.
|
| 136 |
+
* Normalisation is `[-1, 1]`.
|
| 137 |
+
* CycleGAN is **unpaired**: it never saw a SAR/EO correspondence during
|
| 138 |
+
training, only the two marginal distributions.
|
| 139 |
+
|
| 140 |
+
## Licence — stated factually
|
| 141 |
+
|
| 142 |
+
Same upstream, same LICENSE file as pix2pix; reproduced verbatim here as
|
| 143 |
+
`licenses/LICENSE-cyclegan.txt` (byte-identical to `licenses/LICENSE-pix2pix.txt`
|
| 144 |
+
— one file covers both methods upstream). It is three stacked notices:
|
| 145 |
+
|
| 146 |
+
* CycleGAN — `Copyright (c) 2017, Jun-Yan Zhu and Taesung Park` (BSD-2-Clause in
|
| 147 |
+
form);
|
| 148 |
+
* pix2pix — `Copyright (c) 2016, Phillip Isola and Jun-Yan Zhu`;
|
| 149 |
+
* DCGAN — `Copyright (c) 2015, Facebook, Inc.` (BSD-3-Clause, with a
|
| 150 |
+
no-endorsement clause).
|
| 151 |
+
|
| 152 |
+
**Redistribution of these weights is permitted** with the notice reproduced. No
|
| 153 |
+
non-commercial and no share-alike clause. Do not use Facebook's name to promote
|
| 154 |
+
this release.
|
| 155 |
+
|
| 156 |
+
Attribution: *trained with `junyanz/pytorch-CycleGAN-and-pix2pix`. Copyright (c)
|
| 157 |
+
2017, Jun-Yan Zhu and Taesung Park (CycleGAN); Copyright (c) 2016, Phillip Isola
|
| 158 |
+
and Jun-Yan Zhu (pix2pix); Copyright (c) 2015, Facebook, Inc. (DCGAN). All rights
|
| 159 |
+
reserved. Redistributed under the BSD terms reproduced in
|
| 160 |
+
`licenses/LICENSE-cyclegan.txt`.*
|
| 161 |
+
|
| 162 |
+
---
|
| 163 |
+
|
| 164 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 165 |
+
checkpoint that we retrained ourselves on QXS-SAROPT; it is not ReFlowSET
|
| 166 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 167 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 168 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 169 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/qxs-saropt/ddpm/README.md
ADDED
|
@@ -0,0 +1,174 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# DDPM (SR3-class) — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
The **SR3 method class** — an eps-prediction conditional DDPM that conditions on
|
| 4 |
+
the source image — as introduced in *Image Super-Resolution via Iterative
|
| 5 |
+
Refinement*, Saharia, Ho, Chan, Salimans, Fleet and Norouzi,
|
| 6 |
+
*IEEE TPAMI 2022* ([arXiv:2104.07636](https://arxiv.org/abs/2104.07636)).
|
| 7 |
+
|
| 8 |
+
**Read this before citing the row: the code is not the SR3 authors'.** No code
|
| 9 |
+
from any SR3 release was used. This cell is
|
| 10 |
+
[`DeepSARRS/E3Diff`](https://github.com/DeepSARRS/E3Diff) at commit
|
| 11 |
+
`38601093ab8f8e4b478144621f20890b100a3b74`, run in its **stage-1**
|
| 12 |
+
configuration (`"stage": 1`), which is an SR3-class conditional DDPM. We
|
| 13 |
+
retrained it ourselves on QXS-SAROPT and it is the checkpoint behind the DDPM
|
| 14 |
+
row of the ReFlowSET comparison table.
|
| 15 |
+
|
| 16 |
+
## What is in this folder
|
| 17 |
+
|
| 18 |
+
| file | bytes | what it is |
|
| 19 |
+
|---|---|---|
|
| 20 |
+
| `gen.pth` | 768,426,944 | the `sr3` UNet at iteration 250,000 — the whole model |
|
| 21 |
+
|
| 22 |
+
`gen.pth` is E3Diff's **`sr3` UNet**: `inner_channel` 64,
|
| 23 |
+
`channel_multiplier` [1, 2, 4, 8, 16], `res_blocks` 1, `norm_groups` 16, **no
|
| 24 |
+
attention** (`attn_res` is empty), 3 input channels, 3 output channels and a
|
| 25 |
+
3-channel condition.
|
| 26 |
+
|
| 27 |
+
The condition is **not the raw SAR image alone**. It is a three-channel stack
|
| 28 |
+
`[PPB, Canny, SAR]`:
|
| 29 |
+
|
| 30 |
+
* channel 0 — the SAR image after **FAST_PPB** speckle filtering
|
| 31 |
+
(Deledalle 2009, P = 3, W = 10, h = 0.5), a GPU port of E3Diff's own
|
| 32 |
+
`FAST_PPB.m`;
|
| 33 |
+
* channel 1 — `cv2.Canny(ppb_uint8, 50, 150, L2gradient=True)` on that filtered
|
| 34 |
+
image;
|
| 35 |
+
* channel 2 — the raw SAR image.
|
| 36 |
+
|
| 37 |
+
**This row cannot be run from a SAR PNG alone**: the PPB and Canny images must
|
| 38 |
+
be computed first.
|
| 39 |
+
|
| 40 |
+
## Training budget we used
|
| 41 |
+
|
| 42 |
+
| | |
|
| 43 |
+
|---|---|
|
| 44 |
+
| **generator updates released** | **250,000** |
|
| 45 |
+
| batch size / resolution | 16 @ 256 px |
|
| 46 |
+
| optimizer | Adam, lr 5e-5, constant (`MultiStepLR` with no milestones) |
|
| 47 |
+
| EMA | decay 0.9999, started at step 5,000, updated every step |
|
| 48 |
+
| diffusion (train) | T = 1,000, linear β from 1e-6 to 1e-2 |
|
| 49 |
+
| sampler (test) | **DDIM, 50 steps** |
|
| 50 |
+
| loss | plain eps-prediction MSE — the stage-2 auxiliary losses are all zero here (`lpips_w` 0, `fft_w` 0, `lambda_gan` 0) |
|
| 51 |
+
| augmentation | flips, 90° rotations, and a random brightness jitter applied to the SAR/PPB pair only (upstream's own `transform_augment`) |
|
| 52 |
+
|
| 53 |
+
Both datasets get the same 250,000 updates; the batch differs so that the
|
| 54 |
+
pixels-per-update budget matches (16 × 256² = 4 × 512²).
|
| 55 |
+
|
| 56 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 57 |
+
|
| 58 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 59 |
+
|---|---|---|---|---|
|
| 60 |
+
| 43.8 | 0.311 | 0.620 | 0.359 | 14.04 |
|
| 61 |
+
|
| 62 |
+
Evaluated on the official split's 3,999 test chips at their native 256 px.
|
| 63 |
+
No subsampling: every metric on this page is measured over all 3,999 pairs.
|
| 64 |
+
|
| 65 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 66 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 67 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 68 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 69 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 70 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 71 |
+
convention gives a systematically different number and the two must never be
|
| 72 |
+
mixed, or compared against a paper that used the other one.
|
| 73 |
+
|
| 74 |
+
## Load it and translate one SAR image
|
| 75 |
+
|
| 76 |
+
This model cannot be driven from a bare SAR file: it needs the PPB and Canny
|
| 77 |
+
condition channels, and the E3Diff loader looks them up by filename. Build the
|
| 78 |
+
condition first, then run the repository's own entry point.
|
| 79 |
+
|
| 80 |
+
```python
|
| 81 |
+
# 1. Build the condition channels for each SAR image, once.
|
| 82 |
+
import cv2, numpy as np
|
| 83 |
+
# ppb = FAST_PPB(sar, P=3, W=10, h=0.5) # Deledalle 2009; E3Diff ships FAST_PPB.m
|
| 84 |
+
# canny = cv2.Canny(ppb.astype(np.uint8), 50, 150, L2gradient=True)
|
| 85 |
+
# Write ppb and canny under <root>/val/SAR-PPB/<name> and <root>/val/SAR-canny/<name>,
|
| 86 |
+
# alongside <root>/val/SAR/<name> and <root>/val/EO/<name>. Names must match exactly.
|
| 87 |
+
|
| 88 |
+
# 2. Run the repository, phase 'val', with a config whose
|
| 89 |
+
# path.resume_state is the checkpoint PREFIX -- no '_gen.pth' suffix:
|
| 90 |
+
# "path": {"resume_state": "<dir>/I250000_E<epoch>"} # place gen.pth as I250000_E<epoch>_gen.pth
|
| 91 |
+
# "stage": 1,
|
| 92 |
+
# "model": {"beta_schedule": {"val": {"n_timestep": 50, "ddim": 1, ...}}},
|
| 93 |
+
# "datasets": {"val": {"data_len": -1, "r_resolution": 256}}
|
| 94 |
+
# python main.py -c <val.json> -p val -enable_wandb "" --seed 1
|
| 95 |
+
```
|
| 96 |
+
|
| 97 |
+
Two mechanical details that will otherwise cost you a full inference pass:
|
| 98 |
+
|
| 99 |
+
* the RGB loader. Upstream's `SAR2EODataset` hard-codes a **grayscale** EO
|
| 100 |
+
target and a two-channel `[PPB, Canny]` condition. QXS-SAROPT's optical side is
|
| 101 |
+
RGB, and with three channels upstream's `ddim_sample()` slices the condition to
|
| 102 |
+
`condition_x[:, :3]` and concatenates it with 3-channel predictions, which
|
| 103 |
+
raises on the first validation image. We ran through a small wrapper that
|
| 104 |
+
patches `SAR2EODataset.__getitem__` **in memory** — returning
|
| 105 |
+
`HR = EO[0:3]`, `LR = SAR[0:3]`, `SR = cat(PPB[0:1], Canny[0:1], SAR[0:1])` —
|
| 106 |
+
and then executes the repository's `main.py` verbatim, so no upstream file is
|
| 107 |
+
touched and every flag behaves the same. Reproduce that patch or write your own
|
| 108 |
+
three-channel loader;
|
| 109 |
+
* `-enable_wandb ""` is mandatory. The flag defaults to the string `'false'`,
|
| 110 |
+
which is truthy.
|
| 111 |
+
|
| 112 |
+
## Read before using this checkpoint
|
| 113 |
+
|
| 114 |
+
* **The checkpoint loads with `strict=False`.** `model/model.py` calls
|
| 115 |
+
`load_state_dict(..., strict=False)`, so a configuration that does not match
|
| 116 |
+
the file **loads nothing and raises no error** — you get a fully random model
|
| 117 |
+
and a plausible-looking run. Verify the `Loading pretrained model for G [...]`
|
| 118 |
+
line in the log and eyeball the first output.
|
| 119 |
+
* **`-p val` writes next to the checkpoint and then renames.** Images land in
|
| 120 |
+
`<resume_state>/sample/`, and at the end the directory is renamed to
|
| 121 |
+
`<resume_state>_S<ssim>_P<psnr>_l2<l2>_Lp<lpips>`. That rename **raises if the
|
| 122 |
+
target already exists** — after the entire inference has been paid for. Guard
|
| 123 |
+
for an existing `<prefix>_S*/sample` before starting.
|
| 124 |
+
* **SoftPool.** The E3Diff code imports the SoftPool CUDA extension
|
| 125 |
+
unconditionally. We used a pure-PyTorch drop-in with identical maths
|
| 126 |
+
(`softpool(x) = avgpool(x·eˣ) / avgpool(eˣ)`) and identical autograd, which
|
| 127 |
+
needs no build step. Either build the extension or write the equivalent.
|
| 128 |
+
* **A known tone offset, inherited deliberately.** On another dataset in the
|
| 129 |
+
wider campaign this exact configuration produced generations that were, on
|
| 130 |
+
average, tens of gray levels brighter than the ground truth, at a substantial
|
| 131 |
+
cost in PSNR. We did **not** correct it per dataset, because comparability
|
| 132 |
+
across the row was judged worth more than per-cell tuning. Read this row's
|
| 133 |
+
PSNR with that in mind.
|
| 134 |
+
* Passes the identity-collapse audit on both datasets.
|
| 135 |
+
|
| 136 |
+
## Licence — stated factually ⚠ no upstream licence exists
|
| 137 |
+
|
| 138 |
+
**The upstream code base publishes no licence.**
|
| 139 |
+
[`DeepSARRS/E3Diff`](https://github.com/DeepSARRS/E3Diff), whose stage-1
|
| 140 |
+
configuration this row is, has no LICENSE, LICENCE, COPYING or NOTICE file at
|
| 141 |
+
any depth in its tree; only 1 of its 58 tracked files is licence-shaped and that
|
| 142 |
+
one is `SoftPool/LICENSE.txt`, the MIT licence of a **vendored third-party
|
| 143 |
+
dependency** (`Copyright (c) 2020 Alexandros Stergiou`, reproduced here as
|
| 144 |
+
`licenses/LICENSE-SoftPool-MIT.txt`) — not a grant for E3Diff. Its README has no
|
| 145 |
+
licence section, and the GitHub API reports no declared licence, with the
|
| 146 |
+
`/license` endpoint returning 404. Checked 2026-08-28.
|
| 147 |
+
|
| 148 |
+
Under default copyright that means **all rights are reserved by the authors and
|
| 149 |
+
no express permission to redistribute derived work has been granted** to us or
|
| 150 |
+
to you. We publish this checkpoint anyway, so that the benchmark is reproducible
|
| 151 |
+
end to end, and we state the position plainly rather than implying a permission
|
| 152 |
+
that does not exist. If you intend to redistribute this checkpoint or build on
|
| 153 |
+
it, assess that for yourself, and consider asking the authors directly.
|
| 154 |
+
|
| 155 |
+
The code base credits SR3
|
| 156 |
+
(`Janspiry/Image-Super-Resolution-via-Iterative-Refinement`, Apache-2.0),
|
| 157 |
+
`GaParmar/img2img-turbo` (MIT) and `alexandrosstergiou/SoftPool` (MIT). Those
|
| 158 |
+
licences cover the borrowed parts, not the authors' own contribution.
|
| 159 |
+
|
| 160 |
+
Please cite the SR3 paper for the method class, and E3Diff for the code that
|
| 161 |
+
produced these weights: Qin, Zou, Li and Zhang, *Efficient End-to-End Diffusion
|
| 162 |
+
Model for One-Step SAR-to-Optical Translation*, IEEE Geoscience and Remote
|
| 163 |
+
Sensing Letters, [doi:10.1109/LGRS.2024.3506566](https://doi.org/10.1109/LGRS.2024.3506566).
|
| 164 |
+
The full record of what we checked, per method, is in
|
| 165 |
+
`licenses/NO-UPSTREAM-LICENSE.md` at the root of this repository.
|
| 166 |
+
|
| 167 |
+
---
|
| 168 |
+
|
| 169 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 170 |
+
checkpoint that we retrained ourselves on QXS-SAROPT; it is not ReFlowSET
|
| 171 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 172 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 173 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 174 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/qxs-saropt/e3diff/README.md
ADDED
|
@@ -0,0 +1,177 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# E3Diff — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**Efficient End-to-End Diffusion Model for One-Step SAR-to-Optical
|
| 4 |
+
Translation**, Qin, Zou, Li and Zhang, *IEEE Geoscience and Remote Sensing
|
| 5 |
+
Letters*, 2024
|
| 6 |
+
([doi:10.1109/LGRS.2024.3506566](https://doi.org/10.1109/LGRS.2024.3506566); the
|
| 7 |
+
year is the one in the authors' own citation block — a table that labels this row
|
| 8 |
+
with a later issue year is referring to the same paper). Upstream code:
|
| 9 |
+
[`DeepSARRS/E3Diff`](https://github.com/DeepSARRS/E3Diff) at commit
|
| 10 |
+
`38601093ab8f8e4b478144621f20890b100a3b74`.
|
| 11 |
+
|
| 12 |
+
This is the authors' two-stage method, **retrained by us end to end** on
|
| 13 |
+
QXS-SAROPT, and it is the checkpoint behind the E3Diff row of the ReFlowSET
|
| 14 |
+
comparison table.
|
| 15 |
+
|
| 16 |
+
**These are our weights, not the authors' released checkpoint.** The authors'
|
| 17 |
+
own weights exist for a different dataset and are not what this row measures.
|
| 18 |
+
|
| 19 |
+
## What is in this folder
|
| 20 |
+
|
| 21 |
+
| file | bytes | what it is |
|
| 22 |
+
|---|---|---|
|
| 23 |
+
| `gen.pth` | 768,426,944 | the `sr3` UNet after stage 2, at absolute iteration 310,000 |
|
| 24 |
+
|
| 25 |
+
Architecturally this is **the same network as the DDPM (SR3-class) row** in this
|
| 26 |
+
repository — E3Diff's two stages share one UNet. What differs is the training:
|
| 27 |
+
|
| 28 |
+
* **stage 1** (250,000 updates) is the eps-prediction conditional DDPM. Its
|
| 29 |
+
checkpoint is the `ddpm/` folder in this repository.
|
| 30 |
+
* **stage 2** (60,000 further updates, absolute iteration 310,000) fine-tunes
|
| 31 |
+
that same network into a **one-step generator**: the sampler is run *with
|
| 32 |
+
gradients* from pure noise for `ddim_steps = 1`, and the loss is taken directly
|
| 33 |
+
on pixels.
|
| 34 |
+
|
| 35 |
+
UNet: `inner_channel` 64, `channel_multiplier` [1, 2, 4, 8, 16], `res_blocks` 1,
|
| 36 |
+
`norm_groups` 16, no attention, 3 in / 3 out, 3-channel condition.
|
| 37 |
+
|
| 38 |
+
The condition is the same three-channel `[PPB, Canny, SAR]` stack described on
|
| 39 |
+
the DDPM card: FAST_PPB speckle filtering (Deledalle 2009, P = 3, W = 10,
|
| 40 |
+
h = 0.5), `cv2.Canny(ppb, 50, 150, L2gradient=True)`, and the raw SAR image.
|
| 41 |
+
**This row cannot be run from a SAR PNG alone.**
|
| 42 |
+
|
| 43 |
+
## Training budget we used
|
| 44 |
+
|
| 45 |
+
| | |
|
| 46 |
+
|---|---|
|
| 47 |
+
| **generator updates released** | **250,000 (stage 1, inherited) + 60,000 (stage 2) = 310,000 absolute** |
|
| 48 |
+
| batch size / resolution | 16 @ 256 px |
|
| 49 |
+
| optimizer | Adam, lr 5e-5 |
|
| 50 |
+
| stage-2 sampler during training | DDIM, **1 step**, run with gradients from pure noise |
|
| 51 |
+
| stage-2 losses | L1 on pixels, + LPIPS (weight 5), + focal-frequency (weight 10), + a vision-aided CLIP GAN (λ_gan 0.5) |
|
| 52 |
+
| stage-1 losses | plain eps-prediction MSE (all auxiliary weights zero) |
|
| 53 |
+
| sampler (test) | **DDIM, 1 step** |
|
| 54 |
+
| EMA | decay 0.9999 (stage 1) |
|
| 55 |
+
| augmentation | flips, 90° rotations, and a random brightness jitter on the SAR/PPB pair (upstream's `transform_augment`) |
|
| 56 |
+
|
| 57 |
+
The stage-2 budget is 24 % of the stage-1 budget, which follows the authors' own
|
| 58 |
+
ratio (they resume at 640,000 and run to 800,000). Note that `n_iter` in this
|
| 59 |
+
code base is an **absolute** step count continuing stage 1, not a stage-2
|
| 60 |
+
budget — a config that reads 310,000 buys 60,000 new iterations.
|
| 61 |
+
|
| 62 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 63 |
+
|
| 64 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 65 |
+
|---|---|---|---|---|
|
| 66 |
+
| 47.8 | 0.278 | 0.530 | 0.302 | 16.44 |
|
| 67 |
+
|
| 68 |
+
Evaluated on the official split's 3,999 test chips at their native 256 px.
|
| 69 |
+
No subsampling: every metric on this page is measured over all 3,999 pairs.
|
| 70 |
+
|
| 71 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 72 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 73 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 74 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 75 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 76 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 77 |
+
convention gives a systematically different number and the two must never be
|
| 78 |
+
mixed, or compared against a paper that used the other one.
|
| 79 |
+
|
| 80 |
+
## Load it and translate one SAR image
|
| 81 |
+
|
| 82 |
+
Same entry point and same mechanics as the DDPM row, with `"stage": 2` and
|
| 83 |
+
`ddim_steps: 1`.
|
| 84 |
+
|
| 85 |
+
```python
|
| 86 |
+
# 1. Build the condition channels for each SAR image, once:
|
| 87 |
+
# ppb = FAST_PPB(sar, P=3, W=10, h=0.5)
|
| 88 |
+
# canny = cv2.Canny(ppb.astype('uint8'), 50, 150, L2gradient=True)
|
| 89 |
+
# written to <root>/val/SAR-PPB/<name> and <root>/val/SAR-canny/<name>,
|
| 90 |
+
# alongside <root>/val/SAR/<name> and <root>/val/EO/<name>.
|
| 91 |
+
|
| 92 |
+
# 2. Run phase 'val' with a config whose path.resume_state is the checkpoint
|
| 93 |
+
# PREFIX -- no '_gen.pth' suffix -- and place gen.pth as <prefix>_gen.pth:
|
| 94 |
+
# "stage": 2, "ddim_steps": 1,
|
| 95 |
+
# "model": {"beta_schedule": {"val": {"n_timestep": 1, "ddim": 1, ...}}},
|
| 96 |
+
# "datasets": {"val": {"data_len": -1, "r_resolution": 256}}
|
| 97 |
+
# python main.py -c <val.json> -p val -enable_wandb "" --seed 1
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
Sampling is genuinely one network evaluation per image, which is the point of
|
| 101 |
+
the method.
|
| 102 |
+
|
| 103 |
+
As on the DDPM card, the RGB loader is the one thing you must sort out: upstream
|
| 104 |
+
hard-codes a grayscale EO target and a two-channel condition, and with three
|
| 105 |
+
channels its `ddim_sample()` slices the condition and concatenates mismatched
|
| 106 |
+
shapes. We patched `SAR2EODataset.__getitem__` in memory to return
|
| 107 |
+
`HR = EO[0:3]`, `LR = SAR[0:3]`, `SR = cat(PPB[0:1], Canny[0:1], SAR[0:1])` and
|
| 108 |
+
then ran the repository's `main.py` verbatim.
|
| 109 |
+
|
| 110 |
+
## Read before using this checkpoint
|
| 111 |
+
|
| 112 |
+
* **This is not an oracle.** Worth stating, because a one-step generator invites
|
| 113 |
+
the question: the sampler's `condition_x` is the `[PPB, Canny, SAR]` stack, and
|
| 114 |
+
the ground truth never enters it. The cell also passes our leak audit on both
|
| 115 |
+
datasets.
|
| 116 |
+
* **Stage 2 needs the *real* vision-aided CLIP discriminator.** Stage 1 has
|
| 117 |
+
`lambda_gan = 0` and can import a stub; stage 2 cannot. If a constant-output
|
| 118 |
+
stub shadows the real `vision_aided_loss` package, stage 2 trains against a
|
| 119 |
+
constant GAN loss and **silently is not the authors' method**. Assert on
|
| 120 |
+
`vision_aided_loss.__file__` before starting a run.
|
| 121 |
+
* **SoftPool.** The code imports the SoftPool CUDA extension unconditionally. We
|
| 122 |
+
used a pure-PyTorch drop-in with identical maths
|
| 123 |
+
(`softpool(x) = avgpool(x·eˣ) / avgpool(eˣ)`) and identical autograd, which
|
| 124 |
+
needs no build step.
|
| 125 |
+
* **The checkpoint loads with `strict=False`.** A mismatched configuration
|
| 126 |
+
**loads nothing and raises no error**. Verify the `Loading pretrained model for
|
| 127 |
+
G [...]` log line and look at the first output.
|
| 128 |
+
* **`-p val` writes next to the checkpoint and then renames** the sample
|
| 129 |
+
directory to `<prefix>_S<ssim>_P<psnr>_l2<l2>_Lp<lpips>`. That rename raises if
|
| 130 |
+
the target already exists — after the whole inference has been paid for.
|
| 131 |
+
* **`-enable_wandb ""` is mandatory**; the flag defaults to the truthy string
|
| 132 |
+
`'false'`.
|
| 133 |
+
* One change we made to the repository: `core/logger.py` no longer
|
| 134 |
+
unconditionally overwrites `CUDA_VISIBLE_DEVICES` from the config's `gpu_ids`,
|
| 135 |
+
which on a shared machine remapped every job onto another user's device. It is
|
| 136 |
+
a scheduling fix and does not affect the model.
|
| 137 |
+
|
| 138 |
+
## Licence — stated factually ⚠ no upstream licence exists
|
| 139 |
+
|
| 140 |
+
**The upstream code base publishes no licence.**
|
| 141 |
+
[`DeepSARRS/E3Diff`](https://github.com/DeepSARRS/E3Diff) has no LICENSE,
|
| 142 |
+
LICENCE, COPYING or NOTICE file at the repository root or at any depth; **1 of
|
| 143 |
+
its 58 tracked files** is licence-shaped, and it is `SoftPool/LICENSE.txt`, the
|
| 144 |
+
MIT licence of a **vendored third-party dependency** (`Copyright (c) 2020
|
| 145 |
+
Alexandros Stergiou`, reproduced here as `licenses/LICENSE-SoftPool-MIT.txt`) —
|
| 146 |
+
not a grant covering E3Diff. Its README has no licence section, and the GitHub
|
| 147 |
+
API reports no declared licence, with the `/license` endpoint returning 404.
|
| 148 |
+
Checked 2026-08-28.
|
| 149 |
+
|
| 150 |
+
Under default copyright that means **all rights are reserved by the authors and
|
| 151 |
+
no express permission to redistribute derived work has been granted** to us or
|
| 152 |
+
to you. We publish this checkpoint anyway, so that the benchmark is reproducible
|
| 153 |
+
end to end, and we state the position plainly rather than implying a permission
|
| 154 |
+
that does not exist. If you intend to redistribute this checkpoint or build on
|
| 155 |
+
it, assess that for yourself, and consider asking the authors directly.
|
| 156 |
+
|
| 157 |
+
**Lineage.** E3Diff's README credits SR3
|
| 158 |
+
(`Janspiry/Image-Super-Resolution-via-Iterative-Refinement`, Apache-2.0),
|
| 159 |
+
`GaParmar/img2img-turbo` (MIT) and `alexandrosstergiou/SoftPool` (MIT). Those
|
| 160 |
+
licences cover the borrowed parts only, not the authors' own two-stage
|
| 161 |
+
contribution. We ship only the SoftPool text, because it is the only one of the
|
| 162 |
+
three that is vendored in the tree we trained from.
|
| 163 |
+
|
| 164 |
+
Please cite: Qin, Zou, Li and Zhang, *Efficient End-to-End Diffusion Model for
|
| 165 |
+
One-Step SAR-to-Optical Translation*, IEEE Geoscience and Remote Sensing
|
| 166 |
+
Letters, [doi:10.1109/LGRS.2024.3506566](https://doi.org/10.1109/LGRS.2024.3506566).
|
| 167 |
+
The full record of what we checked, per method, is in
|
| 168 |
+
`licenses/NO-UPSTREAM-LICENSE.md` at the root of this repository.
|
| 169 |
+
|
| 170 |
+
---
|
| 171 |
+
|
| 172 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 173 |
+
checkpoint that we retrained ourselves on QXS-SAROPT; it is not ReFlowSET
|
| 174 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 175 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 176 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 177 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/qxs-saropt/hidiff/README.md
ADDED
|
@@ -0,0 +1,162 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# HI-Diff — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**Hierarchical Integration Diffusion Model for Realistic Image Deblurring**,
|
| 4 |
+
Chen, Zhang, Liu, Kong and Yuan, *NeurIPS 2023*
|
| 5 |
+
([arXiv:2305.12966](https://arxiv.org/abs/2305.12966)). Upstream code:
|
| 6 |
+
[`zhengchen1999/HI-Diff`](https://github.com/zhengchen1999/HI-Diff) at commit
|
| 7 |
+
`b3bfd167997e27f8edd57681cf70e5031a0e35f2`.
|
| 8 |
+
|
| 9 |
+
This is the two-stage latent-prior restoration transformer we retrained
|
| 10 |
+
ourselves on QXS-SAROPT (SAR as the degraded input, EO as the target), and it is
|
| 11 |
+
the checkpoint behind the HI-Diff row of the ReFlowSET comparison table.
|
| 12 |
+
|
| 13 |
+
## What is in this folder
|
| 14 |
+
|
| 15 |
+
| file | bytes | what it is |
|
| 16 |
+
|---|---|---|
|
| 17 |
+
| `S1_net_g_latest.pth` | 101,401,457 | stage-1 restoration transformer — for reproducing stage-2 *training*, not for inference |
|
| 18 |
+
| `S1_net_le_latest.pth` | 2,357,509 | stage-1 latent encoder, `in_chans` **6** (it sees LQ ‖ GT) — **not usable at test time** |
|
| 19 |
+
| `S2_net_d_latest.pth` | 10,530,491 | **the 8-step latent denoiser** — needed at inference |
|
| 20 |
+
| `S2_net_g_latest.pth` | 101,401,457 | **the final restoration transformer** — needed at inference |
|
| 21 |
+
| `S2_net_le_dm_latest.pth` | 2,247,061 | **the stage-2 latent encoder**, `in_chans` 3 (LQ only) — needed at inference |
|
| 22 |
+
|
| 23 |
+
**Only the three `S2_*` files are needed to run this model.** The repository's
|
| 24 |
+
own test configuration loads exactly `pretrain_network_g`,
|
| 25 |
+
`pretrain_network_le_dm` and `pretrain_network_d`, and nothing else. The two
|
| 26 |
+
`S1_*` files are shipped so that stage-2 *training* can be reproduced: stage 1
|
| 27 |
+
learns a latent prior from the LQ **and** the ground truth together, which is why
|
| 28 |
+
its encoder takes 6 input channels and cannot be used at deployment.
|
| 29 |
+
|
| 30 |
+
Architecture: the transformer is Restormer-shaped — `dim` 48, blocks
|
| 31 |
+
[3, 5, 6, 6], 4 refinement blocks, heads [1, 2, 4, 8], FFN expansion 2.66,
|
| 32 |
+
`embed_dim` 64, `group` 4. The latent denoiser runs **8 timesteps** on the
|
| 33 |
+
compressed prior (`linear_start` 0.1, `linear_end` 0.99) — the diffusion here
|
| 34 |
+
acts on a small latent prior, not on the image.
|
| 35 |
+
|
| 36 |
+
## Training budget we used
|
| 37 |
+
|
| 38 |
+
| | |
|
| 39 |
+
|---|---|
|
| 40 |
+
| **generator updates released** | **25,000 (stage 1) + 25,000 (stage 2) = 50,000** |
|
| 41 |
+
| batch size / patch | 8 @ 256 px |
|
| 42 |
+
| optimizer | AdamW, lr 2e-4, weight decay 1e-4, β (0.9, 0.999), gradient clipping on |
|
| 43 |
+
| schedule | `CosineAnnealingRestartCyclicLR`, periods [8,000 / 17,000], η_min [2e-4, 1e-6] |
|
| 44 |
+
| loss | L1 |
|
| 45 |
+
| progressive training | **disabled** |
|
| 46 |
+
| test resolution | tested at native 256 px |
|
| 47 |
+
|
| 48 |
+
The upstream recipe is 300,000 + 300,000 iterations with a progressive
|
| 49 |
+
patch/batch schedule from 128 to 384 px. We disabled the progressive schedule
|
| 50 |
+
(a single fixed 256 px group) and matched the budget to the rest of this
|
| 51 |
+
benchmark's diffusion band at 50,000 total updates. That is a substantial
|
| 52 |
+
reduction from the authors' recipe and is stated here so nobody reads this row
|
| 53 |
+
as HI-Diff at full budget.
|
| 54 |
+
|
| 55 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 56 |
+
|
| 57 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 58 |
+
|---|---|---|---|---|
|
| 59 |
+
| 324.3 | 0.539 | 0.692 | 0.457 | 17.10 |
|
| 60 |
+
|
| 61 |
+
Evaluated on the official split's 3,999 test chips at their native 256 px.
|
| 62 |
+
No subsampling: every metric on this page is measured over all 3,999 pairs.
|
| 63 |
+
|
| 64 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 65 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 66 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 67 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 68 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 69 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 70 |
+
convention gives a systematically different number and the two must never be
|
| 71 |
+
mixed, or compared against a paper that used the other one.
|
| 72 |
+
|
| 73 |
+
## Load it and translate one SAR image
|
| 74 |
+
|
| 75 |
+
The three networks must be assembled by hand; the repository's own `test.py`
|
| 76 |
+
with a `HI_Diff_S2` option file is the path of least resistance.
|
| 77 |
+
|
| 78 |
+
```yaml
|
| 79 |
+
# options/test/<your>.yml -- the three paths below are the three S2_ files
|
| 80 |
+
model_type: HI_Diff_S2
|
| 81 |
+
network_g: {type: Transformer, inp_channels: 3, out_channels: 3, dim: 48,
|
| 82 |
+
num_blocks: [3,5,6,6], num_refinement_blocks: 4, heads: [1,2,4,8],
|
| 83 |
+
ffn_expansion_factor: 2.66, bias: False, LayerNorm_type: WithBias,
|
| 84 |
+
dual_pixel_task: False, embed_dim: 64, group: 4}
|
| 85 |
+
network_le_dm: {type: latent_encoder_gelu, in_chans: 3, embed_dim: 64, block_num: 6,
|
| 86 |
+
group: 4, stage: 2, patch_expansion: 0.5, channel_expansion: 4}
|
| 87 |
+
network_d: {type: denoising, in_channel: 256, out_channel: 256, inner_channel: 512,
|
| 88 |
+
block_num: 4, group: 4, patch_expansion: 0.5, channel_expansion: 2}
|
| 89 |
+
diffusion_schedule: {apply_ldm: False, schedule: linear, timesteps: 8,
|
| 90 |
+
linear_start: 0.1, linear_end: 0.99}
|
| 91 |
+
path:
|
| 92 |
+
pretrain_network_g: S2_net_g_latest.pth
|
| 93 |
+
pretrain_network_le_dm: S2_net_le_dm_latest.pth
|
| 94 |
+
pretrain_network_d: S2_net_d_latest.pth
|
| 95 |
+
param_key_g: params
|
| 96 |
+
```
|
| 97 |
+
|
| 98 |
+
```bash
|
| 99 |
+
python test.py -opt options/test/<your>.yml
|
| 100 |
+
```
|
| 101 |
+
|
| 102 |
+
The SAR input is read with `cv2.IMREAD_COLOR`, so a single-channel SAR image is
|
| 103 |
+
replicated to three channels automatically — that is what training saw. At
|
| 104 |
+
inference `le_dm` produces the prior from the degraded input alone, `net_d`
|
| 105 |
+
denoises that prior over 8 steps, and `net_g` is conditioned on it.
|
| 106 |
+
|
| 107 |
+
## Read before using this checkpoint
|
| 108 |
+
|
| 109 |
+
* **The `S1_*` files are not an inference path.** `S1_net_le_latest.pth` has
|
| 110 |
+
`in_chans: 6` because stage 1 encodes LQ concatenated with the ground truth.
|
| 111 |
+
Wiring it into a test run would make the model an oracle. Use the `S2_*` trio.
|
| 112 |
+
* **These are the `latest` checkpoints — iteration 25,000 — not a best-validation
|
| 113 |
+
selection.** That is deliberate and benchmark-wide: every other method here
|
| 114 |
+
publishes its last checkpoint, so selecting on validation for this one would
|
| 115 |
+
be an asymmetry in its favour. Do not repoint at a best-val file and then
|
| 116 |
+
compare against this table.
|
| 117 |
+
* **Retraining is destructive.** The upstream launcher path is not idempotent:
|
| 118 |
+
the training configs set `resume_state: ~`, no auto-resume flag is passed, and
|
| 119 |
+
basicsr's `mkdir_and_rename` archives a finished experiment directory and
|
| 120 |
+
starts from scratch. Re-running a training script over a finished experiment
|
| 121 |
+
destroys it.
|
| 122 |
+
* **`torch.load` needs `weights_only=False`** under PyTorch ≥ 2.6; we made that
|
| 123 |
+
change in two places in the repository (`hi_diff/utils/base_model.py` and
|
| 124 |
+
`train.py`). Both are compatibility fixes.
|
| 125 |
+
* **How to read this row's numbers.** HI-Diff takes the best PSNR and SSIM of
|
| 126 |
+
any method in this benchmark on both datasets, while sitting at or near the
|
| 127 |
+
worst FID, DISTS and LPIPS. That combination is the signature of
|
| 128 |
+
regression-to-the-mean: a blurred, low-variance prediction minimises squared
|
| 129 |
+
error and is perceptually poor. Reporting its PSNR without its FID would badly
|
| 130 |
+
misrepresent it — and so would the reverse.
|
| 131 |
+
* Passes the identity-collapse audit on both datasets.
|
| 132 |
+
|
| 133 |
+
## Licence — stated factually
|
| 134 |
+
|
| 135 |
+
Upstream `zhengchen1999/HI-Diff` ships a LICENSE file, reproduced verbatim here
|
| 136 |
+
as `licenses/LICENSE-hidiff.txt`: the **Apache License 2.0**, appendix copyright
|
| 137 |
+
line `Copyright 2023 HI-Diff Authors`.
|
| 138 |
+
|
| 139 |
+
**Redistribution of these weights is permitted.** Apache-2.0 §4 obligations
|
| 140 |
+
apply: ship the licence, keep the notices, and **state that you modified the
|
| 141 |
+
files**. We did modify them — the two `torch.load(..., weights_only=False)`
|
| 142 |
+
changes noted above, plus our own option files — and this sentence is that
|
| 143 |
+
statement. Upstream ships no NOTICE file, so there is none to carry. Apache-2.0
|
| 144 |
+
§6 withholds trademark rights.
|
| 145 |
+
|
| 146 |
+
HI-Diff itself builds on BasicSR, Restormer and DiffIR, which its README
|
| 147 |
+
credits; those projects' own licences cover their code.
|
| 148 |
+
|
| 149 |
+
Attribution: *trained with `zhengchen1999/HI-Diff`, Copyright 2023 HI-Diff
|
| 150 |
+
Authors, licensed under the Apache License 2.0; our training and test
|
| 151 |
+
configurations modify the upstream configuration, and two source files were
|
| 152 |
+
modified for PyTorch ≥ 2.6 compatibility. HI-Diff builds on BasicSR, Restormer
|
| 153 |
+
and DiffIR. Full text in `licenses/LICENSE-hidiff.txt`.*
|
| 154 |
+
|
| 155 |
+
---
|
| 156 |
+
|
| 157 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 158 |
+
checkpoint that we retrained ourselves on QXS-SAROPT; it is not ReFlowSET
|
| 159 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 160 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 161 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 162 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/qxs-saropt/p2phd/README.md
ADDED
|
@@ -0,0 +1,142 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# pix2pixHD — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**High-Resolution Image Synthesis and Semantic Manipulation with Conditional
|
| 4 |
+
GANs**, Wang, Liu, Zhu, Tao, Kautz and Catanzaro, *CVPR 2018*
|
| 5 |
+
([arXiv:1711.11585](https://arxiv.org/abs/1711.11585)). Upstream code:
|
| 6 |
+
[`NVIDIA/pix2pixHD`](https://github.com/NVIDIA/pix2pixHD) at commit
|
| 7 |
+
`14b3b3c7fff413086e3b58df52096f16b6891172`.
|
| 8 |
+
|
| 9 |
+
This is the coarse-to-fine paired GAN we retrained ourselves on QXS-SAROPT, and
|
| 10 |
+
it is the checkpoint behind the pix2pixHD row of the ReFlowSET comparison
|
| 11 |
+
table.
|
| 12 |
+
|
| 13 |
+
## What is in this folder
|
| 14 |
+
|
| 15 |
+
| file | bytes | what it is |
|
| 16 |
+
|---|---|---|
|
| 17 |
+
| `net_G.pth` | 729,795,505 | the global generator, and the only file inference needs |
|
| 18 |
+
|
| 19 |
+
`net_G.pth` is the **`global` generator** (`GlobalGenerator`):
|
| 20 |
+
`n_downsample_global` 4, `n_blocks_global` 9 residual blocks, `ngf = 64`,
|
| 21 |
+
instance normalisation, 3 in / 3 out. The run was configured with
|
| 22 |
+
`n_local_enhancers 1`, but at `netG=global` the local enhancer is never
|
| 23 |
+
constructed and is not in this file — this is the single-scale generator.
|
| 24 |
+
|
| 25 |
+
The run used `--label_nc 0 --no_instance`, i.e. the "label" input is a real
|
| 26 |
+
image (the SAR chip) rather than a segmentation map, and no instance-edge
|
| 27 |
+
channel is concatenated. The two discriminators (`num_D` 2) are not released.
|
| 28 |
+
|
| 29 |
+
## Training budget we used
|
| 30 |
+
|
| 31 |
+
| | |
|
| 32 |
+
|---|---|
|
| 33 |
+
| **generator updates released** | **120,000** |
|
| 34 |
+
| batch size | 16 |
|
| 35 |
+
| epochs (`niter` + `niter_decay`) | 60 + 60 |
|
| 36 |
+
| iterations per epoch | 1,000 (16,001 training pairs; this loader floor-rounds, unlike the junyanz loaders which round up) |
|
| 37 |
+
| optimizer | Adam, lr 2e-4, β₁ 0.5, linear decay over the second half |
|
| 38 |
+
| adversarial loss | LSGAN, multi-scale discriminator (`num_D` 2, `n_layers_D` 3) |
|
| 39 |
+
| feature-matching loss | λ_feat 10 |
|
| 40 |
+
| perceptual loss | VGG loss enabled |
|
| 41 |
+
| augmentation | horizontal flip |
|
| 42 |
+
| input | 256 px (`resize_and_crop`, `loadSize` 256, `fineSize` 256) |
|
| 43 |
+
|
| 44 |
+
The floor-vs-ceiling detail is not pedantry: pix2pixHD and SPADE drop the last
|
| 45 |
+
partial batch while the junyanz-family loaders do not, so the same "epochs"
|
| 46 |
+
figure yields slightly different update counts across the table. The update
|
| 47 |
+
counts above are what actually ran.
|
| 48 |
+
|
| 49 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 50 |
+
|
| 51 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 52 |
+
|---|---|---|---|---|
|
| 53 |
+
| 85.7 | 0.298 | 0.573 | 0.358 | 16.13 |
|
| 54 |
+
|
| 55 |
+
Evaluated on the official split's 3,999 test chips at their native 256 px.
|
| 56 |
+
No subsampling: every metric on this page is measured over all 3,999 pairs.
|
| 57 |
+
|
| 58 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 59 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 60 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 61 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 62 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 63 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 64 |
+
convention gives a systematically different number and the two must never be
|
| 65 |
+
mixed, or compared against a paper that used the other one.
|
| 66 |
+
|
| 67 |
+
## Load it and translate one SAR image
|
| 68 |
+
|
| 69 |
+
`GlobalGenerator` can be constructed directly, without the repository's option
|
| 70 |
+
machinery:
|
| 71 |
+
|
| 72 |
+
```python
|
| 73 |
+
import numpy as np, torch, torch.nn as nn
|
| 74 |
+
from PIL import Image
|
| 75 |
+
from models.networks import GlobalGenerator # from the NVIDIA/pix2pixHD repo
|
| 76 |
+
|
| 77 |
+
G = GlobalGenerator(3, 3, 64, 4, 9, nn.InstanceNorm2d)
|
| 78 |
+
G.load_state_dict(torch.load('net_G.pth', map_location='cpu'))
|
| 79 |
+
G.eval()
|
| 80 |
+
|
| 81 |
+
sar = Image.open('sar.png').convert('RGB')
|
| 82 |
+
x = torch.from_numpy(np.asarray(sar, np.float32) / 127.5 - 1).permute(2, 0, 1)[None]
|
| 83 |
+
|
| 84 |
+
with torch.no_grad():
|
| 85 |
+
y = G(x)
|
| 86 |
+
|
| 87 |
+
eo = ((y[0].permute(1, 2, 0).numpy() + 1) * 127.5).clip(0, 255).astype(np.uint8)
|
| 88 |
+
Image.fromarray(eo).save('eo.png')
|
| 89 |
+
```
|
| 90 |
+
|
| 91 |
+
Through the repository's own `test.py` the equivalent invocation is
|
| 92 |
+
`--label_nc 0 --no_instance --which_epoch latest`, and the output lands as
|
| 93 |
+
`<stem>_synthesized_image.png`.
|
| 94 |
+
|
| 95 |
+
## Read before using this checkpoint
|
| 96 |
+
|
| 97 |
+
* **Upstream writes its test outputs as JPEG.** `util/visualizer.py` hard-codes
|
| 98 |
+
`'%s_%s.jpg'`. We changed that one line to `.png` before scoring, because
|
| 99 |
+
every other method in this benchmark writes PNG and JPEG re-compression would
|
| 100 |
+
have biased FID and LPIPS for this row alone. **If you clone upstream
|
| 101 |
+
unmodified and evaluate, your numbers will not match ours**, and the gap is an
|
| 102 |
+
artefact of the file format, not of the model.
|
| 103 |
+
* Three further changes we made to the repository are pure Python-3.12 /
|
| 104 |
+
modern-library compatibility and do not affect the weights:
|
| 105 |
+
`transforms.Scale` → `transforms.Resize`; `fractions.gcd` → `math.gcd` with
|
| 106 |
+
integer division; and a guarded `import scipy.misc` (needed only for
|
| 107 |
+
`--tf_log`, which we never pass).
|
| 108 |
+
* Normalisation is `[-1, 1]`.
|
| 109 |
+
* Passes the identity-collapse audit on both datasets.
|
| 110 |
+
|
| 111 |
+
## Licence — stated factually
|
| 112 |
+
|
| 113 |
+
Upstream `NVIDIA/pix2pixHD` ships `LICENSE.txt`, reproduced verbatim here as
|
| 114 |
+
`licenses/LICENSE-p2phd.txt`. It contains two notices:
|
| 115 |
+
|
| 116 |
+
* `Copyright (C) 2019 NVIDIA Corporation. Ting-Chun Wang, Ming-Yu Liu,
|
| 117 |
+
Jun-Yan Zhu. BSD License. All rights reserved.` — two conditions and a
|
| 118 |
+
disclaimer;
|
| 119 |
+
* the bundled `pytorch-CycleGAN-and-pix2pix` notice,
|
| 120 |
+
`Copyright (c) 2017, Jun-Yan Zhu and Taesung Park`, because pix2pixHD's code
|
| 121 |
+
borrows heavily from it.
|
| 122 |
+
|
| 123 |
+
**Redistribution of these weights is permitted** with both notices reproduced.
|
| 124 |
+
Despite NVIDIA's non-commercial licensing of some of its other releases, this
|
| 125 |
+
one carries **no non-commercial clause and no share-alike clause** — we checked
|
| 126 |
+
the file and the README, and the README has no licence section at all, so
|
| 127 |
+
`LICENSE.txt` is the entire grant.
|
| 128 |
+
|
| 129 |
+
Attribution: *trained with `NVIDIA/pix2pixHD`. Copyright (C) 2019 NVIDIA
|
| 130 |
+
Corporation — Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu. BSD License, all rights
|
| 131 |
+
reserved; includes the bundled pytorch-CycleGAN-and-pix2pix notice (Copyright
|
| 132 |
+
(c) 2017, Jun-Yan Zhu and Taesung Park). Full text in
|
| 133 |
+
`licenses/LICENSE-p2phd.txt`.*
|
| 134 |
+
|
| 135 |
+
---
|
| 136 |
+
|
| 137 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 138 |
+
checkpoint that we retrained ourselves on QXS-SAROPT; it is not ReFlowSET
|
| 139 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 140 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 141 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 142 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/qxs-saropt/pix2pix/README.md
ADDED
|
@@ -0,0 +1,141 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# pix2pix — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**Image-to-Image Translation with Conditional Adversarial Networks**, Isola, Zhu,
|
| 4 |
+
Zhou and Efros, *CVPR 2017* ([arXiv:1611.07004](https://arxiv.org/abs/1611.07004)).
|
| 5 |
+
Upstream code:
|
| 6 |
+
[`junyanz/pytorch-CycleGAN-and-pix2pix`](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix)
|
| 7 |
+
at commit `2a7afba2895d52556dd5dfe07e8555ef657ced6f`.
|
| 8 |
+
|
| 9 |
+
This is the paired conditional GAN we retrained ourselves on QXS-SAROPT, and it
|
| 10 |
+
is the checkpoint behind the pix2pix row of the ReFlowSET comparison table.
|
| 11 |
+
|
| 12 |
+
## What is in this folder
|
| 13 |
+
|
| 14 |
+
| file | bytes | what it is |
|
| 15 |
+
|---|---|---|
|
| 16 |
+
| `net_G.pth` | 217,728,293 | the generator, and the only file inference needs |
|
| 17 |
+
|
| 18 |
+
`net_G.pth` is a **`unet_256` generator**: a U-Net with batch normalisation,
|
| 19 |
+
`ngf = 64`, 3 input channels and 3 output channels, saved from a single-GPU run
|
| 20 |
+
so the state-dict keys are bare (no `module.` prefix). It maps **SAR → EO**:
|
| 21 |
+
the training set was built as side-by-side `A|B` tiles with SAR on the left, and
|
| 22 |
+
the run used `--direction AtoB`.
|
| 23 |
+
|
| 24 |
+
The discriminator is not part of this release — upstream's `test.py` constructs
|
| 25 |
+
only the generator, so `net_G.pth` is sufficient to reproduce every image we
|
| 26 |
+
scored.
|
| 27 |
+
|
| 28 |
+
## Training budget we used
|
| 29 |
+
|
| 30 |
+
| | |
|
| 31 |
+
|---|---|
|
| 32 |
+
| **generator updates released** | **120,120** |
|
| 33 |
+
| batch size | 16 |
|
| 34 |
+
| epochs (constant LR + linear decay) | 60 + 60 |
|
| 35 |
+
| iterations per epoch | 1,001 (16,001 training pairs, the loader does not drop the last batch) |
|
| 36 |
+
| optimizer | Adam, lr 2e-4, β₁ 0.5, linear decay to 0 over the second half |
|
| 37 |
+
| adversarial loss | vanilla GAN; discriminator `basic` (70×70 PatchGAN, `n_layers_D` 3) |
|
| 38 |
+
| reconstruction loss | L1, weight 100 |
|
| 39 |
+
| augmentation | horizontal flip |
|
| 40 |
+
| input | 256 px, no resize (the chips are already 256 px) |
|
| 41 |
+
|
| 42 |
+
We quote budgets in **optimizer updates**, never in epochs: the two datasets
|
| 43 |
+
differ by more than 10× in size, so "200 epochs" means something completely
|
| 44 |
+
different on each and would silently favour one of them.
|
| 45 |
+
|
| 46 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 47 |
+
|
| 48 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 49 |
+
|---|---|---|---|---|
|
| 50 |
+
| 174.6 | 0.373 | 0.665 | 0.203 | 12.33 |
|
| 51 |
+
|
| 52 |
+
Evaluated on the official split's 3,999 test chips at their native 256 px.
|
| 53 |
+
No subsampling: every metric on this page is measured over all 3,999 pairs.
|
| 54 |
+
|
| 55 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 56 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 57 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 58 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 59 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 60 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 61 |
+
convention gives a systematically different number and the two must never be
|
| 62 |
+
mixed, or compared against a paper that used the other one.
|
| 63 |
+
|
| 64 |
+
## Load it and translate one SAR image
|
| 65 |
+
|
| 66 |
+
The snippet uses the upstream repository's own `define_G`, so clone
|
| 67 |
+
`junyanz/pytorch-CycleGAN-and-pix2pix` at the commit above and run this from its
|
| 68 |
+
root. No patch is needed for this method — our only change to that repository was
|
| 69 |
+
an import fix in the dataset-preparation helper, which does not affect the model.
|
| 70 |
+
|
| 71 |
+
```python
|
| 72 |
+
import numpy as np, torch
|
| 73 |
+
from PIL import Image
|
| 74 |
+
from models.networks import define_G # from the junyanz repo
|
| 75 |
+
|
| 76 |
+
G = define_G(3, 3, 64, 'unet_256', 'batch', use_dropout=False,
|
| 77 |
+
init_type='normal', init_gain=0.02, gpu_ids=[])
|
| 78 |
+
sd = torch.load('net_G.pth', map_location='cpu')
|
| 79 |
+
if hasattr(sd, '_metadata'):
|
| 80 |
+
del sd._metadata # upstream strips this too
|
| 81 |
+
G.load_state_dict(sd)
|
| 82 |
+
G.eval()
|
| 83 |
+
|
| 84 |
+
# SAR input -> RGB -> [-1, 1]. QXS-SAROPT SAR is single-channel; .convert('RGB')
|
| 85 |
+
# replicates it to three channels, which is what training saw.
|
| 86 |
+
sar = Image.open('sar.png').convert('RGB')
|
| 87 |
+
x = torch.from_numpy(np.asarray(sar, np.float32) / 127.5 - 1).permute(2, 0, 1)[None]
|
| 88 |
+
|
| 89 |
+
with torch.no_grad():
|
| 90 |
+
y = G(x)
|
| 91 |
+
|
| 92 |
+
eo = ((y[0].permute(1, 2, 0).numpy() + 1) * 127.5).clip(0, 255).astype(np.uint8)
|
| 93 |
+
Image.fromarray(eo).save('eo.png')
|
| 94 |
+
```
|
| 95 |
+
|
| 96 |
+
The image this produces is `<stem>_fake_B.png` in upstream's own `test.py`
|
| 97 |
+
output tree, which is exactly what we scored.
|
| 98 |
+
|
| 99 |
+
## Read before using this checkpoint
|
| 100 |
+
|
| 101 |
+
* **`define_G` must be called with `gpu_ids=[]`.** Our checkpoints were saved
|
| 102 |
+
from single-GPU runs and have bare keys; building the network with a non-empty
|
| 103 |
+
`gpu_ids` wraps it in `DataParallel` and the load then fails on a `module.`
|
| 104 |
+
prefix mismatch.
|
| 105 |
+
* **The normalisation is `[-1, 1]`, not `[0, 1]`.** Feeding `[0, 1]` produces a
|
| 106 |
+
plausible-looking but systematically wrong image.
|
| 107 |
+
* This row is a **paired** method and was trained on aligned SAR/EO pairs. It
|
| 108 |
+
passes the identity-collapse audit on both datasets.
|
| 109 |
+
|
| 110 |
+
## Licence — stated factually
|
| 111 |
+
|
| 112 |
+
Upstream `junyanz/pytorch-CycleGAN-and-pix2pix` ships a LICENSE file, reproduced
|
| 113 |
+
verbatim in this repository as `licenses/LICENSE-pix2pix.txt`. It is **not a
|
| 114 |
+
single SPDX licence** but three stacked notices, and the whole file is what
|
| 115 |
+
travels with these weights:
|
| 116 |
+
|
| 117 |
+
* CycleGAN — `Copyright (c) 2017, Jun-Yan Zhu and Taesung Park`, two conditions
|
| 118 |
+
and a disclaimer (BSD-2-Clause in form);
|
| 119 |
+
* pix2pix — `Copyright (c) 2016, Phillip Isola and Jun-Yan Zhu`, two conditions;
|
| 120 |
+
* DCGAN — `Copyright (c) 2015, Facebook, Inc.`, three conditions including a
|
| 121 |
+
no-endorsement clause (BSD-3-Clause).
|
| 122 |
+
|
| 123 |
+
**Redistribution of these weights is permitted** with that notice reproduced.
|
| 124 |
+
There is no non-commercial and no share-alike clause. The DCGAN section's
|
| 125 |
+
no-endorsement clause means Facebook's name must not be used to promote this
|
| 126 |
+
release.
|
| 127 |
+
|
| 128 |
+
Attribution: *trained with `junyanz/pytorch-CycleGAN-and-pix2pix`. Copyright (c)
|
| 129 |
+
2017, Jun-Yan Zhu and Taesung Park (CycleGAN); Copyright (c) 2016, Phillip Isola
|
| 130 |
+
and Jun-Yan Zhu (pix2pix); Copyright (c) 2015, Facebook, Inc. (DCGAN). All rights
|
| 131 |
+
reserved. Redistributed under the BSD terms reproduced in
|
| 132 |
+
`licenses/LICENSE-pix2pix.txt`.*
|
| 133 |
+
|
| 134 |
+
---
|
| 135 |
+
|
| 136 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 137 |
+
checkpoint that we retrained ourselves on QXS-SAROPT; it is not ReFlowSET
|
| 138 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 139 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 140 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 141 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/qxs-saropt/resshift/README.md
ADDED
|
@@ -0,0 +1,144 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ResShift — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**ResShift: Efficient Diffusion Model for Image Super-Resolution by Residual
|
| 4 |
+
Shifting**, Yue, Wang and Loy, *NeurIPS 2023*
|
| 5 |
+
([arXiv:2307.12348](https://arxiv.org/abs/2307.12348)). Upstream code:
|
| 6 |
+
[`zsyOAOA/ResShift`](https://github.com/zsyOAOA/ResShift) at commit
|
| 7 |
+
`bb03b7d21614cace01787e097c8a6ab6b945227d`.
|
| 8 |
+
|
| 9 |
+
This is the residual-shifting latent diffusion model we retrained ourselves on
|
| 10 |
+
QXS-SAROPT, run as a **same-size** mapping (`sf = 1`: SAR in, EO out, no
|
| 11 |
+
upscaling), and it is the checkpoint behind the ResShift row of the ReFlowSET
|
| 12 |
+
comparison table.
|
| 13 |
+
|
| 14 |
+
## What is in this folder
|
| 15 |
+
|
| 16 |
+
| file | bytes | what it is |
|
| 17 |
+
|---|---|---|
|
| 18 |
+
| `ema_model.pth` | 478,392,663 | EMA weights of the denoising UNet at iteration 50,000 |
|
| 19 |
+
|
| 20 |
+
`ema_model.pth` is the EMA (rate 0.999) of a **`models.unet.UNetModelSwin`**:
|
| 21 |
+
latent resolution 64, `model_channels` 160, `channel_mult` [1, 2, 2, 4],
|
| 22 |
+
`num_res_blocks` [2, 2, 2, 2], attention at [64, 32, 16, 8], 32 channels per
|
| 23 |
+
head, Swin blocks of depth 2 with `swin_embed_dim` 192 and window size 8,
|
| 24 |
+
`cond_lq = True`, `lq_size = 256`.
|
| 25 |
+
|
| 26 |
+
**This is a latent-space model and this file is not sufficient to run it.** It
|
| 27 |
+
additionally needs the external CompVis **vq-f4** autoencoder
|
| 28 |
+
(`ldm.models.autoencoder.VQModelTorch`, 221,364,711 bytes), which is **not**
|
| 29 |
+
redistributed here.
|
| 30 |
+
|
| 31 |
+
## Training budget we used
|
| 32 |
+
|
| 33 |
+
| | |
|
| 34 |
+
|---|---|
|
| 35 |
+
| **generator updates released** | **50,000** |
|
| 36 |
+
| batch size | 16 (micro-batch 8, i.e. 2 gradient-accumulation steps per update) |
|
| 37 |
+
| optimizer | Adam, lr 5e-5 → 2e-5 cosine, 2,000-step warmup, no weight decay |
|
| 38 |
+
| EMA | rate 0.999 |
|
| 39 |
+
| diffusion | **4 steps**, exponential noise schedule (power 0.3), `etas_end` 0.99, `min_noise_level` 0.2, κ = 2.0, `predict_type` xstart, `sf` 1 |
|
| 40 |
+
| loss weights | latent MSE 1.0, pixel LPIPS (AlexNet) 4.0, pixel MSE 0.0 |
|
| 41 |
+
| precision | AMP on |
|
| 42 |
+
| training crops | random 256 px, normalised with mean 0.5 / std 0.5 |
|
| 43 |
+
| seed | 123456 |
|
| 44 |
+
|
| 45 |
+
ResShift is a **4-step** sampler by construction; that is the method, not a
|
| 46 |
+
shortcut we took.
|
| 47 |
+
|
| 48 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 49 |
+
|
| 50 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 51 |
+
|---|---|---|---|---|
|
| 52 |
+
| 140.2 | 0.334 | 0.607 | 0.217 | 14.20 |
|
| 53 |
+
|
| 54 |
+
Evaluated on the official split's 3,999 test chips at their native 256 px.
|
| 55 |
+
No subsampling: every metric on this page is measured over all 3,999 pairs.
|
| 56 |
+
|
| 57 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 58 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 59 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 60 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 61 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 62 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 63 |
+
convention gives a systematically different number and the two must never be
|
| 64 |
+
mixed, or compared against a paper that used the other one.
|
| 65 |
+
|
| 66 |
+
## Load it and translate one SAR image
|
| 67 |
+
|
| 68 |
+
Upstream's `inference_resshift.py` is a super-resolution CLI keyed to the
|
| 69 |
+
authors' own released tasks and does not accept a SAR→EO configuration. We wrote
|
| 70 |
+
a small driver against the repository's `ResShiftSampler`; the equivalent is:
|
| 71 |
+
|
| 72 |
+
```python
|
| 73 |
+
from omegaconf import OmegaConf
|
| 74 |
+
from sampler import ResShiftSampler # from the zsyOAOA/ResShift repo
|
| 75 |
+
|
| 76 |
+
cfg = OmegaConf.load('<your>_sar2eo_256.yaml') # sf 1, steps 4, predict_type xstart
|
| 77 |
+
cfg.model.ckpt_path = 'ema_model.pth'
|
| 78 |
+
cfg.autoencoder.ckpt_path = '<path to>/autoencoder_vq_f4.pth' # NOT shipped here
|
| 79 |
+
|
| 80 |
+
sampler = ResShiftSampler(cfg, sf=1, chop_size=256, chop_stride=256,
|
| 81 |
+
chop_bs=1, use_amp=True, seed=12345, padding_offset=None)
|
| 82 |
+
sampler.inference('<dir of SAR images>', '<output dir>', bs=16, noise_repeat=False)
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
At 256 px no tiling happens. Outputs keep the input file stems.
|
| 86 |
+
|
| 87 |
+
## Read before using this checkpoint
|
| 88 |
+
|
| 89 |
+
* **The external vq-f4 autoencoder is mandatory and is not in this repository.**
|
| 90 |
+
Without it the model cannot be constructed, let alone run. It is CompVis
|
| 91 |
+
latent-diffusion material under its own (MIT) licence.
|
| 92 |
+
* **Upstream's own inference CLI will not drive these weights.** It is written
|
| 93 |
+
around the authors' released super-resolution tasks. Use the sampler directly,
|
| 94 |
+
as above.
|
| 95 |
+
* **xFormers must be disabled.** On the accelerator we trained on, the installed
|
| 96 |
+
xFormers build has no usable kernel for these shapes (attention head dimension
|
| 97 |
+
512 exceeds its limit, and fp32 is unsupported), so we set
|
| 98 |
+
`XFORMERS_IS_AVAILBLE = False` in three files
|
| 99 |
+
(`ldm/modules/attention.py`, `ldm/modules/diffusionmodules/model.py`,
|
| 100 |
+
`models/unet.py`). If your hardware differs, this is a performance switch, not
|
| 101 |
+
a correctness one.
|
| 102 |
+
* **One numerical guard in the image logger.** fp16 diffused latents decoded
|
| 103 |
+
through the fp32 VQGAN produce NaN/Inf at early iterations and crashed training
|
| 104 |
+
*inside the logging path*. We added a `nan_to_num` + `clamp` before
|
| 105 |
+
`make_grid`. It is logging-only; the loss path is untouched. (A separate
|
| 106 |
+
`nan_to_num` on the LPIPS loss is upstream's, not ours.)
|
| 107 |
+
* **There is no auto-resume.** A crash mid-run needs a manual relaunch pointing
|
| 108 |
+
at the last saved checkpoint.
|
| 109 |
+
* Passes the identity-collapse audit on both datasets.
|
| 110 |
+
|
| 111 |
+
## Licence — stated factually ⚠ non-commercial
|
| 112 |
+
|
| 113 |
+
Upstream `zsyOAOA/ResShift` ships a LICENSE file, reproduced verbatim here as
|
| 114 |
+
`licenses/LICENSE-resshift.txt`: the **S-Lab License 1.0**,
|
| 115 |
+
`Copyright 2022 S-Lab`.
|
| 116 |
+
|
| 117 |
+
**This licence is non-commercial.** Its opening clause reads *"Redistribution
|
| 118 |
+
and use **for non-commercial purpose** in source and binary forms, with or
|
| 119 |
+
without modification, are permitted provided that the following conditions are
|
| 120 |
+
met"*, and it adds that commercial use requires contacting the contributors. The
|
| 121 |
+
README repeats it: *"This project is licensed under NTU S-Lab License 1.0.
|
| 122 |
+
Redistribution and use should follow this license."*
|
| 123 |
+
|
| 124 |
+
So: **redistribution of these weights is permitted for non-commercial purposes
|
| 125 |
+
only**, in binary form, with the copyright notice, the conditions and the
|
| 126 |
+
disclaimer reproduced — which is what `licenses/LICENSE-resshift.txt` is for.
|
| 127 |
+
Clause 3 is a **no-endorsement** clause: neither S-Lab's name nor its
|
| 128 |
+
contributors' names may be used to endorse or promote this release.
|
| 129 |
+
|
| 130 |
+
Attribution: *trained with `zsyOAOA/ResShift`, Copyright 2022 S-Lab, **S-Lab
|
| 131 |
+
License 1.0 — redistribution and use for non-commercial purposes only**; for
|
| 132 |
+
commercial use, contact the contributors. Requires the CompVis vq-f4
|
| 133 |
+
autoencoder (MIT), not redistributed here. Neither S-Lab's name nor its
|
| 134 |
+
contributors' names may be used to endorse this release. Full text in
|
| 135 |
+
`licenses/LICENSE-resshift.txt`.*
|
| 136 |
+
|
| 137 |
+
---
|
| 138 |
+
|
| 139 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 140 |
+
checkpoint that we retrained ourselves on QXS-SAROPT; it is not ReFlowSET
|
| 141 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 142 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 143 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 144 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/qxs-saropt/sd21ft/README.md
ADDED
|
@@ -0,0 +1,156 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# SD2.1 fine-tune only — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
The **"SD2.1 fine-tune only"** ablation row of the ReFlowSET comparison table:
|
| 4 |
+
Stable Diffusion 2.1-base fine-tuned for SAR-to-EO translation with a plain MSE
|
| 5 |
+
epsilon loss and **no confidence head**.
|
| 6 |
+
|
| 7 |
+
> **This is not a stock SD 2.1 UNet and it will not load into a stock SD 2.1
|
| 8 |
+
> pipeline.** It is C-DiffSET **stage 1**: the input convolution is widened to
|
| 9 |
+
> 8 channels so the SAR latent can be concatenated to the noisy EO latent. Only
|
| 10 |
+
> the output side is unmodified.
|
| 11 |
+
|
| 12 |
+
Venue: there is no separate paper for this row. The backbone is Stable
|
| 13 |
+
Diffusion 2.1-base (latent diffusion, Rombach et al., CVPR 2022); the trainer
|
| 14 |
+
is the stage-1 trainer of
|
| 15 |
+
[`KAIST-VICLab/C-DiffSET`](https://github.com/KAIST-VICLab/C-DiffSET)
|
| 16 |
+
(*IEEE TCSVT* 2026, [doi:10.1109/TCSVT.2026.3701447](https://doi.org/10.1109/TCSVT.2026.3701447)).
|
| 17 |
+
|
| 18 |
+
## What is in this folder
|
| 19 |
+
|
| 20 |
+
A single `UNet2DConditionModel` (865,922,244 parameters, fp32, 686 tensors):
|
| 21 |
+
|
| 22 |
+
* **`conv_in` 4 -> 8 channels**, exactly as in the C-DiffSET folder — the input
|
| 23 |
+
is `torch.cat([sar_latent, noisy_eo_latent], dim=1)`, **SAR first**. In this
|
| 24 |
+
trained checkpoint the two halves have diverged (mean |w| 0.01443 on channels
|
| 25 |
+
0-3 vs 0.02727 on channels 4-7).
|
| 26 |
+
* **`conv_out` unchanged at 4 channels.** No variance head, no confidence map.
|
| 27 |
+
The stage-1 trainer explicitly overrides the channel surgery to a no-op, and
|
| 28 |
+
the whole 4-channel prediction goes to the scheduler.
|
| 29 |
+
|
| 30 |
+
That 5-vs-4 output channel count is the only architectural difference from the
|
| 31 |
+
`cdiffset` folder in this repository; the parameter counts differ by exactly
|
| 32 |
+
2,881 (one 320x3x3 filter plus one bias).
|
| 33 |
+
|
| 34 |
+
## Training budget we used
|
| 35 |
+
|
| 36 |
+
| | |
|
| 37 |
+
|---|---|
|
| 38 |
+
| optimizer updates released | **40,000** (fixed-step snapshot `checkpoint-40000`) |
|
| 39 |
+
| updates the run was configured for | 50,000 |
|
| 40 |
+
| initialised from | the SD 2.1-base UNet with the duplicated-and-halved 8-channel `conv_in` |
|
| 41 |
+
| batch size / resolution | 64 @ 256 px |
|
| 42 |
+
| epochs the 40,000 updates work out to | ~160 (16,001 training pairs, 250 updates/epoch) |
|
| 43 |
+
| optimizer | AdamW, lr 3e-5, weight decay 0.01, cosine schedule, 100-step warmup |
|
| 44 |
+
| precision | fp32 (no mixed precision) |
|
| 45 |
+
| seed | 2024 |
|
| 46 |
+
| loss | plain MSE on the epsilon prediction |
|
| 47 |
+
| prompt | the fixed string `"electro-optical image"` |
|
| 48 |
+
| augmentation | horizontal flip, vertical flip, 90-degree rotation |
|
| 49 |
+
| test-time sampler | **DDIM, 50 steps**, fp32, one image at a time |
|
| 50 |
+
|
| 51 |
+
Same two caveats as the C-DiffSET folder: the snapshot is taken at update 40,000
|
| 52 |
+
of a cosine schedule laid out for 50,000, so the learning rate had not finished
|
| 53 |
+
decaying; and it is a fixed-step snapshot, not the repository's
|
| 54 |
+
validation-selected `best/`. For this stage the fixed-step choice is also the
|
| 55 |
+
better one on the merits — validation LPIPS bottoms out near update 41,000 and
|
| 56 |
+
then degrades toward 50,000.
|
| 57 |
+
|
| 58 |
+
Data: 256 px, no crop and no resize (the chips are already 256 px). SAR chips come from the `sar_256_oc_0.2` directory of QXS-SAROPT; the EO target is the same stem under `opt_256_oc_0.2`.
|
| 59 |
+
|
| 60 |
+
## Base model
|
| 61 |
+
|
| 62 |
+
This checkpoint is a derivative of **Stable Diffusion 2.1-base**. Because the
|
| 63 |
+
original `stabilityai/stable-diffusion-2-1-base` repository is no longer
|
| 64 |
+
available on the Hub, our training and every snippet below load the community
|
| 65 |
+
mirror [`Manojb/stable-diffusion-2-1-base`](https://huggingface.co/Manojb/stable-diffusion-2-1-base).
|
| 66 |
+
That is what we actually trained against — verified in the training
|
| 67 |
+
configuration files, the launch scripts and the training logs, none of which
|
| 68 |
+
ever reference `stabilityai/...`. The frozen VAE, CLIP text encoder, tokenizer
|
| 69 |
+
and scheduler all come from that repo. A different SD 2.1-base mirror is
|
| 70 |
+
expected to work but has not been verified by us.
|
| 71 |
+
|
| 72 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 73 |
+
|
| 74 |
+
| FID↓ | KID↓ | DISTS↓ | LPIPS↓ | PSNR↑ | SSIM↑ |
|
| 75 |
+
|---|---|---|---|---|---|
|
| 76 |
+
| 19.1 | 0.0042 | 0.257 | 0.561 | 15.40 | 0.348 |
|
| 77 |
+
|
| 78 |
+
These are our own re-evaluation numbers, not numbers copied from any paper.
|
| 79 |
+
LPIPS is **LPIPS-VGG on inputs mapped to [-1, 1]** (the `normalize=False`
|
| 80 |
+
convention); the alternative [0, 1] convention gives a systematically different
|
| 81 |
+
number and the two must never be mixed. PSNR/SSIM are per-image torchmetrics
|
| 82 |
+
with `data_range=1`; FID is `pytorch-fid` against the size-matched ground truth.
|
| 83 |
+
|
| 84 |
+
## Load it and translate one SAR PNG
|
| 85 |
+
|
| 86 |
+
```python
|
| 87 |
+
import numpy as np
|
| 88 |
+
import torch
|
| 89 |
+
from PIL import Image
|
| 90 |
+
from diffusers import UNet2DConditionModel, AutoencoderKL, DDIMScheduler
|
| 91 |
+
from transformers import CLIPTextModel, CLIPTokenizer
|
| 92 |
+
|
| 93 |
+
REPO = "JeonghyeokDo/ReFlowSET" # comparison cells live under baselines/
|
| 94 |
+
BASE = "Manojb/stable-diffusion-2-1-base" # frozen VAE / text encoder / scheduler
|
| 95 |
+
DEV, DT = "cuda", torch.float32
|
| 96 |
+
|
| 97 |
+
unet = UNet2DConditionModel.from_pretrained(
|
| 98 |
+
REPO, subfolder="baselines/qxs-saropt/sd21ft", torch_dtype=DT).to(DEV).eval()
|
| 99 |
+
vae = AutoencoderKL.from_pretrained(BASE, subfolder="vae", torch_dtype=DT).to(DEV).eval()
|
| 100 |
+
tok = CLIPTokenizer.from_pretrained(BASE, subfolder="tokenizer")
|
| 101 |
+
txt = CLIPTextModel.from_pretrained(BASE, subfolder="text_encoder", torch_dtype=DT).to(DEV).eval()
|
| 102 |
+
sch = DDIMScheduler.from_pretrained(BASE, subfolder="scheduler")
|
| 103 |
+
sch.set_timesteps(50, device=DEV)
|
| 104 |
+
|
| 105 |
+
# The fixed training prompt, tokenized exactly as in training (no padding).
|
| 106 |
+
ids = tok("electro-optical image", padding="do_not_pad",
|
| 107 |
+
max_length=tok.model_max_length, truncation=True,
|
| 108 |
+
return_tensors="pt").input_ids.to(DEV)
|
| 109 |
+
with torch.no_grad():
|
| 110 |
+
embed = txt(ids)[0]
|
| 111 |
+
|
| 112 |
+
# SAR input: 256x256 PNG -> RGB -> [-1, 1]. Do not resize; crop if needed.
|
| 113 |
+
sar = Image.open("sar.png").convert("RGB")
|
| 114 |
+
x = torch.from_numpy(np.array(sar)).permute(2, 0, 1)[None]
|
| 115 |
+
x = (x.to(DEV, DT) / 255.0) * 2 - 1
|
| 116 |
+
|
| 117 |
+
with torch.no_grad():
|
| 118 |
+
sar_lat = vae.encode(x).latent_dist.mean * vae.config.scaling_factor # .mean, not .sample()
|
| 119 |
+
eo_lat = torch.randn_like(sar_lat)
|
| 120 |
+
for t in sch.timesteps:
|
| 121 |
+
out = unet(torch.cat([sar_lat, eo_lat], 1), # SAR first, then the noisy EO latent
|
| 122 |
+
t, encoder_hidden_states=embed).sample
|
| 123 |
+
eo_lat = sch.step(out, t, eo_lat).prev_sample # all 4 channels; no variance head
|
| 124 |
+
eo = vae.decode(eo_lat / vae.config.scaling_factor).sample
|
| 125 |
+
eo = (eo * 0.5 + 0.5).clamp(0, 1)[0].cpu()
|
| 126 |
+
|
| 127 |
+
Image.fromarray((eo.permute(1, 2, 0).numpy() * 255).round().astype(np.uint8)).save("eo.png")
|
| 128 |
+
```
|
| 129 |
+
|
| 130 |
+
The two things that silently produce a plausible-but-wrong image if you get
|
| 131 |
+
them backwards: the concatenation order is **SAR first**, and the VAE must be
|
| 132 |
+
read as `latent_dist.mean`, not `.sample()`.
|
| 133 |
+
|
| 134 |
+
## Licence status — stated factually
|
| 135 |
+
|
| 136 |
+
* **Training code:** MIT, `Copyright (c) 2026 KAIST VICLab` (the stage-1
|
| 137 |
+
trainer of [`KAIST-VICLab/C-DiffSET`](https://github.com/KAIST-VICLab/C-DiffSET)).
|
| 138 |
+
The row is named after Stable Diffusion, but no stabilityai code was used and
|
| 139 |
+
no stabilityai repository was loaded.
|
| 140 |
+
* **These weights:** a fine-tune of Stable Diffusion 2.1-base, hence a
|
| 141 |
+
*Derivative of the Model* under the **CreativeML Open RAIL++-M** licence
|
| 142 |
+
(`openrail++`). The **Attachment A use restrictions travel with these
|
| 143 |
+
weights** and must be passed on to anyone you redistribute to. Full text,
|
| 144 |
+
including Attachment A, ships in this repository as `LICENSE-WEIGHTS.md`.
|
| 145 |
+
|
| 146 |
+
Attribution: *fine-tuned from Stable Diffusion 2.1-base (obtained as
|
| 147 |
+
`Manojb/stable-diffusion-2-1-base`, a re-upload of
|
| 148 |
+
`stabilityai/stable-diffusion-2-1-base`) using the stage-1 trainer of
|
| 149 |
+
`KAIST-VICLab/C-DiffSET`.*
|
| 150 |
+
|
| 151 |
+
---
|
| 152 |
+
|
| 153 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 154 |
+
checkpoint retrained by us on QXS-SAROPT; it is not ReFlowSET itself. Every
|
| 155 |
+
comparison method in the release was retrained by us on the same splits at the
|
| 156 |
+
same resolution, so the rows are directly comparable to each other.
|
baselines/qxs-saropt/sd21ft/config.json
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_class_name": "UNet2DConditionModel",
|
| 3 |
+
"_diffusers_version": "0.10.0.dev0",
|
| 4 |
+
"act_fn": "silu",
|
| 5 |
+
"attention_head_dim": [
|
| 6 |
+
5,
|
| 7 |
+
10,
|
| 8 |
+
20,
|
| 9 |
+
20
|
| 10 |
+
],
|
| 11 |
+
"block_out_channels": [
|
| 12 |
+
320,
|
| 13 |
+
640,
|
| 14 |
+
1280,
|
| 15 |
+
1280
|
| 16 |
+
],
|
| 17 |
+
"center_input_sample": false,
|
| 18 |
+
"cross_attention_dim": 1024,
|
| 19 |
+
"down_block_types": [
|
| 20 |
+
"CrossAttnDownBlock2D",
|
| 21 |
+
"CrossAttnDownBlock2D",
|
| 22 |
+
"CrossAttnDownBlock2D",
|
| 23 |
+
"DownBlock2D"
|
| 24 |
+
],
|
| 25 |
+
"downsample_padding": 1,
|
| 26 |
+
"dual_cross_attention": false,
|
| 27 |
+
"flip_sin_to_cos": true,
|
| 28 |
+
"freq_shift": 0,
|
| 29 |
+
"in_channels": 8,
|
| 30 |
+
"layers_per_block": 2,
|
| 31 |
+
"mid_block_scale_factor": 1,
|
| 32 |
+
"norm_eps": 1e-05,
|
| 33 |
+
"norm_num_groups": 32,
|
| 34 |
+
"num_class_embeds": null,
|
| 35 |
+
"only_cross_attention": false,
|
| 36 |
+
"out_channels": 4,
|
| 37 |
+
"sample_size": 64,
|
| 38 |
+
"up_block_types": [
|
| 39 |
+
"UpBlock2D",
|
| 40 |
+
"CrossAttnUpBlock2D",
|
| 41 |
+
"CrossAttnUpBlock2D",
|
| 42 |
+
"CrossAttnUpBlock2D"
|
| 43 |
+
],
|
| 44 |
+
"use_linear_projection": true
|
| 45 |
+
}
|
baselines/qxs-saropt/spade/README.md
ADDED
|
@@ -0,0 +1,176 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# SPADE — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**Semantic Image Synthesis with Spatially-Adaptive Normalization**, Park, Liu,
|
| 4 |
+
Wang and Zhu, *CVPR 2019* ([arXiv:1903.07291](https://arxiv.org/abs/1903.07291)).
|
| 5 |
+
Upstream code: [`NVlabs/SPADE`](https://github.com/NVlabs/SPADE) at commit
|
| 6 |
+
`fecacc920c1367a038995c45a39c15f6521ca64f`.
|
| 7 |
+
|
| 8 |
+
This is the spatially-adaptive-normalisation generator we retrained ourselves on
|
| 9 |
+
QXS-SAROPT, and it is the checkpoint behind the SPADE row of the ReFlowSET
|
| 10 |
+
comparison table.
|
| 11 |
+
|
| 12 |
+
**This checkpoint will not load into a clean clone of `NVlabs/SPADE`.** Read the
|
| 13 |
+
next two sections before trying.
|
| 14 |
+
|
| 15 |
+
## What is in this folder
|
| 16 |
+
|
| 17 |
+
| file | bytes | what it is |
|
| 18 |
+
|---|---|---|
|
| 19 |
+
| `net_G.pth` | 368,775,707 | the SPADE generator, and the only file inference needs |
|
| 20 |
+
|
| 21 |
+
`net_G.pth` is a **`SPADEGenerator`**: `ngf = 64`,
|
| 22 |
+
`norm_G = spectralspadesyncbatch3x3`, `num_upsampling_layers = normal`,
|
| 23 |
+
`use_vae = False` (no style encoder — the generator is deterministic given the
|
| 24 |
+
input), and **`semantic_nc = 3`**.
|
| 25 |
+
|
| 26 |
+
That `semantic_nc = 3` is the whole story of this cell. SPADE is a *semantic
|
| 27 |
+
image synthesis* model: upstream one-hot-encodes its "label" input into
|
| 28 |
+
`label_nc` channels. A SAR chip is a real image, not a segmentation map, so we
|
| 29 |
+
run with `--label_nc 0 --no_instance`, which upstream treats as an error
|
| 30 |
+
condition. Stock SPADE therefore never constructs a generator with a 3-channel
|
| 31 |
+
spatially-adaptive input and cannot load this file. The two discriminators
|
| 32 |
+
(`num_D` 2) are not released.
|
| 33 |
+
|
| 34 |
+
## Training budget we used
|
| 35 |
+
|
| 36 |
+
| | |
|
| 37 |
+
|---|---|
|
| 38 |
+
| **generator updates released** | **120,000** |
|
| 39 |
+
| batch size | 16 |
|
| 40 |
+
| epochs (`niter` + `niter_decay`) | 60 + 60 |
|
| 41 |
+
| iterations per epoch | 1,000 (16,001 training pairs; this loader drops the last partial batch) |
|
| 42 |
+
| optimizer | Adam, lr 2e-4 with TTUR (`no_TTUR` off, so G runs at lr/2 and D at 2·lr), β₁ 0.0, β₂ 0.9 |
|
| 43 |
+
| adversarial loss | hinge GAN, multi-scale discriminator (`num_D` 2, `n_layers_D` 4) |
|
| 44 |
+
| feature-matching loss | λ_feat 10 |
|
| 45 |
+
| perceptual loss | VGG loss, λ_vgg 10 |
|
| 46 |
+
| initialisation | Xavier |
|
| 47 |
+
| augmentation | horizontal flip |
|
| 48 |
+
| input | 256 px (`resize_and_crop`, `load_size` 256, `crop_size` 256) |
|
| 49 |
+
|
| 50 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 51 |
+
|
| 52 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 53 |
+
|---|---|---|---|---|
|
| 54 |
+
| 90.7 | 0.292 | 0.599 | 0.320 | 14.53 |
|
| 55 |
+
|
| 56 |
+
Evaluated on the official split's 3,999 test chips at their native 256 px.
|
| 57 |
+
No subsampling: every metric on this page is measured over all 3,999 pairs.
|
| 58 |
+
|
| 59 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 60 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 61 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 62 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 63 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 64 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 65 |
+
convention gives a systematically different number and the two must never be
|
| 66 |
+
mixed, or compared against a paper that used the other one.
|
| 67 |
+
|
| 68 |
+
## Load it and translate one SAR image
|
| 69 |
+
|
| 70 |
+
**You need the image-to-image adaptation described below**, applied to the
|
| 71 |
+
upstream repository, before this will run. With that in place, the generator can
|
| 72 |
+
be constructed directly:
|
| 73 |
+
|
| 74 |
+
```python
|
| 75 |
+
import numpy as np, torch
|
| 76 |
+
from argparse import Namespace
|
| 77 |
+
from PIL import Image
|
| 78 |
+
from models.networks.generator import SPADEGenerator # from the NVlabs/SPADE repo
|
| 79 |
+
|
| 80 |
+
opt = Namespace(semantic_nc=3, label_nc=0, no_instance=True, ngf=64,
|
| 81 |
+
num_upsampling_layers='normal',
|
| 82 |
+
norm_G='spectralspadesyncbatch3x3',
|
| 83 |
+
crop_size=256, aspect_ratio=1.0, use_vae=False)
|
| 84 |
+
G = SPADEGenerator(opt)
|
| 85 |
+
G.load_state_dict(torch.load('net_G.pth', map_location='cpu'))
|
| 86 |
+
G.eval()
|
| 87 |
+
|
| 88 |
+
sar = Image.open('sar.png').convert('RGB')
|
| 89 |
+
x = torch.from_numpy(np.asarray(sar, np.float32) / 127.5 - 1).permute(2, 0, 1)[None]
|
| 90 |
+
|
| 91 |
+
with torch.no_grad():
|
| 92 |
+
y = G(seg=x, z=None) # the SAR image goes in the 'seg' slot
|
| 93 |
+
|
| 94 |
+
eo = ((y[0].permute(1, 2, 0).numpy() + 1) * 127.5).clip(0, 255).astype(np.uint8)
|
| 95 |
+
Image.fromarray(eo).save('eo.png')
|
| 96 |
+
```
|
| 97 |
+
|
| 98 |
+
Through the repository's own `test.py`, the invocation is
|
| 99 |
+
`--dataset_mode custom --label_nc 0 --no_instance --no_pairing_check` with the
|
| 100 |
+
SAR directory as `--label_dir`. Note that the paired loader also demands an
|
| 101 |
+
`--image_dir` even at test time, so a SAR-only run needs either a dummy EO
|
| 102 |
+
directory or the direct generator call above.
|
| 103 |
+
|
| 104 |
+
## Read before using this checkpoint
|
| 105 |
+
|
| 106 |
+
**The five changes that make this cell exist.** These are a functional
|
| 107 |
+
adaptation, not compatibility fixes, and a reproducer must apply all of them:
|
| 108 |
+
|
| 109 |
+
1. `data/pix2pix_dataset.py` — when `label_nc == 0`, load the "label" through
|
| 110 |
+
the *image* transform (`.convert('RGB')`, normalised to `[-1, 1]`) instead of
|
| 111 |
+
nearest-neighbour resampling, ×255 and the `==255 → label_nc` remap. The SAR
|
| 112 |
+
PNG is a real image.
|
| 113 |
+
2. `models/pix2pix_model.py` — an early return in `preprocess_input()`: when
|
| 114 |
+
`label_nc == 0`, skip the `.long()` cast and the `scatter_` one-hot encoding
|
| 115 |
+
and pass the float image straight through as `input_semantics`. One-hot
|
| 116 |
+
encoding a normalised float image is undefined.
|
| 117 |
+
3. `options/base_options.py` — when `label_nc == 0`, force
|
| 118 |
+
`opt.semantic_nc = 3 + (0 if no_instance else 1)`. Upstream computes
|
| 119 |
+
`semantic_nc = label_nc + …`, i.e. 0 or 1, and would build a generator with a
|
| 120 |
+
one-channel spatially-adaptive input.
|
| 121 |
+
4. `models/networks/discriminator.py` — the discriminator input width becomes
|
| 122 |
+
`(label_nc if label_nc > 0 else 3) + output_nc`, for the same reason on the
|
| 123 |
+
discriminator side.
|
| 124 |
+
5. `util/visualizer.py` — `tensor2label` is given `n_label=0` when
|
| 125 |
+
`label_nc == 0` so it falls through to `tensor2im` instead of colour-mapping
|
| 126 |
+
a real image as a label map; and `import scipy.misc` is guarded, since it was
|
| 127 |
+
removed in SciPy ≥ 1.12.
|
| 128 |
+
|
| 129 |
+
Changes 1–4 change what the network *is*. **Without them this file does not
|
| 130 |
+
load.**
|
| 131 |
+
|
| 132 |
+
* Normalisation is `[-1, 1]`.
|
| 133 |
+
* `norm_G` uses synchronised batch normalisation; on a single device this
|
| 134 |
+
behaves as ordinary batch norm, and `.eval()` is required.
|
| 135 |
+
* Passes the identity-collapse audit on both datasets.
|
| 136 |
+
|
| 137 |
+
## Licence — stated factually ⚠ non-commercial and share-alike
|
| 138 |
+
|
| 139 |
+
Upstream `NVlabs/SPADE` ships `LICENSE.md`, reproduced verbatim here as
|
| 140 |
+
`licenses/LICENSE-spade.txt`. It is the full legal code of
|
| 141 |
+
**Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International
|
| 142 |
+
(CC BY-NC-SA 4.0)**. The copyright holder is named in the README rather than in
|
| 143 |
+
the licence file: *"Copyright (C) 2019 NVIDIA Corporation. All rights reserved.
|
| 144 |
+
… The code is released for academic research use only. For commercial use or
|
| 145 |
+
business inquiries, please contact researchinquiries@nvidia.com."*
|
| 146 |
+
|
| 147 |
+
**This is the most restrictive licence in this repository, and two of its terms
|
| 148 |
+
travel with this checkpoint:**
|
| 149 |
+
|
| 150 |
+
* **NonCommercial.** You may not use this checkpoint for commercial advantage or
|
| 151 |
+
monetary compensation.
|
| 152 |
+
* **ShareAlike.** If a trained checkpoint is Adapted Material — and we treat it
|
| 153 |
+
as such — then any licence you apply when you pass it on must be a Creative
|
| 154 |
+
Commons licence with the same elements, this version or later, or a BY-NC-SA
|
| 155 |
+
compatible licence. **This cell is therefore distributed under CC BY-NC-SA
|
| 156 |
+
4.0**; it cannot sit under a permissive repository-wide weight licence, and
|
| 157 |
+
neither can anything you derive from it.
|
| 158 |
+
* The licence also states that patent and trademark rights are not licensed.
|
| 159 |
+
* NVIDIA's academic-use-only statement in the README is an additional assertion
|
| 160 |
+
beyond the CC text.
|
| 161 |
+
|
| 162 |
+
Attribution: *trained with `NVlabs/SPADE`. Copyright (C) 2019 NVIDIA
|
| 163 |
+
Corporation, all rights reserved. Licensed under CC BY-NC-SA 4.0; NVIDIA
|
| 164 |
+
releases the code for academic research use only (commercial enquiries:
|
| 165 |
+
researchinquiries@nvidia.com). This checkpoint is therefore distributed under
|
| 166 |
+
CC BY-NC-SA 4.0 — non-commercial, share-alike. Full text in
|
| 167 |
+
`licenses/LICENSE-spade.txt`.*
|
| 168 |
+
|
| 169 |
+
---
|
| 170 |
+
|
| 171 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 172 |
+
checkpoint that we retrained ourselves on QXS-SAROPT; it is not ReFlowSET
|
| 173 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 174 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 175 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 176 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/qxs-saropt/stegogan/README.md
ADDED
|
@@ -0,0 +1,200 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# StegoGAN — QXS-SAROPT
|
| 2 |
+
|
| 3 |
+
**StegoGAN: Leveraging Steganography for Non-Bijective Image-to-Image
|
| 4 |
+
Translation**, Wu, Chen, Mermet, Hurni, Schindler, Gonthier and Landrieu,
|
| 5 |
+
*CVPR 2024*. Upstream code:
|
| 6 |
+
[`sian-wusidi/StegoGAN`](https://github.com/sian-wusidi/StegoGAN) (the official
|
| 7 |
+
CVPR 2024 repository) at commit `cad61997c0f82793444f60f81298142b80cdf3c1`.
|
| 8 |
+
|
| 9 |
+
This is the mismatch-aware unpaired GAN we retrained ourselves on QXS-SAROPT,
|
| 10 |
+
and it is the checkpoint behind the StegoGAN row of the ReFlowSET comparison
|
| 11 |
+
table. **Two things on this page are load-bearing: there is an oracle output you
|
| 12 |
+
must not use, and this cell fails our input-copy audit.**
|
| 13 |
+
|
| 14 |
+
We applied **no patches** to this repository. It is stock upstream at that
|
| 15 |
+
commit.
|
| 16 |
+
|
| 17 |
+
## What is in this folder
|
| 18 |
+
|
| 19 |
+
| file | bytes | what it is |
|
| 20 |
+
|---|---|---|
|
| 21 |
+
| `net_G_A.pth` | 50,254,749 | **SAR → EO** (`resnet_9blocks_maskv1`) — the generator that produces the reported row |
|
| 22 |
+
| `net_G_B.pth` | 52,615,525 | EO → SAR (`resnet_9blocks_maskv3`), which also returns the mismatch mask; used to close the cycle in training, and **not needed for a prediction** |
|
| 23 |
+
|
| 24 |
+
Both generators are ResNet-style with instance normalisation, `ngf = 64`, built
|
| 25 |
+
with `--resnet_layer 8 --fusionblock`. **Those two are architecture flags, not
|
| 26 |
+
training options**: build the network without them and the state dict will not
|
| 27 |
+
load.
|
| 28 |
+
|
| 29 |
+
`net_G_A` takes an **optional second argument**. That option is the whole reason
|
| 30 |
+
this card has a warning section — see below.
|
| 31 |
+
|
| 32 |
+
## Training budget we used
|
| 33 |
+
|
| 34 |
+
| | |
|
| 35 |
+
|---|---|
|
| 36 |
+
| **data iterations released** | **120,030** |
|
| 37 |
+
| generator optimizer steps | 240,060 — see the note below |
|
| 38 |
+
| batch size | 4 |
|
| 39 |
+
| epochs (constant LR + linear decay) | 15 + 15 |
|
| 40 |
+
| iterations per epoch | 4,001 (16,001 training images, last batch not dropped) |
|
| 41 |
+
| optimizer | Adam, lr 2e-4, β₁ 0.5, linear decay to 0 over the second half |
|
| 42 |
+
| adversarial loss | LSGAN; discriminator `basic` (70×70 PatchGAN) |
|
| 43 |
+
| cycle losses | λ_A 10, λ_B 10 |
|
| 44 |
+
| identity loss | λ_identity 0.5 |
|
| 45 |
+
| consistency / regularisation | λ_consistency 1.0, λ_reg 0.3 |
|
| 46 |
+
| augmentation | horizontal flip |
|
| 47 |
+
| input | 256 px, no resize |
|
| 48 |
+
|
| 49 |
+
**Why two numbers.** StegoGAN's training step calls `optimizer_G.step()`
|
| 50 |
+
**twice** off a single backward pass, so its optimizer-step count is exactly
|
| 51 |
+
double its data-iteration count. We quote StegoGAN budgets in *data iterations*
|
| 52 |
+
throughout this benchmark and note the 2× explicitly. Do not re-derive one cell
|
| 53 |
+
the other way — the family stops being comparable with itself.
|
| 54 |
+
|
| 55 |
+
## Measured on the QXS-SAROPT test set (n = 3,999, 256 px)
|
| 56 |
+
|
| 57 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 58 |
+
|---|---|---|---|---|
|
| 59 |
+
| 106.8 | 0.384 | 0.658 | 0.254 | 12.96 |
|
| 60 |
+
|
| 61 |
+
Evaluated on the official split's 3,999 test chips at their native 256 px.
|
| 62 |
+
No subsampling: every metric on this page is measured over all 3,999 pairs.
|
| 63 |
+
|
| 64 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 65 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 66 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 67 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 68 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 69 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 70 |
+
convention gives a systematically different number and the two must never be
|
| 71 |
+
mixed, or compared against a paper that used the other one.
|
| 72 |
+
|
| 73 |
+
## Load it and translate one SAR image
|
| 74 |
+
|
| 75 |
+
Clone `sian-wusidi/StegoGAN` at the commit above and run this from its root.
|
| 76 |
+
Check `define_G`'s signature in that repository first: StegoGAN adds
|
| 77 |
+
`resnet_layer` and `fusionblock` keyword arguments that the junyanz original does
|
| 78 |
+
not have.
|
| 79 |
+
|
| 80 |
+
```python
|
| 81 |
+
import numpy as np, torch
|
| 82 |
+
from PIL import Image
|
| 83 |
+
from models.networks import define_G # from the StegoGAN repo
|
| 84 |
+
|
| 85 |
+
G_A = define_G(3, 3, 64, 'resnet_9blocks_maskv1', 'instance', use_dropout=False,
|
| 86 |
+
init_type='normal', init_gain=0.02, gpu_ids=[],
|
| 87 |
+
resnet_layer=8, fusionblock=True)
|
| 88 |
+
G_A.load_state_dict(torch.load('net_G_A.pth', map_location='cpu'))
|
| 89 |
+
G_A.eval()
|
| 90 |
+
|
| 91 |
+
sar = Image.open('sar.png').convert('RGB')
|
| 92 |
+
x = torch.from_numpy(np.asarray(sar, np.float32) / 127.5 - 1).permute(2, 0, 1)[None]
|
| 93 |
+
|
| 94 |
+
with torch.no_grad():
|
| 95 |
+
fake_B_clean = G_A(x) # ONE argument. Passing a second makes it an oracle.
|
| 96 |
+
|
| 97 |
+
eo = ((fake_B_clean[0].permute(1, 2, 0).numpy() + 1) * 127.5).clip(0, 255).astype(np.uint8)
|
| 98 |
+
Image.fromarray(eo).save('eo.png')
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
Through the repository's own `test.py` (`--model stego_gan --no_dropout
|
| 102 |
+
--resnet_layer 8 --fusionblock`), outputs are written one directory per visual;
|
| 103 |
+
**`fake_B_clean/` is the row on this page.**
|
| 104 |
+
|
| 105 |
+
## ⚠ `fake_B` is an oracle. `fake_B_clean` is the prediction.
|
| 106 |
+
|
| 107 |
+
The model computes two outputs:
|
| 108 |
+
|
| 109 |
+
```
|
| 110 |
+
fake_B_clean = netG_A(real_A) # SAR only
|
| 111 |
+
fake_B = netG_A(real_A, latent_real_B.detach()) # + a feature map of the GROUND TRUTH
|
| 112 |
+
```
|
| 113 |
+
|
| 114 |
+
`latent_real_B` comes from `netG_B(real_B)` — the *target* EO image. `fake_B` is
|
| 115 |
+
therefore conditioned on the answer and **cannot be produced at deployment**.
|
| 116 |
+
Only `fake_B_clean` is a prediction, and every number on this page is
|
| 117 |
+
`fake_B_clean`.
|
| 118 |
+
|
| 119 |
+
This is not hypothetical. Earlier in our own benchmarking the oracle was scored as
|
| 120 |
+
"StegoGAN" for two days on a different dataset and topped every column at
|
| 121 |
+
23.19 dB, against a true deployable value of 12.74 dB. **Any published StegoGAN
|
| 122 |
+
number must state which of the two it is.**
|
| 123 |
+
|
| 124 |
+
## ⚠ This cell fails the input-copy audit
|
| 125 |
+
|
| 126 |
+
We audit every cell for **identity collapse**: a model that has learned to pass
|
| 127 |
+
its input through rather than translate it. The test compares the mean absolute
|
| 128 |
+
distance from the generated image to the SAR **input** against the distance to
|
| 129 |
+
the EO **target**; a ratio below 1.0 means the output is closer to the input it
|
| 130 |
+
was given than to the answer it was asked for.
|
| 131 |
+
|
| 132 |
+
| quantity | value |
|
| 133 |
+
|---|---|
|
| 134 |
+
| mean \|generated − ground truth\| | 46.908 |
|
| 135 |
+
| mean \|generated − SAR input\| | 43.087 |
|
| 136 |
+
| **ratio** (SAR distance ÷ GT distance) | **0.919** — below 1.0 |
|
| 137 |
+
| verdict | **COLLAPSED** |
|
| 138 |
+
| cross-pair separation | 0.976 (cohort median 1.160) — no oracle behaviour |
|
| 139 |
+
|
| 140 |
+
**Read the metrics on this page with that in mind.** They are reported unchanged
|
| 141 |
+
because they are what the released implementation does at its own published
|
| 142 |
+
protocol, and a benchmark that quietly substitutes a better-behaved variant is
|
| 143 |
+
not reporting the method. But they should not be read as translation quality.
|
| 144 |
+
The separation figure is the companion check for the *opposite* failure: it is
|
| 145 |
+
the mean distance from a generated image to **other** images' ground truth
|
| 146 |
+
divided by the distance to its **own**, so a value well above the cohort median
|
| 147 |
+
would indicate that ground-truth information had leaked into the generation.
|
| 148 |
+
This cell sits at or below the median, so there is no leak — the problem here is
|
| 149 |
+
input-copying, and only input-copying.
|
| 150 |
+
|
| 151 |
+
## Read before using this checkpoint
|
| 152 |
+
|
| 153 |
+
* **`--resnet_layer 8` and `--fusionblock` must match training.** They change the
|
| 154 |
+
architecture; omit them and the load fails.
|
| 155 |
+
* **`test.py` rejects `--display_id`.** That flag is declared only in the train
|
| 156 |
+
options. Passing it to `test.py` aborts with an unrecognised argument — and in
|
| 157 |
+
a shell `&&` chain the cell then reports success with an empty output
|
| 158 |
+
directory. This silently destroyed several of our own test passes once.
|
| 159 |
+
* Build with `gpu_ids=[]`; the checkpoints have bare keys.
|
| 160 |
+
* Normalisation is `[-1, 1]`.
|
| 161 |
+
* StegoGAN is **unpaired**.
|
| 162 |
+
|
| 163 |
+
## Licence — stated factually ⚠ no upstream licence exists
|
| 164 |
+
|
| 165 |
+
**The upstream code base publishes no licence.**
|
| 166 |
+
[`sian-wusidi/StegoGAN`](https://github.com/sian-wusidi/StegoGAN) — the official
|
| 167 |
+
CVPR 2024 repository — has no LICENSE, LICENCE, COPYING or NOTICE file at any
|
| 168 |
+
depth; **0 of its 52 tracked files** at the pinned commit are licence-shaped; its
|
| 169 |
+
README has no licence section; and the GitHub API reports no declared licence,
|
| 170 |
+
with the `/license` endpoint returning 404. Checked 2026-08-28.
|
| 171 |
+
|
| 172 |
+
Under default copyright that means **all rights are reserved by the authors and
|
| 173 |
+
no express permission to redistribute derived work has been granted** to us or
|
| 174 |
+
to you. We publish this checkpoint anyway, so that the benchmark is reproducible
|
| 175 |
+
end to end, and we state the position plainly rather than implying a permission
|
| 176 |
+
that does not exist. If you intend to redistribute this checkpoint or build on
|
| 177 |
+
it, assess that for yourself, and consider asking the authors directly.
|
| 178 |
+
|
| 179 |
+
**What is known about the lineage.** StegoGAN's README thanks
|
| 180 |
+
`junyanz/pytorch-CycleGAN-and-pix2pix`, and the parts of the tree that are
|
| 181 |
+
unmodified CycleGAN carry that project's BSD notice, reproduced here as
|
| 182 |
+
`licenses/LICENSE-cyclegan.txt`. **That notice does not extend to StegoGAN's own
|
| 183 |
+
contributions** — the mismatch-mask generators, the fusion block and the
|
| 184 |
+
consistency and regularisation losses, which are exactly what makes
|
| 185 |
+
`net_G_A.pth` what it is.
|
| 186 |
+
|
| 187 |
+
Please cite: Wu, Chen, Mermet, Hurni, Schindler, Gonthier and Landrieu,
|
| 188 |
+
*StegoGAN: Leveraging Steganography for Non-Bijective Image-to-Image
|
| 189 |
+
Translation*, CVPR 2024.
|
| 190 |
+
The full record of what we checked, per method, is in
|
| 191 |
+
`licenses/NO-UPSTREAM-LICENSE.md` at the root of this repository.
|
| 192 |
+
|
| 193 |
+
---
|
| 194 |
+
|
| 195 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 196 |
+
checkpoint that we retrained ourselves on QXS-SAROPT; it is not ReFlowSET
|
| 197 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 198 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 199 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 200 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/sar2opt/bbdm/README.md
ADDED
|
@@ -0,0 +1,138 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# BBDM — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**BBDM: Image-to-Image Translation with Brownian Bridge Diffusion Models**, Li,
|
| 4 |
+
Xue, Liu and Lai, *CVPR 2023*
|
| 5 |
+
([arXiv:2205.07680](https://arxiv.org/abs/2205.07680)). Upstream code:
|
| 6 |
+
[`xuekt98/BBDM`](https://github.com/xuekt98/BBDM) at commit
|
| 7 |
+
`02c3b13c9f9dfab0853e32123100680a0640c4ed`.
|
| 8 |
+
|
| 9 |
+
This is the latent Brownian-bridge diffusion model we retrained ourselves on
|
| 10 |
+
SAR2Opt, and it is the checkpoint behind the BBDM row of the ReFlowSET
|
| 11 |
+
comparison table.
|
| 12 |
+
|
| 13 |
+
## What is in this folder
|
| 14 |
+
|
| 15 |
+
| file | bytes | what it is |
|
| 16 |
+
|---|---|---|
|
| 17 |
+
| `last_model.pth` | 2,118,317,140 | the whole trained system — see below |
|
| 18 |
+
|
| 19 |
+
`last_model.pth` is a pickled dictionary, not a bare state dict. It holds:
|
| 20 |
+
|
| 21 |
+
* `['model']` — the full `LatentBrownianBridgeModel` state dict: the denoising
|
| 22 |
+
UNet **and**, embedded inside it, the frozen vq-f4 VQGAN submodule;
|
| 23 |
+
* `['ema']` — the EMA shadow weights. **Sampling uses these**, not `['model']`;
|
| 24 |
+
the runner calls `apply_ema()` before generating;
|
| 25 |
+
* `['step']` and `['epoch']`.
|
| 26 |
+
|
| 27 |
+
UNet: latent resolution 128×128, `model_channels` 128, `num_res_blocks` 2,
|
| 28 |
+
attention at resolutions 32/16/8, `channel_mult` (1, 4, 8), 8 heads,
|
| 29 |
+
64 channels per head, scale-shift normalisation, resblock up/downsampling.
|
| 30 |
+
|
| 31 |
+
The denoising UNet has `condition_key: nocond` — there is no separate
|
| 32 |
+
conditioning branch, because in BBDM the SAR image *is* one endpoint of the
|
| 33 |
+
Brownian bridge and the EO image is the other. The translation is the bridge,
|
| 34 |
+
not a conditioner.
|
| 35 |
+
|
| 36 |
+
## Training budget we used
|
| 37 |
+
|
| 38 |
+
| | |
|
| 39 |
+
|---|---|
|
| 40 |
+
| **generator updates released** | **50,137** |
|
| 41 |
+
| batch size / resolution | 8 @ 512 px |
|
| 42 |
+
| iterations per epoch | 181 |
|
| 43 |
+
| epochs | 280 configured, capped by the step limit |
|
| 44 |
+
| optimizer | Adam, lr 1e-4, β₁ 0.9, no weight decay; `ReduceLROnPlateau` (factor 0.5, patience 3,000, min lr 5e-7) |
|
| 45 |
+
| EMA | decay 0.995, started at step 30,000, updated every 8 steps |
|
| 46 |
+
| bridge | `mt_type` linear, objective `grad`, L1 loss, T = 1,000, `max_var` 1.0 |
|
| 47 |
+
| sampler (test) | 200 skip-sampled steps, η = 1.0, `clip_denoised` off |
|
| 48 |
+
| latent space | CompVis **vq-f4** VQGAN, frozen (`embed_dim` 3, `n_embed` 8192) |
|
| 49 |
+
| horizontal flip | enabled (horizontal) |
|
| 50 |
+
|
| 51 |
+
The 50,137 rather than a round 50,000 is not a typo: the runner stops at the first epoch boundary past the step cap.
|
| 52 |
+
|
| 53 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 54 |
+
|
| 55 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 56 |
+
|---|---|---|---|---|
|
| 57 |
+
| 143.1 | 0.290 | 0.590 | 0.276 | 15.29 |
|
| 58 |
+
|
| 59 |
+
Evaluated on the official split's 627 test tiles, centre-cropped to 512 px.
|
| 60 |
+
The tiles are 600 px natively; this benchmark crops and never resizes, in every
|
| 61 |
+
method's training and in the evaluation. No subsampling: every metric on this
|
| 62 |
+
page is measured over all 627 pairs.
|
| 63 |
+
|
| 64 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 65 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 66 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 67 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 68 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 69 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 70 |
+
convention gives a systematically different number and the two must never be
|
| 71 |
+
mixed, or compared against a paper that used the other one.
|
| 72 |
+
|
| 73 |
+
## Load it and translate one SAR image
|
| 74 |
+
|
| 75 |
+
There is no smaller entry point than the repository's own runner, because the
|
| 76 |
+
model is constructed with a VQGAN whose weights are loaded from a path in the
|
| 77 |
+
config **even though the same weights are already inside `last_model.pth`**.
|
| 78 |
+
You must have the CompVis latent-diffusion **vq-f4** checkpoint
|
| 79 |
+
(`model.ckpt`, 756,175,527 bytes) on disk and pointed at by
|
| 80 |
+
`model.VQGAN.params.ckpt_path`, or construction fails before any of this runs.
|
| 81 |
+
|
| 82 |
+
```bash
|
| 83 |
+
# from the BBDM repository root, with a config whose data.dataset_config
|
| 84 |
+
# points at your SAR / EO directories and whose VQGAN ckpt_path exists:
|
| 85 |
+
python main.py -c configs/<your>-LBBDM-f4.yaml --gpu_ids 0 \
|
| 86 |
+
-r <output dir> --sample_to_eval --resume_model last_model.pth
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
`--resume_model` sets `model.model_load_path`; the runner loads it with
|
| 90 |
+
`torch.load(..., weights_only=False)`, restores the EMA shadow and applies it
|
| 91 |
+
before sampling.
|
| 92 |
+
|
| 93 |
+
Outputs land in
|
| 94 |
+
`<output dir>/<dataset_name>/LBBDM-f4/sample_to_eval/`,
|
| 95 |
+
in three sibling directories: **`200/` holds the generated EO** (the name is the
|
| 96 |
+
`sample_step` count), `condition/` holds the SAR inputs and `ground_truth/` holds
|
| 97 |
+
the targets. Score `200/`.
|
| 98 |
+
|
| 99 |
+
## Read before using this checkpoint
|
| 100 |
+
|
| 101 |
+
* **Sampling must use the EMA weights.** `['model']` and `['ema']` are both in
|
| 102 |
+
the file and they are different networks. The reported numbers are the EMA.
|
| 103 |
+
* **You need the external vq-f4 VQGAN.** It is not redistributed here — it is
|
| 104 |
+
CompVis latent-diffusion material with its own (MIT) licence, and the config
|
| 105 |
+
requires it at construction time regardless.
|
| 106 |
+
* **`torch.load` needs `weights_only=False`.** PyTorch �� 2.6 flipped that
|
| 107 |
+
default; these are pickled Lightning-style checkpoints. We made that change in
|
| 108 |
+
four places in the repository (`model/VQGAN/vqgan.py`, `runners/BaseRunner.py`
|
| 109 |
+
twice, `runners/.../BBDMRunner.py`) and also removed a `verbose=True` kwarg
|
| 110 |
+
from `ReduceLROnPlateau` that PyTorch 2.x no longer accepts. All four are
|
| 111 |
+
compatibility fixes; none changes the maths.
|
| 112 |
+
* Passes the identity-collapse audit on both datasets.
|
| 113 |
+
|
| 114 |
+
## Licence — stated factually
|
| 115 |
+
|
| 116 |
+
Upstream `xuekt98/BBDM` ships a LICENSE file, reproduced verbatim here as
|
| 117 |
+
`licenses/LICENSE-bbdm.txt`: the **MIT licence**, `Copyright (c) 2023 xuekt98`.
|
| 118 |
+
|
| 119 |
+
**Redistribution of these weights is permitted.** No non-commercial clause, no
|
| 120 |
+
share-alike clause.
|
| 121 |
+
|
| 122 |
+
One dependency note that is attribution, not restriction: this checkpoint cannot
|
| 123 |
+
be run without the CompVis latent-diffusion **vq-f4** VQGAN, which is MIT and is
|
| 124 |
+
**not** redistributed here.
|
| 125 |
+
|
| 126 |
+
Attribution: *trained with `xuekt98/BBDM` (MIT, Copyright (c) 2023 xuekt98).
|
| 127 |
+
Requires the CompVis latent-diffusion vq-f4 VQGAN (MIT,
|
| 128 |
+
https://github.com/CompVis/latent-diffusion), which is not redistributed here.
|
| 129 |
+
Full text in `licenses/LICENSE-bbdm.txt`.*
|
| 130 |
+
|
| 131 |
+
---
|
| 132 |
+
|
| 133 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 134 |
+
checkpoint that we retrained ourselves on SAR2Opt; it is not ReFlowSET
|
| 135 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 136 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 137 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 138 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/sar2opt/cbbdm/README.md
ADDED
|
@@ -0,0 +1,155 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# cBBDM — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**Conditional Brownian Bridge Diffusion Model for VHR SAR to Optical Image
|
| 4 |
+
Translation**, Kim and Chung, *IEEE Geoscience and Remote Sensing Letters*, 2025
|
| 5 |
+
([arXiv:2408.07947](https://arxiv.org/abs/2408.07947); the venue and year are
|
| 6 |
+
taken from the authors' own citation block — the repository records no DOI).
|
| 7 |
+
Upstream code:
|
| 8 |
+
[`egshkim/ConditionalBBDM-for-VHR-SAR-to-Optical`](https://github.com/egshkim/ConditionalBBDM-for-VHR-SAR-to-Optical)
|
| 9 |
+
at commit `8ce15934f4d4e3f01efe70d11e2d9b9e0859210c`, a fork of
|
| 10 |
+
[`xuekt98/BBDM`](https://github.com/xuekt98/BBDM).
|
| 11 |
+
|
| 12 |
+
This is the conditional Brownian-bridge latent diffusion model we retrained
|
| 13 |
+
ourselves on SAR2Opt, and it is the checkpoint behind the cBBDM row of the
|
| 14 |
+
ReFlowSET comparison table.
|
| 15 |
+
|
| 16 |
+
## What is in this folder
|
| 17 |
+
|
| 18 |
+
| file | bytes | what it is |
|
| 19 |
+
|---|---|---|
|
| 20 |
+
| `last_model.pth` | 2,118,345,622 | the whole trained system — see below |
|
| 21 |
+
|
| 22 |
+
`last_model.pth` is a pickled dictionary, not a bare state dict. It holds:
|
| 23 |
+
|
| 24 |
+
* `['model']` — the full `LatentBrownianBridgeModel` state dict: the denoising
|
| 25 |
+
UNet **and**, embedded inside it, the frozen vq-f4 VQGAN submodule;
|
| 26 |
+
* `['ema']` — the EMA shadow weights. **Sampling uses these**, not `['model']`;
|
| 27 |
+
the runner calls `apply_ema()` before generating;
|
| 28 |
+
* `['step']` and `['epoch']`.
|
| 29 |
+
|
| 30 |
+
UNet: latent resolution 128×128, `model_channels` 128, `num_res_blocks` 2,
|
| 31 |
+
attention at resolutions 32/16/8, `channel_mult` (1, 4, 8), 8 heads,
|
| 32 |
+
64 channels per head, scale-shift normalisation, resblock up/downsampling.
|
| 33 |
+
|
| 34 |
+
The denoising UNet differs from plain BBDM in exactly two ways, and they are
|
| 35 |
+
the method: **`in_channels` is 6, not 3**, and **`condition_key` is
|
| 36 |
+
`SpatialRescaler`** — a small trained module that rescales the SAR latent and
|
| 37 |
+
concatenates it to the bridge state at every step, so the SAR conditioning is
|
| 38 |
+
explicit rather than implicit in the bridge endpoints. The SpatialRescaler is
|
| 39 |
+
trained (it is in the model's parameter list) and is inside this file.
|
| 40 |
+
|
| 41 |
+
## Training budget we used
|
| 42 |
+
|
| 43 |
+
| | |
|
| 44 |
+
|---|---|
|
| 45 |
+
| **generator updates released** | **50,137** |
|
| 46 |
+
| batch size / resolution | 8 @ 512 px |
|
| 47 |
+
| iterations per epoch | 181 |
|
| 48 |
+
| epochs | 280 configured, capped by the step limit |
|
| 49 |
+
| optimizer | Adam, lr 1e-4, β₁ 0.9, no weight decay; `ReduceLROnPlateau` (factor 0.5, patience 3,000, min lr 5e-7) |
|
| 50 |
+
| EMA | decay 0.995, started at step 30,000, updated every 8 steps |
|
| 51 |
+
| bridge | `mt_type` linear, objective `grad`, L1 loss, T = 1,000, `max_var` 1.0 |
|
| 52 |
+
| sampler (test) | 200 skip-sampled steps, η = 1.0, `clip_denoised` off |
|
| 53 |
+
| latent space | CompVis **vq-f4** VQGAN, frozen (`embed_dim` 3, `n_embed` 8192) |
|
| 54 |
+
| horizontal flip | enabled (horizontal) |
|
| 55 |
+
|
| 56 |
+
The 50,137 rather than a round 50,000 is not a typo: the runner stops at the first epoch boundary past the step cap.
|
| 57 |
+
|
| 58 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 59 |
+
|
| 60 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 61 |
+
|---|---|---|---|---|
|
| 62 |
+
| 222.3 | 0.377 | 0.571 | 0.361 | 17.05 |
|
| 63 |
+
|
| 64 |
+
Evaluated on the official split's 627 test tiles, centre-cropped to 512 px.
|
| 65 |
+
The tiles are 600 px natively; this benchmark crops and never resizes, in every
|
| 66 |
+
method's training and in the evaluation. No subsampling: every metric on this
|
| 67 |
+
page is measured over all 627 pairs.
|
| 68 |
+
|
| 69 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 70 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 71 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 72 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 73 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 74 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 75 |
+
convention gives a systematically different number and the two must never be
|
| 76 |
+
mixed, or compared against a paper that used the other one.
|
| 77 |
+
|
| 78 |
+
## Load it and translate one SAR image
|
| 79 |
+
|
| 80 |
+
There is no smaller entry point than the repository's own runner, because the
|
| 81 |
+
model is constructed with a VQGAN whose weights are loaded from a path in the
|
| 82 |
+
config **even though the same weights are already inside `last_model.pth`**.
|
| 83 |
+
You must have the CompVis latent-diffusion **vq-f4** checkpoint
|
| 84 |
+
(`model.ckpt`, 756,175,527 bytes) on disk and pointed at by
|
| 85 |
+
`model.VQGAN.params.ckpt_path`, or construction fails before any of this runs.
|
| 86 |
+
|
| 87 |
+
```bash
|
| 88 |
+
# from the cBBDM repository root, with a config whose data.dataset_config
|
| 89 |
+
# points at your SAR / EO directories and whose VQGAN ckpt_path exists:
|
| 90 |
+
python main.py -c configs/<your>-cBBDM-f4.yaml --gpu_ids 0 \
|
| 91 |
+
-r <output dir> --sample_to_eval --resume_model last_model.pth
|
| 92 |
+
```
|
| 93 |
+
|
| 94 |
+
`--resume_model` sets `model.model_load_path`; the runner loads it with
|
| 95 |
+
`torch.load(..., weights_only=False)`, restores the EMA shadow and applies it
|
| 96 |
+
before sampling.
|
| 97 |
+
|
| 98 |
+
Outputs land in
|
| 99 |
+
`<output dir>/<dataset_name>/cBBDM-f4/sample_to_eval/`,
|
| 100 |
+
in three sibling directories: **`200/` holds the generated EO** (the name is the
|
| 101 |
+
`sample_step` count), `condition/` holds the SAR inputs and `ground_truth/` holds
|
| 102 |
+
the targets. Score `200/`.
|
| 103 |
+
|
| 104 |
+
## Read before using this checkpoint
|
| 105 |
+
|
| 106 |
+
* **Sampling must use the EMA weights.** `['model']` and `['ema']` are both in
|
| 107 |
+
the file and they are different networks. The reported numbers are the EMA.
|
| 108 |
+
* **You need the external vq-f4 VQGAN.** It is not redistributed here — it is
|
| 109 |
+
CompVis latent-diffusion material with its own (MIT) licence, and the config
|
| 110 |
+
requires it at construction time regardless.
|
| 111 |
+
* **`torch.load` needs `weights_only=False`.** PyTorch ≥ 2.6 flipped that
|
| 112 |
+
default; these are pickled Lightning-style checkpoints. We made that change in
|
| 113 |
+
four places in the repository (`model/VQGAN/vqgan.py`, `runners/BaseRunner.py`
|
| 114 |
+
twice, `runners/.../BBDMRunner.py`) and also removed a `verbose=True` kwarg
|
| 115 |
+
from `ReduceLROnPlateau` that PyTorch 2.x no longer accepts. All four are
|
| 116 |
+
compatibility fixes; none changes the maths.
|
| 117 |
+
* **A performance patch we applied, and when.** This fork regressed the
|
| 118 |
+
DataLoader worker count from upstream BBDM's 8 to 0, which dominated wall-clock
|
| 119 |
+
time (one 32-pair batch: 6.5 s serial versus 1.1 s across 8 threads). We
|
| 120 |
+
restored `num_workers=8`. **Both cells published here were trained *before*
|
| 121 |
+
that change, i.e. with 0 workers** — same data, same order, same step count,
|
| 122 |
+
only slower. The numbers on this page are unaffected either way: worker count
|
| 123 |
+
cannot change which samples the model sees.
|
| 124 |
+
* Passes the identity-collapse audit on both datasets.
|
| 125 |
+
|
| 126 |
+
## Licence — stated factually
|
| 127 |
+
|
| 128 |
+
Upstream `egshkim/ConditionalBBDM-for-VHR-SAR-to-Optical` ships a LICENSE file,
|
| 129 |
+
reproduced verbatim here as `licenses/LICENSE-cbbdm.txt`: the **MIT licence**,
|
| 130 |
+
`Copyright (c) 2025 egshkim`. Its README adds: *"The original source code is
|
| 131 |
+
licensed under the MIT License, and this repository continues to comply with its
|
| 132 |
+
terms"* — the original being `xuekt98/BBDM`, also MIT
|
| 133 |
+
(`licenses/LICENSE-bbdm.txt`).
|
| 134 |
+
|
| 135 |
+
**Redistribution of these weights is permitted.** No non-commercial clause, no
|
| 136 |
+
share-alike clause.
|
| 137 |
+
|
| 138 |
+
One dependency note that is attribution, not restriction: this checkpoint cannot
|
| 139 |
+
be run without the CompVis latent-diffusion **vq-f4** VQGAN, which is MIT and is
|
| 140 |
+
**not** redistributed here.
|
| 141 |
+
|
| 142 |
+
Attribution: *trained with `egshkim/ConditionalBBDM-for-VHR-SAR-to-Optical`
|
| 143 |
+
(MIT, Copyright (c) 2025 egshkim), a fork of `xuekt98/BBDM` (MIT, Copyright (c)
|
| 144 |
+
2023 xuekt98). Requires the CompVis vq-f4 VQGAN (MIT,
|
| 145 |
+
https://github.com/CompVis/latent-diffusion), not redistributed here. Full texts
|
| 146 |
+
in `licenses/LICENSE-cbbdm.txt` and `licenses/LICENSE-bbdm.txt`.*
|
| 147 |
+
|
| 148 |
+
---
|
| 149 |
+
|
| 150 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 151 |
+
checkpoint that we retrained ourselves on SAR2Opt; it is not ReFlowSET
|
| 152 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 153 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 154 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 155 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/sar2opt/cdiffset/README.md
ADDED
|
@@ -0,0 +1,166 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# C-DiffSET — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**C-DiffSET: Leveraging Latent Diffusion for SAR-to-EO Image Translation with
|
| 4 |
+
Confidence-Guided Reliable Object Generation**, Do, Lee, Lee and Kim,
|
| 5 |
+
*IEEE Transactions on Circuits and Systems for Video Technology*, 2026
|
| 6 |
+
([doi:10.1109/TCSVT.2026.3701447](https://doi.org/10.1109/TCSVT.2026.3701447),
|
| 7 |
+
[arXiv:2411.10788](https://arxiv.org/abs/2411.10788)). Upstream code:
|
| 8 |
+
[`KAIST-VICLab/C-DiffSET`](https://github.com/KAIST-VICLab/C-DiffSET).
|
| 9 |
+
|
| 10 |
+
This is the **stage-2 (confidence-guided) UNet** we retrained ourselves on
|
| 11 |
+
SAR2Opt, and it is the checkpoint behind the C-DiffSET row of the ReFlowSET
|
| 12 |
+
comparison table.
|
| 13 |
+
|
| 14 |
+
## What is in this folder
|
| 15 |
+
|
| 16 |
+
A single `UNet2DConditionModel` (865,925,125 parameters, fp32, 686 tensors) with
|
| 17 |
+
two channel modifications relative to stock SD 2.1-base:
|
| 18 |
+
|
| 19 |
+
* **`conv_in` 4 -> 8 channels.** The UNet is conditioned by channel
|
| 20 |
+
concatenation in latent space, **SAR first**:
|
| 21 |
+
`torch.cat([sar_latent, noisy_eo_latent], dim=1)`. The extra half was
|
| 22 |
+
initialised by duplicating the pretrained kernel and halving it; in this
|
| 23 |
+
trained checkpoint the two halves have clearly diverged
|
| 24 |
+
(mean |w| 0.01517 on channels 0-3 vs 0.02702 on channels 4-7).
|
| 25 |
+
* **`conv_out` 4 -> 5 channels.** Channels 0-3 are the usual epsilon
|
| 26 |
+
prediction; channel 4 is a **raw spatial variance** used by the
|
| 27 |
+
confidence-guided loss. It was zero-initialised and is no longer zero
|
| 28 |
+
(mean |w| 0.03187), i.e. the head really trained. To turn it into a variance:
|
| 29 |
+
`var = clamp(softplus(raw + c) + 1e-6, 1e-6, 10)` where the offset
|
| 30 |
+
`c = log(exp(1 - 1e-6) - 1)` is the inverse softplus of the initial variance,
|
| 31 |
+
and confidence is `1 / var`. **The scheduler must only ever see channels 0-3.**
|
| 32 |
+
|
| 33 |
+
There is no VAE, no text encoder and no scheduler here — take those from the
|
| 34 |
+
base model (below).
|
| 35 |
+
|
| 36 |
+
## Training budget we used
|
| 37 |
+
|
| 38 |
+
| | |
|
| 39 |
+
|---|---|
|
| 40 |
+
| optimizer updates released | **40,000** (fixed-step snapshot `checkpoint-40000`) |
|
| 41 |
+
| updates the run was configured for | 50,000 |
|
| 42 |
+
| initialised from | our own stage-1 checkpoint (the `sd21ft` folder's run, at its final 50,000-update state) |
|
| 43 |
+
| batch size / resolution | 16 @ 512 px |
|
| 44 |
+
| epochs the 40,000 updates work out to | ~444 (1,450 training pairs, 90 updates/epoch) |
|
| 45 |
+
| optimizer | AdamW, lr 3e-5, weight decay 0.01, cosine schedule, 100-step warmup |
|
| 46 |
+
| precision | fp32 (no mixed precision) |
|
| 47 |
+
| seed | 2024 |
|
| 48 |
+
| prediction type | epsilon, with the confidence-guided (Gaussian-NLL) loss |
|
| 49 |
+
| prompt | the fixed string `"electro-optical image"` |
|
| 50 |
+
| augmentation | horizontal flip, vertical flip, 90-degree rotation |
|
| 51 |
+
| test-time sampler | **DDIM, 50 steps**, fp32, one image at a time |
|
| 52 |
+
|
| 53 |
+
The budget is quoted in **optimizer updates**, not epochs, because the two
|
| 54 |
+
datasets have very different training-set sizes. Two things a reader
|
| 55 |
+
reproducing this needs to know: the released snapshot is taken at update 40,000
|
| 56 |
+
of a cosine schedule that was laid out for 50,000, so the learning rate had
|
| 57 |
+
**not** finished decaying at that point; and we publish the fixed-step snapshot
|
| 58 |
+
rather than the repository's validation-selected `best/`, because every other
|
| 59 |
+
method in our comparison publishes its last checkpoint and selecting on
|
| 60 |
+
validation PSNR would have been an asymmetry in C-DiffSET's favour.
|
| 61 |
+
|
| 62 |
+
Data: 512 px. The source images are 600 px and are **never resized**: training takes a random 512 crop, evaluation takes the centre 512 crop (offset 44). The EO/SAR pairing follows the official SAR2Opt layout (`trainB`/`testB` are EO, `trainA`/`testA` are SAR).
|
| 63 |
+
|
| 64 |
+
## Base model
|
| 65 |
+
|
| 66 |
+
This checkpoint is a derivative of **Stable Diffusion 2.1-base**. Because the
|
| 67 |
+
original `stabilityai/stable-diffusion-2-1-base` repository is no longer
|
| 68 |
+
available on the Hub, our training and every snippet below load the community
|
| 69 |
+
mirror [`Manojb/stable-diffusion-2-1-base`](https://huggingface.co/Manojb/stable-diffusion-2-1-base).
|
| 70 |
+
That is what we actually trained against — verified in the training
|
| 71 |
+
configuration files, the launch scripts and the training logs, none of which
|
| 72 |
+
ever reference `stabilityai/...`. The frozen VAE, CLIP text encoder, tokenizer
|
| 73 |
+
and scheduler all come from that repo. A different SD 2.1-base mirror is
|
| 74 |
+
expected to work but has not been verified by us.
|
| 75 |
+
|
| 76 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 77 |
+
|
| 78 |
+
| FID↓ | KID↓ | DISTS↓ | LPIPS↓ | PSNR↑ | SSIM↑ |
|
| 79 |
+
|---|---|---|---|---|---|
|
| 80 |
+
| 78.1 | 0.0139 | 0.214 | 0.529 | 16.81 | 0.314 |
|
| 81 |
+
|
| 82 |
+
These are our own re-evaluation numbers, not numbers copied from any paper.
|
| 83 |
+
LPIPS is **LPIPS-VGG on inputs mapped to [-1, 1]** (the `normalize=False`
|
| 84 |
+
convention); the alternative [0, 1] convention gives a systematically different
|
| 85 |
+
number and the two must never be mixed. PSNR/SSIM are per-image torchmetrics
|
| 86 |
+
with `data_range=1`; FID is `pytorch-fid` against the size-matched ground truth.
|
| 87 |
+
|
| 88 |
+
> **DISTS on this cell was re-measured.** This cell's prediction dump was regenerated after the only extended-metric pass that scored it, and that pass resumes on the *number* of images rather than their content, so its stored DISTS went stale. A full re-measurement of the current dump at n=627 gives the value printed above; the harness was validated on a cell that is not stale, whose DISTS it reproduces to 6e-6. PSNR, SSIM, FID and LPIPS come from the headline pass, which rescores every cell on every run.
|
| 89 |
+
|
| 90 |
+
## Load it and translate one SAR PNG
|
| 91 |
+
|
| 92 |
+
```python
|
| 93 |
+
import numpy as np
|
| 94 |
+
import torch
|
| 95 |
+
from PIL import Image
|
| 96 |
+
from diffusers import UNet2DConditionModel, AutoencoderKL, DDIMScheduler
|
| 97 |
+
from transformers import CLIPTextModel, CLIPTokenizer
|
| 98 |
+
|
| 99 |
+
REPO = "JeonghyeokDo/ReFlowSET" # comparison cells live under baselines/
|
| 100 |
+
BASE = "Manojb/stable-diffusion-2-1-base" # frozen VAE / text encoder / scheduler
|
| 101 |
+
DEV, DT = "cuda", torch.float32
|
| 102 |
+
|
| 103 |
+
unet = UNet2DConditionModel.from_pretrained(
|
| 104 |
+
REPO, subfolder="baselines/sar2opt/cdiffset", torch_dtype=DT).to(DEV).eval()
|
| 105 |
+
vae = AutoencoderKL.from_pretrained(BASE, subfolder="vae", torch_dtype=DT).to(DEV).eval()
|
| 106 |
+
tok = CLIPTokenizer.from_pretrained(BASE, subfolder="tokenizer")
|
| 107 |
+
txt = CLIPTextModel.from_pretrained(BASE, subfolder="text_encoder", torch_dtype=DT).to(DEV).eval()
|
| 108 |
+
sch = DDIMScheduler.from_pretrained(BASE, subfolder="scheduler")
|
| 109 |
+
sch.set_timesteps(50, device=DEV)
|
| 110 |
+
|
| 111 |
+
# The fixed training prompt, tokenized exactly as in training (no padding).
|
| 112 |
+
ids = tok("electro-optical image", padding="do_not_pad",
|
| 113 |
+
max_length=tok.model_max_length, truncation=True,
|
| 114 |
+
return_tensors="pt").input_ids.to(DEV)
|
| 115 |
+
with torch.no_grad():
|
| 116 |
+
embed = txt(ids)[0]
|
| 117 |
+
|
| 118 |
+
# SAR input: 512x512 PNG -> RGB -> [-1, 1]. Do not resize; crop if needed.
|
| 119 |
+
sar = Image.open("sar.png").convert("RGB")
|
| 120 |
+
x = torch.from_numpy(np.array(sar)).permute(2, 0, 1)[None]
|
| 121 |
+
x = (x.to(DEV, DT) / 255.0) * 2 - 1
|
| 122 |
+
|
| 123 |
+
with torch.no_grad():
|
| 124 |
+
sar_lat = vae.encode(x).latent_dist.mean * vae.config.scaling_factor # .mean, not .sample()
|
| 125 |
+
eo_lat = torch.randn_like(sar_lat)
|
| 126 |
+
for t in sch.timesteps:
|
| 127 |
+
out = unet(torch.cat([sar_lat, eo_lat], 1), # SAR first, then the noisy EO latent
|
| 128 |
+
t, encoder_hidden_states=embed).sample
|
| 129 |
+
# channels 0-3 are epsilon; channel 4 is the raw variance and never
|
| 130 |
+
# goes to the scheduler.
|
| 131 |
+
eo_lat = sch.step(out[:, :4], t, eo_lat).prev_sample
|
| 132 |
+
eo = vae.decode(eo_lat / vae.config.scaling_factor).sample
|
| 133 |
+
eo = (eo * 0.5 + 0.5).clamp(0, 1)[0].cpu()
|
| 134 |
+
|
| 135 |
+
Image.fromarray((eo.permute(1, 2, 0).numpy() * 255).round().astype(np.uint8)).save("eo.png")
|
| 136 |
+
```
|
| 137 |
+
|
| 138 |
+
The three things that silently produce a plausible-but-wrong image if you get
|
| 139 |
+
them backwards: the concatenation order is **SAR first**, only `out[:, :4]`
|
| 140 |
+
reaches the scheduler, and the VAE must be read as `latent_dist.mean`, not
|
| 141 |
+
`.sample()`.
|
| 142 |
+
|
| 143 |
+
## Licence status — stated factually
|
| 144 |
+
|
| 145 |
+
* **Training code:** MIT, `Copyright (c) 2026 KAIST VICLab`
|
| 146 |
+
([`KAIST-VICLab/C-DiffSET`](https://github.com/KAIST-VICLab/C-DiffSET) ships a
|
| 147 |
+
LICENSE file).
|
| 148 |
+
* **These weights:** they are a fine-tune of Stable Diffusion 2.1-base and are
|
| 149 |
+
therefore a *Derivative of the Model* under the **CreativeML Open RAIL++-M**
|
| 150 |
+
licence. The base repository we loaded declares `openrail++`.
|
| 151 |
+
The **Attachment A use restrictions travel with these weights** — they bind
|
| 152 |
+
you, and you must pass them on to anyone you redistribute to. The full text,
|
| 153 |
+
including Attachment A, ships in this repository as `LICENSE-WEIGHTS.md`.
|
| 154 |
+
|
| 155 |
+
Attribution: *fine-tuned from Stable Diffusion 2.1-base (obtained as
|
| 156 |
+
`Manojb/stable-diffusion-2-1-base`, a re-upload of
|
| 157 |
+
`stabilityai/stable-diffusion-2-1-base`) with the training code of
|
| 158 |
+
`KAIST-VICLab/C-DiffSET`. Do, Lee, Lee and Kim, "C-DiffSET", IEEE TCSVT 2026,
|
| 159 |
+
doi:10.1109/TCSVT.2026.3701447.*
|
| 160 |
+
|
| 161 |
+
---
|
| 162 |
+
|
| 163 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 164 |
+
checkpoint retrained by us on SAR2Opt; it is not ReFlowSET itself. Every
|
| 165 |
+
comparison method in the release was retrained by us on the same splits at the
|
| 166 |
+
same resolution, so the rows are directly comparable to each other.
|
baselines/sar2opt/cdiffset/config.json
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_class_name": "UNet2DConditionModel",
|
| 3 |
+
"_diffusers_version": "0.10.0.dev0",
|
| 4 |
+
"act_fn": "silu",
|
| 5 |
+
"attention_head_dim": [
|
| 6 |
+
5,
|
| 7 |
+
10,
|
| 8 |
+
20,
|
| 9 |
+
20
|
| 10 |
+
],
|
| 11 |
+
"block_out_channels": [
|
| 12 |
+
320,
|
| 13 |
+
640,
|
| 14 |
+
1280,
|
| 15 |
+
1280
|
| 16 |
+
],
|
| 17 |
+
"center_input_sample": false,
|
| 18 |
+
"cross_attention_dim": 1024,
|
| 19 |
+
"down_block_types": [
|
| 20 |
+
"CrossAttnDownBlock2D",
|
| 21 |
+
"CrossAttnDownBlock2D",
|
| 22 |
+
"CrossAttnDownBlock2D",
|
| 23 |
+
"DownBlock2D"
|
| 24 |
+
],
|
| 25 |
+
"downsample_padding": 1,
|
| 26 |
+
"dual_cross_attention": false,
|
| 27 |
+
"flip_sin_to_cos": true,
|
| 28 |
+
"freq_shift": 0,
|
| 29 |
+
"in_channels": 8,
|
| 30 |
+
"layers_per_block": 2,
|
| 31 |
+
"mid_block_scale_factor": 1,
|
| 32 |
+
"norm_eps": 1e-05,
|
| 33 |
+
"norm_num_groups": 32,
|
| 34 |
+
"num_class_embeds": null,
|
| 35 |
+
"only_cross_attention": false,
|
| 36 |
+
"out_channels": 5,
|
| 37 |
+
"sample_size": 64,
|
| 38 |
+
"up_block_types": [
|
| 39 |
+
"UpBlock2D",
|
| 40 |
+
"CrossAttnUpBlock2D",
|
| 41 |
+
"CrossAttnUpBlock2D",
|
| 42 |
+
"CrossAttnUpBlock2D"
|
| 43 |
+
],
|
| 44 |
+
"use_linear_projection": true
|
| 45 |
+
}
|
baselines/sar2opt/conddiff/README.md
ADDED
|
@@ -0,0 +1,175 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Conditional Diffusion — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**Conditional Diffusion for SAR to Optical Image Translation**, Bai, Pu and Xu,
|
| 4 |
+
*IEEE Geoscience and Remote Sensing Letters*, 2023
|
| 5 |
+
([doi:10.1109/LGRS.2023.3337143](https://doi.org/10.1109/LGRS.2023.3337143); the
|
| 6 |
+
year is the one in the authors' own citation block — a table that labels this
|
| 7 |
+
row with a later issue year is referring to the same paper).
|
| 8 |
+
|
| 9 |
+
Upstream code:
|
| 10 |
+
[`Coordi777/Conditional-Diffusion-for-SAR-to-Optical-Image-Translation`](https://github.com/Coordi777/Conditional-Diffusion-for-SAR-to-Optical-Image-Translation),
|
| 11 |
+
a modified copy of [`openai/guided-diffusion`](https://github.com/openai/guided-diffusion).
|
| 12 |
+
**The exact upstream commit is not recoverable** — see the licence section.
|
| 13 |
+
|
| 14 |
+
This is the pixel-space conditional diffusion model we retrained ourselves on
|
| 15 |
+
SAR2Opt, and it is the checkpoint behind the Conditional Diffusion row of the
|
| 16 |
+
ReFlowSET comparison table.
|
| 17 |
+
|
| 18 |
+
## What is in this folder
|
| 19 |
+
|
| 20 |
+
| file | bytes | what it is |
|
| 21 |
+
|---|---|---|
|
| 22 |
+
| `ema_final.pt` | 662,459,343 | EMA (decay 0.9999) of the UNet at update 50,000 — a **bare state dict**, no wrapper key |
|
| 23 |
+
|
| 24 |
+
`ema_final.pt` is the EMA of a **164.3 M-parameter guided-diffusion UNet**:
|
| 25 |
+
`num_channels` 128, `num_res_blocks` 3, `learn_sigma` **False**, attention at
|
| 26 |
+
resolutions 16 and 8. It is a plain `state_dict` — load it directly, with no
|
| 27 |
+
`['model']` or `['ema']` indirection.
|
| 28 |
+
|
| 29 |
+
The QXS-SAROPT and SAR2Opt files differ in size (657,495,287 vs 662,459,343
|
| 30 |
+
bytes) purely because of the 256 px versus 512 px positional and attention
|
| 31 |
+
buffers; the parameter count is otherwise the same.
|
| 32 |
+
|
| 33 |
+
**Conditioning.** The SAR image is concatenated to the noisy state **noise-free**
|
| 34 |
+
at every reverse step and at training time — that is the paper's claim, and the
|
| 35 |
+
code does exactly that.
|
| 36 |
+
|
| 37 |
+
## Training budget we used
|
| 38 |
+
|
| 39 |
+
| | |
|
| 40 |
+
|---|---|
|
| 41 |
+
| **generator updates released** | **50,000** |
|
| 42 |
+
| batch size / resolution | 6 @ 512 px |
|
| 43 |
+
| optimizer | Adam, **constant** lr 1e-4 with `--lr_anneal_steps 50000` (linear decay to zero, which is also the only stop mechanism in the released code) |
|
| 44 |
+
| EMA | decay 0.9999 |
|
| 45 |
+
| diffusion | T = 2,000, linear β schedule, eps-prediction, `learn_sigma` False |
|
| 46 |
+
| sampler (test) | **respaced DDPM, 250 steps**, `clip_denoised` on |
|
| 47 |
+
| augmentation | none — the released code has none |
|
| 48 |
+
| input | deterministic centre 512 crops of the 600 px tiles |
|
| 49 |
+
|
| 50 |
+
**The paper and the released code disagree about the learning-rate schedule.**
|
| 51 |
+
The paper describes warmup plus cosine; the released code implements neither. We
|
| 52 |
+
ran the code.
|
| 53 |
+
|
| 54 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 55 |
+
|
| 56 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 57 |
+
|---|---|---|---|---|
|
| 58 |
+
| 211.8 | 0.415 | 0.686 | 0.248 | 12.48 |
|
| 59 |
+
|
| 60 |
+
Evaluated on the official split's 627 test tiles, centre-cropped to 512 px.
|
| 61 |
+
The tiles are 600 px natively; this benchmark crops and never resizes, in every
|
| 62 |
+
method's training and in the evaluation. No subsampling: every metric on this
|
| 63 |
+
page is measured over all 627 pairs.
|
| 64 |
+
|
| 65 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 66 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 67 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 68 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 69 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 70 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 71 |
+
convention gives a systematically different number and the two must never be
|
| 72 |
+
mixed, or compared against a paper that used the other one.
|
| 73 |
+
|
| 74 |
+
## Load it and translate one SAR image
|
| 75 |
+
|
| 76 |
+
```python
|
| 77 |
+
import torch
|
| 78 |
+
from guided_diffusion.script_util import (create_model_and_diffusion,
|
| 79 |
+
model_and_diffusion_defaults)
|
| 80 |
+
|
| 81 |
+
d = model_and_diffusion_defaults()
|
| 82 |
+
d.update(image_size=512, num_channels=128, num_res_blocks=3, learn_sigma=False,
|
| 83 |
+
diffusion_steps=2000, noise_schedule='linear', timestep_respacing='250')
|
| 84 |
+
model, diffusion = create_model_and_diffusion(**d)
|
| 85 |
+
model.load_state_dict(torch.load('ema_final.pt', map_location='cpu'))
|
| 86 |
+
model.cuda().eval()
|
| 87 |
+
|
| 88 |
+
# sar: (1, 3, 512, 512) float tensor in [-1, 1]
|
| 89 |
+
sample = diffusion.p_sample_loop(model, (1, 3, 512, 512),
|
| 90 |
+
clip_denoised=True, model_kwargs={},
|
| 91 |
+
noise=None, condition=sar)
|
| 92 |
+
```
|
| 93 |
+
|
| 94 |
+
The repository imports `blobfile` and `mpi4py` unconditionally; a single-process
|
| 95 |
+
run on a local filesystem needs either those packages or small local stand-ins on
|
| 96 |
+
`PYTHONPATH`.
|
| 97 |
+
|
| 98 |
+
**Never pass `use_ddim=True`.** See below.
|
| 99 |
+
|
| 100 |
+
## Read before using this checkpoint
|
| 101 |
+
|
| 102 |
+
* **DDIM is broken upstream, and that is not a choice we made.**
|
| 103 |
+
`p_sample_loop(..., condition=None, ...)` accepts and threads the SAR
|
| 104 |
+
condition; `ddim_sample_loop(...)` has **no `condition` parameter at all**, so
|
| 105 |
+
passing `--use_ddim True` raises `TypeError`. Sampling is respaced DDPM with
|
| 106 |
+
250 steps, which is also what the authors' own `sample.sh` uses.
|
| 107 |
+
* **A correctness bug in the released sampler, which we fixed.** Upstream paired
|
| 108 |
+
each SAR image with an EO image by **unsorted `os.listdir` position** — i.e. by
|
| 109 |
+
filesystem order. Any number produced with the unpatched sampler is measured
|
| 110 |
+
against effectively arbitrary ground truth. We sort both listings. If you
|
| 111 |
+
reproduce this row from a clean upstream checkout, apply that fix or your
|
| 112 |
+
metrics are meaningless.
|
| 113 |
+
* **Three further changes we made**, all commented in place: the
|
| 114 |
+
distributed-init helper no longer overwrites `CUDA_VISIBLE_DEVICES` (upstream
|
| 115 |
+
pins rank % 8, which on a shared machine hijacks another user's device); the
|
| 116 |
+
noise tensor for a partial last batch is shaped from the batch rather than from
|
| 117 |
+
the `--batch_size` flag; and the sampler takes explicit input/output directories
|
| 118 |
+
and builds its resize transform at run time, because the module-level transform
|
| 119 |
+
hard-codes 256 px and would silently downsize the 512 px cell.
|
| 120 |
+
* **The training loader requires integer filenames.** It sorts with
|
| 121 |
+
`int(stem)`, so any non-numeric stem raises `ValueError`. Feed it an
|
| 122 |
+
integer-named adapter directory and keep a manifest to map back to the real
|
| 123 |
+
stems.
|
| 124 |
+
* **Do not let the released 512 px path resize.** The fork's `center_crop_arr`
|
| 125 |
+
*resizes* 600 → 512. We wrote deterministic centre-512 crops instead, to keep a
|
| 126 |
+
crop-not-resize protocol across the whole benchmark.
|
| 127 |
+
* Passes the identity-collapse audit on both datasets.
|
| 128 |
+
|
| 129 |
+
## Licence — stated factually ⚠ no upstream licence exists
|
| 130 |
+
|
| 131 |
+
**The upstream code base publishes no licence.**
|
| 132 |
+
[`Coordi777/Conditional-Diffusion-for-SAR-to-Optical-Image-Translation`](https://github.com/Coordi777/Conditional-Diffusion-for-SAR-to-Optical-Image-Translation)
|
| 133 |
+
has no LICENSE, LICENCE, COPYING or NOTICE file anywhere in the tree we trained
|
| 134 |
+
from, no licence section in its README, and the GitHub API reports no declared
|
| 135 |
+
licence, with the `/license` endpoint returning 404. Checked 2026-08-28.
|
| 136 |
+
|
| 137 |
+
Under default copyright that means **all rights are reserved by the authors and
|
| 138 |
+
no express permission to redistribute derived work has been granted** to us or
|
| 139 |
+
to you. We publish this checkpoint anyway and state the position plainly. Assess
|
| 140 |
+
redistribution for yourself; consider asking the authors.
|
| 141 |
+
|
| 142 |
+
**A second gap, which is about reproducibility rather than licensing.** Our
|
| 143 |
+
vendored copy of this repository carries no version control and records no
|
| 144 |
+
upstream URL inside its tree, so **the exact commit these weights were trained
|
| 145 |
+
from cannot be recovered**. A "clone upstream, then apply our patch" recipe is
|
| 146 |
+
therefore not possible for this row.
|
| 147 |
+
|
| 148 |
+
**Lineage.** The README states the repository is based on
|
| 149 |
+
`openai/guided-diffusion` with modifications, and the tree is visibly that code
|
| 150 |
+
base — OpenAI provenance comments survive in `guided_diffusion/logger.py` and
|
| 151 |
+
`guided_diffusion/unet.py`. `openai/guided-diffusion` is **MIT**; the unmodified
|
| 152 |
+
guided-diffusion parts carry that licence, which does **not** extend to the
|
| 153 |
+
authors' modifications. We do not ship the guided-diffusion licence text here
|
| 154 |
+
because it is not vendored in the tree we trained from — take it from that
|
| 155 |
+
repository if you need it, and make sure the copyright line you carry is
|
| 156 |
+
OpenAI's rather than another project's MIT file.
|
| 157 |
+
|
| 158 |
+
**We modified the code further**, and say so as a matter of discipline rather
|
| 159 |
+
than because any licence compels it: the four changes listed in the section
|
| 160 |
+
above, of which the sorted-listing fix is a correctness fix.
|
| 161 |
+
|
| 162 |
+
Please cite: Bai, Pu and Xu, *Conditional Diffusion for SAR to Optical Image
|
| 163 |
+
Translation*, IEEE Geoscience and Remote Sensing Letters,
|
| 164 |
+
[doi:10.1109/LGRS.2023.3337143](https://doi.org/10.1109/LGRS.2023.3337143).
|
| 165 |
+
The full record of what we checked, per method, is in
|
| 166 |
+
`licenses/NO-UPSTREAM-LICENSE.md` at the root of this repository.
|
| 167 |
+
|
| 168 |
+
---
|
| 169 |
+
|
| 170 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 171 |
+
checkpoint that we retrained ourselves on SAR2Opt; it is not ReFlowSET
|
| 172 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 173 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 174 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 175 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/sar2opt/controlnet/README.md
ADDED
|
@@ -0,0 +1,128 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ControlNet — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**Adding Conditional Control to Text-to-Image Diffusion Models**, Zhang, Rao and
|
| 4 |
+
Agrawala, *ICCV* 2023. The SAR image is the control signal and the base
|
| 5 |
+
Stable Diffusion 2.1-base UNet stays frozen.
|
| 6 |
+
|
| 7 |
+
> **How our row was produced, precisely.** We trained with the
|
| 8 |
+
> `examples/controlnet/train_controlnet.py` script from
|
| 9 |
+
> [`huggingface/diffusers`](https://github.com/huggingface/diffusers)
|
| 10 |
+
> (Apache-2.0, Copyright 2025 The HuggingFace Inc. team), initialising the
|
| 11 |
+
> adapter with `ControlNetModel.from_unet` on SD 2.1-base. **No code from
|
| 12 |
+
> `lllyasviel/ControlNet` was used.** The method is theirs; the implementation
|
| 13 |
+
> is the diffusers one.
|
| 14 |
+
|
| 15 |
+
## What is in this folder
|
| 16 |
+
|
| 17 |
+
A complete diffusers `ControlNetModel` folder — `config.json` plus
|
| 18 |
+
`diffusion_pytorch_model.safetensors` (364,228,240 parameters, fp32, 340
|
| 19 |
+
tensors). It is the encoder half of the UNet plus the zero-convolutions and the
|
| 20 |
+
conditioning embedder; it has no `up_blocks` and no `conv_out`, so **it cannot
|
| 21 |
+
generate anything on its own** and must be paired with the base pipeline.
|
| 22 |
+
|
| 23 |
+
The conditioning input is 3-channel RGB (`conditioning_channels: 3`) with
|
| 24 |
+
embedder widths `[16, 32, 96, 256]`; the SAR PNG is opened as RGB, so the three
|
| 25 |
+
channels are identical.
|
| 26 |
+
|
| 27 |
+
## Training budget we used
|
| 28 |
+
|
| 29 |
+
| | |
|
| 30 |
+
|---|---|
|
| 31 |
+
| optimizer updates | **50,000** |
|
| 32 |
+
| batch size / resolution | 8 @ 512 px |
|
| 33 |
+
| epochs the 50,000 updates work out to | ~276 (1,450 training pairs, 181 updates/epoch) |
|
| 34 |
+
| optimizer | AdamW, lr 1e-5, **constant** schedule (the script's default) |
|
| 35 |
+
| precision | bf16 mixed precision — the saved tensors are the fp32 master weights |
|
| 36 |
+
| seed | 42 |
|
| 37 |
+
| trainable parameters | the adapter only; the SD 2.1-base UNet, VAE and text encoder stayed frozen |
|
| 38 |
+
| prompt | the fixed string `"electro-optical image"` for every pair |
|
| 39 |
+
| augmentation | **none** — the diffusers trainer applies only `Resize(res)` + `CenterCrop(res)`, both no-ops at our input size, and no flips or rotations |
|
| 40 |
+
| test-time sampler | **UniPCMultistepScheduler, 50 steps, guidance scale 7.5**, bf16 |
|
| 41 |
+
|
| 42 |
+
Unlike the two UNet folders in this repository, this is **not** an SD 2.1
|
| 43 |
+
fine-tune: the base weights were never updated, which is why only the 364 M
|
| 44 |
+
adapter ships.
|
| 45 |
+
|
| 46 |
+
Reproduction note: our inference script creates **one** `torch.Generator`
|
| 47 |
+
seeded 42 and reuses it across the whole test set in batches of 8,
|
| 48 |
+
so a single image regenerated on its own will not be bit-identical to our dump
|
| 49 |
+
even at the same seed. The conditioning image is
|
| 50 |
+
the SAR PNG opened as RGB and bilinearly resized to 512x512 (a no-op at our
|
| 51 |
+
evaluation resolution, since the SAR inputs are already 512 px).
|
| 52 |
+
|
| 53 |
+
Data: 512 px. The 600 px source images were pre-cropped to a deterministic **centre** 512 crop (offset 44) for both training and evaluation, and are **never resized**. So, unlike the C-DiffSET rows in this repository, this row trained on a fixed crop rather than a random one. The EO/SAR pairing follows the official SAR2Opt layout (`trainB`/`testB` are EO, `trainA`/`testA` are SAR).
|
| 54 |
+
|
| 55 |
+
## Base model
|
| 56 |
+
|
| 57 |
+
This checkpoint is a derivative of **Stable Diffusion 2.1-base**. Because the
|
| 58 |
+
original `stabilityai/stable-diffusion-2-1-base` repository is no longer
|
| 59 |
+
available on the Hub, our training and every snippet below load the community
|
| 60 |
+
mirror [`Manojb/stable-diffusion-2-1-base`](https://huggingface.co/Manojb/stable-diffusion-2-1-base).
|
| 61 |
+
That is what we actually trained against — verified in the training
|
| 62 |
+
configuration files, the launch scripts and the training logs, none of which
|
| 63 |
+
ever reference `stabilityai/...`. The frozen VAE, CLIP text encoder, tokenizer
|
| 64 |
+
and scheduler all come from that repo. A different SD 2.1-base mirror is
|
| 65 |
+
expected to work but has not been verified by us.
|
| 66 |
+
|
| 67 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 68 |
+
|
| 69 |
+
| FID↓ | KID↓ | DISTS↓ | LPIPS↓ | PSNR↑ | SSIM↑ |
|
| 70 |
+
|---|---|---|---|---|---|
|
| 71 |
+
| 140.5 | 0.0480 | 0.350 | 0.643 | 11.73 | 0.217 |
|
| 72 |
+
|
| 73 |
+
These are our own re-evaluation numbers, not numbers copied from any paper.
|
| 74 |
+
LPIPS is **LPIPS-VGG on inputs mapped to [-1, 1]** (the `normalize=False`
|
| 75 |
+
convention); the alternative [0, 1] convention gives a systematically different
|
| 76 |
+
number and the two must never be mixed. PSNR/SSIM are per-image torchmetrics
|
| 77 |
+
with `data_range=1`; FID is `pytorch-fid` against the size-matched ground truth.
|
| 78 |
+
{}
|
| 79 |
+
|
| 80 |
+
## Load it and translate one SAR PNG
|
| 81 |
+
|
| 82 |
+
```python
|
| 83 |
+
import torch
|
| 84 |
+
from PIL import Image
|
| 85 |
+
from diffusers import (ControlNetModel, StableDiffusionControlNetPipeline,
|
| 86 |
+
UniPCMultistepScheduler)
|
| 87 |
+
|
| 88 |
+
REPO = "JeonghyeokDo/ReFlowSET" # comparison cells live under baselines/
|
| 89 |
+
BASE = "Manojb/stable-diffusion-2-1-base"
|
| 90 |
+
|
| 91 |
+
controlnet = ControlNetModel.from_pretrained(
|
| 92 |
+
REPO, subfolder="baselines/sar2opt/controlnet", torch_dtype=torch.bfloat16)
|
| 93 |
+
pipe = StableDiffusionControlNetPipeline.from_pretrained(
|
| 94 |
+
BASE, controlnet=controlnet, torch_dtype=torch.bfloat16,
|
| 95 |
+
safety_checker=None).to("cuda")
|
| 96 |
+
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
|
| 97 |
+
|
| 98 |
+
sar = Image.open("sar.png").convert("RGB").resize((512, 512), Image.BILINEAR)
|
| 99 |
+
eo = pipe("electro-optical image", image=sar, height=512, width=512,
|
| 100 |
+
num_inference_steps=50, guidance_scale=7.5,
|
| 101 |
+
generator=torch.Generator("cuda").manual_seed(42)).images[0]
|
| 102 |
+
eo.save("eo.png")
|
| 103 |
+
```
|
| 104 |
+
|
| 105 |
+
## Licence status — stated factually
|
| 106 |
+
|
| 107 |
+
* **Training code:** Apache-2.0. The trainer is
|
| 108 |
+
`examples/controlnet/train_controlnet.py` from
|
| 109 |
+
[`huggingface/diffusers`](https://github.com/huggingface/diffusers),
|
| 110 |
+
`Copyright 2025 The HuggingFace Inc. team`; the Apache-2.0 grant is in the
|
| 111 |
+
file header. Our copy of that script is four loose files rather than a full
|
| 112 |
+
clone, so it carries no separate repository LICENSE file — the header is the
|
| 113 |
+
grant.
|
| 114 |
+
* **Method credit:** ControlNet (Zhang et al., ICCV 2023),
|
| 115 |
+
[`lllyasviel/ControlNet`](https://github.com/lllyasviel/ControlNet),
|
| 116 |
+
Apache-2.0. **No code from that repository was used in this row.**
|
| 117 |
+
* **These weights:** initialised from Stable Diffusion 2.1-base via
|
| 118 |
+
`ControlNetModel.from_unet`, so they are an SD 2.1 derivative under the
|
| 119 |
+
**CreativeML Open RAIL++-M** licence (`openrail++`). The **Attachment A use
|
| 120 |
+
restrictions travel with these weights**. Full text, including Attachment A,
|
| 121 |
+
ships in this repository as `LICENSE-WEIGHTS.md`.
|
| 122 |
+
|
| 123 |
+
---
|
| 124 |
+
|
| 125 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 126 |
+
checkpoint retrained by us on SAR2Opt; it is not ReFlowSET itself. Every
|
| 127 |
+
comparison method in the release was retrained by us on the same splits at the
|
| 128 |
+
same resolution, so the rows are directly comparable to each other.
|
baselines/sar2opt/controlnet/config.json
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_class_name": "ControlNetModel",
|
| 3 |
+
"_diffusers_version": "0.37.1",
|
| 4 |
+
"act_fn": "silu",
|
| 5 |
+
"addition_embed_type": null,
|
| 6 |
+
"addition_embed_type_num_heads": 64,
|
| 7 |
+
"addition_time_embed_dim": null,
|
| 8 |
+
"attention_head_dim": [
|
| 9 |
+
5,
|
| 10 |
+
10,
|
| 11 |
+
20,
|
| 12 |
+
20
|
| 13 |
+
],
|
| 14 |
+
"block_out_channels": [
|
| 15 |
+
320,
|
| 16 |
+
640,
|
| 17 |
+
1280,
|
| 18 |
+
1280
|
| 19 |
+
],
|
| 20 |
+
"class_embed_type": null,
|
| 21 |
+
"conditioning_channels": 3,
|
| 22 |
+
"conditioning_embedding_out_channels": [
|
| 23 |
+
16,
|
| 24 |
+
32,
|
| 25 |
+
96,
|
| 26 |
+
256
|
| 27 |
+
],
|
| 28 |
+
"controlnet_conditioning_channel_order": "rgb",
|
| 29 |
+
"cross_attention_dim": 1024,
|
| 30 |
+
"down_block_types": [
|
| 31 |
+
"CrossAttnDownBlock2D",
|
| 32 |
+
"CrossAttnDownBlock2D",
|
| 33 |
+
"CrossAttnDownBlock2D",
|
| 34 |
+
"DownBlock2D"
|
| 35 |
+
],
|
| 36 |
+
"downsample_padding": 1,
|
| 37 |
+
"encoder_hid_dim": null,
|
| 38 |
+
"encoder_hid_dim_type": null,
|
| 39 |
+
"flip_sin_to_cos": true,
|
| 40 |
+
"freq_shift": 0,
|
| 41 |
+
"global_pool_conditions": false,
|
| 42 |
+
"in_channels": 4,
|
| 43 |
+
"layers_per_block": 2,
|
| 44 |
+
"mid_block_scale_factor": 1,
|
| 45 |
+
"mid_block_type": "UNetMidBlock2DCrossAttn",
|
| 46 |
+
"norm_eps": 1e-05,
|
| 47 |
+
"norm_num_groups": 32,
|
| 48 |
+
"num_attention_heads": null,
|
| 49 |
+
"num_class_embeds": null,
|
| 50 |
+
"only_cross_attention": false,
|
| 51 |
+
"projection_class_embeddings_input_dim": null,
|
| 52 |
+
"resnet_time_scale_shift": "default",
|
| 53 |
+
"transformer_layers_per_block": 1,
|
| 54 |
+
"upcast_attention": false,
|
| 55 |
+
"use_linear_projection": true
|
| 56 |
+
}
|
baselines/sar2opt/cyclegan/README.md
ADDED
|
@@ -0,0 +1,172 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# CycleGAN — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial
|
| 4 |
+
Networks**, Zhu, Park, Isola and Efros, *ICCV 2017*
|
| 5 |
+
([arXiv:1703.10593](https://arxiv.org/abs/1703.10593)). Upstream code:
|
| 6 |
+
[`junyanz/pytorch-CycleGAN-and-pix2pix`](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix)
|
| 7 |
+
at commit `2a7afba2895d52556dd5dfe07e8555ef657ced6f`.
|
| 8 |
+
|
| 9 |
+
This is the unpaired cycle-consistent GAN we retrained ourselves on SAR2Opt,
|
| 10 |
+
and it is the checkpoint behind the CycleGAN row of the ReFlowSET comparison
|
| 11 |
+
table. **It fails our input-copy audit — see the section below before you use
|
| 12 |
+
or quote it.**
|
| 13 |
+
|
| 14 |
+
## What is in this folder
|
| 15 |
+
|
| 16 |
+
| file | bytes | what it is |
|
| 17 |
+
|---|---|---|
|
| 18 |
+
| `net_G_A.pth` | 45,533,133 | **SAR → EO** — this is the translation direction the table reports |
|
| 19 |
+
| `net_G_B.pth` | 45,533,133 | EO → SAR, the reverse generator that closes the cycle; not used to produce any reported number |
|
| 20 |
+
|
| 21 |
+
Both are **`resnet_9blocks` generators** with instance normalisation,
|
| 22 |
+
`ngf = 64`, 3 in / 3 out, dropout disabled. The direction assignment is not a
|
| 23 |
+
convention we chose: upstream's unaligned loader binds `A_paths` to the
|
| 24 |
+
`trainA` / `testA` directory, which holds the SAR side, and the model computes
|
| 25 |
+
`fake_B = netG_A(real_A)`. So **`net_G_A` is the SAR → EO generator**.
|
| 26 |
+
|
| 27 |
+
`net_G_B` is shipped because it is half of the trained system and someone
|
| 28 |
+
reproducing the cycle needs it; it is not needed to reproduce our numbers. The
|
| 29 |
+
two discriminators are not released.
|
| 30 |
+
|
| 31 |
+
## Training budget we used
|
| 32 |
+
|
| 33 |
+
| | |
|
| 34 |
+
|---|---|
|
| 35 |
+
| **generator updates released** | **72,600** |
|
| 36 |
+
| batch size | 4 |
|
| 37 |
+
| epochs (constant LR + linear decay) | 100 + 100 |
|
| 38 |
+
| iterations per epoch | 363 (1,450 training images, the loader does not drop the last batch) |
|
| 39 |
+
| optimizer | Adam, lr 2e-4, β₁ 0.5, linear decay to 0 over the second half |
|
| 40 |
+
| adversarial loss | LSGAN; discriminator `basic` (70×70 PatchGAN) |
|
| 41 |
+
| cycle losses | λ_A 10, λ_B 10 |
|
| 42 |
+
| identity loss | λ_identity 0.5 |
|
| 43 |
+
| augmentation | horizontal flip |
|
| 44 |
+
| input | random 512 crops of the 600 px tiles |
|
| 45 |
+
|
| 46 |
+
Budgets are quoted in **optimizer updates**, never epochs. This is the single
|
| 47 |
+
most important comparability rule for the unpaired methods in this benchmark:
|
| 48 |
+
"100 + 100 epochs" is 72,600 updates on 1,450 images and 400,000+ on
|
| 49 |
+
16,001, a difference that reads as budget-matched and is not.
|
| 50 |
+
|
| 51 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 52 |
+
|
| 53 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 54 |
+
|---|---|---|---|---|
|
| 55 |
+
| 143.5 | 0.330 | 0.650 | 0.178 | 12.90 |
|
| 56 |
+
|
| 57 |
+
Evaluated on the official split's 627 test tiles, centre-cropped to 512 px.
|
| 58 |
+
The tiles are 600 px natively; this benchmark crops and never resizes, in every
|
| 59 |
+
method's training and in the evaluation. No subsampling: every metric on this
|
| 60 |
+
page is measured over all 627 pairs.
|
| 61 |
+
|
| 62 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 63 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 64 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 65 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 66 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 67 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 68 |
+
convention gives a systematically different number and the two must never be
|
| 69 |
+
mixed, or compared against a paper that used the other one.
|
| 70 |
+
|
| 71 |
+
## Load it and translate one SAR image
|
| 72 |
+
|
| 73 |
+
Clone `junyanz/pytorch-CycleGAN-and-pix2pix` at the commit above and run this
|
| 74 |
+
from its root. The model code is stock — we applied no patch to it.
|
| 75 |
+
|
| 76 |
+
```python
|
| 77 |
+
import numpy as np, torch
|
| 78 |
+
from PIL import Image
|
| 79 |
+
from models.networks import define_G # from the junyanz repo
|
| 80 |
+
|
| 81 |
+
G_A = define_G(3, 3, 64, 'resnet_9blocks', 'instance', use_dropout=False,
|
| 82 |
+
init_type='normal', init_gain=0.02, gpu_ids=[])
|
| 83 |
+
sd = torch.load('net_G_A.pth', map_location='cpu')
|
| 84 |
+
if hasattr(sd, '_metadata'):
|
| 85 |
+
del sd._metadata
|
| 86 |
+
G_A.load_state_dict(sd)
|
| 87 |
+
G_A.eval()
|
| 88 |
+
|
| 89 |
+
sar = Image.open('sar.png').convert('RGB')
|
| 90 |
+
sar = sar.crop(((600-512)//2, (600-512)//2, (600+512)//2, (600+512)//2))
|
| 91 |
+
x = torch.from_numpy(np.asarray(sar, np.float32) / 127.5 - 1).permute(2, 0, 1)[None]
|
| 92 |
+
|
| 93 |
+
with torch.no_grad():
|
| 94 |
+
y = G_A(x) # this is fake_B
|
| 95 |
+
|
| 96 |
+
eo = ((y[0].permute(1, 2, 0).numpy() + 1) * 127.5).clip(0, 255).astype(np.uint8)
|
| 97 |
+
Image.fromarray(eo).save('eo.png')
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
Upstream also supports a SAR-only run without this snippet:
|
| 101 |
+
`test.py --model test --dataset_mode single --no_dropout --model_suffix _A`,
|
| 102 |
+
which loads `latest_net_G_A.pth` directly.
|
| 103 |
+
|
| 104 |
+
## ⚠ This cell fails the input-copy audit
|
| 105 |
+
|
| 106 |
+
We audit every cell for **identity collapse**: a model that has learned to pass
|
| 107 |
+
its input through rather than translate it. The test compares the mean absolute
|
| 108 |
+
distance from the generated image to the SAR **input** against the distance to
|
| 109 |
+
the EO **target**; a ratio below 1.0 means the output is closer to the input it
|
| 110 |
+
was given than to the answer it was asked for.
|
| 111 |
+
|
| 112 |
+
| quantity | value |
|
| 113 |
+
|---|---|
|
| 114 |
+
| mean \|generated − ground truth\| | 50.812 |
|
| 115 |
+
| mean \|generated − SAR input\| | 40.415 |
|
| 116 |
+
| **ratio** (SAR distance ÷ GT distance) | **0.795** — below 1.0 |
|
| 117 |
+
| verdict | **COLLAPSED** |
|
| 118 |
+
| cross-pair separation | 0.885 (cohort median 1.088) — no oracle behaviour |
|
| 119 |
+
|
| 120 |
+
**Read the metrics on this page with that in mind.** They are reported unchanged
|
| 121 |
+
because they are what the released implementation does at its own published
|
| 122 |
+
protocol, and a benchmark that quietly substitutes a better-behaved variant is
|
| 123 |
+
not reporting the method. But they should not be read as translation quality.
|
| 124 |
+
The separation figure is the companion check for the *opposite* failure: it is
|
| 125 |
+
the mean distance from a generated image to **other** images' ground truth
|
| 126 |
+
divided by the distance to its **own**, so a value well above the cohort median
|
| 127 |
+
would indicate that ground-truth information had leaked into the generation.
|
| 128 |
+
This cell sits at or below the median, so there is no leak — the problem here is
|
| 129 |
+
input-copying, and only input-copying.
|
| 130 |
+
|
| 131 |
+
This is not a budget artefact on this dataset: a diagnostic control run of the same code at roughly 2.2× the budget still collapses (ratio 0.767). That control is not published here.
|
| 132 |
+
|
| 133 |
+
## Read before using this checkpoint
|
| 134 |
+
|
| 135 |
+
* **`net_G_A` is SAR → EO.** Loading `net_G_B` and expecting an EO image gives
|
| 136 |
+
you the reverse mapping and a confusing result.
|
| 137 |
+
* Build the network with `gpu_ids=[]` — the checkpoints have bare keys and a
|
| 138 |
+
non-empty `gpu_ids` introduces a `module.` prefix the load will reject.
|
| 139 |
+
* Normalisation is `[-1, 1]`.
|
| 140 |
+
* CycleGAN is **unpaired**: it never saw a SAR/EO correspondence during
|
| 141 |
+
training, only the two marginal distributions.
|
| 142 |
+
|
| 143 |
+
## Licence — stated factually
|
| 144 |
+
|
| 145 |
+
Same upstream, same LICENSE file as pix2pix; reproduced verbatim here as
|
| 146 |
+
`licenses/LICENSE-cyclegan.txt` (byte-identical to `licenses/LICENSE-pix2pix.txt`
|
| 147 |
+
— one file covers both methods upstream). It is three stacked notices:
|
| 148 |
+
|
| 149 |
+
* CycleGAN — `Copyright (c) 2017, Jun-Yan Zhu and Taesung Park` (BSD-2-Clause in
|
| 150 |
+
form);
|
| 151 |
+
* pix2pix — `Copyright (c) 2016, Phillip Isola and Jun-Yan Zhu`;
|
| 152 |
+
* DCGAN — `Copyright (c) 2015, Facebook, Inc.` (BSD-3-Clause, with a
|
| 153 |
+
no-endorsement clause).
|
| 154 |
+
|
| 155 |
+
**Redistribution of these weights is permitted** with the notice reproduced. No
|
| 156 |
+
non-commercial and no share-alike clause. Do not use Facebook's name to promote
|
| 157 |
+
this release.
|
| 158 |
+
|
| 159 |
+
Attribution: *trained with `junyanz/pytorch-CycleGAN-and-pix2pix`. Copyright (c)
|
| 160 |
+
2017, Jun-Yan Zhu and Taesung Park (CycleGAN); Copyright (c) 2016, Phillip Isola
|
| 161 |
+
and Jun-Yan Zhu (pix2pix); Copyright (c) 2015, Facebook, Inc. (DCGAN). All rights
|
| 162 |
+
reserved. Redistributed under the BSD terms reproduced in
|
| 163 |
+
`licenses/LICENSE-cyclegan.txt`.*
|
| 164 |
+
|
| 165 |
+
---
|
| 166 |
+
|
| 167 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 168 |
+
checkpoint that we retrained ourselves on SAR2Opt; it is not ReFlowSET
|
| 169 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 170 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 171 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 172 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/sar2opt/ddpm/README.md
ADDED
|
@@ -0,0 +1,176 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# DDPM (SR3-class) — SAR2Opt
|
| 2 |
+
|
| 3 |
+
The **SR3 method class** — an eps-prediction conditional DDPM that conditions on
|
| 4 |
+
the source image — as introduced in *Image Super-Resolution via Iterative
|
| 5 |
+
Refinement*, Saharia, Ho, Chan, Salimans, Fleet and Norouzi,
|
| 6 |
+
*IEEE TPAMI 2022* ([arXiv:2104.07636](https://arxiv.org/abs/2104.07636)).
|
| 7 |
+
|
| 8 |
+
**Read this before citing the row: the code is not the SR3 authors'.** No code
|
| 9 |
+
from any SR3 release was used. This cell is
|
| 10 |
+
[`DeepSARRS/E3Diff`](https://github.com/DeepSARRS/E3Diff) at commit
|
| 11 |
+
`38601093ab8f8e4b478144621f20890b100a3b74`, run in its **stage-1**
|
| 12 |
+
configuration (`"stage": 1`), which is an SR3-class conditional DDPM. We
|
| 13 |
+
retrained it ourselves on SAR2Opt and it is the checkpoint behind the DDPM
|
| 14 |
+
row of the ReFlowSET comparison table.
|
| 15 |
+
|
| 16 |
+
## What is in this folder
|
| 17 |
+
|
| 18 |
+
| file | bytes | what it is |
|
| 19 |
+
|---|---|---|
|
| 20 |
+
| `gen.pth` | 768,426,944 | the `sr3` UNet at iteration 250,000 — the whole model |
|
| 21 |
+
|
| 22 |
+
`gen.pth` is E3Diff's **`sr3` UNet**: `inner_channel` 64,
|
| 23 |
+
`channel_multiplier` [1, 2, 4, 8, 16], `res_blocks` 1, `norm_groups` 16, **no
|
| 24 |
+
attention** (`attn_res` is empty), 3 input channels, 3 output channels and a
|
| 25 |
+
3-channel condition.
|
| 26 |
+
|
| 27 |
+
The condition is **not the raw SAR image alone**. It is a three-channel stack
|
| 28 |
+
`[PPB, Canny, SAR]`:
|
| 29 |
+
|
| 30 |
+
* channel 0 — the SAR image after **FAST_PPB** speckle filtering
|
| 31 |
+
(Deledalle 2009, P = 3, W = 10, h = 0.5), a GPU port of E3Diff's own
|
| 32 |
+
`FAST_PPB.m`;
|
| 33 |
+
* channel 1 — `cv2.Canny(ppb_uint8, 50, 150, L2gradient=True)` on that filtered
|
| 34 |
+
image;
|
| 35 |
+
* channel 2 — the raw SAR image.
|
| 36 |
+
|
| 37 |
+
**This row cannot be run from a SAR PNG alone**: the PPB and Canny images must
|
| 38 |
+
be computed first.
|
| 39 |
+
|
| 40 |
+
## Training budget we used
|
| 41 |
+
|
| 42 |
+
| | |
|
| 43 |
+
|---|---|
|
| 44 |
+
| **generator updates released** | **250,000** |
|
| 45 |
+
| batch size / resolution | 4 @ 512 px |
|
| 46 |
+
| optimizer | Adam, lr 5e-5, constant (`MultiStepLR` with no milestones) |
|
| 47 |
+
| EMA | decay 0.9999, started at step 5,000, updated every step |
|
| 48 |
+
| diffusion (train) | T = 1,000, linear β from 1e-6 to 1e-2 |
|
| 49 |
+
| sampler (test) | **DDIM, 50 steps** |
|
| 50 |
+
| loss | plain eps-prediction MSE — the stage-2 auxiliary losses are all zero here (`lpips_w` 0, `fft_w` 0, `lambda_gan` 0) |
|
| 51 |
+
| augmentation | flips, 90° rotations, and a random brightness jitter applied to the SAR/PPB pair only (upstream's own `transform_augment`) |
|
| 52 |
+
|
| 53 |
+
Both datasets get the same 250,000 updates; the batch differs so that the
|
| 54 |
+
pixels-per-update budget matches (16 × 256² = 4 × 512²).
|
| 55 |
+
|
| 56 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 57 |
+
|
| 58 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 59 |
+
|---|---|---|---|---|
|
| 60 |
+
| 122.5 | 0.295 | 0.610 | 0.313 | 13.65 |
|
| 61 |
+
|
| 62 |
+
Evaluated on the official split's 627 test tiles, centre-cropped to 512 px.
|
| 63 |
+
The tiles are 600 px natively; this benchmark crops and never resizes, in every
|
| 64 |
+
method's training and in the evaluation. No subsampling: every metric on this
|
| 65 |
+
page is measured over all 627 pairs.
|
| 66 |
+
|
| 67 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 68 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 69 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 70 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 71 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 72 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 73 |
+
convention gives a systematically different number and the two must never be
|
| 74 |
+
mixed, or compared against a paper that used the other one.
|
| 75 |
+
|
| 76 |
+
## Load it and translate one SAR image
|
| 77 |
+
|
| 78 |
+
This model cannot be driven from a bare SAR file: it needs the PPB and Canny
|
| 79 |
+
condition channels, and the E3Diff loader looks them up by filename. Build the
|
| 80 |
+
condition first, then run the repository's own entry point.
|
| 81 |
+
|
| 82 |
+
```python
|
| 83 |
+
# 1. Build the condition channels for each SAR image, once.
|
| 84 |
+
import cv2, numpy as np
|
| 85 |
+
# ppb = FAST_PPB(sar, P=3, W=10, h=0.5) # Deledalle 2009; E3Diff ships FAST_PPB.m
|
| 86 |
+
# canny = cv2.Canny(ppb.astype(np.uint8), 50, 150, L2gradient=True)
|
| 87 |
+
# Write ppb and canny under <root>/val/SAR-PPB/<name> and <root>/val/SAR-canny/<name>,
|
| 88 |
+
# alongside <root>/val/SAR/<name> and <root>/val/EO/<name>. Names must match exactly.
|
| 89 |
+
|
| 90 |
+
# 2. Run the repository, phase 'val', with a config whose
|
| 91 |
+
# path.resume_state is the checkpoint PREFIX -- no '_gen.pth' suffix:
|
| 92 |
+
# "path": {"resume_state": "<dir>/I250000_E<epoch>"} # place gen.pth as I250000_E<epoch>_gen.pth
|
| 93 |
+
# "stage": 1,
|
| 94 |
+
# "model": {"beta_schedule": {"val": {"n_timestep": 50, "ddim": 1, ...}}},
|
| 95 |
+
# "datasets": {"val": {"data_len": -1, "r_resolution": 512}}
|
| 96 |
+
# python main.py -c <val.json> -p val -enable_wandb "" --seed 1
|
| 97 |
+
```
|
| 98 |
+
|
| 99 |
+
Two mechanical details that will otherwise cost you a full inference pass:
|
| 100 |
+
|
| 101 |
+
* the RGB loader. Upstream's `SAR2EODataset` hard-codes a **grayscale** EO
|
| 102 |
+
target and a two-channel `[PPB, Canny]` condition. SAR2Opt's optical side is
|
| 103 |
+
RGB, and with three channels upstream's `ddim_sample()` slices the condition to
|
| 104 |
+
`condition_x[:, :3]` and concatenates it with 3-channel predictions, which
|
| 105 |
+
raises on the first validation image. We ran through a small wrapper that
|
| 106 |
+
patches `SAR2EODataset.__getitem__` **in memory** — returning
|
| 107 |
+
`HR = EO[0:3]`, `LR = SAR[0:3]`, `SR = cat(PPB[0:1], Canny[0:1], SAR[0:1])` —
|
| 108 |
+
and then executes the repository's `main.py` verbatim, so no upstream file is
|
| 109 |
+
touched and every flag behaves the same. Reproduce that patch or write your own
|
| 110 |
+
three-channel loader;
|
| 111 |
+
* `-enable_wandb ""` is mandatory. The flag defaults to the string `'false'`,
|
| 112 |
+
which is truthy.
|
| 113 |
+
|
| 114 |
+
## Read before using this checkpoint
|
| 115 |
+
|
| 116 |
+
* **The checkpoint loads with `strict=False`.** `model/model.py` calls
|
| 117 |
+
`load_state_dict(..., strict=False)`, so a configuration that does not match
|
| 118 |
+
the file **loads nothing and raises no error** — you get a fully random model
|
| 119 |
+
and a plausible-looking run. Verify the `Loading pretrained model for G [...]`
|
| 120 |
+
line in the log and eyeball the first output.
|
| 121 |
+
* **`-p val` writes next to the checkpoint and then renames.** Images land in
|
| 122 |
+
`<resume_state>/sample/`, and at the end the directory is renamed to
|
| 123 |
+
`<resume_state>_S<ssim>_P<psnr>_l2<l2>_Lp<lpips>`. That rename **raises if the
|
| 124 |
+
target already exists** — after the entire inference has been paid for. Guard
|
| 125 |
+
for an existing `<prefix>_S*/sample` before starting.
|
| 126 |
+
* **SoftPool.** The E3Diff code imports the SoftPool CUDA extension
|
| 127 |
+
unconditionally. We used a pure-PyTorch drop-in with identical maths
|
| 128 |
+
(`softpool(x) = avgpool(x·eˣ) / avgpool(eˣ)`) and identical autograd, which
|
| 129 |
+
needs no build step. Either build the extension or write the equivalent.
|
| 130 |
+
* **A known tone offset, inherited deliberately.** On another dataset in the
|
| 131 |
+
wider campaign this exact configuration produced generations that were, on
|
| 132 |
+
average, tens of gray levels brighter than the ground truth, at a substantial
|
| 133 |
+
cost in PSNR. We did **not** correct it per dataset, because comparability
|
| 134 |
+
across the row was judged worth more than per-cell tuning. Read this row's
|
| 135 |
+
PSNR with that in mind.
|
| 136 |
+
* Passes the identity-collapse audit on both datasets.
|
| 137 |
+
|
| 138 |
+
## Licence — stated factually ⚠ no upstream licence exists
|
| 139 |
+
|
| 140 |
+
**The upstream code base publishes no licence.**
|
| 141 |
+
[`DeepSARRS/E3Diff`](https://github.com/DeepSARRS/E3Diff), whose stage-1
|
| 142 |
+
configuration this row is, has no LICENSE, LICENCE, COPYING or NOTICE file at
|
| 143 |
+
any depth in its tree; only 1 of its 58 tracked files is licence-shaped and that
|
| 144 |
+
one is `SoftPool/LICENSE.txt`, the MIT licence of a **vendored third-party
|
| 145 |
+
dependency** (`Copyright (c) 2020 Alexandros Stergiou`, reproduced here as
|
| 146 |
+
`licenses/LICENSE-SoftPool-MIT.txt`) — not a grant for E3Diff. Its README has no
|
| 147 |
+
licence section, and the GitHub API reports no declared licence, with the
|
| 148 |
+
`/license` endpoint returning 404. Checked 2026-08-28.
|
| 149 |
+
|
| 150 |
+
Under default copyright that means **all rights are reserved by the authors and
|
| 151 |
+
no express permission to redistribute derived work has been granted** to us or
|
| 152 |
+
to you. We publish this checkpoint anyway, so that the benchmark is reproducible
|
| 153 |
+
end to end, and we state the position plainly rather than implying a permission
|
| 154 |
+
that does not exist. If you intend to redistribute this checkpoint or build on
|
| 155 |
+
it, assess that for yourself, and consider asking the authors directly.
|
| 156 |
+
|
| 157 |
+
The code base credits SR3
|
| 158 |
+
(`Janspiry/Image-Super-Resolution-via-Iterative-Refinement`, Apache-2.0),
|
| 159 |
+
`GaParmar/img2img-turbo` (MIT) and `alexandrosstergiou/SoftPool` (MIT). Those
|
| 160 |
+
licences cover the borrowed parts, not the authors' own contribution.
|
| 161 |
+
|
| 162 |
+
Please cite the SR3 paper for the method class, and E3Diff for the code that
|
| 163 |
+
produced these weights: Qin, Zou, Li and Zhang, *Efficient End-to-End Diffusion
|
| 164 |
+
Model for One-Step SAR-to-Optical Translation*, IEEE Geoscience and Remote
|
| 165 |
+
Sensing Letters, [doi:10.1109/LGRS.2024.3506566](https://doi.org/10.1109/LGRS.2024.3506566).
|
| 166 |
+
The full record of what we checked, per method, is in
|
| 167 |
+
`licenses/NO-UPSTREAM-LICENSE.md` at the root of this repository.
|
| 168 |
+
|
| 169 |
+
---
|
| 170 |
+
|
| 171 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 172 |
+
checkpoint that we retrained ourselves on SAR2Opt; it is not ReFlowSET
|
| 173 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 174 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 175 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 176 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/sar2opt/e3diff/README.md
ADDED
|
@@ -0,0 +1,179 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# E3Diff — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**Efficient End-to-End Diffusion Model for One-Step SAR-to-Optical
|
| 4 |
+
Translation**, Qin, Zou, Li and Zhang, *IEEE Geoscience and Remote Sensing
|
| 5 |
+
Letters*, 2024
|
| 6 |
+
([doi:10.1109/LGRS.2024.3506566](https://doi.org/10.1109/LGRS.2024.3506566); the
|
| 7 |
+
year is the one in the authors' own citation block — a table that labels this row
|
| 8 |
+
with a later issue year is referring to the same paper). Upstream code:
|
| 9 |
+
[`DeepSARRS/E3Diff`](https://github.com/DeepSARRS/E3Diff) at commit
|
| 10 |
+
`38601093ab8f8e4b478144621f20890b100a3b74`.
|
| 11 |
+
|
| 12 |
+
This is the authors' two-stage method, **retrained by us end to end** on
|
| 13 |
+
SAR2Opt, and it is the checkpoint behind the E3Diff row of the ReFlowSET
|
| 14 |
+
comparison table.
|
| 15 |
+
|
| 16 |
+
**These are our weights, not the authors' released checkpoint.** The authors'
|
| 17 |
+
own weights exist for a different dataset and are not what this row measures.
|
| 18 |
+
|
| 19 |
+
## What is in this folder
|
| 20 |
+
|
| 21 |
+
| file | bytes | what it is |
|
| 22 |
+
|---|---|---|
|
| 23 |
+
| `gen.pth` | 768,426,944 | the `sr3` UNet after stage 2, at absolute iteration 310,000 |
|
| 24 |
+
|
| 25 |
+
Architecturally this is **the same network as the DDPM (SR3-class) row** in this
|
| 26 |
+
repository — E3Diff's two stages share one UNet. What differs is the training:
|
| 27 |
+
|
| 28 |
+
* **stage 1** (250,000 updates) is the eps-prediction conditional DDPM. Its
|
| 29 |
+
checkpoint is the `ddpm/` folder in this repository.
|
| 30 |
+
* **stage 2** (60,000 further updates, absolute iteration 310,000) fine-tunes
|
| 31 |
+
that same network into a **one-step generator**: the sampler is run *with
|
| 32 |
+
gradients* from pure noise for `ddim_steps = 1`, and the loss is taken directly
|
| 33 |
+
on pixels.
|
| 34 |
+
|
| 35 |
+
UNet: `inner_channel` 64, `channel_multiplier` [1, 2, 4, 8, 16], `res_blocks` 1,
|
| 36 |
+
`norm_groups` 16, no attention, 3 in / 3 out, 3-channel condition.
|
| 37 |
+
|
| 38 |
+
The condition is the same three-channel `[PPB, Canny, SAR]` stack described on
|
| 39 |
+
the DDPM card: FAST_PPB speckle filtering (Deledalle 2009, P = 3, W = 10,
|
| 40 |
+
h = 0.5), `cv2.Canny(ppb, 50, 150, L2gradient=True)`, and the raw SAR image.
|
| 41 |
+
**This row cannot be run from a SAR PNG alone.**
|
| 42 |
+
|
| 43 |
+
## Training budget we used
|
| 44 |
+
|
| 45 |
+
| | |
|
| 46 |
+
|---|---|
|
| 47 |
+
| **generator updates released** | **250,000 (stage 1, inherited) + 60,000 (stage 2) = 310,000 absolute** |
|
| 48 |
+
| batch size / resolution | 4 @ 512 px |
|
| 49 |
+
| optimizer | Adam, lr 5e-5 |
|
| 50 |
+
| stage-2 sampler during training | DDIM, **1 step**, run with gradients from pure noise |
|
| 51 |
+
| stage-2 losses | L1 on pixels, + LPIPS (weight 5), + focal-frequency (weight 10), + a vision-aided CLIP GAN (λ_gan 0.5) |
|
| 52 |
+
| stage-1 losses | plain eps-prediction MSE (all auxiliary weights zero) |
|
| 53 |
+
| sampler (test) | **DDIM, 1 step** |
|
| 54 |
+
| EMA | decay 0.9999 (stage 1) |
|
| 55 |
+
| augmentation | flips, 90° rotations, and a random brightness jitter on the SAR/PPB pair (upstream's `transform_augment`) |
|
| 56 |
+
|
| 57 |
+
The stage-2 budget is 24 % of the stage-1 budget, which follows the authors' own
|
| 58 |
+
ratio (they resume at 640,000 and run to 800,000). Note that `n_iter` in this
|
| 59 |
+
code base is an **absolute** step count continuing stage 1, not a stage-2
|
| 60 |
+
budget — a config that reads 310,000 buys 60,000 new iterations.
|
| 61 |
+
|
| 62 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 63 |
+
|
| 64 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 65 |
+
|---|---|---|---|---|
|
| 66 |
+
| 104.7 | 0.232 | 0.529 | 0.249 | 16.09 |
|
| 67 |
+
|
| 68 |
+
Evaluated on the official split's 627 test tiles, centre-cropped to 512 px.
|
| 69 |
+
The tiles are 600 px natively; this benchmark crops and never resizes, in every
|
| 70 |
+
method's training and in the evaluation. No subsampling: every metric on this
|
| 71 |
+
page is measured over all 627 pairs.
|
| 72 |
+
|
| 73 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 74 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 75 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 76 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 77 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 78 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 79 |
+
convention gives a systematically different number and the two must never be
|
| 80 |
+
mixed, or compared against a paper that used the other one.
|
| 81 |
+
|
| 82 |
+
## Load it and translate one SAR image
|
| 83 |
+
|
| 84 |
+
Same entry point and same mechanics as the DDPM row, with `"stage": 2` and
|
| 85 |
+
`ddim_steps: 1`.
|
| 86 |
+
|
| 87 |
+
```python
|
| 88 |
+
# 1. Build the condition channels for each SAR image, once:
|
| 89 |
+
# ppb = FAST_PPB(sar, P=3, W=10, h=0.5)
|
| 90 |
+
# canny = cv2.Canny(ppb.astype('uint8'), 50, 150, L2gradient=True)
|
| 91 |
+
# written to <root>/val/SAR-PPB/<name> and <root>/val/SAR-canny/<name>,
|
| 92 |
+
# alongside <root>/val/SAR/<name> and <root>/val/EO/<name>.
|
| 93 |
+
|
| 94 |
+
# 2. Run phase 'val' with a config whose path.resume_state is the checkpoint
|
| 95 |
+
# PREFIX -- no '_gen.pth' suffix -- and place gen.pth as <prefix>_gen.pth:
|
| 96 |
+
# "stage": 2, "ddim_steps": 1,
|
| 97 |
+
# "model": {"beta_schedule": {"val": {"n_timestep": 1, "ddim": 1, ...}}},
|
| 98 |
+
# "datasets": {"val": {"data_len": -1, "r_resolution": 512}}
|
| 99 |
+
# python main.py -c <val.json> -p val -enable_wandb "" --seed 1
|
| 100 |
+
```
|
| 101 |
+
|
| 102 |
+
Sampling is genuinely one network evaluation per image, which is the point of
|
| 103 |
+
the method.
|
| 104 |
+
|
| 105 |
+
As on the DDPM card, the RGB loader is the one thing you must sort out: upstream
|
| 106 |
+
hard-codes a grayscale EO target and a two-channel condition, and with three
|
| 107 |
+
channels its `ddim_sample()` slices the condition and concatenates mismatched
|
| 108 |
+
shapes. We patched `SAR2EODataset.__getitem__` in memory to return
|
| 109 |
+
`HR = EO[0:3]`, `LR = SAR[0:3]`, `SR = cat(PPB[0:1], Canny[0:1], SAR[0:1])` and
|
| 110 |
+
then ran the repository's `main.py` verbatim.
|
| 111 |
+
|
| 112 |
+
## Read before using this checkpoint
|
| 113 |
+
|
| 114 |
+
* **This is not an oracle.** Worth stating, because a one-step generator invites
|
| 115 |
+
the question: the sampler's `condition_x` is the `[PPB, Canny, SAR]` stack, and
|
| 116 |
+
the ground truth never enters it. The cell also passes our leak audit on both
|
| 117 |
+
datasets.
|
| 118 |
+
* **Stage 2 needs the *real* vision-aided CLIP discriminator.** Stage 1 has
|
| 119 |
+
`lambda_gan = 0` and can import a stub; stage 2 cannot. If a constant-output
|
| 120 |
+
stub shadows the real `vision_aided_loss` package, stage 2 trains against a
|
| 121 |
+
constant GAN loss and **silently is not the authors' method**. Assert on
|
| 122 |
+
`vision_aided_loss.__file__` before starting a run.
|
| 123 |
+
* **SoftPool.** The code imports the SoftPool CUDA extension unconditionally. We
|
| 124 |
+
used a pure-PyTorch drop-in with identical maths
|
| 125 |
+
(`softpool(x) = avgpool(x·eˣ) / avgpool(eˣ)`) and identical autograd, which
|
| 126 |
+
needs no build step.
|
| 127 |
+
* **The checkpoint loads with `strict=False`.** A mismatched configuration
|
| 128 |
+
**loads nothing and raises no error**. Verify the `Loading pretrained model for
|
| 129 |
+
G [...]` log line and look at the first output.
|
| 130 |
+
* **`-p val` writes next to the checkpoint and then renames** the sample
|
| 131 |
+
directory to `<prefix>_S<ssim>_P<psnr>_l2<l2>_Lp<lpips>`. That rename raises if
|
| 132 |
+
the target already exists — after the whole inference has been paid for.
|
| 133 |
+
* **`-enable_wandb ""` is mandatory**; the flag defaults to the truthy string
|
| 134 |
+
`'false'`.
|
| 135 |
+
* One change we made to the repository: `core/logger.py` no longer
|
| 136 |
+
unconditionally overwrites `CUDA_VISIBLE_DEVICES` from the config's `gpu_ids`,
|
| 137 |
+
which on a shared machine remapped every job onto another user's device. It is
|
| 138 |
+
a scheduling fix and does not affect the model.
|
| 139 |
+
|
| 140 |
+
## Licence — stated factually ⚠ no upstream licence exists
|
| 141 |
+
|
| 142 |
+
**The upstream code base publishes no licence.**
|
| 143 |
+
[`DeepSARRS/E3Diff`](https://github.com/DeepSARRS/E3Diff) has no LICENSE,
|
| 144 |
+
LICENCE, COPYING or NOTICE file at the repository root or at any depth; **1 of
|
| 145 |
+
its 58 tracked files** is licence-shaped, and it is `SoftPool/LICENSE.txt`, the
|
| 146 |
+
MIT licence of a **vendored third-party dependency** (`Copyright (c) 2020
|
| 147 |
+
Alexandros Stergiou`, reproduced here as `licenses/LICENSE-SoftPool-MIT.txt`) —
|
| 148 |
+
not a grant covering E3Diff. Its README has no licence section, and the GitHub
|
| 149 |
+
API reports no declared licence, with the `/license` endpoint returning 404.
|
| 150 |
+
Checked 2026-08-28.
|
| 151 |
+
|
| 152 |
+
Under default copyright that means **all rights are reserved by the authors and
|
| 153 |
+
no express permission to redistribute derived work has been granted** to us or
|
| 154 |
+
to you. We publish this checkpoint anyway, so that the benchmark is reproducible
|
| 155 |
+
end to end, and we state the position plainly rather than implying a permission
|
| 156 |
+
that does not exist. If you intend to redistribute this checkpoint or build on
|
| 157 |
+
it, assess that for yourself, and consider asking the authors directly.
|
| 158 |
+
|
| 159 |
+
**Lineage.** E3Diff's README credits SR3
|
| 160 |
+
(`Janspiry/Image-Super-Resolution-via-Iterative-Refinement`, Apache-2.0),
|
| 161 |
+
`GaParmar/img2img-turbo` (MIT) and `alexandrosstergiou/SoftPool` (MIT). Those
|
| 162 |
+
licences cover the borrowed parts only, not the authors' own two-stage
|
| 163 |
+
contribution. We ship only the SoftPool text, because it is the only one of the
|
| 164 |
+
three that is vendored in the tree we trained from.
|
| 165 |
+
|
| 166 |
+
Please cite: Qin, Zou, Li and Zhang, *Efficient End-to-End Diffusion Model for
|
| 167 |
+
One-Step SAR-to-Optical Translation*, IEEE Geoscience and Remote Sensing
|
| 168 |
+
Letters, [doi:10.1109/LGRS.2024.3506566](https://doi.org/10.1109/LGRS.2024.3506566).
|
| 169 |
+
The full record of what we checked, per method, is in
|
| 170 |
+
`licenses/NO-UPSTREAM-LICENSE.md` at the root of this repository.
|
| 171 |
+
|
| 172 |
+
---
|
| 173 |
+
|
| 174 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 175 |
+
checkpoint that we retrained ourselves on SAR2Opt; it is not ReFlowSET
|
| 176 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 177 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 178 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 179 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/sar2opt/hidiff/README.md
ADDED
|
@@ -0,0 +1,164 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# HI-Diff — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**Hierarchical Integration Diffusion Model for Realistic Image Deblurring**,
|
| 4 |
+
Chen, Zhang, Liu, Kong and Yuan, *NeurIPS 2023*
|
| 5 |
+
([arXiv:2305.12966](https://arxiv.org/abs/2305.12966)). Upstream code:
|
| 6 |
+
[`zhengchen1999/HI-Diff`](https://github.com/zhengchen1999/HI-Diff) at commit
|
| 7 |
+
`b3bfd167997e27f8edd57681cf70e5031a0e35f2`.
|
| 8 |
+
|
| 9 |
+
This is the two-stage latent-prior restoration transformer we retrained
|
| 10 |
+
ourselves on SAR2Opt (SAR as the degraded input, EO as the target), and it is
|
| 11 |
+
the checkpoint behind the HI-Diff row of the ReFlowSET comparison table.
|
| 12 |
+
|
| 13 |
+
## What is in this folder
|
| 14 |
+
|
| 15 |
+
| file | bytes | what it is |
|
| 16 |
+
|---|---|---|
|
| 17 |
+
| `S1_net_g_latest.pth` | 101,401,457 | stage-1 restoration transformer — for reproducing stage-2 *training*, not for inference |
|
| 18 |
+
| `S1_net_le_latest.pth` | 2,357,509 | stage-1 latent encoder, `in_chans` **6** (it sees LQ ‖ GT) — **not usable at test time** |
|
| 19 |
+
| `S2_net_d_latest.pth` | 10,530,491 | **the 8-step latent denoiser** — needed at inference |
|
| 20 |
+
| `S2_net_g_latest.pth` | 101,401,457 | **the final restoration transformer** — needed at inference |
|
| 21 |
+
| `S2_net_le_dm_latest.pth` | 2,247,061 | **the stage-2 latent encoder**, `in_chans` 3 (LQ only) — needed at inference |
|
| 22 |
+
|
| 23 |
+
**Only the three `S2_*` files are needed to run this model.** The repository's
|
| 24 |
+
own test configuration loads exactly `pretrain_network_g`,
|
| 25 |
+
`pretrain_network_le_dm` and `pretrain_network_d`, and nothing else. The two
|
| 26 |
+
`S1_*` files are shipped so that stage-2 *training* can be reproduced: stage 1
|
| 27 |
+
learns a latent prior from the LQ **and** the ground truth together, which is why
|
| 28 |
+
its encoder takes 6 input channels and cannot be used at deployment.
|
| 29 |
+
|
| 30 |
+
Architecture: the transformer is Restormer-shaped — `dim` 48, blocks
|
| 31 |
+
[3, 5, 6, 6], 4 refinement blocks, heads [1, 2, 4, 8], FFN expansion 2.66,
|
| 32 |
+
`embed_dim` 64, `group` 4. The latent denoiser runs **8 timesteps** on the
|
| 33 |
+
compressed prior (`linear_start` 0.1, `linear_end` 0.99) — the diffusion here
|
| 34 |
+
acts on a small latent prior, not on the image.
|
| 35 |
+
|
| 36 |
+
## Training budget we used
|
| 37 |
+
|
| 38 |
+
| | |
|
| 39 |
+
|---|---|
|
| 40 |
+
| **generator updates released** | **25,000 (stage 1) + 25,000 (stage 2) = 50,000** |
|
| 41 |
+
| batch size / patch | 8 @ 256 px |
|
| 42 |
+
| optimizer | AdamW, lr 2e-4, weight decay 1e-4, β (0.9, 0.999), gradient clipping on |
|
| 43 |
+
| schedule | `CosineAnnealingRestartCyclicLR`, periods [8,000 / 17,000], η_min [2e-4, 1e-6] |
|
| 44 |
+
| loss | L1 |
|
| 45 |
+
| progressive training | **disabled** |
|
| 46 |
+
| test resolution | trained on random 256 px crops and **tested at 512 px** — the network is fully convolutional, so the test resolution is not the training resolution for this cell |
|
| 47 |
+
|
| 48 |
+
The upstream recipe is 300,000 + 300,000 iterations with a progressive
|
| 49 |
+
patch/batch schedule from 128 to 384 px. We disabled the progressive schedule
|
| 50 |
+
(a single fixed 256 px group) and matched the budget to the rest of this
|
| 51 |
+
benchmark's diffusion band at 50,000 total updates. That is a substantial
|
| 52 |
+
reduction from the authors' recipe and is stated here so nobody reads this row
|
| 53 |
+
as HI-Diff at full budget.
|
| 54 |
+
|
| 55 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 56 |
+
|
| 57 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 58 |
+
|---|---|---|---|---|
|
| 59 |
+
| 319.8 | 0.473 | 0.692 | 0.384 | 17.36 |
|
| 60 |
+
|
| 61 |
+
Evaluated on the official split's 627 test tiles, centre-cropped to 512 px.
|
| 62 |
+
The tiles are 600 px natively; this benchmark crops and never resizes, in every
|
| 63 |
+
method's training and in the evaluation. No subsampling: every metric on this
|
| 64 |
+
page is measured over all 627 pairs.
|
| 65 |
+
|
| 66 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 67 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 68 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 69 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 70 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 71 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 72 |
+
convention gives a systematically different number and the two must never be
|
| 73 |
+
mixed, or compared against a paper that used the other one.
|
| 74 |
+
|
| 75 |
+
## Load it and translate one SAR image
|
| 76 |
+
|
| 77 |
+
The three networks must be assembled by hand; the repository's own `test.py`
|
| 78 |
+
with a `HI_Diff_S2` option file is the path of least resistance.
|
| 79 |
+
|
| 80 |
+
```yaml
|
| 81 |
+
# options/test/<your>.yml -- the three paths below are the three S2_ files
|
| 82 |
+
model_type: HI_Diff_S2
|
| 83 |
+
network_g: {type: Transformer, inp_channels: 3, out_channels: 3, dim: 48,
|
| 84 |
+
num_blocks: [3,5,6,6], num_refinement_blocks: 4, heads: [1,2,4,8],
|
| 85 |
+
ffn_expansion_factor: 2.66, bias: False, LayerNorm_type: WithBias,
|
| 86 |
+
dual_pixel_task: False, embed_dim: 64, group: 4}
|
| 87 |
+
network_le_dm: {type: latent_encoder_gelu, in_chans: 3, embed_dim: 64, block_num: 6,
|
| 88 |
+
group: 4, stage: 2, patch_expansion: 0.5, channel_expansion: 4}
|
| 89 |
+
network_d: {type: denoising, in_channel: 256, out_channel: 256, inner_channel: 512,
|
| 90 |
+
block_num: 4, group: 4, patch_expansion: 0.5, channel_expansion: 2}
|
| 91 |
+
diffusion_schedule: {apply_ldm: False, schedule: linear, timesteps: 8,
|
| 92 |
+
linear_start: 0.1, linear_end: 0.99}
|
| 93 |
+
path:
|
| 94 |
+
pretrain_network_g: S2_net_g_latest.pth
|
| 95 |
+
pretrain_network_le_dm: S2_net_le_dm_latest.pth
|
| 96 |
+
pretrain_network_d: S2_net_d_latest.pth
|
| 97 |
+
param_key_g: params
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
```bash
|
| 101 |
+
python test.py -opt options/test/<your>.yml
|
| 102 |
+
```
|
| 103 |
+
|
| 104 |
+
The SAR input is read with `cv2.IMREAD_COLOR`, so a single-channel SAR image is
|
| 105 |
+
replicated to three channels automatically — that is what training saw. At
|
| 106 |
+
inference `le_dm` produces the prior from the degraded input alone, `net_d`
|
| 107 |
+
denoises that prior over 8 steps, and `net_g` is conditioned on it.
|
| 108 |
+
|
| 109 |
+
## Read before using this checkpoint
|
| 110 |
+
|
| 111 |
+
* **The `S1_*` files are not an inference path.** `S1_net_le_latest.pth` has
|
| 112 |
+
`in_chans: 6` because stage 1 encodes LQ concatenated with the ground truth.
|
| 113 |
+
Wiring it into a test run would make the model an oracle. Use the `S2_*` trio.
|
| 114 |
+
* **These are the `latest` checkpoints — iteration 25,000 — not a best-validation
|
| 115 |
+
selection.** That is deliberate and benchmark-wide: every other method here
|
| 116 |
+
publishes its last checkpoint, so selecting on validation for this one would
|
| 117 |
+
be an asymmetry in its favour. Do not repoint at a best-val file and then
|
| 118 |
+
compare against this table.
|
| 119 |
+
* **Retraining is destructive.** The upstream launcher path is not idempotent:
|
| 120 |
+
the training configs set `resume_state: ~`, no auto-resume flag is passed, and
|
| 121 |
+
basicsr's `mkdir_and_rename` archives a finished experiment directory and
|
| 122 |
+
starts from scratch. Re-running a training script over a finished experiment
|
| 123 |
+
destroys it.
|
| 124 |
+
* **`torch.load` needs `weights_only=False`** under PyTorch ≥ 2.6; we made that
|
| 125 |
+
change in two places in the repository (`hi_diff/utils/base_model.py` and
|
| 126 |
+
`train.py`). Both are compatibility fixes.
|
| 127 |
+
* **How to read this row's numbers.** HI-Diff takes the best PSNR and SSIM of
|
| 128 |
+
any method in this benchmark on both datasets, while sitting at or near the
|
| 129 |
+
worst FID, DISTS and LPIPS. That combination is the signature of
|
| 130 |
+
regression-to-the-mean: a blurred, low-variance prediction minimises squared
|
| 131 |
+
error and is perceptually poor. Reporting its PSNR without its FID would badly
|
| 132 |
+
misrepresent it — and so would the reverse.
|
| 133 |
+
* Passes the identity-collapse audit on both datasets.
|
| 134 |
+
|
| 135 |
+
## Licence — stated factually
|
| 136 |
+
|
| 137 |
+
Upstream `zhengchen1999/HI-Diff` ships a LICENSE file, reproduced verbatim here
|
| 138 |
+
as `licenses/LICENSE-hidiff.txt`: the **Apache License 2.0**, appendix copyright
|
| 139 |
+
line `Copyright 2023 HI-Diff Authors`.
|
| 140 |
+
|
| 141 |
+
**Redistribution of these weights is permitted.** Apache-2.0 §4 obligations
|
| 142 |
+
apply: ship the licence, keep the notices, and **state that you modified the
|
| 143 |
+
files**. We did modify them — the two `torch.load(..., weights_only=False)`
|
| 144 |
+
changes noted above, plus our own option files — and this sentence is that
|
| 145 |
+
statement. Upstream ships no NOTICE file, so there is none to carry. Apache-2.0
|
| 146 |
+
§6 withholds trademark rights.
|
| 147 |
+
|
| 148 |
+
HI-Diff itself builds on BasicSR, Restormer and DiffIR, which its README
|
| 149 |
+
credits; those projects' own licences cover their code.
|
| 150 |
+
|
| 151 |
+
Attribution: *trained with `zhengchen1999/HI-Diff`, Copyright 2023 HI-Diff
|
| 152 |
+
Authors, licensed under the Apache License 2.0; our training and test
|
| 153 |
+
configurations modify the upstream configuration, and two source files were
|
| 154 |
+
modified for PyTorch ≥ 2.6 compatibility. HI-Diff builds on BasicSR, Restormer
|
| 155 |
+
and DiffIR. Full text in `licenses/LICENSE-hidiff.txt`.*
|
| 156 |
+
|
| 157 |
+
---
|
| 158 |
+
|
| 159 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 160 |
+
checkpoint that we retrained ourselves on SAR2Opt; it is not ReFlowSET
|
| 161 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 162 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 163 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 164 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/sar2opt/p2phd/README.md
ADDED
|
@@ -0,0 +1,145 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# pix2pixHD — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**High-Resolution Image Synthesis and Semantic Manipulation with Conditional
|
| 4 |
+
GANs**, Wang, Liu, Zhu, Tao, Kautz and Catanzaro, *CVPR 2018*
|
| 5 |
+
([arXiv:1711.11585](https://arxiv.org/abs/1711.11585)). Upstream code:
|
| 6 |
+
[`NVIDIA/pix2pixHD`](https://github.com/NVIDIA/pix2pixHD) at commit
|
| 7 |
+
`14b3b3c7fff413086e3b58df52096f16b6891172`.
|
| 8 |
+
|
| 9 |
+
This is the coarse-to-fine paired GAN we retrained ourselves on SAR2Opt, and
|
| 10 |
+
it is the checkpoint behind the pix2pixHD row of the ReFlowSET comparison
|
| 11 |
+
table.
|
| 12 |
+
|
| 13 |
+
## What is in this folder
|
| 14 |
+
|
| 15 |
+
| file | bytes | what it is |
|
| 16 |
+
|---|---|---|
|
| 17 |
+
| `net_G.pth` | 729,795,505 | the global generator, and the only file inference needs |
|
| 18 |
+
|
| 19 |
+
`net_G.pth` is the **`global` generator** (`GlobalGenerator`):
|
| 20 |
+
`n_downsample_global` 4, `n_blocks_global` 9 residual blocks, `ngf = 64`,
|
| 21 |
+
instance normalisation, 3 in / 3 out. The run was configured with
|
| 22 |
+
`n_local_enhancers 1`, but at `netG=global` the local enhancer is never
|
| 23 |
+
constructed and is not in this file — this is the single-scale generator.
|
| 24 |
+
|
| 25 |
+
The run used `--label_nc 0 --no_instance`, i.e. the "label" input is a real
|
| 26 |
+
image (the SAR chip) rather than a segmentation map, and no instance-edge
|
| 27 |
+
channel is concatenated. The two discriminators (`num_D` 2) are not released.
|
| 28 |
+
|
| 29 |
+
## Training budget we used
|
| 30 |
+
|
| 31 |
+
| | |
|
| 32 |
+
|---|---|
|
| 33 |
+
| **generator updates released** | **36,200** |
|
| 34 |
+
| batch size | 8 |
|
| 35 |
+
| epochs (`niter` + `niter_decay`) | 100 + 100 |
|
| 36 |
+
| iterations per epoch | 181 (1,450 training pairs; this loader floor-rounds, unlike the junyanz loaders which round up) |
|
| 37 |
+
| optimizer | Adam, lr 2e-4, β₁ 0.5, linear decay over the second half |
|
| 38 |
+
| adversarial loss | LSGAN, multi-scale discriminator (`num_D` 2, `n_layers_D` 3) |
|
| 39 |
+
| feature-matching loss | λ_feat 10 |
|
| 40 |
+
| perceptual loss | VGG loss enabled |
|
| 41 |
+
| augmentation | horizontal flip |
|
| 42 |
+
| input | random 512 crops of the 600 px tiles (`crop`, `loadSize` 600, `fineSize` 512) |
|
| 43 |
+
|
| 44 |
+
The floor-vs-ceiling detail is not pedantry: pix2pixHD and SPADE drop the last
|
| 45 |
+
partial batch while the junyanz-family loaders do not, so the same "epochs"
|
| 46 |
+
figure yields slightly different update counts across the table. The update
|
| 47 |
+
counts above are what actually ran.
|
| 48 |
+
|
| 49 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 50 |
+
|
| 51 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 52 |
+
|---|---|---|---|---|
|
| 53 |
+
| 146.3 | 0.283 | 0.567 | 0.268 | 15.95 |
|
| 54 |
+
|
| 55 |
+
Evaluated on the official split's 627 test tiles, centre-cropped to 512 px.
|
| 56 |
+
The tiles are 600 px natively; this benchmark crops and never resizes, in every
|
| 57 |
+
method's training and in the evaluation. No subsampling: every metric on this
|
| 58 |
+
page is measured over all 627 pairs.
|
| 59 |
+
|
| 60 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 61 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 62 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 63 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 64 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 65 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 66 |
+
convention gives a systematically different number and the two must never be
|
| 67 |
+
mixed, or compared against a paper that used the other one.
|
| 68 |
+
|
| 69 |
+
## Load it and translate one SAR image
|
| 70 |
+
|
| 71 |
+
`GlobalGenerator` can be constructed directly, without the repository's option
|
| 72 |
+
machinery:
|
| 73 |
+
|
| 74 |
+
```python
|
| 75 |
+
import numpy as np, torch, torch.nn as nn
|
| 76 |
+
from PIL import Image
|
| 77 |
+
from models.networks import GlobalGenerator # from the NVIDIA/pix2pixHD repo
|
| 78 |
+
|
| 79 |
+
G = GlobalGenerator(3, 3, 64, 4, 9, nn.InstanceNorm2d)
|
| 80 |
+
G.load_state_dict(torch.load('net_G.pth', map_location='cpu'))
|
| 81 |
+
G.eval()
|
| 82 |
+
|
| 83 |
+
sar = Image.open('sar.png').convert('RGB')
|
| 84 |
+
sar = sar.crop(((600-512)//2, (600-512)//2, (600+512)//2, (600+512)//2))
|
| 85 |
+
x = torch.from_numpy(np.asarray(sar, np.float32) / 127.5 - 1).permute(2, 0, 1)[None]
|
| 86 |
+
|
| 87 |
+
with torch.no_grad():
|
| 88 |
+
y = G(x)
|
| 89 |
+
|
| 90 |
+
eo = ((y[0].permute(1, 2, 0).numpy() + 1) * 127.5).clip(0, 255).astype(np.uint8)
|
| 91 |
+
Image.fromarray(eo).save('eo.png')
|
| 92 |
+
```
|
| 93 |
+
|
| 94 |
+
Through the repository's own `test.py` the equivalent invocation is
|
| 95 |
+
`--label_nc 0 --no_instance --which_epoch latest`, and the output lands as
|
| 96 |
+
`<stem>_synthesized_image.png`.
|
| 97 |
+
|
| 98 |
+
## Read before using this checkpoint
|
| 99 |
+
|
| 100 |
+
* **Upstream writes its test outputs as JPEG.** `util/visualizer.py` hard-codes
|
| 101 |
+
`'%s_%s.jpg'`. We changed that one line to `.png` before scoring, because
|
| 102 |
+
every other method in this benchmark writes PNG and JPEG re-compression would
|
| 103 |
+
have biased FID and LPIPS for this row alone. **If you clone upstream
|
| 104 |
+
unmodified and evaluate, your numbers will not match ours**, and the gap is an
|
| 105 |
+
artefact of the file format, not of the model.
|
| 106 |
+
* Three further changes we made to the repository are pure Python-3.12 /
|
| 107 |
+
modern-library compatibility and do not affect the weights:
|
| 108 |
+
`transforms.Scale` → `transforms.Resize`; `fractions.gcd` → `math.gcd` with
|
| 109 |
+
integer division; and a guarded `import scipy.misc` (needed only for
|
| 110 |
+
`--tf_log`, which we never pass).
|
| 111 |
+
* Normalisation is `[-1, 1]`.
|
| 112 |
+
* Passes the identity-collapse audit on both datasets.
|
| 113 |
+
|
| 114 |
+
## Licence — stated factually
|
| 115 |
+
|
| 116 |
+
Upstream `NVIDIA/pix2pixHD` ships `LICENSE.txt`, reproduced verbatim here as
|
| 117 |
+
`licenses/LICENSE-p2phd.txt`. It contains two notices:
|
| 118 |
+
|
| 119 |
+
* `Copyright (C) 2019 NVIDIA Corporation. Ting-Chun Wang, Ming-Yu Liu,
|
| 120 |
+
Jun-Yan Zhu. BSD License. All rights reserved.` — two conditions and a
|
| 121 |
+
disclaimer;
|
| 122 |
+
* the bundled `pytorch-CycleGAN-and-pix2pix` notice,
|
| 123 |
+
`Copyright (c) 2017, Jun-Yan Zhu and Taesung Park`, because pix2pixHD's code
|
| 124 |
+
borrows heavily from it.
|
| 125 |
+
|
| 126 |
+
**Redistribution of these weights is permitted** with both notices reproduced.
|
| 127 |
+
Despite NVIDIA's non-commercial licensing of some of its other releases, this
|
| 128 |
+
one carries **no non-commercial clause and no share-alike clause** — we checked
|
| 129 |
+
the file and the README, and the README has no licence section at all, so
|
| 130 |
+
`LICENSE.txt` is the entire grant.
|
| 131 |
+
|
| 132 |
+
Attribution: *trained with `NVIDIA/pix2pixHD`. Copyright (C) 2019 NVIDIA
|
| 133 |
+
Corporation — Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu. BSD License, all rights
|
| 134 |
+
reserved; includes the bundled pytorch-CycleGAN-and-pix2pix notice (Copyright
|
| 135 |
+
(c) 2017, Jun-Yan Zhu and Taesung Park). Full text in
|
| 136 |
+
`licenses/LICENSE-p2phd.txt`.*
|
| 137 |
+
|
| 138 |
+
---
|
| 139 |
+
|
| 140 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 141 |
+
checkpoint that we retrained ourselves on SAR2Opt; it is not ReFlowSET
|
| 142 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 143 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 144 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 145 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/sar2opt/pix2pix/README.md
ADDED
|
@@ -0,0 +1,144 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# pix2pix — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**Image-to-Image Translation with Conditional Adversarial Networks**, Isola, Zhu,
|
| 4 |
+
Zhou and Efros, *CVPR 2017* ([arXiv:1611.07004](https://arxiv.org/abs/1611.07004)).
|
| 5 |
+
Upstream code:
|
| 6 |
+
[`junyanz/pytorch-CycleGAN-and-pix2pix`](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix)
|
| 7 |
+
at commit `2a7afba2895d52556dd5dfe07e8555ef657ced6f`.
|
| 8 |
+
|
| 9 |
+
This is the paired conditional GAN we retrained ourselves on SAR2Opt, and it
|
| 10 |
+
is the checkpoint behind the pix2pix row of the ReFlowSET comparison table.
|
| 11 |
+
|
| 12 |
+
## What is in this folder
|
| 13 |
+
|
| 14 |
+
| file | bytes | what it is |
|
| 15 |
+
|---|---|---|
|
| 16 |
+
| `net_G.pth` | 217,728,293 | the generator, and the only file inference needs |
|
| 17 |
+
|
| 18 |
+
`net_G.pth` is a **`unet_256` generator**: a U-Net with batch normalisation,
|
| 19 |
+
`ngf = 64`, 3 input channels and 3 output channels, saved from a single-GPU run
|
| 20 |
+
so the state-dict keys are bare (no `module.` prefix). It maps **SAR → EO**:
|
| 21 |
+
the training set was built as side-by-side `A|B` tiles with SAR on the left, and
|
| 22 |
+
the run used `--direction AtoB`.
|
| 23 |
+
|
| 24 |
+
The discriminator is not part of this release — upstream's `test.py` constructs
|
| 25 |
+
only the generator, so `net_G.pth` is sufficient to reproduce every image we
|
| 26 |
+
scored.
|
| 27 |
+
|
| 28 |
+
## Training budget we used
|
| 29 |
+
|
| 30 |
+
| | |
|
| 31 |
+
|---|---|
|
| 32 |
+
| **generator updates released** | **36,400** |
|
| 33 |
+
| batch size | 8 |
|
| 34 |
+
| epochs (constant LR + linear decay) | 100 + 100 |
|
| 35 |
+
| iterations per epoch | 182 (1,450 training pairs, the loader does not drop the last batch) |
|
| 36 |
+
| optimizer | Adam, lr 2e-4, β₁ 0.5, linear decay to 0 over the second half |
|
| 37 |
+
| adversarial loss | vanilla GAN; discriminator `basic` (70×70 PatchGAN, `n_layers_D` 3) |
|
| 38 |
+
| reconstruction loss | L1, weight 100 |
|
| 39 |
+
| augmentation | horizontal flip |
|
| 40 |
+
| input | random 512 crops of the 600 px tiles (`resize_and_crop`, `load_size` 600, `crop_size` 512) |
|
| 41 |
+
|
| 42 |
+
We quote budgets in **optimizer updates**, never in epochs: the two datasets
|
| 43 |
+
differ by more than 10× in size, so "200 epochs" means something completely
|
| 44 |
+
different on each and would silently favour one of them.
|
| 45 |
+
|
| 46 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 47 |
+
|
| 48 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 49 |
+
|---|---|---|---|---|
|
| 50 |
+
| 261.9 | 0.347 | 0.657 | 0.199 | 13.39 |
|
| 51 |
+
|
| 52 |
+
Evaluated on the official split's 627 test tiles, centre-cropped to 512 px.
|
| 53 |
+
The tiles are 600 px natively; this benchmark crops and never resizes, in every
|
| 54 |
+
method's training and in the evaluation. No subsampling: every metric on this
|
| 55 |
+
page is measured over all 627 pairs.
|
| 56 |
+
|
| 57 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 58 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 59 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 60 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 61 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 62 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 63 |
+
convention gives a systematically different number and the two must never be
|
| 64 |
+
mixed, or compared against a paper that used the other one.
|
| 65 |
+
|
| 66 |
+
## Load it and translate one SAR image
|
| 67 |
+
|
| 68 |
+
The snippet uses the upstream repository's own `define_G`, so clone
|
| 69 |
+
`junyanz/pytorch-CycleGAN-and-pix2pix` at the commit above and run this from its
|
| 70 |
+
root. No patch is needed for this method — our only change to that repository was
|
| 71 |
+
an import fix in the dataset-preparation helper, which does not affect the model.
|
| 72 |
+
|
| 73 |
+
```python
|
| 74 |
+
import numpy as np, torch
|
| 75 |
+
from PIL import Image
|
| 76 |
+
from models.networks import define_G # from the junyanz repo
|
| 77 |
+
|
| 78 |
+
G = define_G(3, 3, 64, 'unet_256', 'batch', use_dropout=False,
|
| 79 |
+
init_type='normal', init_gain=0.02, gpu_ids=[])
|
| 80 |
+
sd = torch.load('net_G.pth', map_location='cpu')
|
| 81 |
+
if hasattr(sd, '_metadata'):
|
| 82 |
+
del sd._metadata # upstream strips this too
|
| 83 |
+
G.load_state_dict(sd)
|
| 84 |
+
G.eval()
|
| 85 |
+
|
| 86 |
+
# SAR input -> RGB -> [-1, 1]. SAR2Opt SAR is single-channel; .convert('RGB')
|
| 87 |
+
# replicates it to three channels, which is what training saw.
|
| 88 |
+
sar = Image.open('sar.png').convert('RGB')
|
| 89 |
+
sar = sar.crop(((600-512)//2, (600-512)//2, (600+512)//2, (600+512)//2))
|
| 90 |
+
x = torch.from_numpy(np.asarray(sar, np.float32) / 127.5 - 1).permute(2, 0, 1)[None]
|
| 91 |
+
|
| 92 |
+
with torch.no_grad():
|
| 93 |
+
y = G(x)
|
| 94 |
+
|
| 95 |
+
eo = ((y[0].permute(1, 2, 0).numpy() + 1) * 127.5).clip(0, 255).astype(np.uint8)
|
| 96 |
+
Image.fromarray(eo).save('eo.png')
|
| 97 |
+
```
|
| 98 |
+
|
| 99 |
+
The image this produces is `<stem>_fake_B.png` in upstream's own `test.py`
|
| 100 |
+
output tree, which is exactly what we scored.
|
| 101 |
+
|
| 102 |
+
## Read before using this checkpoint
|
| 103 |
+
|
| 104 |
+
* **`define_G` must be called with `gpu_ids=[]`.** Our checkpoints were saved
|
| 105 |
+
from single-GPU runs and have bare keys; building the network with a non-empty
|
| 106 |
+
`gpu_ids` wraps it in `DataParallel` and the load then fails on a `module.`
|
| 107 |
+
prefix mismatch.
|
| 108 |
+
* **The normalisation is `[-1, 1]`, not `[0, 1]`.** Feeding `[0, 1]` produces a
|
| 109 |
+
plausible-looking but systematically wrong image.
|
| 110 |
+
* This row is a **paired** method and was trained on aligned SAR/EO pairs. It
|
| 111 |
+
passes the identity-collapse audit on both datasets.
|
| 112 |
+
|
| 113 |
+
## Licence — stated factually
|
| 114 |
+
|
| 115 |
+
Upstream `junyanz/pytorch-CycleGAN-and-pix2pix` ships a LICENSE file, reproduced
|
| 116 |
+
verbatim in this repository as `licenses/LICENSE-pix2pix.txt`. It is **not a
|
| 117 |
+
single SPDX licence** but three stacked notices, and the whole file is what
|
| 118 |
+
travels with these weights:
|
| 119 |
+
|
| 120 |
+
* CycleGAN — `Copyright (c) 2017, Jun-Yan Zhu and Taesung Park`, two conditions
|
| 121 |
+
and a disclaimer (BSD-2-Clause in form);
|
| 122 |
+
* pix2pix — `Copyright (c) 2016, Phillip Isola and Jun-Yan Zhu`, two conditions;
|
| 123 |
+
* DCGAN — `Copyright (c) 2015, Facebook, Inc.`, three conditions including a
|
| 124 |
+
no-endorsement clause (BSD-3-Clause).
|
| 125 |
+
|
| 126 |
+
**Redistribution of these weights is permitted** with that notice reproduced.
|
| 127 |
+
There is no non-commercial and no share-alike clause. The DCGAN section's
|
| 128 |
+
no-endorsement clause means Facebook's name must not be used to promote this
|
| 129 |
+
release.
|
| 130 |
+
|
| 131 |
+
Attribution: *trained with `junyanz/pytorch-CycleGAN-and-pix2pix`. Copyright (c)
|
| 132 |
+
2017, Jun-Yan Zhu and Taesung Park (CycleGAN); Copyright (c) 2016, Phillip Isola
|
| 133 |
+
and Jun-Yan Zhu (pix2pix); Copyright (c) 2015, Facebook, Inc. (DCGAN). All rights
|
| 134 |
+
reserved. Redistributed under the BSD terms reproduced in
|
| 135 |
+
`licenses/LICENSE-pix2pix.txt`.*
|
| 136 |
+
|
| 137 |
+
---
|
| 138 |
+
|
| 139 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 140 |
+
checkpoint that we retrained ourselves on SAR2Opt; it is not ReFlowSET
|
| 141 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 142 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 143 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 144 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/sar2opt/resshift/README.md
ADDED
|
@@ -0,0 +1,146 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ResShift — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**ResShift: Efficient Diffusion Model for Image Super-Resolution by Residual
|
| 4 |
+
Shifting**, Yue, Wang and Loy, *NeurIPS 2023*
|
| 5 |
+
([arXiv:2307.12348](https://arxiv.org/abs/2307.12348)). Upstream code:
|
| 6 |
+
[`zsyOAOA/ResShift`](https://github.com/zsyOAOA/ResShift) at commit
|
| 7 |
+
`bb03b7d21614cace01787e097c8a6ab6b945227d`.
|
| 8 |
+
|
| 9 |
+
This is the residual-shifting latent diffusion model we retrained ourselves on
|
| 10 |
+
SAR2Opt, run as a **same-size** mapping (`sf = 1`: SAR in, EO out, no
|
| 11 |
+
upscaling), and it is the checkpoint behind the ResShift row of the ReFlowSET
|
| 12 |
+
comparison table.
|
| 13 |
+
|
| 14 |
+
## What is in this folder
|
| 15 |
+
|
| 16 |
+
| file | bytes | what it is |
|
| 17 |
+
|---|---|---|
|
| 18 |
+
| `ema_model.pth` | 478,392,663 | EMA weights of the denoising UNet at iteration 50,000 |
|
| 19 |
+
|
| 20 |
+
`ema_model.pth` is the EMA (rate 0.999) of a **`models.unet.UNetModelSwin`**:
|
| 21 |
+
latent resolution 64, `model_channels` 160, `channel_mult` [1, 2, 2, 4],
|
| 22 |
+
`num_res_blocks` [2, 2, 2, 2], attention at [64, 32, 16, 8], 32 channels per
|
| 23 |
+
head, Swin blocks of depth 2 with `swin_embed_dim` 192 and window size 8,
|
| 24 |
+
`cond_lq = True`, `lq_size = 256`.
|
| 25 |
+
|
| 26 |
+
**This is a latent-space model and this file is not sufficient to run it.** It
|
| 27 |
+
additionally needs the external CompVis **vq-f4** autoencoder
|
| 28 |
+
(`ldm.models.autoencoder.VQModelTorch`, 221,364,711 bytes), which is **not**
|
| 29 |
+
redistributed here.
|
| 30 |
+
|
| 31 |
+
## Training budget we used
|
| 32 |
+
|
| 33 |
+
| | |
|
| 34 |
+
|---|---|
|
| 35 |
+
| **generator updates released** | **50,000** |
|
| 36 |
+
| batch size | 16 (micro-batch 8, i.e. 2 gradient-accumulation steps per update) |
|
| 37 |
+
| optimizer | Adam, lr 5e-5 → 2e-5 cosine, 2,000-step warmup, no weight decay |
|
| 38 |
+
| EMA | rate 0.999 |
|
| 39 |
+
| diffusion | **4 steps**, exponential noise schedule (power 0.3), `etas_end` 0.99, `min_noise_level` 0.2, κ = 2.0, `predict_type` xstart, `sf` 1 |
|
| 40 |
+
| loss weights | latent MSE 1.0, pixel LPIPS (AlexNet) 4.0, pixel MSE 0.0 |
|
| 41 |
+
| precision | AMP on |
|
| 42 |
+
| training crops | random 256 px, normalised with mean 0.5 / std 0.5 |
|
| 43 |
+
| seed | 123456 |
|
| 44 |
+
|
| 45 |
+
ResShift is a **4-step** sampler by construction; that is the method, not a
|
| 46 |
+
shortcut we took.
|
| 47 |
+
|
| 48 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 49 |
+
|
| 50 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 51 |
+
|---|---|---|---|---|
|
| 52 |
+
| 141.7 | 0.304 | 0.597 | 0.177 | 14.31 |
|
| 53 |
+
|
| 54 |
+
Evaluated on the official split's 627 test tiles, centre-cropped to 512 px.
|
| 55 |
+
The tiles are 600 px natively; this benchmark crops and never resizes, in every
|
| 56 |
+
method's training and in the evaluation. No subsampling: every metric on this
|
| 57 |
+
page is measured over all 627 pairs.
|
| 58 |
+
|
| 59 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 60 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 61 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 62 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 63 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 64 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 65 |
+
convention gives a systematically different number and the two must never be
|
| 66 |
+
mixed, or compared against a paper that used the other one.
|
| 67 |
+
|
| 68 |
+
## Load it and translate one SAR image
|
| 69 |
+
|
| 70 |
+
Upstream's `inference_resshift.py` is a super-resolution CLI keyed to the
|
| 71 |
+
authors' own released tasks and does not accept a SAR→EO configuration. We wrote
|
| 72 |
+
a small driver against the repository's `ResShiftSampler`; the equivalent is:
|
| 73 |
+
|
| 74 |
+
```python
|
| 75 |
+
from omegaconf import OmegaConf
|
| 76 |
+
from sampler import ResShiftSampler # from the zsyOAOA/ResShift repo
|
| 77 |
+
|
| 78 |
+
cfg = OmegaConf.load('<your>_sar2eo_256.yaml') # sf 1, steps 4, predict_type xstart
|
| 79 |
+
cfg.model.ckpt_path = 'ema_model.pth'
|
| 80 |
+
cfg.autoencoder.ckpt_path = '<path to>/autoencoder_vq_f4.pth' # NOT shipped here
|
| 81 |
+
|
| 82 |
+
sampler = ResShiftSampler(cfg, sf=1, chop_size=256, chop_stride=256,
|
| 83 |
+
chop_bs=1, use_amp=True, seed=12345, padding_offset=None)
|
| 84 |
+
sampler.inference('<dir of SAR images>', '<output dir>', bs=8, noise_repeat=False)
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
At 512 px each input is processed as four clean 256 px tiles (`chop_size = chop_stride = lq_size = 256`, so the tiles do not overlap). Outputs keep the input file stems.
|
| 88 |
+
|
| 89 |
+
## Read before using this checkpoint
|
| 90 |
+
|
| 91 |
+
* **The external vq-f4 autoencoder is mandatory and is not in this repository.**
|
| 92 |
+
Without it the model cannot be constructed, let alone run. It is CompVis
|
| 93 |
+
latent-diffusion material under its own (MIT) licence.
|
| 94 |
+
* **Upstream's own inference CLI will not drive these weights.** It is written
|
| 95 |
+
around the authors' released super-resolution tasks. Use the sampler directly,
|
| 96 |
+
as above.
|
| 97 |
+
* **xFormers must be disabled.** On the accelerator we trained on, the installed
|
| 98 |
+
xFormers build has no usable kernel for these shapes (attention head dimension
|
| 99 |
+
512 exceeds its limit, and fp32 is unsupported), so we set
|
| 100 |
+
`XFORMERS_IS_AVAILBLE = False` in three files
|
| 101 |
+
(`ldm/modules/attention.py`, `ldm/modules/diffusionmodules/model.py`,
|
| 102 |
+
`models/unet.py`). If your hardware differs, this is a performance switch, not
|
| 103 |
+
a correctness one.
|
| 104 |
+
* **One numerical guard in the image logger.** fp16 diffused latents decoded
|
| 105 |
+
through the fp32 VQGAN produce NaN/Inf at early iterations and crashed training
|
| 106 |
+
*inside the logging path*. We added a `nan_to_num` + `clamp` before
|
| 107 |
+
`make_grid`. It is logging-only; the loss path is untouched. (A separate
|
| 108 |
+
`nan_to_num` on the LPIPS loss is upstream's, not ours.)
|
| 109 |
+
* **There is no auto-resume.** A crash mid-run needs a manual relaunch pointing
|
| 110 |
+
at the last saved checkpoint.
|
| 111 |
+
* Passes the identity-collapse audit on both datasets.
|
| 112 |
+
|
| 113 |
+
## Licence — stated factually ⚠ non-commercial
|
| 114 |
+
|
| 115 |
+
Upstream `zsyOAOA/ResShift` ships a LICENSE file, reproduced verbatim here as
|
| 116 |
+
`licenses/LICENSE-resshift.txt`: the **S-Lab License 1.0**,
|
| 117 |
+
`Copyright 2022 S-Lab`.
|
| 118 |
+
|
| 119 |
+
**This licence is non-commercial.** Its opening clause reads *"Redistribution
|
| 120 |
+
and use **for non-commercial purpose** in source and binary forms, with or
|
| 121 |
+
without modification, are permitted provided that the following conditions are
|
| 122 |
+
met"*, and it adds that commercial use requires contacting the contributors. The
|
| 123 |
+
README repeats it: *"This project is licensed under NTU S-Lab License 1.0.
|
| 124 |
+
Redistribution and use should follow this license."*
|
| 125 |
+
|
| 126 |
+
So: **redistribution of these weights is permitted for non-commercial purposes
|
| 127 |
+
only**, in binary form, with the copyright notice, the conditions and the
|
| 128 |
+
disclaimer reproduced — which is what `licenses/LICENSE-resshift.txt` is for.
|
| 129 |
+
Clause 3 is a **no-endorsement** clause: neither S-Lab's name nor its
|
| 130 |
+
contributors' names may be used to endorse or promote this release.
|
| 131 |
+
|
| 132 |
+
Attribution: *trained with `zsyOAOA/ResShift`, Copyright 2022 S-Lab, **S-Lab
|
| 133 |
+
License 1.0 — redistribution and use for non-commercial purposes only**; for
|
| 134 |
+
commercial use, contact the contributors. Requires the CompVis vq-f4
|
| 135 |
+
autoencoder (MIT), not redistributed here. Neither S-Lab's name nor its
|
| 136 |
+
contributors' names may be used to endorse this release. Full text in
|
| 137 |
+
`licenses/LICENSE-resshift.txt`.*
|
| 138 |
+
|
| 139 |
+
---
|
| 140 |
+
|
| 141 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 142 |
+
checkpoint that we retrained ourselves on SAR2Opt; it is not ReFlowSET
|
| 143 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 144 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 145 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 146 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/sar2opt/sd21ft/README.md
ADDED
|
@@ -0,0 +1,158 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# SD2.1 fine-tune only — SAR2Opt
|
| 2 |
+
|
| 3 |
+
The **"SD2.1 fine-tune only"** ablation row of the ReFlowSET comparison table:
|
| 4 |
+
Stable Diffusion 2.1-base fine-tuned for SAR-to-EO translation with a plain MSE
|
| 5 |
+
epsilon loss and **no confidence head**.
|
| 6 |
+
|
| 7 |
+
> **This is not a stock SD 2.1 UNet and it will not load into a stock SD 2.1
|
| 8 |
+
> pipeline.** It is C-DiffSET **stage 1**: the input convolution is widened to
|
| 9 |
+
> 8 channels so the SAR latent can be concatenated to the noisy EO latent. Only
|
| 10 |
+
> the output side is unmodified.
|
| 11 |
+
|
| 12 |
+
Venue: there is no separate paper for this row. The backbone is Stable
|
| 13 |
+
Diffusion 2.1-base (latent diffusion, Rombach et al., CVPR 2022); the trainer
|
| 14 |
+
is the stage-1 trainer of
|
| 15 |
+
[`KAIST-VICLab/C-DiffSET`](https://github.com/KAIST-VICLab/C-DiffSET)
|
| 16 |
+
(*IEEE TCSVT* 2026, [doi:10.1109/TCSVT.2026.3701447](https://doi.org/10.1109/TCSVT.2026.3701447)).
|
| 17 |
+
|
| 18 |
+
## What is in this folder
|
| 19 |
+
|
| 20 |
+
A single `UNet2DConditionModel` (865,922,244 parameters, fp32, 686 tensors):
|
| 21 |
+
|
| 22 |
+
* **`conv_in` 4 -> 8 channels**, exactly as in the C-DiffSET folder — the input
|
| 23 |
+
is `torch.cat([sar_latent, noisy_eo_latent], dim=1)`, **SAR first**. In this
|
| 24 |
+
trained checkpoint the two halves have diverged (mean |w| 0.01545 on channels
|
| 25 |
+
0-3 vs 0.02596 on channels 4-7).
|
| 26 |
+
* **`conv_out` unchanged at 4 channels.** No variance head, no confidence map.
|
| 27 |
+
The stage-1 trainer explicitly overrides the channel surgery to a no-op, and
|
| 28 |
+
the whole 4-channel prediction goes to the scheduler.
|
| 29 |
+
|
| 30 |
+
That 5-vs-4 output channel count is the only architectural difference from the
|
| 31 |
+
`cdiffset` folder in this repository; the parameter counts differ by exactly
|
| 32 |
+
2,881 (one 320x3x3 filter plus one bias).
|
| 33 |
+
|
| 34 |
+
## Training budget we used
|
| 35 |
+
|
| 36 |
+
| | |
|
| 37 |
+
|---|---|
|
| 38 |
+
| optimizer updates released | **40,000** (fixed-step snapshot `checkpoint-40000`) |
|
| 39 |
+
| updates the run was configured for | 50,000 |
|
| 40 |
+
| initialised from | the SD 2.1-base UNet with the duplicated-and-halved 8-channel `conv_in` |
|
| 41 |
+
| batch size / resolution | 16 @ 512 px |
|
| 42 |
+
| epochs the 40,000 updates work out to | ~444 (1,450 training pairs, 90 updates/epoch) |
|
| 43 |
+
| optimizer | AdamW, lr 3e-5, weight decay 0.01, cosine schedule, 100-step warmup |
|
| 44 |
+
| precision | fp32 (no mixed precision) |
|
| 45 |
+
| seed | 2024 |
|
| 46 |
+
| loss | plain MSE on the epsilon prediction |
|
| 47 |
+
| prompt | the fixed string `"electro-optical image"` |
|
| 48 |
+
| augmentation | horizontal flip, vertical flip, 90-degree rotation |
|
| 49 |
+
| test-time sampler | **DDIM, 50 steps**, fp32, one image at a time |
|
| 50 |
+
|
| 51 |
+
Same two caveats as the C-DiffSET folder: the snapshot is taken at update 40,000
|
| 52 |
+
of a cosine schedule laid out for 50,000, so the learning rate had not finished
|
| 53 |
+
decaying; and it is a fixed-step snapshot, not the repository's
|
| 54 |
+
validation-selected `best/`. For this stage the fixed-step choice is also the
|
| 55 |
+
better one on the merits — validation LPIPS bottoms out near update 41,000 and
|
| 56 |
+
then degrades toward 50,000.
|
| 57 |
+
|
| 58 |
+
Data: 512 px. The source images are 600 px and are **never resized**: training takes a random 512 crop, evaluation takes the centre 512 crop (offset 44). The EO/SAR pairing follows the official SAR2Opt layout (`trainB`/`testB` are EO, `trainA`/`testA` are SAR).
|
| 59 |
+
|
| 60 |
+
## Base model
|
| 61 |
+
|
| 62 |
+
This checkpoint is a derivative of **Stable Diffusion 2.1-base**. Because the
|
| 63 |
+
original `stabilityai/stable-diffusion-2-1-base` repository is no longer
|
| 64 |
+
available on the Hub, our training and every snippet below load the community
|
| 65 |
+
mirror [`Manojb/stable-diffusion-2-1-base`](https://huggingface.co/Manojb/stable-diffusion-2-1-base).
|
| 66 |
+
That is what we actually trained against — verified in the training
|
| 67 |
+
configuration files, the launch scripts and the training logs, none of which
|
| 68 |
+
ever reference `stabilityai/...`. The frozen VAE, CLIP text encoder, tokenizer
|
| 69 |
+
and scheduler all come from that repo. A different SD 2.1-base mirror is
|
| 70 |
+
expected to work but has not been verified by us.
|
| 71 |
+
|
| 72 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 73 |
+
|
| 74 |
+
| FID↓ | KID↓ | DISTS↓ | LPIPS↓ | PSNR↑ | SSIM↑ |
|
| 75 |
+
|---|---|---|---|---|---|
|
| 76 |
+
| 71.8 | 0.0090 | 0.211 | 0.541 | 16.24 | 0.293 |
|
| 77 |
+
|
| 78 |
+
These are our own re-evaluation numbers, not numbers copied from any paper.
|
| 79 |
+
LPIPS is **LPIPS-VGG on inputs mapped to [-1, 1]** (the `normalize=False`
|
| 80 |
+
convention); the alternative [0, 1] convention gives a systematically different
|
| 81 |
+
number and the two must never be mixed. PSNR/SSIM are per-image torchmetrics
|
| 82 |
+
with `data_range=1`; FID is `pytorch-fid` against the size-matched ground truth.
|
| 83 |
+
|
| 84 |
+
> **DISTS on this cell was re-measured.** This cell's prediction dump was regenerated after the only extended-metric pass that scored it, and that pass resumes on the *number* of images rather than their content, so its stored DISTS went stale. A full re-measurement of the current dump at n=627 gives the value printed above; the harness was validated on a cell that is not stale, whose DISTS it reproduces to 6e-6. PSNR, SSIM, FID and LPIPS come from the headline pass, which rescores every cell on every run.
|
| 85 |
+
|
| 86 |
+
## Load it and translate one SAR PNG
|
| 87 |
+
|
| 88 |
+
```python
|
| 89 |
+
import numpy as np
|
| 90 |
+
import torch
|
| 91 |
+
from PIL import Image
|
| 92 |
+
from diffusers import UNet2DConditionModel, AutoencoderKL, DDIMScheduler
|
| 93 |
+
from transformers import CLIPTextModel, CLIPTokenizer
|
| 94 |
+
|
| 95 |
+
REPO = "JeonghyeokDo/ReFlowSET" # comparison cells live under baselines/
|
| 96 |
+
BASE = "Manojb/stable-diffusion-2-1-base" # frozen VAE / text encoder / scheduler
|
| 97 |
+
DEV, DT = "cuda", torch.float32
|
| 98 |
+
|
| 99 |
+
unet = UNet2DConditionModel.from_pretrained(
|
| 100 |
+
REPO, subfolder="baselines/sar2opt/sd21ft", torch_dtype=DT).to(DEV).eval()
|
| 101 |
+
vae = AutoencoderKL.from_pretrained(BASE, subfolder="vae", torch_dtype=DT).to(DEV).eval()
|
| 102 |
+
tok = CLIPTokenizer.from_pretrained(BASE, subfolder="tokenizer")
|
| 103 |
+
txt = CLIPTextModel.from_pretrained(BASE, subfolder="text_encoder", torch_dtype=DT).to(DEV).eval()
|
| 104 |
+
sch = DDIMScheduler.from_pretrained(BASE, subfolder="scheduler")
|
| 105 |
+
sch.set_timesteps(50, device=DEV)
|
| 106 |
+
|
| 107 |
+
# The fixed training prompt, tokenized exactly as in training (no padding).
|
| 108 |
+
ids = tok("electro-optical image", padding="do_not_pad",
|
| 109 |
+
max_length=tok.model_max_length, truncation=True,
|
| 110 |
+
return_tensors="pt").input_ids.to(DEV)
|
| 111 |
+
with torch.no_grad():
|
| 112 |
+
embed = txt(ids)[0]
|
| 113 |
+
|
| 114 |
+
# SAR input: 512x512 PNG -> RGB -> [-1, 1]. Do not resize; crop if needed.
|
| 115 |
+
sar = Image.open("sar.png").convert("RGB")
|
| 116 |
+
x = torch.from_numpy(np.array(sar)).permute(2, 0, 1)[None]
|
| 117 |
+
x = (x.to(DEV, DT) / 255.0) * 2 - 1
|
| 118 |
+
|
| 119 |
+
with torch.no_grad():
|
| 120 |
+
sar_lat = vae.encode(x).latent_dist.mean * vae.config.scaling_factor # .mean, not .sample()
|
| 121 |
+
eo_lat = torch.randn_like(sar_lat)
|
| 122 |
+
for t in sch.timesteps:
|
| 123 |
+
out = unet(torch.cat([sar_lat, eo_lat], 1), # SAR first, then the noisy EO latent
|
| 124 |
+
t, encoder_hidden_states=embed).sample
|
| 125 |
+
eo_lat = sch.step(out, t, eo_lat).prev_sample # all 4 channels; no variance head
|
| 126 |
+
eo = vae.decode(eo_lat / vae.config.scaling_factor).sample
|
| 127 |
+
eo = (eo * 0.5 + 0.5).clamp(0, 1)[0].cpu()
|
| 128 |
+
|
| 129 |
+
Image.fromarray((eo.permute(1, 2, 0).numpy() * 255).round().astype(np.uint8)).save("eo.png")
|
| 130 |
+
```
|
| 131 |
+
|
| 132 |
+
The two things that silently produce a plausible-but-wrong image if you get
|
| 133 |
+
them backwards: the concatenation order is **SAR first**, and the VAE must be
|
| 134 |
+
read as `latent_dist.mean`, not `.sample()`.
|
| 135 |
+
|
| 136 |
+
## Licence status — stated factually
|
| 137 |
+
|
| 138 |
+
* **Training code:** MIT, `Copyright (c) 2026 KAIST VICLab` (the stage-1
|
| 139 |
+
trainer of [`KAIST-VICLab/C-DiffSET`](https://github.com/KAIST-VICLab/C-DiffSET)).
|
| 140 |
+
The row is named after Stable Diffusion, but no stabilityai code was used and
|
| 141 |
+
no stabilityai repository was loaded.
|
| 142 |
+
* **These weights:** a fine-tune of Stable Diffusion 2.1-base, hence a
|
| 143 |
+
*Derivative of the Model* under the **CreativeML Open RAIL++-M** licence
|
| 144 |
+
(`openrail++`). The **Attachment A use restrictions travel with these
|
| 145 |
+
weights** and must be passed on to anyone you redistribute to. Full text,
|
| 146 |
+
including Attachment A, ships in this repository as `LICENSE-WEIGHTS.md`.
|
| 147 |
+
|
| 148 |
+
Attribution: *fine-tuned from Stable Diffusion 2.1-base (obtained as
|
| 149 |
+
`Manojb/stable-diffusion-2-1-base`, a re-upload of
|
| 150 |
+
`stabilityai/stable-diffusion-2-1-base`) using the stage-1 trainer of
|
| 151 |
+
`KAIST-VICLab/C-DiffSET`.*
|
| 152 |
+
|
| 153 |
+
---
|
| 154 |
+
|
| 155 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 156 |
+
checkpoint retrained by us on SAR2Opt; it is not ReFlowSET itself. Every
|
| 157 |
+
comparison method in the release was retrained by us on the same splits at the
|
| 158 |
+
same resolution, so the rows are directly comparable to each other.
|
baselines/sar2opt/sd21ft/config.json
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_class_name": "UNet2DConditionModel",
|
| 3 |
+
"_diffusers_version": "0.10.0.dev0",
|
| 4 |
+
"act_fn": "silu",
|
| 5 |
+
"attention_head_dim": [
|
| 6 |
+
5,
|
| 7 |
+
10,
|
| 8 |
+
20,
|
| 9 |
+
20
|
| 10 |
+
],
|
| 11 |
+
"block_out_channels": [
|
| 12 |
+
320,
|
| 13 |
+
640,
|
| 14 |
+
1280,
|
| 15 |
+
1280
|
| 16 |
+
],
|
| 17 |
+
"center_input_sample": false,
|
| 18 |
+
"cross_attention_dim": 1024,
|
| 19 |
+
"down_block_types": [
|
| 20 |
+
"CrossAttnDownBlock2D",
|
| 21 |
+
"CrossAttnDownBlock2D",
|
| 22 |
+
"CrossAttnDownBlock2D",
|
| 23 |
+
"DownBlock2D"
|
| 24 |
+
],
|
| 25 |
+
"downsample_padding": 1,
|
| 26 |
+
"dual_cross_attention": false,
|
| 27 |
+
"flip_sin_to_cos": true,
|
| 28 |
+
"freq_shift": 0,
|
| 29 |
+
"in_channels": 8,
|
| 30 |
+
"layers_per_block": 2,
|
| 31 |
+
"mid_block_scale_factor": 1,
|
| 32 |
+
"norm_eps": 1e-05,
|
| 33 |
+
"norm_num_groups": 32,
|
| 34 |
+
"num_class_embeds": null,
|
| 35 |
+
"only_cross_attention": false,
|
| 36 |
+
"out_channels": 4,
|
| 37 |
+
"sample_size": 64,
|
| 38 |
+
"up_block_types": [
|
| 39 |
+
"UpBlock2D",
|
| 40 |
+
"CrossAttnUpBlock2D",
|
| 41 |
+
"CrossAttnUpBlock2D",
|
| 42 |
+
"CrossAttnUpBlock2D"
|
| 43 |
+
],
|
| 44 |
+
"use_linear_projection": true
|
| 45 |
+
}
|
baselines/sar2opt/spade/README.md
ADDED
|
@@ -0,0 +1,179 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# SPADE — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**Semantic Image Synthesis with Spatially-Adaptive Normalization**, Park, Liu,
|
| 4 |
+
Wang and Zhu, *CVPR 2019* ([arXiv:1903.07291](https://arxiv.org/abs/1903.07291)).
|
| 5 |
+
Upstream code: [`NVlabs/SPADE`](https://github.com/NVlabs/SPADE) at commit
|
| 6 |
+
`fecacc920c1367a038995c45a39c15f6521ca64f`.
|
| 7 |
+
|
| 8 |
+
This is the spatially-adaptive-normalisation generator we retrained ourselves on
|
| 9 |
+
SAR2Opt, and it is the checkpoint behind the SPADE row of the ReFlowSET
|
| 10 |
+
comparison table.
|
| 11 |
+
|
| 12 |
+
**This checkpoint will not load into a clean clone of `NVlabs/SPADE`.** Read the
|
| 13 |
+
next two sections before trying.
|
| 14 |
+
|
| 15 |
+
## What is in this folder
|
| 16 |
+
|
| 17 |
+
| file | bytes | what it is |
|
| 18 |
+
|---|---|---|
|
| 19 |
+
| `net_G.pth` | 368,775,707 | the SPADE generator, and the only file inference needs |
|
| 20 |
+
|
| 21 |
+
`net_G.pth` is a **`SPADEGenerator`**: `ngf = 64`,
|
| 22 |
+
`norm_G = spectralspadesyncbatch3x3`, `num_upsampling_layers = normal`,
|
| 23 |
+
`use_vae = False` (no style encoder — the generator is deterministic given the
|
| 24 |
+
input), and **`semantic_nc = 3`**.
|
| 25 |
+
|
| 26 |
+
That `semantic_nc = 3` is the whole story of this cell. SPADE is a *semantic
|
| 27 |
+
image synthesis* model: upstream one-hot-encodes its "label" input into
|
| 28 |
+
`label_nc` channels. A SAR chip is a real image, not a segmentation map, so we
|
| 29 |
+
run with `--label_nc 0 --no_instance`, which upstream treats as an error
|
| 30 |
+
condition. Stock SPADE therefore never constructs a generator with a 3-channel
|
| 31 |
+
spatially-adaptive input and cannot load this file. The two discriminators
|
| 32 |
+
(`num_D` 2) are not released.
|
| 33 |
+
|
| 34 |
+
## Training budget we used
|
| 35 |
+
|
| 36 |
+
| | |
|
| 37 |
+
|---|---|
|
| 38 |
+
| **generator updates released** | **36,200** |
|
| 39 |
+
| batch size | 8 |
|
| 40 |
+
| epochs (`niter` + `niter_decay`) | 100 + 100 |
|
| 41 |
+
| iterations per epoch | 181 (1,450 training pairs; this loader drops the last partial batch) |
|
| 42 |
+
| optimizer | Adam, lr 2e-4 with TTUR (`no_TTUR` off, so G runs at lr/2 and D at 2·lr), β₁ 0.0, β₂ 0.9 |
|
| 43 |
+
| adversarial loss | hinge GAN, multi-scale discriminator (`num_D` 2, `n_layers_D` 4) |
|
| 44 |
+
| feature-matching loss | λ_feat 10 |
|
| 45 |
+
| perceptual loss | VGG loss, λ_vgg 10 |
|
| 46 |
+
| initialisation | Xavier |
|
| 47 |
+
| augmentation | horizontal flip |
|
| 48 |
+
| input | random 512 crops of the 600 px tiles (`crop`, `load_size` 600, `crop_size` 512) |
|
| 49 |
+
|
| 50 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 51 |
+
|
| 52 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 53 |
+
|---|---|---|---|---|
|
| 54 |
+
| 142.5 | 0.265 | 0.597 | 0.234 | 14.47 |
|
| 55 |
+
|
| 56 |
+
Evaluated on the official split's 627 test tiles, centre-cropped to 512 px.
|
| 57 |
+
The tiles are 600 px natively; this benchmark crops and never resizes, in every
|
| 58 |
+
method's training and in the evaluation. No subsampling: every metric on this
|
| 59 |
+
page is measured over all 627 pairs.
|
| 60 |
+
|
| 61 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 62 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 63 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 64 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 65 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 66 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 67 |
+
convention gives a systematically different number and the two must never be
|
| 68 |
+
mixed, or compared against a paper that used the other one.
|
| 69 |
+
|
| 70 |
+
## Load it and translate one SAR image
|
| 71 |
+
|
| 72 |
+
**You need the image-to-image adaptation described below**, applied to the
|
| 73 |
+
upstream repository, before this will run. With that in place, the generator can
|
| 74 |
+
be constructed directly:
|
| 75 |
+
|
| 76 |
+
```python
|
| 77 |
+
import numpy as np, torch
|
| 78 |
+
from argparse import Namespace
|
| 79 |
+
from PIL import Image
|
| 80 |
+
from models.networks.generator import SPADEGenerator # from the NVlabs/SPADE repo
|
| 81 |
+
|
| 82 |
+
opt = Namespace(semantic_nc=3, label_nc=0, no_instance=True, ngf=64,
|
| 83 |
+
num_upsampling_layers='normal',
|
| 84 |
+
norm_G='spectralspadesyncbatch3x3',
|
| 85 |
+
crop_size=512, aspect_ratio=1.0, use_vae=False)
|
| 86 |
+
G = SPADEGenerator(opt)
|
| 87 |
+
G.load_state_dict(torch.load('net_G.pth', map_location='cpu'))
|
| 88 |
+
G.eval()
|
| 89 |
+
|
| 90 |
+
sar = Image.open('sar.png').convert('RGB')
|
| 91 |
+
sar = sar.crop(((600-512)//2, (600-512)//2, (600+512)//2, (600+512)//2))
|
| 92 |
+
x = torch.from_numpy(np.asarray(sar, np.float32) / 127.5 - 1).permute(2, 0, 1)[None]
|
| 93 |
+
|
| 94 |
+
with torch.no_grad():
|
| 95 |
+
y = G(seg=x, z=None) # the SAR image goes in the 'seg' slot
|
| 96 |
+
|
| 97 |
+
eo = ((y[0].permute(1, 2, 0).numpy() + 1) * 127.5).clip(0, 255).astype(np.uint8)
|
| 98 |
+
Image.fromarray(eo).save('eo.png')
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
Through the repository's own `test.py`, the invocation is
|
| 102 |
+
`--dataset_mode custom --label_nc 0 --no_instance --no_pairing_check` with the
|
| 103 |
+
SAR directory as `--label_dir`. Note that the paired loader also demands an
|
| 104 |
+
`--image_dir` even at test time, so a SAR-only run needs either a dummy EO
|
| 105 |
+
directory or the direct generator call above.
|
| 106 |
+
|
| 107 |
+
## Read before using this checkpoint
|
| 108 |
+
|
| 109 |
+
**The five changes that make this cell exist.** These are a functional
|
| 110 |
+
adaptation, not compatibility fixes, and a reproducer must apply all of them:
|
| 111 |
+
|
| 112 |
+
1. `data/pix2pix_dataset.py` — when `label_nc == 0`, load the "label" through
|
| 113 |
+
the *image* transform (`.convert('RGB')`, normalised to `[-1, 1]`) instead of
|
| 114 |
+
nearest-neighbour resampling, ×255 and the `==255 → label_nc` remap. The SAR
|
| 115 |
+
PNG is a real image.
|
| 116 |
+
2. `models/pix2pix_model.py` — an early return in `preprocess_input()`: when
|
| 117 |
+
`label_nc == 0`, skip the `.long()` cast and the `scatter_` one-hot encoding
|
| 118 |
+
and pass the float image straight through as `input_semantics`. One-hot
|
| 119 |
+
encoding a normalised float image is undefined.
|
| 120 |
+
3. `options/base_options.py` — when `label_nc == 0`, force
|
| 121 |
+
`opt.semantic_nc = 3 + (0 if no_instance else 1)`. Upstream computes
|
| 122 |
+
`semantic_nc = label_nc + …`, i.e. 0 or 1, and would build a generator with a
|
| 123 |
+
one-channel spatially-adaptive input.
|
| 124 |
+
4. `models/networks/discriminator.py` — the discriminator input width becomes
|
| 125 |
+
`(label_nc if label_nc > 0 else 3) + output_nc`, for the same reason on the
|
| 126 |
+
discriminator side.
|
| 127 |
+
5. `util/visualizer.py` — `tensor2label` is given `n_label=0` when
|
| 128 |
+
`label_nc == 0` so it falls through to `tensor2im` instead of colour-mapping
|
| 129 |
+
a real image as a label map; and `import scipy.misc` is guarded, since it was
|
| 130 |
+
removed in SciPy ≥ 1.12.
|
| 131 |
+
|
| 132 |
+
Changes 1–4 change what the network *is*. **Without them this file does not
|
| 133 |
+
load.**
|
| 134 |
+
|
| 135 |
+
* Normalisation is `[-1, 1]`.
|
| 136 |
+
* `norm_G` uses synchronised batch normalisation; on a single device this
|
| 137 |
+
behaves as ordinary batch norm, and `.eval()` is required.
|
| 138 |
+
* Passes the identity-collapse audit on both datasets.
|
| 139 |
+
|
| 140 |
+
## Licence — stated factually ⚠ non-commercial and share-alike
|
| 141 |
+
|
| 142 |
+
Upstream `NVlabs/SPADE` ships `LICENSE.md`, reproduced verbatim here as
|
| 143 |
+
`licenses/LICENSE-spade.txt`. It is the full legal code of
|
| 144 |
+
**Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International
|
| 145 |
+
(CC BY-NC-SA 4.0)**. The copyright holder is named in the README rather than in
|
| 146 |
+
the licence file: *"Copyright (C) 2019 NVIDIA Corporation. All rights reserved.
|
| 147 |
+
… The code is released for academic research use only. For commercial use or
|
| 148 |
+
business inquiries, please contact researchinquiries@nvidia.com."*
|
| 149 |
+
|
| 150 |
+
**This is the most restrictive licence in this repository, and two of its terms
|
| 151 |
+
travel with this checkpoint:**
|
| 152 |
+
|
| 153 |
+
* **NonCommercial.** You may not use this checkpoint for commercial advantage or
|
| 154 |
+
monetary compensation.
|
| 155 |
+
* **ShareAlike.** If a trained checkpoint is Adapted Material — and we treat it
|
| 156 |
+
as such — then any licence you apply when you pass it on must be a Creative
|
| 157 |
+
Commons licence with the same elements, this version or later, or a BY-NC-SA
|
| 158 |
+
compatible licence. **This cell is therefore distributed under CC BY-NC-SA
|
| 159 |
+
4.0**; it cannot sit under a permissive repository-wide weight licence, and
|
| 160 |
+
neither can anything you derive from it.
|
| 161 |
+
* The licence also states that patent and trademark rights are not licensed.
|
| 162 |
+
* NVIDIA's academic-use-only statement in the README is an additional assertion
|
| 163 |
+
beyond the CC text.
|
| 164 |
+
|
| 165 |
+
Attribution: *trained with `NVlabs/SPADE`. Copyright (C) 2019 NVIDIA
|
| 166 |
+
Corporation, all rights reserved. Licensed under CC BY-NC-SA 4.0; NVIDIA
|
| 167 |
+
releases the code for academic research use only (commercial enquiries:
|
| 168 |
+
researchinquiries@nvidia.com). This checkpoint is therefore distributed under
|
| 169 |
+
CC BY-NC-SA 4.0 — non-commercial, share-alike. Full text in
|
| 170 |
+
`licenses/LICENSE-spade.txt`.*
|
| 171 |
+
|
| 172 |
+
---
|
| 173 |
+
|
| 174 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 175 |
+
checkpoint that we retrained ourselves on SAR2Opt; it is not ReFlowSET
|
| 176 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 177 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 178 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 179 |
+
**not** directly comparable with the numbers in the methods' own papers.
|
baselines/sar2opt/stegogan/README.md
ADDED
|
@@ -0,0 +1,205 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# StegoGAN — SAR2Opt
|
| 2 |
+
|
| 3 |
+
**StegoGAN: Leveraging Steganography for Non-Bijective Image-to-Image
|
| 4 |
+
Translation**, Wu, Chen, Mermet, Hurni, Schindler, Gonthier and Landrieu,
|
| 5 |
+
*CVPR 2024*. Upstream code:
|
| 6 |
+
[`sian-wusidi/StegoGAN`](https://github.com/sian-wusidi/StegoGAN) (the official
|
| 7 |
+
CVPR 2024 repository) at commit `cad61997c0f82793444f60f81298142b80cdf3c1`.
|
| 8 |
+
|
| 9 |
+
This is the mismatch-aware unpaired GAN we retrained ourselves on SAR2Opt,
|
| 10 |
+
and it is the checkpoint behind the StegoGAN row of the ReFlowSET comparison
|
| 11 |
+
table. **Two things on this page are load-bearing: there is an oracle output you
|
| 12 |
+
must not use, and this cell fails our input-copy audit.**
|
| 13 |
+
|
| 14 |
+
We applied **no patches** to this repository. It is stock upstream at that
|
| 15 |
+
commit.
|
| 16 |
+
|
| 17 |
+
## What is in this folder
|
| 18 |
+
|
| 19 |
+
| file | bytes | what it is |
|
| 20 |
+
|---|---|---|
|
| 21 |
+
| `net_G_A.pth` | 50,254,749 | **SAR → EO** (`resnet_9blocks_maskv1`) — the generator that produces the reported row |
|
| 22 |
+
| `net_G_B.pth` | 52,615,525 | EO → SAR (`resnet_9blocks_maskv3`), which also returns the mismatch mask; used to close the cycle in training, and **not needed for a prediction** |
|
| 23 |
+
|
| 24 |
+
Both generators are ResNet-style with instance normalisation, `ngf = 64`, built
|
| 25 |
+
with `--resnet_layer 8 --fusionblock`. **Those two are architecture flags, not
|
| 26 |
+
training options**: build the network without them and the state dict will not
|
| 27 |
+
load.
|
| 28 |
+
|
| 29 |
+
`net_G_A` takes an **optional second argument**. That option is the whole reason
|
| 30 |
+
this card has a warning section — see below.
|
| 31 |
+
|
| 32 |
+
## Training budget we used
|
| 33 |
+
|
| 34 |
+
| | |
|
| 35 |
+
|---|---|
|
| 36 |
+
| **data iterations released** | **87,000** |
|
| 37 |
+
| generator optimizer steps | 174,000 — see the note below |
|
| 38 |
+
| batch size | 2 |
|
| 39 |
+
| epochs (constant LR + linear decay) | 60 + 60 |
|
| 40 |
+
| iterations per epoch | 725 (1,450 training images, last batch not dropped) |
|
| 41 |
+
| optimizer | Adam, lr 2e-4, β₁ 0.5, linear decay to 0 over the second half |
|
| 42 |
+
| adversarial loss | LSGAN; discriminator `basic` (70×70 PatchGAN) |
|
| 43 |
+
| cycle losses | λ_A 10, λ_B 10 |
|
| 44 |
+
| identity loss | λ_identity 0.5 |
|
| 45 |
+
| consistency / regularisation | λ_consistency 1.0, λ_reg 0.3 |
|
| 46 |
+
| augmentation | horizontal flip |
|
| 47 |
+
| input | random 512 crops of the 600 px tiles |
|
| 48 |
+
|
| 49 |
+
**Why two numbers.** StegoGAN's training step calls `optimizer_G.step()`
|
| 50 |
+
**twice** off a single backward pass, so its optimizer-step count is exactly
|
| 51 |
+
double its data-iteration count. We quote StegoGAN budgets in *data iterations*
|
| 52 |
+
throughout this benchmark and note the 2× explicitly. Do not re-derive one cell
|
| 53 |
+
the other way — the family stops being comparable with itself.
|
| 54 |
+
|
| 55 |
+
## Measured on the SAR2Opt test set (n = 627, 512 px)
|
| 56 |
+
|
| 57 |
+
| FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |
|
| 58 |
+
|---|---|---|---|---|
|
| 59 |
+
| 150.1 | 0.347 | 0.655 | 0.158 | 12.47 |
|
| 60 |
+
|
| 61 |
+
Evaluated on the official split's 627 test tiles, centre-cropped to 512 px.
|
| 62 |
+
The tiles are 600 px natively; this benchmark crops and never resizes, in every
|
| 63 |
+
method's training and in the evaluation. No subsampling: every metric on this
|
| 64 |
+
page is measured over all 627 pairs.
|
| 65 |
+
|
| 66 |
+
These are our own re-evaluation numbers, measured by us on the images this
|
| 67 |
+
checkpoint produced. **No number here is copied from any paper.** PSNR and SSIM
|
| 68 |
+
are per-image torchmetrics with `data_range=1`; LPIPS is **LPIPS-VGG on inputs
|
| 69 |
+
mapped to [-1, 1]** (the `normalize=False` convention); DISTS is the standard
|
| 70 |
+
implementation on the same pairs; FID is `pytorch-fid` against the size-matched
|
| 71 |
+
ground-truth test set. The LPIPS convention matters: the alternative [0, 1]
|
| 72 |
+
convention gives a systematically different number and the two must never be
|
| 73 |
+
mixed, or compared against a paper that used the other one.
|
| 74 |
+
|
| 75 |
+
## Load it and translate one SAR image
|
| 76 |
+
|
| 77 |
+
Clone `sian-wusidi/StegoGAN` at the commit above and run this from its root.
|
| 78 |
+
Check `define_G`'s signature in that repository first: StegoGAN adds
|
| 79 |
+
`resnet_layer` and `fusionblock` keyword arguments that the junyanz original does
|
| 80 |
+
not have.
|
| 81 |
+
|
| 82 |
+
```python
|
| 83 |
+
import numpy as np, torch
|
| 84 |
+
from PIL import Image
|
| 85 |
+
from models.networks import define_G # from the StegoGAN repo
|
| 86 |
+
|
| 87 |
+
G_A = define_G(3, 3, 64, 'resnet_9blocks_maskv1', 'instance', use_dropout=False,
|
| 88 |
+
init_type='normal', init_gain=0.02, gpu_ids=[],
|
| 89 |
+
resnet_layer=8, fusionblock=True)
|
| 90 |
+
G_A.load_state_dict(torch.load('net_G_A.pth', map_location='cpu'))
|
| 91 |
+
G_A.eval()
|
| 92 |
+
|
| 93 |
+
sar = Image.open('sar.png').convert('RGB')
|
| 94 |
+
sar = sar.crop(((600-512)//2, (600-512)//2, (600+512)//2, (600+512)//2))
|
| 95 |
+
x = torch.from_numpy(np.asarray(sar, np.float32) / 127.5 - 1).permute(2, 0, 1)[None]
|
| 96 |
+
|
| 97 |
+
with torch.no_grad():
|
| 98 |
+
fake_B_clean = G_A(x) # ONE argument. Passing a second makes it an oracle.
|
| 99 |
+
|
| 100 |
+
eo = ((fake_B_clean[0].permute(1, 2, 0).numpy() + 1) * 127.5).clip(0, 255).astype(np.uint8)
|
| 101 |
+
Image.fromarray(eo).save('eo.png')
|
| 102 |
+
```
|
| 103 |
+
|
| 104 |
+
Through the repository's own `test.py` (`--model stego_gan --no_dropout
|
| 105 |
+
--resnet_layer 8 --fusionblock`), outputs are written one directory per visual;
|
| 106 |
+
**`fake_B_clean/` is the row on this page.**
|
| 107 |
+
|
| 108 |
+
## ⚠ `fake_B` is an oracle. `fake_B_clean` is the prediction.
|
| 109 |
+
|
| 110 |
+
The model computes two outputs:
|
| 111 |
+
|
| 112 |
+
```
|
| 113 |
+
fake_B_clean = netG_A(real_A) # SAR only
|
| 114 |
+
fake_B = netG_A(real_A, latent_real_B.detach()) # + a feature map of the GROUND TRUTH
|
| 115 |
+
```
|
| 116 |
+
|
| 117 |
+
`latent_real_B` comes from `netG_B(real_B)` — the *target* EO image. `fake_B` is
|
| 118 |
+
therefore conditioned on the answer and **cannot be produced at deployment**.
|
| 119 |
+
Only `fake_B_clean` is a prediction, and every number on this page is
|
| 120 |
+
`fake_B_clean`.
|
| 121 |
+
|
| 122 |
+
This is not hypothetical. Earlier in our own benchmarking the oracle was scored as
|
| 123 |
+
"StegoGAN" for two days on a different dataset and topped every column at
|
| 124 |
+
23.19 dB, against a true deployable value of 12.74 dB. **Any published StegoGAN
|
| 125 |
+
number must state which of the two it is.**
|
| 126 |
+
|
| 127 |
+
## ⚠ This cell fails the input-copy audit
|
| 128 |
+
|
| 129 |
+
We audit every cell for **identity collapse**: a model that has learned to pass
|
| 130 |
+
its input through rather than translate it. The test compares the mean absolute
|
| 131 |
+
distance from the generated image to the SAR **input** against the distance to
|
| 132 |
+
the EO **target**; a ratio below 1.0 means the output is closer to the input it
|
| 133 |
+
was given than to the answer it was asked for.
|
| 134 |
+
|
| 135 |
+
| quantity | value |
|
| 136 |
+
|---|---|
|
| 137 |
+
| mean \|generated − ground truth\| | 48.694 |
|
| 138 |
+
| mean \|generated − SAR input\| | 36.915 |
|
| 139 |
+
| **ratio** (SAR distance ÷ GT distance) | **0.758** — below 1.0 |
|
| 140 |
+
| verdict | **COLLAPSED** |
|
| 141 |
+
| cross-pair separation | 0.890 (cohort median 1.088) — no oracle behaviour |
|
| 142 |
+
|
| 143 |
+
**Read the metrics on this page with that in mind.** They are reported unchanged
|
| 144 |
+
because they are what the released implementation does at its own published
|
| 145 |
+
protocol, and a benchmark that quietly substitutes a better-behaved variant is
|
| 146 |
+
not reporting the method. But they should not be read as translation quality.
|
| 147 |
+
The separation figure is the companion check for the *opposite* failure: it is
|
| 148 |
+
the mean distance from a generated image to **other** images' ground truth
|
| 149 |
+
divided by the distance to its **own**, so a value well above the cohort median
|
| 150 |
+
would indicate that ground-truth information had leaked into the generation.
|
| 151 |
+
This cell sits at or below the median, so there is no leak — the problem here is
|
| 152 |
+
input-copying, and only input-copying.
|
| 153 |
+
|
| 154 |
+
A longer control run at roughly 1.85× this budget was also trained as a diagnostic; it collapses harder (ratio 0.697), and it is neither the table's row nor published here.
|
| 155 |
+
|
| 156 |
+
## Read before using this checkpoint
|
| 157 |
+
|
| 158 |
+
* **`--resnet_layer 8` and `--fusionblock` must match training.** They change the
|
| 159 |
+
architecture; omit them and the load fails.
|
| 160 |
+
* **`test.py` rejects `--display_id`.** That flag is declared only in the train
|
| 161 |
+
options. Passing it to `test.py` aborts with an unrecognised argument — and in
|
| 162 |
+
a shell `&&` chain the cell then reports success with an empty output
|
| 163 |
+
directory. This silently destroyed several of our own test passes once.
|
| 164 |
+
* Build with `gpu_ids=[]`; the checkpoints have bare keys.
|
| 165 |
+
* Normalisation is `[-1, 1]`.
|
| 166 |
+
* StegoGAN is **unpaired**.
|
| 167 |
+
|
| 168 |
+
## Licence — stated factually ⚠ no upstream licence exists
|
| 169 |
+
|
| 170 |
+
**The upstream code base publishes no licence.**
|
| 171 |
+
[`sian-wusidi/StegoGAN`](https://github.com/sian-wusidi/StegoGAN) — the official
|
| 172 |
+
CVPR 2024 repository — has no LICENSE, LICENCE, COPYING or NOTICE file at any
|
| 173 |
+
depth; **0 of its 52 tracked files** at the pinned commit are licence-shaped; its
|
| 174 |
+
README has no licence section; and the GitHub API reports no declared licence,
|
| 175 |
+
with the `/license` endpoint returning 404. Checked 2026-08-28.
|
| 176 |
+
|
| 177 |
+
Under default copyright that means **all rights are reserved by the authors and
|
| 178 |
+
no express permission to redistribute derived work has been granted** to us or
|
| 179 |
+
to you. We publish this checkpoint anyway, so that the benchmark is reproducible
|
| 180 |
+
end to end, and we state the position plainly rather than implying a permission
|
| 181 |
+
that does not exist. If you intend to redistribute this checkpoint or build on
|
| 182 |
+
it, assess that for yourself, and consider asking the authors directly.
|
| 183 |
+
|
| 184 |
+
**What is known about the lineage.** StegoGAN's README thanks
|
| 185 |
+
`junyanz/pytorch-CycleGAN-and-pix2pix`, and the parts of the tree that are
|
| 186 |
+
unmodified CycleGAN carry that project's BSD notice, reproduced here as
|
| 187 |
+
`licenses/LICENSE-cyclegan.txt`. **That notice does not extend to StegoGAN's own
|
| 188 |
+
contributions** — the mismatch-mask generators, the fusion block and the
|
| 189 |
+
consistency and regularisation losses, which are exactly what makes
|
| 190 |
+
`net_G_A.pth` what it is.
|
| 191 |
+
|
| 192 |
+
Please cite: Wu, Chen, Mermet, Hurni, Schindler, Gonthier and Landrieu,
|
| 193 |
+
*StegoGAN: Leveraging Steganography for Non-Bijective Image-to-Image
|
| 194 |
+
Translation*, CVPR 2024.
|
| 195 |
+
The full record of what we checked, per method, is in
|
| 196 |
+
`licenses/NO-UPSTREAM-LICENSE.md` at the root of this repository.
|
| 197 |
+
|
| 198 |
+
---
|
| 199 |
+
|
| 200 |
+
Part of the **ReFlowSET** release. This folder holds one comparison-method
|
| 201 |
+
checkpoint that we retrained ourselves on SAR2Opt; it is not ReFlowSET
|
| 202 |
+
itself. Every comparison method in this repository was retrained by us on the
|
| 203 |
+
same splits at the same resolution and scored through one evaluation pipeline,
|
| 204 |
+
so the rows are directly comparable to each other — and, for the same reason,
|
| 205 |
+
**not** directly comparable with the numbers in the methods' own papers.
|