Upload 4 files

a24b840 verified 13 days ago

4.66 kB

---
license: apache-2.0
base_model: stabilityai/stable-diffusion-xl-base-1.0
tags:
  - controlnet
  - lora
  - stable-diffusion-xl
  - text-to-image
  - image-to-image
  - remote-sensing
  - satellite-imagery
  - synthetic-data
  - geospatial
pipeline_tag: text-to-image
language:
  - en
---
# ConLoRA-Sat

**Parameter-Efficient Synthetic Satellite Imagery Generation for Data-Scarce Humanitarian Contexts**

ConLoRA-Sat combines **LoRA** (domain-style adaptation) with **ControlNet** (spatial structure) on a **Stable Diffusion XL 1.0** backbone: one text prompt plus a 5-class land-cover segmentation map generates a realistic 0.5 m satellite image. It targets data-scarce humanitarian use, e.g. UN peacekeeping training, producing fictitious yet geographically plausible terrain rather than identifiable real locations.

```
Sam-Roblex/ConLoRA-Sat/
├── lora/
│   └── satellite_lora.safetensors          ← one file: the LoRA weights
└── controlnet/
    ├── config.json                         ← SDXL ControlNet architecture (required by diffusers)
    └── diffusion_pytorch_model.safetensors ← the ControlNet weights
```

- Trained on **322 curated images** (LoRA) and **4,333 image/segmentation pairs** (ControlNet) from Mogadishu (Somalia) and Addis Ababa (Ethiopia). About **24 A6000-equivalent GPU-hours** total. Runs in 16-20 GB inference VRAM.

## Usage

Both components load from this repo via subfolders:

```python
from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel
import torch
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=ControlNetModel.from_pretrained(
        "Sam-Roblex/ConLoRA-Sat", subfolder="controlnet", torch_dtype=torch.float16),
    torch_dtype=torch.float16, variant="fp16", use_safetensors=True).to("cuda")
pipe.load_lora_weights("Sam-Roblex/ConLoRA-Sat",
                       weight_name="lora/satellite_lora.safetensors")
image = pipe(
    "satellite imagery, aerial view, Mixed-use urban settlement with moderate building density",
    image="seg_map.png",      # 5-class map, 1024×1024
    num_inference_steps=50,
    guidance_scale=7.5,
    controlnet_conditioning_scale=1.0,
).images[0]
```

Notes:
- LoRA trigger phrase is **"satellite imagery, aerial view"**, prepended to a descriptive caption.
- On water-dominated maps lower `controlnet_conditioning_scale` toward **0.8** for smoother coastlines.
- The two components are trained separately and composed at inference, so either can be reused independently.

## Land-cover class legend

| Class      | RGB      |
|------------|----------|
| Bareland   | `#D2B48C` |
| Vegetation | `#228B22` |
| Roads      | `#404040` |
| Buildings  | `#B22222` |
| Water      | `#1E90FF` |

## Model details

- **Backbone**: SDXL 1.0 Base, frozen throughout.
- **LoRA** (`lora/`): cross-attention query/value projections, rank 32, α 16. Prodigy optimizer, Min-SNR weighting (γ 5). ~4,000 steps, ~2 GPU-hours.
- **ControlNet** (`controlnet/`): trainable copy of SDXL's U-Net encoder path, injected via zero-initialised 1×1 convolutions. AdamW (lr 5e-6, 500-step warm-up), DDPM schedule. All 4,333 pairs, ~22 GPU-hours.

## Results

| Config | FID | LPIPS | CMMD | CLIPScore |
|--------|-----|-------|------|-----------|
| LoRA only | 110.98 | 0.640 | 0.034 | 0.249 |
| ControlNet only | 106.44 | 0.490 | 0.063 | 0.246 |
| **ConLoRA-Sat** | **100.88** | **0.474** | 0.061 | 0.248 |

Downstream (DeepLabV3+/ResNet-50): a segmenter trained only on synthetic imagery retains **90.1 %** of a real-data baseline; 200 real + 600 synthetic recovers **92.5 %**. In a 53-expert perceptual study, synthetic images were misclassified as real in **20.8 %** of trials.

## Train / evaluate / cite
Training scripts, evaluation code, and dataset documentation live in the companion GitHub repository. Dataset: [Zenodo: 10.5281/zenodo.22747581](https://doi.org/10.5281/zenodo.22747581) (CC BY-NC 4.0). Paper: *ConLoRA-Sat: Parameter-Efficient Synthetic Satellite Imagery Generation for Data-Scarce Humanitarian Contexts*, EarthArXiv preprint (under review).

```bibtex
@misc{qureshi2026conlorasat,
  title  = {ConLoRA-Sat: Parameter-Efficient Synthetic Satellite Imagery
            Generation for Data-Scarce Humanitarian Contexts},
  author = {Qureshi, Seyed Amir Muhammad and Granell, Carlos and Casteleyn, Sven},
  year   = {2026},
  note   = {EarthArXiv preprint; under review}
}
```

**License**: weights Apache-2.0; base model CreativeML OpenRAIL++-M. Training dataset CC BY-NC 4.0. Generates fictitious terrain, not identifiable real locations.

	---
	license: apache-2.0
	base_model: stabilityai/stable-diffusion-xl-base-1.0
	tags:
	- controlnet
	- lora
	- stable-diffusion-xl
	- text-to-image
	- image-to-image
	- remote-sensing
	- satellite-imagery
	- synthetic-data
	- geospatial
	pipeline_tag: text-to-image
	language:
	- en
	---

	# ConLoRA-Sat

	Parameter-Efficient Synthetic Satellite Imagery Generation for Data-Scarce Humanitarian Contexts

	ConLoRA-Sat combines LoRA (domain-style adaptation) with ControlNet (spatial structure) on a Stable Diffusion XL 1.0 backbone: one text prompt plus a 5-class land-cover segmentation map generates a realistic 0.5 m satellite image. It targets data-scarce humanitarian use, e.g. UN peacekeeping training, producing fictitious yet geographically plausible terrain rather than identifiable real locations.

	```
	Sam-Roblex/ConLoRA-Sat/
	├── lora/
	│ └── satellite_lora.safetensors ← one file: the LoRA weights
	└── controlnet/
	├── config.json ← SDXL ControlNet architecture (required by diffusers)
	└── diffusion_pytorch_model.safetensors ← the ControlNet weights
	```

	- Trained on 322 curated images (LoRA) and 4,333 image/segmentation pairs (ControlNet) from Mogadishu (Somalia) and Addis Ababa (Ethiopia). About 24 A6000-equivalent GPU-hours total. Runs in 16-20 GB inference VRAM.

	## Usage

	Both components load from this repo via subfolders:

	```python
	from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel
	import torch

	pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
	"stabilityai/stable-diffusion-xl-base-1.0",
	controlnet=ControlNetModel.from_pretrained(
	"Sam-Roblex/ConLoRA-Sat", subfolder="controlnet", torch_dtype=torch.float16),
	torch_dtype=torch.float16, variant="fp16", use_safetensors=True).to("cuda")

	pipe.load_lora_weights("Sam-Roblex/ConLoRA-Sat",
	weight_name="lora/satellite_lora.safetensors")

	image = pipe(
	"satellite imagery, aerial view, Mixed-use urban settlement with moderate building density",
	image="seg_map.png", # 5-class map, 1024×1024
	num_inference_steps=50,
	guidance_scale=7.5,
	controlnet_conditioning_scale=1.0,
	).images[0]
	```

	Notes:
	- LoRA trigger phrase is "satellite imagery, aerial view", prepended to a descriptive caption.
	- On water-dominated maps lower `controlnet_conditioning_scale` toward 0.8 for smoother coastlines.
	- The two components are trained separately and composed at inference, so either can be reused independently.

	## Land-cover class legend

	\| Class \| RGB \|
	\|------------\|----------\|
	\| Bareland \| `#D2B48C` \|
	\| Vegetation \| `#228B22` \|
	\| Roads \| `#404040` \|
	\| Buildings \| `#B22222` \|
	\| Water \| `#1E90FF` \|

	## Model details

	- Backbone: SDXL 1.0 Base, frozen throughout.
	- LoRA (`lora/`): cross-attention query/value projections, rank 32, α 16. Prodigy optimizer, Min-SNR weighting (γ 5). ~4,000 steps, ~2 GPU-hours.
	- ControlNet (`controlnet/`): trainable copy of SDXL's U-Net encoder path, injected via zero-initialised 1×1 convolutions. AdamW (lr 5e-6, 500-step warm-up), DDPM schedule. All 4,333 pairs, ~22 GPU-hours.

	## Results

	\| Config \| FID \| LPIPS \| CMMD \| CLIPScore \|
	\|--------\|-----\|-------\|------\|-----------\|
	\| LoRA only \| 110.98 \| 0.640 \| 0.034 \| 0.249 \|
	\| ControlNet only \| 106.44 \| 0.490 \| 0.063 \| 0.246 \|
	\| ConLoRA-Sat \| 100.88 \| 0.474 \| 0.061 \| 0.248 \|

	Downstream (DeepLabV3+/ResNet-50): a segmenter trained only on synthetic imagery retains 90.1 % of a real-data baseline; 200 real + 600 synthetic recovers 92.5 %. In a 53-expert perceptual study, synthetic images were misclassified as real in 20.8 % of trials.

	## Train / evaluate / cite
	Training scripts, evaluation code, and dataset documentation live in the companion GitHub repository. Dataset: [Zenodo: 10.5281/zenodo.22747581](https://doi.org/10.5281/zenodo.22747581) (CC BY-NC 4.0). Paper: ConLoRA-Sat: Parameter-Efficient Synthetic Satellite Imagery Generation for Data-Scarce Humanitarian Contexts, EarthArXiv preprint (under review).

	```bibtex
	@misc{qureshi2026conlorasat,
	title = {ConLoRA-Sat: Parameter-Efficient Synthetic Satellite Imagery
	Generation for Data-Scarce Humanitarian Contexts},
	author = {Qureshi, Seyed Amir Muhammad and Granell, Carlos and Casteleyn, Sven},
	year = {2026},
	note = {EarthArXiv preprint; under review}
	}
	```

	License: weights Apache-2.0; base model CreativeML OpenRAIL++-M. Training dataset CC BY-NC 4.0. Generates fictitious terrain, not identifiable real locations.