File size: 2,096 Bytes
a99d025 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | ---
license: cc-by-nc-sa-4.0
library_name: diffusers
pipeline_tag: text-to-image
base_model: lcybuaa/Text2Earth
tags:
- controlnet
- remote-sensing
- xbd
- building-damage
---
# Text2Earth ControlNet for xBD
ControlNet fine-tuned on [xBD](https://xview2.org/download) post-event imagery. It takes a building-damage mask and samples a matching satellite image from [Text2Earth](https://huggingface.co/lcybuaa/Text2Earth).
The ControlNet is the only trained module. Text2Earth (UNet, VAE, text encoder) stays frozen.
## Conditioning
| Item | Value |
| --- | --- |
| Base model | [lcybuaa/Text2Earth](https://huggingface.co/lcybuaa/Text2Earth) |
| Conditioning | 3-channel binary damage mask, RGB order, values in `{0, 1}` |
| Prompt | `18_GOOGLE_LEVEL_a high resolution satellite image` |
| Scheduler | Euler discrete |
| Steps / guidance | 50 / 1.0 |
Binary channels follow the xBD hues: background black, no-damage green, minor blue, major yellow, destroy red, un-classified white.
## Usage
```python
import torch
from diffusers import (
ControlNetModel,
EulerDiscreteScheduler,
StableDiffusionControlNetPipeline,
)
from PIL import Image
repo = "BiliSakura/Text2Earth-ControlNet-xBD"
base = "lcybuaa/Text2Earth"
dtype = torch.float16
controlnet = ControlNetModel.from_pretrained(
repo, subfolder="controlnet", torch_dtype=dtype
)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
base,
controlnet=controlnet,
scheduler=EulerDiscreteScheduler.from_pretrained(base, subfolder="scheduler"),
torch_dtype=dtype,
safety_checker=None,
).to("cuda")
image = pipe(
"18_GOOGLE_LEVEL_a high resolution satellite image",
image=Image.open("mask.png"),
num_inference_steps=50,
guidance_scale=1.0,
controlnet_conditioning_scale=1.0,
).images[0]
image.save("post_event.png")
```
Weights are under `controlnet/`.
## License
[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/), the license of the xBD dataset these weights were trained on.
|