---
license: cc-by-nc-sa-4.0
library_name: diffusers
pipeline_tag: text-to-image
base_model: lcybuaa/Text2Earth
tags:
  - controlnet
  - remote-sensing
  - xbd
  - building-damage
---

# Text2Earth ControlNet for xBD

ControlNet fine-tuned on [xBD](https://xview2.org/download) post-event imagery. It takes a building-damage mask and samples a matching satellite image from [Text2Earth](https://huggingface.co/lcybuaa/Text2Earth).

The ControlNet is the only trained module. Text2Earth (UNet, VAE, text encoder) stays frozen.

## Conditioning

| Item | Value |
| --- | --- |
| Base model | [lcybuaa/Text2Earth](https://huggingface.co/lcybuaa/Text2Earth) |
| Conditioning | 3-channel binary damage mask, RGB order, values in `{0, 1}` |
| Prompt | `18_GOOGLE_LEVEL_a high resolution satellite image` |
| Scheduler | Euler discrete |
| Steps / guidance | 50 / 1.0 |

Binary channels follow the xBD hues: background black, no-damage green, minor blue, major yellow, destroy red, un-classified white.

## Usage

```python
import torch
from diffusers import (
    ControlNetModel,
    EulerDiscreteScheduler,
    StableDiffusionControlNetPipeline,
)
from PIL import Image

repo = "BiliSakura/Text2Earth-ControlNet-xBD"
base = "lcybuaa/Text2Earth"
dtype = torch.float16

controlnet = ControlNetModel.from_pretrained(
    repo, subfolder="controlnet", torch_dtype=dtype
)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    base,
    controlnet=controlnet,
    scheduler=EulerDiscreteScheduler.from_pretrained(base, subfolder="scheduler"),
    torch_dtype=dtype,
    safety_checker=None,
).to("cuda")

image = pipe(
    "18_GOOGLE_LEVEL_a high resolution satellite image",
    image=Image.open("mask.png"),
    num_inference_steps=50,
    guidance_scale=1.0,
    controlnet_conditioning_scale=1.0,
).images[0]
image.save("post_event.png")
```

Weights are under `controlnet/`.

## License

[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/), the license of the xBD dataset these weights were trained on.