| ---
|
| license: cc-by-nc-sa-4.0
|
| library_name: diffusers
|
| pipeline_tag: text-to-image
|
| base_model: lcybuaa/Text2Earth
|
| tags:
|
| - controlnet
|
| - remote-sensing
|
| - xbd
|
| - building-damage
|
| ---
|
|
|
| # Text2Earth ControlNet for xBD
|
|
|
| ControlNet fine-tuned on [xBD](https://xview2.org/download) post-event imagery. It takes a building-damage mask and samples a matching satellite image from [Text2Earth](https://huggingface.co/lcybuaa/Text2Earth).
|
|
|
| The ControlNet is the only trained module. Text2Earth (UNet, VAE, text encoder) stays frozen.
|
|
|
| ## Conditioning
|
|
|
| | Item | Value |
|
| | --- | --- |
|
| | Base model | [lcybuaa/Text2Earth](https://huggingface.co/lcybuaa/Text2Earth) |
|
| | Conditioning | 3-channel binary damage mask, RGB order, values in `{0, 1}` |
|
| | Prompt | `18_GOOGLE_LEVEL_a high resolution satellite image` |
|
| | Scheduler | Euler discrete |
|
| | Steps / guidance | 50 / 1.0 |
|
|
|
| Binary channels follow the xBD hues: background black, no-damage green, minor blue, major yellow, destroy red, un-classified white.
|
|
|
| ## Usage
|
|
|
| ```python
|
| import torch
|
| from diffusers import (
|
| ControlNetModel,
|
| EulerDiscreteScheduler,
|
| StableDiffusionControlNetPipeline,
|
| )
|
| from PIL import Image
|
|
|
| repo = "BiliSakura/Text2Earth-ControlNet-xBD"
|
| base = "lcybuaa/Text2Earth"
|
| dtype = torch.float16
|
|
|
| controlnet = ControlNetModel.from_pretrained(
|
| repo, subfolder="controlnet", torch_dtype=dtype
|
| )
|
| pipe = StableDiffusionControlNetPipeline.from_pretrained(
|
| base,
|
| controlnet=controlnet,
|
| scheduler=EulerDiscreteScheduler.from_pretrained(base, subfolder="scheduler"),
|
| torch_dtype=dtype,
|
| safety_checker=None,
|
| ).to("cuda")
|
|
|
| image = pipe(
|
| "18_GOOGLE_LEVEL_a high resolution satellite image",
|
| image=Image.open("mask.png"),
|
| num_inference_steps=50,
|
| guidance_scale=1.0,
|
| controlnet_conditioning_scale=1.0,
|
| ).images[0]
|
| image.save("post_event.png")
|
| ```
|
|
|
| Weights are under `controlnet/`.
|
|
|
| ## License
|
|
|
| [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/), the license of the xBD dataset these weights were trained on.
|
|
|
