|
|
---
|
|
|
language: en
|
|
|
library_name: diffusers
|
|
|
pipeline_tag: text-to-image
|
|
|
tags:
|
|
|
- satellite
|
|
|
- controlnet
|
|
|
- diffusers
|
|
|
- text-to-image
|
|
|
---
|
|
|
|
|
|
# DiffusionSat Custom Pipelines
|
|
|
|
|
|
Custom community pipelines for loading DiffusionSat checkpoints directly with `diffusers.DiffusionPipeline.from_pretrained()`.
|
|
|
|
|
|
> See [Diffusers Community Pipeline Documentation](https://huggingface.co/docs/diffusers/using-diffusers/custom_pipeline_overview)
|
|
|
|
|
|
## Model Index
|
|
|
|
|
|
`model_index.json` is set to the default text-to-image pipeline (`DiffusionSatPipeline`) so `DiffusionPipeline.from_pretrained()` works out of the box. The ControlNet variant is loaded via `custom_pipeline` plus the `controlnet` subfolder, as shown below.
|
|
|
|
|
|
## Available Pipelines
|
|
|
|
|
|
This directory contains two custom pipelines:
|
|
|
|
|
|
1. **`pipeline_diffusionsat.py`**: Standard text-to-image pipeline with DiffusionSat metadata support.
|
|
|
2. **`pipeline_diffusionsat_controlnet.py`**: ControlNet pipeline with DiffusionSat metadata and conditional metadata support.
|
|
|
|
|
|
## Setup
|
|
|
|
|
|
The checkpoint folder (`ckpt/diffusionsat/`) should contain the standard diffusers components (unet, vae, scheduler, etc.). You can reference these pipeline files directly from this directory or copy them to your checkpoint folder.
|
|
|
|
|
|
## Usage
|
|
|
|
|
|
### 1. Text-to-Image Pipeline
|
|
|
|
|
|
Use `pipeline_diffusionsat.py` for standard generation.
|
|
|
|
|
|
```python
|
|
|
import torch
|
|
|
from diffusers import DiffusionPipeline
|
|
|
|
|
|
# Load pipeline
|
|
|
pipe = DiffusionPipeline.from_pretrained(
|
|
|
"path/to/ckpt/diffusionsat",
|
|
|
custom_pipeline="./custom_pipelines/pipeline_diffusionsat.py", # Path to this file
|
|
|
torch_dtype=torch.float16,
|
|
|
trust_remote_code=True,
|
|
|
)
|
|
|
pipe = pipe.to("cuda")
|
|
|
|
|
|
# Optional: Metadata (normalized lat, lon, timestamp, GSD, etc.)
|
|
|
# metadata = [0.5, -0.3, 0.7, 0.2, 0.1, 0.0, 0.5]
|
|
|
|
|
|
# Generate
|
|
|
image = pipe(
|
|
|
"satellite image of farmland",
|
|
|
metadata=None, # Optional
|
|
|
num_inference_steps=30,
|
|
|
).images[0]
|
|
|
```
|
|
|
|
|
|
### 2. ControlNet Pipeline
|
|
|
|
|
|
Use `pipeline_diffusionsat_controlnet.py` for ControlNet generation.
|
|
|
|
|
|
```python
|
|
|
import torch
|
|
|
from diffusers import DiffusionPipeline, ControlNetModel
|
|
|
from diffusers.utils import load_image
|
|
|
|
|
|
# 1. Load ControlNet
|
|
|
controlnet = ControlNetModel.from_pretrained(
|
|
|
"path/to/ckpt/diffusionsat/controlnet",
|
|
|
torch_dtype=torch.float16
|
|
|
)
|
|
|
|
|
|
# 2. Load Pipeline with ControlNet
|
|
|
pipe = DiffusionPipeline.from_pretrained(
|
|
|
"path/to/ckpt/diffusionsat",
|
|
|
controlnet=controlnet,
|
|
|
custom_pipeline="./custom_pipelines/pipeline_diffusionsat_controlnet.py", # Path to this file
|
|
|
torch_dtype=torch.float16,
|
|
|
trust_remote_code=True,
|
|
|
)
|
|
|
pipe = pipe.to("cuda")
|
|
|
|
|
|
# 3. Prepare Control Image
|
|
|
control_image = load_image("path/to/conditioning_image.png")
|
|
|
|
|
|
# 4. Generate
|
|
|
# metadata: Target image metadata (optional)
|
|
|
# cond_metadata: Conditioning image metadata (optional)
|
|
|
|
|
|
image = pipe(
|
|
|
"satellite image of farmland",
|
|
|
image=control_image,
|
|
|
metadata=None,
|
|
|
cond_metadata=None,
|
|
|
num_inference_steps=30,
|
|
|
).images[0]
|
|
|
```
|
|
|
|
