Instructions to use cal54321/sd-controlnet-seg with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use cal54321/sd-controlnet-seg with Diffusers:
pip install -U diffusers transformers accelerate
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline controlnet = ControlNetModel.from_pretrained("cal54321/sd-controlnet-seg") pipe = StableDiffusionControlNetPipeline.from_pretrained( "runwayml/stable-diffusion-v1-5", controlnet=controlnet ) - Notebooks
- Google Colab
- Kaggle
Commit ·
0b56c7c
0
Parent(s):
Duplicate from lllyasviel/sd-controlnet-seg
Browse filesCo-authored-by: Will Berman <williamberman@users.noreply.huggingface.co>
- .gitattributes +35 -0
- README.md +304 -0
- config.json +41 -0
- controlnet_utils.py +40 -0
- diffusion_pytorch_model.bin +3 -0
- diffusion_pytorch_model.safetensors +3 -0
- images/house.png +0 -0
- images/house_seg.png +0 -0
- images/house_seg_out.png +0 -0
- sd.png +0 -0
.gitattributes
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
images/bird.png filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,304 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: openrail
|
| 3 |
+
base_model: runwayml/stable-diffusion-v1-5
|
| 4 |
+
tags:
|
| 5 |
+
- art
|
| 6 |
+
- controlnet
|
| 7 |
+
- stable-diffusion
|
| 8 |
+
- image-to-image
|
| 9 |
+
duplicated_from: lllyasviel/sd-controlnet-seg
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# Controlnet - *Image Segmentation Version*
|
| 13 |
+
|
| 14 |
+
ControlNet is a neural network structure to control diffusion models by adding extra conditions.
|
| 15 |
+
This checkpoint corresponds to the ControlNet conditioned on **Image Segmentation**.
|
| 16 |
+
|
| 17 |
+
It can be used in combination with [Stable Diffusion](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/text2img).
|
| 18 |
+
|
| 19 |
+

|
| 20 |
+
|
| 21 |
+
## Model Details
|
| 22 |
+
- **Developed by:** Lvmin Zhang, Maneesh Agrawala
|
| 23 |
+
- **Model type:** Diffusion-based text-to-image generation model
|
| 24 |
+
- **Language(s):** English
|
| 25 |
+
- **License:** [The CreativeML OpenRAIL M license](https://huggingface.co/spaces/CompVis/stable-diffusion-license) is an [Open RAIL M license](https://www.licenses.ai/blog/2022/8/18/naming-convention-of-responsible-ai-licenses), adapted from the work that [BigScience](https://bigscience.huggingface.co/) and [the RAIL Initiative](https://www.licenses.ai/) are jointly carrying in the area of responsible AI licensing. See also [the article about the BLOOM Open RAIL license](https://bigscience.huggingface.co/blog/the-bigscience-rail-license) on which our license is based.
|
| 26 |
+
- **Resources for more information:** [GitHub Repository](https://github.com/lllyasviel/ControlNet), [Paper](https://arxiv.org/abs/2302.05543).
|
| 27 |
+
- **Cite as:**
|
| 28 |
+
|
| 29 |
+
@misc{zhang2023adding,
|
| 30 |
+
title={Adding Conditional Control to Text-to-Image Diffusion Models},
|
| 31 |
+
author={Lvmin Zhang and Maneesh Agrawala},
|
| 32 |
+
year={2023},
|
| 33 |
+
eprint={2302.05543},
|
| 34 |
+
archivePrefix={arXiv},
|
| 35 |
+
primaryClass={cs.CV}
|
| 36 |
+
}
|
| 37 |
+
|
| 38 |
+
## Introduction
|
| 39 |
+
|
| 40 |
+
Controlnet was proposed in [*Adding Conditional Control to Text-to-Image Diffusion Models*](https://arxiv.org/abs/2302.05543) by
|
| 41 |
+
Lvmin Zhang, Maneesh Agrawala.
|
| 42 |
+
|
| 43 |
+
The abstract reads as follows:
|
| 44 |
+
|
| 45 |
+
*We present a neural network structure, ControlNet, to control pretrained large diffusion models to support additional input conditions.
|
| 46 |
+
The ControlNet learns task-specific conditions in an end-to-end way, and the learning is robust even when the training dataset is small (< 50k).
|
| 47 |
+
Moreover, training a ControlNet is as fast as fine-tuning a diffusion model, and the model can be trained on a personal devices.
|
| 48 |
+
Alternatively, if powerful computation clusters are available, the model can scale to large amounts (millions to billions) of data.
|
| 49 |
+
We report that large diffusion models like Stable Diffusion can be augmented with ControlNets to enable conditional inputs like edge maps, segmentation maps, keypoints, etc.
|
| 50 |
+
This may enrich the methods to control large diffusion models and further facilitate related applications.*
|
| 51 |
+
|
| 52 |
+
## Released Checkpoints
|
| 53 |
+
|
| 54 |
+
The authors released 8 different checkpoints, each trained with [Stable Diffusion v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5)
|
| 55 |
+
on a different type of conditioning:
|
| 56 |
+
|
| 57 |
+
| Model Name | Control Image Overview| Control Image Example | Generated Image Example |
|
| 58 |
+
|---|---|---|---|
|
| 59 |
+
|[lllyasviel/sd-controlnet-canny](https://huggingface.co/lllyasviel/sd-controlnet-canny)<br/> *Trained with canny edge detection* | A monochrome image with white edges on a black background.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_bird_canny.png"><img width="64" style="margin:0;padding:0;" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_bird_canny.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_bird_canny_1.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_bird_canny_1.png"/></a>|
|
| 60 |
+
|[lllyasviel/sd-controlnet-depth](https://huggingface.co/lllyasviel/sd-controlnet-depth)<br/> *Trained with Midas depth estimation* |A grayscale image with black representing deep areas and white representing shallow areas.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_vermeer_depth.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_vermeer_depth.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_vermeer_depth_2.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_vermeer_depth_2.png"/></a>|
|
| 61 |
+
|[lllyasviel/sd-controlnet-hed](https://huggingface.co/lllyasviel/sd-controlnet-hed)<br/> *Trained with HED edge detection (soft edge)* |A monochrome image with white soft edges on a black background.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_bird_hed.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_bird_hed.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_bird_hed_1.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_bird_hed_1.png"/></a> |
|
| 62 |
+
|[lllyasviel/sd-controlnet-mlsd](https://huggingface.co/lllyasviel/sd-controlnet-mlsd)<br/> *Trained with M-LSD line detection* |A monochrome image composed only of white straight lines on a black background.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_room_mlsd.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_room_mlsd.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_room_mlsd_0.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_room_mlsd_0.png"/></a>|
|
| 63 |
+
|[lllyasviel/sd-controlnet-normal](https://huggingface.co/lllyasviel/sd-controlnet-normal)<br/> *Trained with normal map* |A [normal mapped](https://en.wikipedia.org/wiki/Normal_mapping) image.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_human_normal.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_human_normal.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_human_normal_1.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_human_normal_1.png"/></a>|
|
| 64 |
+
|[lllyasviel/sd-controlnet_openpose](https://huggingface.co/lllyasviel/sd-controlnet-openpose)<br/> *Trained with OpenPose bone image* |A [OpenPose bone](https://github.com/CMU-Perceptual-Computing-Lab/openpose) image.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_human_openpose.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_human_openpose.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_human_openpose_0.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_human_openpose_0.png"/></a>|
|
| 65 |
+
|[lllyasviel/sd-controlnet_scribble](https://huggingface.co/lllyasviel/sd-controlnet-scribble)<br/> *Trained with human scribbles* |A hand-drawn monochrome image with white outlines on a black background.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_vermeer_scribble.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_vermeer_scribble.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_vermeer_scribble_0.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_vermeer_scribble_0.png"/></a> |
|
| 66 |
+
|[lllyasviel/sd-controlnet_seg](https://huggingface.co/lllyasviel/sd-controlnet-seg)<br/>*Trained with semantic segmentation* |An [ADE20K](https://groups.csail.mit.edu/vision/datasets/ADE20K/)'s segmentation protocol image.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_room_seg.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_room_seg.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_room_seg_1.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_room_seg_1.png"/></a> |
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
## Example
|
| 70 |
+
|
| 71 |
+
It is recommended to use the checkpoint with [Stable Diffusion v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5) as the checkpoint
|
| 72 |
+
has been trained on it.
|
| 73 |
+
Experimentally, the checkpoint can be used with other diffusion models such as dreamboothed stable diffusion.
|
| 74 |
+
|
| 75 |
+
1. Let's install `diffusers` and related packages:
|
| 76 |
+
|
| 77 |
+
```
|
| 78 |
+
$ pip install diffusers transformers accelerate
|
| 79 |
+
```
|
| 80 |
+
|
| 81 |
+
2. We'll need to make use of a color palette here as described in [semantic_segmentation](https://huggingface.co/docs/transformers/tasks/semantic_segmentation):
|
| 82 |
+
|
| 83 |
+
```py
|
| 84 |
+
palette = np.asarray([
|
| 85 |
+
[0, 0, 0],
|
| 86 |
+
[120, 120, 120],
|
| 87 |
+
[180, 120, 120],
|
| 88 |
+
[6, 230, 230],
|
| 89 |
+
[80, 50, 50],
|
| 90 |
+
[4, 200, 3],
|
| 91 |
+
[120, 120, 80],
|
| 92 |
+
[140, 140, 140],
|
| 93 |
+
[204, 5, 255],
|
| 94 |
+
[230, 230, 230],
|
| 95 |
+
[4, 250, 7],
|
| 96 |
+
[224, 5, 255],
|
| 97 |
+
[235, 255, 7],
|
| 98 |
+
[150, 5, 61],
|
| 99 |
+
[120, 120, 70],
|
| 100 |
+
[8, 255, 51],
|
| 101 |
+
[255, 6, 82],
|
| 102 |
+
[143, 255, 140],
|
| 103 |
+
[204, 255, 4],
|
| 104 |
+
[255, 51, 7],
|
| 105 |
+
[204, 70, 3],
|
| 106 |
+
[0, 102, 200],
|
| 107 |
+
[61, 230, 250],
|
| 108 |
+
[255, 6, 51],
|
| 109 |
+
[11, 102, 255],
|
| 110 |
+
[255, 7, 71],
|
| 111 |
+
[255, 9, 224],
|
| 112 |
+
[9, 7, 230],
|
| 113 |
+
[220, 220, 220],
|
| 114 |
+
[255, 9, 92],
|
| 115 |
+
[112, 9, 255],
|
| 116 |
+
[8, 255, 214],
|
| 117 |
+
[7, 255, 224],
|
| 118 |
+
[255, 184, 6],
|
| 119 |
+
[10, 255, 71],
|
| 120 |
+
[255, 41, 10],
|
| 121 |
+
[7, 255, 255],
|
| 122 |
+
[224, 255, 8],
|
| 123 |
+
[102, 8, 255],
|
| 124 |
+
[255, 61, 6],
|
| 125 |
+
[255, 194, 7],
|
| 126 |
+
[255, 122, 8],
|
| 127 |
+
[0, 255, 20],
|
| 128 |
+
[255, 8, 41],
|
| 129 |
+
[255, 5, 153],
|
| 130 |
+
[6, 51, 255],
|
| 131 |
+
[235, 12, 255],
|
| 132 |
+
[160, 150, 20],
|
| 133 |
+
[0, 163, 255],
|
| 134 |
+
[140, 140, 140],
|
| 135 |
+
[250, 10, 15],
|
| 136 |
+
[20, 255, 0],
|
| 137 |
+
[31, 255, 0],
|
| 138 |
+
[255, 31, 0],
|
| 139 |
+
[255, 224, 0],
|
| 140 |
+
[153, 255, 0],
|
| 141 |
+
[0, 0, 255],
|
| 142 |
+
[255, 71, 0],
|
| 143 |
+
[0, 235, 255],
|
| 144 |
+
[0, 173, 255],
|
| 145 |
+
[31, 0, 255],
|
| 146 |
+
[11, 200, 200],
|
| 147 |
+
[255, 82, 0],
|
| 148 |
+
[0, 255, 245],
|
| 149 |
+
[0, 61, 255],
|
| 150 |
+
[0, 255, 112],
|
| 151 |
+
[0, 255, 133],
|
| 152 |
+
[255, 0, 0],
|
| 153 |
+
[255, 163, 0],
|
| 154 |
+
[255, 102, 0],
|
| 155 |
+
[194, 255, 0],
|
| 156 |
+
[0, 143, 255],
|
| 157 |
+
[51, 255, 0],
|
| 158 |
+
[0, 82, 255],
|
| 159 |
+
[0, 255, 41],
|
| 160 |
+
[0, 255, 173],
|
| 161 |
+
[10, 0, 255],
|
| 162 |
+
[173, 255, 0],
|
| 163 |
+
[0, 255, 153],
|
| 164 |
+
[255, 92, 0],
|
| 165 |
+
[255, 0, 255],
|
| 166 |
+
[255, 0, 245],
|
| 167 |
+
[255, 0, 102],
|
| 168 |
+
[255, 173, 0],
|
| 169 |
+
[255, 0, 20],
|
| 170 |
+
[255, 184, 184],
|
| 171 |
+
[0, 31, 255],
|
| 172 |
+
[0, 255, 61],
|
| 173 |
+
[0, 71, 255],
|
| 174 |
+
[255, 0, 204],
|
| 175 |
+
[0, 255, 194],
|
| 176 |
+
[0, 255, 82],
|
| 177 |
+
[0, 10, 255],
|
| 178 |
+
[0, 112, 255],
|
| 179 |
+
[51, 0, 255],
|
| 180 |
+
[0, 194, 255],
|
| 181 |
+
[0, 122, 255],
|
| 182 |
+
[0, 255, 163],
|
| 183 |
+
[255, 153, 0],
|
| 184 |
+
[0, 255, 10],
|
| 185 |
+
[255, 112, 0],
|
| 186 |
+
[143, 255, 0],
|
| 187 |
+
[82, 0, 255],
|
| 188 |
+
[163, 255, 0],
|
| 189 |
+
[255, 235, 0],
|
| 190 |
+
[8, 184, 170],
|
| 191 |
+
[133, 0, 255],
|
| 192 |
+
[0, 255, 92],
|
| 193 |
+
[184, 0, 255],
|
| 194 |
+
[255, 0, 31],
|
| 195 |
+
[0, 184, 255],
|
| 196 |
+
[0, 214, 255],
|
| 197 |
+
[255, 0, 112],
|
| 198 |
+
[92, 255, 0],
|
| 199 |
+
[0, 224, 255],
|
| 200 |
+
[112, 224, 255],
|
| 201 |
+
[70, 184, 160],
|
| 202 |
+
[163, 0, 255],
|
| 203 |
+
[153, 0, 255],
|
| 204 |
+
[71, 255, 0],
|
| 205 |
+
[255, 0, 163],
|
| 206 |
+
[255, 204, 0],
|
| 207 |
+
[255, 0, 143],
|
| 208 |
+
[0, 255, 235],
|
| 209 |
+
[133, 255, 0],
|
| 210 |
+
[255, 0, 235],
|
| 211 |
+
[245, 0, 255],
|
| 212 |
+
[255, 0, 122],
|
| 213 |
+
[255, 245, 0],
|
| 214 |
+
[10, 190, 212],
|
| 215 |
+
[214, 255, 0],
|
| 216 |
+
[0, 204, 255],
|
| 217 |
+
[20, 0, 255],
|
| 218 |
+
[255, 255, 0],
|
| 219 |
+
[0, 153, 255],
|
| 220 |
+
[0, 41, 255],
|
| 221 |
+
[0, 255, 204],
|
| 222 |
+
[41, 0, 255],
|
| 223 |
+
[41, 255, 0],
|
| 224 |
+
[173, 0, 255],
|
| 225 |
+
[0, 245, 255],
|
| 226 |
+
[71, 0, 255],
|
| 227 |
+
[122, 0, 255],
|
| 228 |
+
[0, 255, 184],
|
| 229 |
+
[0, 92, 255],
|
| 230 |
+
[184, 255, 0],
|
| 231 |
+
[0, 133, 255],
|
| 232 |
+
[255, 214, 0],
|
| 233 |
+
[25, 194, 194],
|
| 234 |
+
[102, 255, 0],
|
| 235 |
+
[92, 0, 255],
|
| 236 |
+
])
|
| 237 |
+
```
|
| 238 |
+
|
| 239 |
+
3. Having defined the color palette we can now run the whole segmentation + controlnet generation code:
|
| 240 |
+
|
| 241 |
+
```py
|
| 242 |
+
from transformers import AutoImageProcessor, UperNetForSemanticSegmentation
|
| 243 |
+
from PIL import Image
|
| 244 |
+
import numpy as np
|
| 245 |
+
import torch
|
| 246 |
+
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
|
| 247 |
+
from diffusers.utils import load_image
|
| 248 |
+
|
| 249 |
+
image_processor = AutoImageProcessor.from_pretrained("openmmlab/upernet-convnext-small")
|
| 250 |
+
image_segmentor = UperNetForSemanticSegmentation.from_pretrained("openmmlab/upernet-convnext-small")
|
| 251 |
+
|
| 252 |
+
image = load_image("https://huggingface.co/lllyasviel/sd-controlnet-seg/resolve/main/images/house.png").convert('RGB')
|
| 253 |
+
|
| 254 |
+
pixel_values = image_processor(image, return_tensors="pt").pixel_values
|
| 255 |
+
|
| 256 |
+
with torch.no_grad():
|
| 257 |
+
outputs = image_segmentor(pixel_values)
|
| 258 |
+
|
| 259 |
+
seg = image_processor.post_process_semantic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]
|
| 260 |
+
|
| 261 |
+
color_seg = np.zeros((seg.shape[0], seg.shape[1], 3), dtype=np.uint8) # height, width, 3
|
| 262 |
+
|
| 263 |
+
for label, color in enumerate(palette):
|
| 264 |
+
color_seg[seg == label, :] = color
|
| 265 |
+
|
| 266 |
+
color_seg = color_seg.astype(np.uint8)
|
| 267 |
+
|
| 268 |
+
image = Image.fromarray(color_seg)
|
| 269 |
+
|
| 270 |
+
controlnet = ControlNetModel.from_pretrained(
|
| 271 |
+
"lllyasviel/sd-controlnet-seg", torch_dtype=torch.float16
|
| 272 |
+
)
|
| 273 |
+
|
| 274 |
+
pipe = StableDiffusionControlNetPipeline.from_pretrained(
|
| 275 |
+
"runwayml/stable-diffusion-v1-5", controlnet=controlnet, safety_checker=None, torch_dtype=torch.float16
|
| 276 |
+
)
|
| 277 |
+
|
| 278 |
+
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
|
| 279 |
+
|
| 280 |
+
# Remove if you do not have xformers installed
|
| 281 |
+
# see https://huggingface.co/docs/diffusers/v0.13.0/en/optimization/xformers#installing-xformers
|
| 282 |
+
# for installation instructions
|
| 283 |
+
pipe.enable_xformers_memory_efficient_attention()
|
| 284 |
+
|
| 285 |
+
pipe.enable_model_cpu_offload()
|
| 286 |
+
|
| 287 |
+
image = pipe("house", image, num_inference_steps=20).images[0]
|
| 288 |
+
|
| 289 |
+
image.save('./images/house_seg_out.png')
|
| 290 |
+
```
|
| 291 |
+
|
| 292 |
+

|
| 293 |
+
|
| 294 |
+

|
| 295 |
+
|
| 296 |
+

|
| 297 |
+
|
| 298 |
+
### Training
|
| 299 |
+
|
| 300 |
+
The semantic segmentation model was trained on 164K segmentation-image, caption pairs from ADE20K. The model was trained for 200 GPU-hours with Nvidia A100 80G using Stable Diffusion 1.5 as a base model.
|
| 301 |
+
|
| 302 |
+
### Blog post
|
| 303 |
+
|
| 304 |
+
For more information, please also have a look at the [official ControlNet Blog Post](https://huggingface.co/blog/controlnet).
|
config.json
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_class_name": "ControlNetModel",
|
| 3 |
+
"_diffusers_version": "0.14.0.dev0",
|
| 4 |
+
"act_fn": "silu",
|
| 5 |
+
"attention_head_dim": 8,
|
| 6 |
+
"block_out_channels": [
|
| 7 |
+
320,
|
| 8 |
+
640,
|
| 9 |
+
1280,
|
| 10 |
+
1280
|
| 11 |
+
],
|
| 12 |
+
"class_embed_type": null,
|
| 13 |
+
"conditioning_embedding_out_channels": [
|
| 14 |
+
16,
|
| 15 |
+
32,
|
| 16 |
+
96,
|
| 17 |
+
256
|
| 18 |
+
],
|
| 19 |
+
"controlnet_conditioning_channel_order": "rgb",
|
| 20 |
+
"cross_attention_dim": 768,
|
| 21 |
+
"down_block_types": [
|
| 22 |
+
"CrossAttnDownBlock2D",
|
| 23 |
+
"CrossAttnDownBlock2D",
|
| 24 |
+
"CrossAttnDownBlock2D",
|
| 25 |
+
"DownBlock2D"
|
| 26 |
+
],
|
| 27 |
+
"downsample_padding": 1,
|
| 28 |
+
"flip_sin_to_cos": true,
|
| 29 |
+
"freq_shift": 0,
|
| 30 |
+
"in_channels": 4,
|
| 31 |
+
"layers_per_block": 2,
|
| 32 |
+
"mid_block_scale_factor": 1,
|
| 33 |
+
"norm_eps": 1e-05,
|
| 34 |
+
"norm_num_groups": 32,
|
| 35 |
+
"num_class_embeds": null,
|
| 36 |
+
"only_cross_attention": false,
|
| 37 |
+
"projection_class_embeddings_input_dim": null,
|
| 38 |
+
"resnet_time_scale_shift": "default",
|
| 39 |
+
"upcast_attention": false,
|
| 40 |
+
"use_linear_projection": false
|
| 41 |
+
}
|
controlnet_utils.py
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
def ade_palette():
|
| 2 |
+
"""ADE20K palette that maps each class to RGB values."""
|
| 3 |
+
return [[120, 120, 120], [180, 120, 120], [6, 230, 230], [80, 50, 50],
|
| 4 |
+
[4, 200, 3], [120, 120, 80], [140, 140, 140], [204, 5, 255],
|
| 5 |
+
[230, 230, 230], [4, 250, 7], [224, 5, 255], [235, 255, 7],
|
| 6 |
+
[150, 5, 61], [120, 120, 70], [8, 255, 51], [255, 6, 82],
|
| 7 |
+
[143, 255, 140], [204, 255, 4], [255, 51, 7], [204, 70, 3],
|
| 8 |
+
[0, 102, 200], [61, 230, 250], [255, 6, 51], [11, 102, 255],
|
| 9 |
+
[255, 7, 71], [255, 9, 224], [9, 7, 230], [220, 220, 220],
|
| 10 |
+
[255, 9, 92], [112, 9, 255], [8, 255, 214], [7, 255, 224],
|
| 11 |
+
[255, 184, 6], [10, 255, 71], [255, 41, 10], [7, 255, 255],
|
| 12 |
+
[224, 255, 8], [102, 8, 255], [255, 61, 6], [255, 194, 7],
|
| 13 |
+
[255, 122, 8], [0, 255, 20], [255, 8, 41], [255, 5, 153],
|
| 14 |
+
[6, 51, 255], [235, 12, 255], [160, 150, 20], [0, 163, 255],
|
| 15 |
+
[140, 140, 140], [250, 10, 15], [20, 255, 0], [31, 255, 0],
|
| 16 |
+
[255, 31, 0], [255, 224, 0], [153, 255, 0], [0, 0, 255],
|
| 17 |
+
[255, 71, 0], [0, 235, 255], [0, 173, 255], [31, 0, 255],
|
| 18 |
+
[11, 200, 200], [255, 82, 0], [0, 255, 245], [0, 61, 255],
|
| 19 |
+
[0, 255, 112], [0, 255, 133], [255, 0, 0], [255, 163, 0],
|
| 20 |
+
[255, 102, 0], [194, 255, 0], [0, 143, 255], [51, 255, 0],
|
| 21 |
+
[0, 82, 255], [0, 255, 41], [0, 255, 173], [10, 0, 255],
|
| 22 |
+
[173, 255, 0], [0, 255, 153], [255, 92, 0], [255, 0, 255],
|
| 23 |
+
[255, 0, 245], [255, 0, 102], [255, 173, 0], [255, 0, 20],
|
| 24 |
+
[255, 184, 184], [0, 31, 255], [0, 255, 61], [0, 71, 255],
|
| 25 |
+
[255, 0, 204], [0, 255, 194], [0, 255, 82], [0, 10, 255],
|
| 26 |
+
[0, 112, 255], [51, 0, 255], [0, 194, 255], [0, 122, 255],
|
| 27 |
+
[0, 255, 163], [255, 153, 0], [0, 255, 10], [255, 112, 0],
|
| 28 |
+
[143, 255, 0], [82, 0, 255], [163, 255, 0], [255, 235, 0],
|
| 29 |
+
[8, 184, 170], [133, 0, 255], [0, 255, 92], [184, 0, 255],
|
| 30 |
+
[255, 0, 31], [0, 184, 255], [0, 214, 255], [255, 0, 112],
|
| 31 |
+
[92, 255, 0], [0, 224, 255], [112, 224, 255], [70, 184, 160],
|
| 32 |
+
[163, 0, 255], [153, 0, 255], [71, 255, 0], [255, 0, 163],
|
| 33 |
+
[255, 204, 0], [255, 0, 143], [0, 255, 235], [133, 255, 0],
|
| 34 |
+
[255, 0, 235], [245, 0, 255], [255, 0, 122], [255, 245, 0],
|
| 35 |
+
[10, 190, 212], [214, 255, 0], [0, 204, 255], [20, 0, 255],
|
| 36 |
+
[255, 255, 0], [0, 153, 255], [0, 41, 255], [0, 255, 204],
|
| 37 |
+
[41, 0, 255], [41, 255, 0], [173, 0, 255], [0, 245, 255],
|
| 38 |
+
[71, 0, 255], [122, 0, 255], [0, 255, 184], [0, 92, 255],
|
| 39 |
+
[184, 255, 0], [0, 133, 255], [255, 214, 0], [25, 194, 194],
|
| 40 |
+
[102, 255, 0], [92, 0, 255]]
|
diffusion_pytorch_model.bin
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:79accbfedbad0439ffe78a0c220fc4eac7d26ee6b3bc7e5e0458c0af1c79ff3c
|
| 3 |
+
size 1445254969
|
diffusion_pytorch_model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a623efef98586c1337a0c6dd7ba45af209d2ae00d5c563432cfbfd4ec8a4d617
|
| 3 |
+
size 1445157124
|
images/house.png
ADDED
|
images/house_seg.png
ADDED
|
images/house_seg_out.png
ADDED
|
sd.png
ADDED
|