Calum Diprose williamberman commited on
Commit
0b56c7c
·
0 Parent(s):

Duplicate from lllyasviel/sd-controlnet-seg

Browse files

Co-authored-by: Will Berman <williamberman@users.noreply.huggingface.co>

.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tflite filter=lfs diff=lfs merge=lfs -text
29
+ *.tgz filter=lfs diff=lfs merge=lfs -text
30
+ *.wasm filter=lfs diff=lfs merge=lfs -text
31
+ *.xz filter=lfs diff=lfs merge=lfs -text
32
+ *.zip filter=lfs diff=lfs merge=lfs -text
33
+ *.zst filter=lfs diff=lfs merge=lfs -text
34
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
35
+ images/bird.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,304 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: openrail
3
+ base_model: runwayml/stable-diffusion-v1-5
4
+ tags:
5
+ - art
6
+ - controlnet
7
+ - stable-diffusion
8
+ - image-to-image
9
+ duplicated_from: lllyasviel/sd-controlnet-seg
10
+ ---
11
+
12
+ # Controlnet - *Image Segmentation Version*
13
+
14
+ ControlNet is a neural network structure to control diffusion models by adding extra conditions.
15
+ This checkpoint corresponds to the ControlNet conditioned on **Image Segmentation**.
16
+
17
+ It can be used in combination with [Stable Diffusion](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/text2img).
18
+
19
+ ![img](./sd.png)
20
+
21
+ ## Model Details
22
+ - **Developed by:** Lvmin Zhang, Maneesh Agrawala
23
+ - **Model type:** Diffusion-based text-to-image generation model
24
+ - **Language(s):** English
25
+ - **License:** [The CreativeML OpenRAIL M license](https://huggingface.co/spaces/CompVis/stable-diffusion-license) is an [Open RAIL M license](https://www.licenses.ai/blog/2022/8/18/naming-convention-of-responsible-ai-licenses), adapted from the work that [BigScience](https://bigscience.huggingface.co/) and [the RAIL Initiative](https://www.licenses.ai/) are jointly carrying in the area of responsible AI licensing. See also [the article about the BLOOM Open RAIL license](https://bigscience.huggingface.co/blog/the-bigscience-rail-license) on which our license is based.
26
+ - **Resources for more information:** [GitHub Repository](https://github.com/lllyasviel/ControlNet), [Paper](https://arxiv.org/abs/2302.05543).
27
+ - **Cite as:**
28
+
29
+ @misc{zhang2023adding,
30
+ title={Adding Conditional Control to Text-to-Image Diffusion Models},
31
+ author={Lvmin Zhang and Maneesh Agrawala},
32
+ year={2023},
33
+ eprint={2302.05543},
34
+ archivePrefix={arXiv},
35
+ primaryClass={cs.CV}
36
+ }
37
+
38
+ ## Introduction
39
+
40
+ Controlnet was proposed in [*Adding Conditional Control to Text-to-Image Diffusion Models*](https://arxiv.org/abs/2302.05543) by
41
+ Lvmin Zhang, Maneesh Agrawala.
42
+
43
+ The abstract reads as follows:
44
+
45
+ *We present a neural network structure, ControlNet, to control pretrained large diffusion models to support additional input conditions.
46
+ The ControlNet learns task-specific conditions in an end-to-end way, and the learning is robust even when the training dataset is small (< 50k).
47
+ Moreover, training a ControlNet is as fast as fine-tuning a diffusion model, and the model can be trained on a personal devices.
48
+ Alternatively, if powerful computation clusters are available, the model can scale to large amounts (millions to billions) of data.
49
+ We report that large diffusion models like Stable Diffusion can be augmented with ControlNets to enable conditional inputs like edge maps, segmentation maps, keypoints, etc.
50
+ This may enrich the methods to control large diffusion models and further facilitate related applications.*
51
+
52
+ ## Released Checkpoints
53
+
54
+ The authors released 8 different checkpoints, each trained with [Stable Diffusion v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5)
55
+ on a different type of conditioning:
56
+
57
+ | Model Name | Control Image Overview| Control Image Example | Generated Image Example |
58
+ |---|---|---|---|
59
+ |[lllyasviel/sd-controlnet-canny](https://huggingface.co/lllyasviel/sd-controlnet-canny)<br/> *Trained with canny edge detection* | A monochrome image with white edges on a black background.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_bird_canny.png"><img width="64" style="margin:0;padding:0;" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_bird_canny.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_bird_canny_1.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_bird_canny_1.png"/></a>|
60
+ |[lllyasviel/sd-controlnet-depth](https://huggingface.co/lllyasviel/sd-controlnet-depth)<br/> *Trained with Midas depth estimation* |A grayscale image with black representing deep areas and white representing shallow areas.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_vermeer_depth.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_vermeer_depth.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_vermeer_depth_2.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_vermeer_depth_2.png"/></a>|
61
+ |[lllyasviel/sd-controlnet-hed](https://huggingface.co/lllyasviel/sd-controlnet-hed)<br/> *Trained with HED edge detection (soft edge)* |A monochrome image with white soft edges on a black background.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_bird_hed.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_bird_hed.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_bird_hed_1.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_bird_hed_1.png"/></a> |
62
+ |[lllyasviel/sd-controlnet-mlsd](https://huggingface.co/lllyasviel/sd-controlnet-mlsd)<br/> *Trained with M-LSD line detection* |A monochrome image composed only of white straight lines on a black background.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_room_mlsd.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_room_mlsd.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_room_mlsd_0.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_room_mlsd_0.png"/></a>|
63
+ |[lllyasviel/sd-controlnet-normal](https://huggingface.co/lllyasviel/sd-controlnet-normal)<br/> *Trained with normal map* |A [normal mapped](https://en.wikipedia.org/wiki/Normal_mapping) image.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_human_normal.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_human_normal.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_human_normal_1.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_human_normal_1.png"/></a>|
64
+ |[lllyasviel/sd-controlnet_openpose](https://huggingface.co/lllyasviel/sd-controlnet-openpose)<br/> *Trained with OpenPose bone image* |A [OpenPose bone](https://github.com/CMU-Perceptual-Computing-Lab/openpose) image.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_human_openpose.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_human_openpose.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_human_openpose_0.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_human_openpose_0.png"/></a>|
65
+ |[lllyasviel/sd-controlnet_scribble](https://huggingface.co/lllyasviel/sd-controlnet-scribble)<br/> *Trained with human scribbles* |A hand-drawn monochrome image with white outlines on a black background.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_vermeer_scribble.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_vermeer_scribble.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_vermeer_scribble_0.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_vermeer_scribble_0.png"/></a> |
66
+ |[lllyasviel/sd-controlnet_seg](https://huggingface.co/lllyasviel/sd-controlnet-seg)<br/>*Trained with semantic segmentation* |An [ADE20K](https://groups.csail.mit.edu/vision/datasets/ADE20K/)'s segmentation protocol image.|<a href="https://huggingface.co/takuma104/controlnet_dev/blob/main/gen_compare/control_images/converted/control_room_seg.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/control_images/converted/control_room_seg.png"/></a>|<a href="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_room_seg_1.png"><img width="64" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/gen_compare/output_images/diffusers/output_room_seg_1.png"/></a> |
67
+
68
+
69
+ ## Example
70
+
71
+ It is recommended to use the checkpoint with [Stable Diffusion v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5) as the checkpoint
72
+ has been trained on it.
73
+ Experimentally, the checkpoint can be used with other diffusion models such as dreamboothed stable diffusion.
74
+
75
+ 1. Let's install `diffusers` and related packages:
76
+
77
+ ```
78
+ $ pip install diffusers transformers accelerate
79
+ ```
80
+
81
+ 2. We'll need to make use of a color palette here as described in [semantic_segmentation](https://huggingface.co/docs/transformers/tasks/semantic_segmentation):
82
+
83
+ ```py
84
+ palette = np.asarray([
85
+ [0, 0, 0],
86
+ [120, 120, 120],
87
+ [180, 120, 120],
88
+ [6, 230, 230],
89
+ [80, 50, 50],
90
+ [4, 200, 3],
91
+ [120, 120, 80],
92
+ [140, 140, 140],
93
+ [204, 5, 255],
94
+ [230, 230, 230],
95
+ [4, 250, 7],
96
+ [224, 5, 255],
97
+ [235, 255, 7],
98
+ [150, 5, 61],
99
+ [120, 120, 70],
100
+ [8, 255, 51],
101
+ [255, 6, 82],
102
+ [143, 255, 140],
103
+ [204, 255, 4],
104
+ [255, 51, 7],
105
+ [204, 70, 3],
106
+ [0, 102, 200],
107
+ [61, 230, 250],
108
+ [255, 6, 51],
109
+ [11, 102, 255],
110
+ [255, 7, 71],
111
+ [255, 9, 224],
112
+ [9, 7, 230],
113
+ [220, 220, 220],
114
+ [255, 9, 92],
115
+ [112, 9, 255],
116
+ [8, 255, 214],
117
+ [7, 255, 224],
118
+ [255, 184, 6],
119
+ [10, 255, 71],
120
+ [255, 41, 10],
121
+ [7, 255, 255],
122
+ [224, 255, 8],
123
+ [102, 8, 255],
124
+ [255, 61, 6],
125
+ [255, 194, 7],
126
+ [255, 122, 8],
127
+ [0, 255, 20],
128
+ [255, 8, 41],
129
+ [255, 5, 153],
130
+ [6, 51, 255],
131
+ [235, 12, 255],
132
+ [160, 150, 20],
133
+ [0, 163, 255],
134
+ [140, 140, 140],
135
+ [250, 10, 15],
136
+ [20, 255, 0],
137
+ [31, 255, 0],
138
+ [255, 31, 0],
139
+ [255, 224, 0],
140
+ [153, 255, 0],
141
+ [0, 0, 255],
142
+ [255, 71, 0],
143
+ [0, 235, 255],
144
+ [0, 173, 255],
145
+ [31, 0, 255],
146
+ [11, 200, 200],
147
+ [255, 82, 0],
148
+ [0, 255, 245],
149
+ [0, 61, 255],
150
+ [0, 255, 112],
151
+ [0, 255, 133],
152
+ [255, 0, 0],
153
+ [255, 163, 0],
154
+ [255, 102, 0],
155
+ [194, 255, 0],
156
+ [0, 143, 255],
157
+ [51, 255, 0],
158
+ [0, 82, 255],
159
+ [0, 255, 41],
160
+ [0, 255, 173],
161
+ [10, 0, 255],
162
+ [173, 255, 0],
163
+ [0, 255, 153],
164
+ [255, 92, 0],
165
+ [255, 0, 255],
166
+ [255, 0, 245],
167
+ [255, 0, 102],
168
+ [255, 173, 0],
169
+ [255, 0, 20],
170
+ [255, 184, 184],
171
+ [0, 31, 255],
172
+ [0, 255, 61],
173
+ [0, 71, 255],
174
+ [255, 0, 204],
175
+ [0, 255, 194],
176
+ [0, 255, 82],
177
+ [0, 10, 255],
178
+ [0, 112, 255],
179
+ [51, 0, 255],
180
+ [0, 194, 255],
181
+ [0, 122, 255],
182
+ [0, 255, 163],
183
+ [255, 153, 0],
184
+ [0, 255, 10],
185
+ [255, 112, 0],
186
+ [143, 255, 0],
187
+ [82, 0, 255],
188
+ [163, 255, 0],
189
+ [255, 235, 0],
190
+ [8, 184, 170],
191
+ [133, 0, 255],
192
+ [0, 255, 92],
193
+ [184, 0, 255],
194
+ [255, 0, 31],
195
+ [0, 184, 255],
196
+ [0, 214, 255],
197
+ [255, 0, 112],
198
+ [92, 255, 0],
199
+ [0, 224, 255],
200
+ [112, 224, 255],
201
+ [70, 184, 160],
202
+ [163, 0, 255],
203
+ [153, 0, 255],
204
+ [71, 255, 0],
205
+ [255, 0, 163],
206
+ [255, 204, 0],
207
+ [255, 0, 143],
208
+ [0, 255, 235],
209
+ [133, 255, 0],
210
+ [255, 0, 235],
211
+ [245, 0, 255],
212
+ [255, 0, 122],
213
+ [255, 245, 0],
214
+ [10, 190, 212],
215
+ [214, 255, 0],
216
+ [0, 204, 255],
217
+ [20, 0, 255],
218
+ [255, 255, 0],
219
+ [0, 153, 255],
220
+ [0, 41, 255],
221
+ [0, 255, 204],
222
+ [41, 0, 255],
223
+ [41, 255, 0],
224
+ [173, 0, 255],
225
+ [0, 245, 255],
226
+ [71, 0, 255],
227
+ [122, 0, 255],
228
+ [0, 255, 184],
229
+ [0, 92, 255],
230
+ [184, 255, 0],
231
+ [0, 133, 255],
232
+ [255, 214, 0],
233
+ [25, 194, 194],
234
+ [102, 255, 0],
235
+ [92, 0, 255],
236
+ ])
237
+ ```
238
+
239
+ 3. Having defined the color palette we can now run the whole segmentation + controlnet generation code:
240
+
241
+ ```py
242
+ from transformers import AutoImageProcessor, UperNetForSemanticSegmentation
243
+ from PIL import Image
244
+ import numpy as np
245
+ import torch
246
+ from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
247
+ from diffusers.utils import load_image
248
+
249
+ image_processor = AutoImageProcessor.from_pretrained("openmmlab/upernet-convnext-small")
250
+ image_segmentor = UperNetForSemanticSegmentation.from_pretrained("openmmlab/upernet-convnext-small")
251
+
252
+ image = load_image("https://huggingface.co/lllyasviel/sd-controlnet-seg/resolve/main/images/house.png").convert('RGB')
253
+
254
+ pixel_values = image_processor(image, return_tensors="pt").pixel_values
255
+
256
+ with torch.no_grad():
257
+ outputs = image_segmentor(pixel_values)
258
+
259
+ seg = image_processor.post_process_semantic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]
260
+
261
+ color_seg = np.zeros((seg.shape[0], seg.shape[1], 3), dtype=np.uint8) # height, width, 3
262
+
263
+ for label, color in enumerate(palette):
264
+ color_seg[seg == label, :] = color
265
+
266
+ color_seg = color_seg.astype(np.uint8)
267
+
268
+ image = Image.fromarray(color_seg)
269
+
270
+ controlnet = ControlNetModel.from_pretrained(
271
+ "lllyasviel/sd-controlnet-seg", torch_dtype=torch.float16
272
+ )
273
+
274
+ pipe = StableDiffusionControlNetPipeline.from_pretrained(
275
+ "runwayml/stable-diffusion-v1-5", controlnet=controlnet, safety_checker=None, torch_dtype=torch.float16
276
+ )
277
+
278
+ pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
279
+
280
+ # Remove if you do not have xformers installed
281
+ # see https://huggingface.co/docs/diffusers/v0.13.0/en/optimization/xformers#installing-xformers
282
+ # for installation instructions
283
+ pipe.enable_xformers_memory_efficient_attention()
284
+
285
+ pipe.enable_model_cpu_offload()
286
+
287
+ image = pipe("house", image, num_inference_steps=20).images[0]
288
+
289
+ image.save('./images/house_seg_out.png')
290
+ ```
291
+
292
+ ![house](images/house.png)
293
+
294
+ ![house_seg](images/house_seg.png)
295
+
296
+ ![house_seg_out](images/house_seg_out.png)
297
+
298
+ ### Training
299
+
300
+ The semantic segmentation model was trained on 164K segmentation-image, caption pairs from ADE20K. The model was trained for 200 GPU-hours with Nvidia A100 80G using Stable Diffusion 1.5 as a base model.
301
+
302
+ ### Blog post
303
+
304
+ For more information, please also have a look at the [official ControlNet Blog Post](https://huggingface.co/blog/controlnet).
config.json ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "ControlNetModel",
3
+ "_diffusers_version": "0.14.0.dev0",
4
+ "act_fn": "silu",
5
+ "attention_head_dim": 8,
6
+ "block_out_channels": [
7
+ 320,
8
+ 640,
9
+ 1280,
10
+ 1280
11
+ ],
12
+ "class_embed_type": null,
13
+ "conditioning_embedding_out_channels": [
14
+ 16,
15
+ 32,
16
+ 96,
17
+ 256
18
+ ],
19
+ "controlnet_conditioning_channel_order": "rgb",
20
+ "cross_attention_dim": 768,
21
+ "down_block_types": [
22
+ "CrossAttnDownBlock2D",
23
+ "CrossAttnDownBlock2D",
24
+ "CrossAttnDownBlock2D",
25
+ "DownBlock2D"
26
+ ],
27
+ "downsample_padding": 1,
28
+ "flip_sin_to_cos": true,
29
+ "freq_shift": 0,
30
+ "in_channels": 4,
31
+ "layers_per_block": 2,
32
+ "mid_block_scale_factor": 1,
33
+ "norm_eps": 1e-05,
34
+ "norm_num_groups": 32,
35
+ "num_class_embeds": null,
36
+ "only_cross_attention": false,
37
+ "projection_class_embeddings_input_dim": null,
38
+ "resnet_time_scale_shift": "default",
39
+ "upcast_attention": false,
40
+ "use_linear_projection": false
41
+ }
controlnet_utils.py ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ def ade_palette():
2
+ """ADE20K palette that maps each class to RGB values."""
3
+ return [[120, 120, 120], [180, 120, 120], [6, 230, 230], [80, 50, 50],
4
+ [4, 200, 3], [120, 120, 80], [140, 140, 140], [204, 5, 255],
5
+ [230, 230, 230], [4, 250, 7], [224, 5, 255], [235, 255, 7],
6
+ [150, 5, 61], [120, 120, 70], [8, 255, 51], [255, 6, 82],
7
+ [143, 255, 140], [204, 255, 4], [255, 51, 7], [204, 70, 3],
8
+ [0, 102, 200], [61, 230, 250], [255, 6, 51], [11, 102, 255],
9
+ [255, 7, 71], [255, 9, 224], [9, 7, 230], [220, 220, 220],
10
+ [255, 9, 92], [112, 9, 255], [8, 255, 214], [7, 255, 224],
11
+ [255, 184, 6], [10, 255, 71], [255, 41, 10], [7, 255, 255],
12
+ [224, 255, 8], [102, 8, 255], [255, 61, 6], [255, 194, 7],
13
+ [255, 122, 8], [0, 255, 20], [255, 8, 41], [255, 5, 153],
14
+ [6, 51, 255], [235, 12, 255], [160, 150, 20], [0, 163, 255],
15
+ [140, 140, 140], [250, 10, 15], [20, 255, 0], [31, 255, 0],
16
+ [255, 31, 0], [255, 224, 0], [153, 255, 0], [0, 0, 255],
17
+ [255, 71, 0], [0, 235, 255], [0, 173, 255], [31, 0, 255],
18
+ [11, 200, 200], [255, 82, 0], [0, 255, 245], [0, 61, 255],
19
+ [0, 255, 112], [0, 255, 133], [255, 0, 0], [255, 163, 0],
20
+ [255, 102, 0], [194, 255, 0], [0, 143, 255], [51, 255, 0],
21
+ [0, 82, 255], [0, 255, 41], [0, 255, 173], [10, 0, 255],
22
+ [173, 255, 0], [0, 255, 153], [255, 92, 0], [255, 0, 255],
23
+ [255, 0, 245], [255, 0, 102], [255, 173, 0], [255, 0, 20],
24
+ [255, 184, 184], [0, 31, 255], [0, 255, 61], [0, 71, 255],
25
+ [255, 0, 204], [0, 255, 194], [0, 255, 82], [0, 10, 255],
26
+ [0, 112, 255], [51, 0, 255], [0, 194, 255], [0, 122, 255],
27
+ [0, 255, 163], [255, 153, 0], [0, 255, 10], [255, 112, 0],
28
+ [143, 255, 0], [82, 0, 255], [163, 255, 0], [255, 235, 0],
29
+ [8, 184, 170], [133, 0, 255], [0, 255, 92], [184, 0, 255],
30
+ [255, 0, 31], [0, 184, 255], [0, 214, 255], [255, 0, 112],
31
+ [92, 255, 0], [0, 224, 255], [112, 224, 255], [70, 184, 160],
32
+ [163, 0, 255], [153, 0, 255], [71, 255, 0], [255, 0, 163],
33
+ [255, 204, 0], [255, 0, 143], [0, 255, 235], [133, 255, 0],
34
+ [255, 0, 235], [245, 0, 255], [255, 0, 122], [255, 245, 0],
35
+ [10, 190, 212], [214, 255, 0], [0, 204, 255], [20, 0, 255],
36
+ [255, 255, 0], [0, 153, 255], [0, 41, 255], [0, 255, 204],
37
+ [41, 0, 255], [41, 255, 0], [173, 0, 255], [0, 245, 255],
38
+ [71, 0, 255], [122, 0, 255], [0, 255, 184], [0, 92, 255],
39
+ [184, 255, 0], [0, 133, 255], [255, 214, 0], [25, 194, 194],
40
+ [102, 255, 0], [92, 0, 255]]
diffusion_pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:79accbfedbad0439ffe78a0c220fc4eac7d26ee6b3bc7e5e0458c0af1c79ff3c
3
+ size 1445254969
diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a623efef98586c1337a0c6dd7ba45af209d2ae00d5c563432cfbfd4ec8a4d617
3
+ size 1445157124
images/house.png ADDED
images/house_seg.png ADDED
images/house_seg_out.png ADDED
sd.png ADDED