| | --- |
| | license: openrail |
| | base_model: runwayml/stable-diffusion-v1-5 |
| | tags: |
| | - art |
| | - controlnet |
| | - stable-diffusion |
| | --- |
| | |
| | # Controlnet |
| |
|
| | Controlnet is an auxiliary model which augments pre-trained diffusion models with an additional conditioning. |
| |
|
| | Controlnet comes with multiple auxiliary models, each which allows a different type of conditioning |
| |
|
| | Controlnet's auxiliary models are trained with stable diffusion 1.5. Experimentally, the auxiliary models can be used with other diffusion models such as dreamboothed stable diffusion. |
| |
|
| | The auxiliary conditioning is passed directly to the diffusers pipeline. If you want to process an image to create the auxiliary conditioning, external dependencies are required. |
| |
|
| | Some of the additional conditionings can be extracted from images via additional models. We extracted these |
| | additional models from the original controlnet repo into a separate package that can be found on [github](https://github.com/patrickvonplaten/controlnet_aux.git). |
| |
|
| | ## Normal map |
| |
|
| | ### Diffusers |
| |
|
| | ```py |
| | from PIL import Image |
| | from transformers import pipeline |
| | import numpy as np |
| | import cv2 |
| | from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler |
| | import torch |
| | |
| | image = Image.open("images/toy.png").convert("RGB") |
| | |
| | depth_estimator = pipeline("depth-estimation", model ="Intel/dpt-hybrid-midas" ) |
| | |
| | image = depth_estimator(image)['predicted_depth'][0] |
| | |
| | image = image.numpy() |
| | |
| | image_depth = image.copy() |
| | image_depth -= np.min(image_depth) |
| | image_depth /= np.max(image_depth) |
| | |
| | bg_threhold = 0.4 |
| | |
| | x = cv2.Sobel(image, cv2.CV_32F, 1, 0, ksize=3) |
| | x[image_depth < bg_threhold] = 0 |
| | |
| | y = cv2.Sobel(image, cv2.CV_32F, 0, 1, ksize=3) |
| | y[image_depth < bg_threhold] = 0 |
| | |
| | z = np.ones_like(x) * np.pi * 2.0 |
| | |
| | image = np.stack([x, y, z], axis=2) |
| | image /= np.sum(image ** 2.0, axis=2, keepdims=True) ** 0.5 |
| | image = (image * 127.5 + 127.5).clip(0, 255).astype(np.uint8) |
| | image = Image.fromarray(image) |
| | |
| | controlnet = ControlNetModel.from_pretrained( |
| | "fusing/stable-diffusion-v1-5-controlnet-normal", torch_dtype=torch.float16 |
| | ) |
| | |
| | pipe = StableDiffusionControlNetPipeline.from_pretrained( |
| | "runwayml/stable-diffusion-v1-5", controlnet=controlnet, safety_checker=None, torch_dtype=torch.float16 |
| | ) |
| | |
| | pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config) |
| | |
| | # Remove if you do not have xformers installed |
| | # see https://huggingface.co/docs/diffusers/v0.13.0/en/optimization/xformers#installing-xformers |
| | # for installation instructions |
| | pipe.enable_xformers_memory_efficient_attention() |
| | |
| | pipe.enable_model_cpu_offload() |
| | |
| | image = pipe("cute toy", image, num_inference_steps=20).images[0] |
| | |
| | image.save('images/toy_normal_out.png') |
| | ``` |
| |
|
| |  |
| |
|
| |  |
| |
|
| |  |
| |
|
| | ### Training |
| |
|
| | The normal model was trained from an initial model and then a further extended model. |
| |
|
| | The initial normal model was trained on 25,452 normal-image, caption pairs from DIODE. The image captions were generated by BLIP. The model was trained for 100 GPU-hours with Nvidia A100 80G using Stable Diffusion 1.5 as a base model. |
| |
|
| | The extended normal model further trained the initial normal model on "coarse" normal maps. The coarse normal maps were generated using Midas to compute a depth map and then performing normal-from-distance. The model was trained for 200 GPU-hours with Nvidia A100 80G using the initial normal model as a base model. |
| |
|
