Update README.md

README.md

@@ -98,6 +98,69 @@ image = pipe(
    <img style="width:400px;" src="images/corgi.jpg">
 </p>
 
+# Combining Flash Diffusion with Existing ControlNets 🎨
+
+FlashSDXL can also be combined with existing ControlNets to unlock few steps generation in a **training free** manner. It can be integrated straight to Hugging Face pipelines. See an example below.
+
+```python
+import torch
+import cv2
+import numpy as np
+from PIL import Image
+
+from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel, LCMScheduler
+from diffusers.utils import load_image, make_image_grid
+
+adapter_id = "jasperai/flash-sdxl"
+
+image = load_image(
+    "https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png"
+).resize((1024, 1024))
+
+image = np.array(image)
+
+image = cv2.Canny(image, 100, 200)
+image = image[:, :, None].repeat(3, 2)
+canny_image = Image.fromarray(image)
+
+# Load ControlNet
+controlnet = ControlNetModel.from_pretrained(
+    "diffusers/controlnet-canny-sdxl-1.0",
+    torch_dtype=torch.float16,
+    variant="fp16"
+)
+pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
+    "stabilityai/stable-diffusion-xl-base-1.0",
+    controlnet=controlnet,
+    torch_dtype=torch.float16,
+    safety_checker=None,
+    variant="fp16"
+).to("cuda")
+
+# Set scheduler
+pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
+
+# Load LoRA
+pipe.load_lora_weights("jasperai/flash-sdxl")
+pipe.fuse_lora()
+
+generator = torch.manual_seed(0)
+image = pipe(
+    "picture of the mona lisa",
+    image=canny_image,
+    num_inference_steps=4,
+    guidance_scale=0,
+    controlnet_conditioning_scale=0.5,
+    cross_attention_kwargs={"scale": 1},
+    generator=generator,
+).images[0]
+make_image_grid([canny_image, image], rows=1, cols=2)
+```
+<p align="center">
+   <img style="width:400px;" src="images/controlnet.jpg">
+</p>
+
+
 # Training Details
 The model was trained for 20k iterations on 4 H100 GPUs (representing approximately a total of 176 GPU hours of training). Please refer to the [paper](http://arxiv.org/abs/2406.02347) for further parameters details.