Uploading weights, images and Readme

README.md

@@ -1,3 +1,92 @@
 ---
 license: openrail
 ---
+
+
+# ControlNet-XS
+
+![](./teaser_small.gif)
+
+These are ControlNet-XS weights trained on [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) and  [stabilityai/stable-diffusion-2-1](https://huggingface.co/stabilityai/stable-diffusion-2-1) on edge and depthmap conditioning respectively. You can find more details and further visual examples on the project page [ControlNet-XS](https://vislearn.github.io/ControlNet-XS/).
+
+## The codebase
+The code is based on on the StableDiffusion frameworks. To use the ControlNet-XS, you need to access the weights for the StableDiffusion version that you want to control separately.
+We provide the weights with both depth and edge control for StableDiffusion2.1 and StableDiffusion-XL.
+
+After obtaining the weights, you need the replace the paths to the weights of StableDiffusion and ControlNet-XS in the config files.
+
+## Usage
+
+
+Example for StableDiffusion-XL with Canny Edges
+
+```python
+import scripts.control_utils as cu
+import matplotlib.pyplot as plt
+import torch
+from PIL import Image
+
+path_to_config = 'ControlNet-XS-main/configs/inference/sdxl/sdxl_encD_canny_48m.yaml'
+model = cu.create_model(config_path_depth)
+
+image_path = 'PATH/TO/IMAGES/Shoe.png'
+
+canny_high_th = 250
+canny_low_th = 100
+size = 768
+num_samples=2
+
+image = cu.get_image(image_path, size=size)
+edges = cu.get_canny_edges(image, low_th=canny_low_th, high_th=canny_high_th)
+
+samples, controls = cu.get_sdxl_sample(
+    guidance=edges,
+    ddim_steps=10,
+    num_samples=2,
+    model=model,
+    shape=[4, size // 8, size // 8],
+    control_scale=0.95,
+    prompt='cinematic, shoe in the streets, made from meat, photorealistic shoe, highly detailed',
+    n_prompt='lowres, bad anatomy, worst quality, low quality',
+)
+
+
+Image.fromarray(cu.create_image_grid(samples)).save('SDXL_MyShoe.png')
+```
+![images_1)](./SDXL_MyShoe.png)
+
+Example for StableDiffusion2.1 with depth maps
+
+
+```python
+import scripts.control_utils as cu
+import matplotlib.pyplot as plt
+import torch
+from PIL import Image
+
+path_to_config = 'PATH/TO/CONFIG/sd21_encD_depth_14m.yaml'
+model = cu.create_model(path_to_config)
+
+size = 768
+image_path = 'PATH/TO/IMAGES/Shoe.png'
+
+
+image = cu.get_image(image_path, size=size)
+depth = cu.get_midas_depth(image, max_resolution=size)
+num_samples = 2
+
+samples, controls = cu.get_sd_sample(
+    guidance=depth,
+    ddim_steps=10,
+    num_samples=num_samples,
+    model=model,
+    shape=[4, size // 8, size // 8],
+    control_scale=0.95,
+    prompt='cinematic, advertising shot, shoe in a city street, photorealistic shoe, colourful, highly detailed',
+    n_prompt='low quality, bad quality, sketches'
+)
+
+
+Image.fromarray(cu.create_image_grid(samples)).save('SD_MyShoe.png')
+```
+![images_2)](./SD_MyShoe.png)

sd21_encD_canny_14m.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb0b63002b89247a770da8a534be12ee239d17dac6840eb8d7536cd9d9d1a385
+size 56916738

sd21_encD_depth_14m.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bb6747e7a8c7fd9f7cbf222a537676b8b8ea00c24edc65aeb575385cc642b255
+size 56916738

sdxl_encD_canny_48m.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:394eac155a33f9bb3c2d8a419e876cc45ab8662abebdcb08cdb81e3fa7827e39
+size 190796044

sdxl_encD_depth_48m.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43be7925d9bff95dae5bbc53109898287f3844d676e74e9ae307cb36cc641151
+size 190796044