Duplicate from Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro

Co-authored-by: Haofan Wang <wanghaofan@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/poster.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,111 @@
+---
+license: other
+license_name: flux-1-dev-non-commercial-license
+license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md
+
+language:
+  - en
+library_name: diffusers
+pipeline_tag: text-to-image
+
+tags:
+- Text-to-Image
+- ControlNet
+- Diffusers
+- Flux.1-dev
+- image-generation
+- Stable Diffusion
+base_model: black-forest-labs/FLUX.1-dev
+---
+
+# FLUX.1-dev-ControlNet-Union-Pro
+
+This repository contains a unified ControlNet for FLUX.1-dev model jointly released by researchers from [InstantX Team](https://huggingface.co/InstantX) and [Shakker Labs](https://huggingface.co/Shakker-Labs).
+
+<div class="container">
+  <img src="./assets/poster.png" width="1024"/>
+</div>
+
+
+# Model Cards
+- This checkpoint is a Pro version of [FLUX.1-dev-Controlnet-Union](https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Union) trained with more steps and datasets.
+- This model supports 7 control modes, including canny (0), tile (1), depth (2), blur (3), pose (4), gray (5), low quality (6).
+- The recommended controlnet_conditioning_scale is 0.3-0.8.
+- This model can be jointly used with other ControlNets.
+
+
+# Showcases
+
+<div class="container">
+  <img src="./assets/teaser1.png" width="1024"/>
+  <img src="./assets/teaser2.png" width="1024"/>
+  <img src="./assets/teaser3.png" width="1024"/>
+</div>
+
+
+# Inference
+Please install `diffusers` from [the source](https://github.com/huggingface/diffusers), as [the PR](https://github.com/huggingface/diffusers/pull/9175) has not been included in currently released version yet.
+
+# Multi-Controls Inference
+```python
+import torch
+from diffusers.utils import load_image
+
+from diffusers import FluxControlNetPipeline, FluxControlNetModel
+from diffusers.models import FluxMultiControlNetModel
+
+base_model = 'black-forest-labs/FLUX.1-dev'
+controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro'
+
+controlnet_union = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
+controlnet = FluxMultiControlNetModel([controlnet_union]) # we always recommend loading via FluxMultiControlNetModel
+
+pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)
+pipe.to("cuda")
+
+prompt = 'A bohemian-style female travel blogger with sun-kissed skin and messy beach waves.'
+control_image_depth = load_image("https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/resolve/main/assets/depth.jpg")
+control_mode_depth = 2
+
+control_image_canny = load_image("https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/resolve/main/assets/canny.jpg")
+control_mode_canny = 0
+
+width, height = control_image_depth.size
+
+image = pipe(
+    prompt, 
+    control_image=[control_image_depth, control_image_canny],
+    control_mode=[control_mode_depth, control_mode_canny],
+    width=width,
+    height=height,
+    controlnet_conditioning_scale=[0.2, 0.4],
+    num_inference_steps=24, 
+    guidance_scale=3.5,
+    generator=torch.manual_seed(42),
+).images[0]
+```
+
+We also support loading multiple ControlNets as before, you can load as
+```python
+from diffusers import FluxControlNetModel
+from diffusers.models import FluxMultiControlNetModel
+
+controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro'
+controlnet_union = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
+
+controlnet_model_depth = 'Shakker-Labs/FLUX.1-dev-Controlnet-Depth'
+controlnet_depth = FluxControlNetModel.from_pretrained(controlnet_model_depth, torch_dtype=torch.bfloat16)
+
+controlnet = FluxMultiControlNetModel([controlnet_union, controlnet_depth])
+
+# set mode to None for other ControlNets
+control_mode=[2, None]
+```
+
+# Resources
+- [InstantX/FLUX.1-dev-Controlnet-Canny](https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny)
+- [Shakker-Labs/FLUX.1-dev-ControlNet-Depth](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Depth)
+- [Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro)
+
+# Acknowledgements
+This project is trained by [InstantX Team](https://huggingface.co/InstantX) and sponsored by [Shakker AI](https://www.shakker.ai/). The original idea is inspired by [xinsir/controlnet-union-sdxl-1.0](https://huggingface.co/xinsir/controlnet-union-sdxl-1.0). All copyright reserved.

config.json

@@ -0,0 +1,19 @@
+{
+  "_class_name": "FluxControlNetModel",
+  "_diffusers_version": "0.30.0.dev0",
+  "attention_head_dim": 128,
+  "axes_dims_rope": [
+    16,
+    56,
+    56
+  ],
+  "guidance_embeds": true,
+  "in_channels": 64,
+  "joint_attention_dim": 4096,
+  "num_attention_heads": 24,
+  "num_layers": 5,
+  "num_mode": 10,
+  "num_single_layers": 10,
+  "patch_size": 1,
+  "pooled_projection_dim": 768
+}

diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:981a01d6a9575e90820275eda61b33d4ecab0928c68a4f31b132b9687930f90a
+size 6603953920