Update Weights

.gitattributes

@@ -1,3 +1,5 @@
+*.png filter=lfs diff=lfs merge=lfs -text
+*.jpg filter=lfs diff=lfs merge=lfs -text
 *.7z filter=lfs diff=lfs merge=lfs -text
 *.arrow filter=lfs diff=lfs merge=lfs -text
 *.bin filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,150 @@
 ---
 license: apache-2.0
 ---
+
+# Z-Image-Fun-Controlnet-Union-2.1
+
+[![Github](https://img.shields.io/badge/🎬%20Code-VideoX_Fun-blue)](https://github.com/aigc-apps/VideoX-Fun)
+
+## Model Card
+
+| Name | Description |
+|--|--|
+| Z-Image-Fun-Controlnet-Union-2.1.safetensors | ControlNet weights for Z-Image. The model supports multiple control conditions such as Canny, Depth, Pose, MLSD, Scribble, Hed and Gray. This ControlNet is added on 15 layer blocks and 2 refiner layer blocks. |
+| Z-Image-Fun-Controlnet-Union-2.1-lite.safetensors | Uses the same training scheme as the 2601 version, but compared to the large version of the model, fewer layers have control added, resulting in weaker control conditions. This makes it suitable for larger control_context_scale values, and the generation results appear more natural. It is also suitable for lower-spec machines. |
+
+## Model Features
+- This ControlNet is added on 15 layer blocks and 2 refiner layer blocks (Lite models are added on 3 layer blocks and 2 refiner blocks). It supports multiple control conditions—including Canny, Depth, Pose, MLSD, Scribble, Hed and Gray can be used like a standard ControlNet. 
+- Inpainting mode is also supported.
+- You can adjust control_context_scale for stronger control and better detail preservation. For better stability, we highly recommend using a detailed prompt. The optimal range for control_context_scale is from 0.65 to 0.90. 
+
+## Results
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr>
+    <td>Pose + Inpaint</td>
+    <td>Output</td>
+  </tr>
+  <tr>
+    <td><img src="asset/inpaint.jpg" width="100%" /><img src="asset/mask.jpg" width="100%" /></td>
+    <td><img src="results/inpaint.png" width="100%" /></td>
+  </tr>
+</table>
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr>
+    <td>Pose + Inpaint</td>
+    <td>Output</td>
+  </tr>
+  <tr>
+    <td><img src="asset/inpaint.jpg" width="100%" /><img src="asset/mask.jpg" width="100%" /><img src="asset/pose.jpg" width="100%" /></td>
+    <td><img src="results/pose_inpaint.png" width="100%" /></td>
+  </tr>
+</table>
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr>
+    <td>Pose</td>
+    <td>Output</td>
+  </tr>
+  <tr>
+    <td><img src="asset/pose2.jpg" width="100%" /></td>
+    <td><img src="results/pose2.png" width="100%" /></td>
+  </tr>
+</table>
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr>
+    <td>Pose</td>
+    <td>Output</td>
+  </tr>
+  <tr>
+    <td><img src="asset/pose.jpg" width="100%" /></td>
+    <td><img src="results/pose.png" width="100%" /></td>
+  </tr>
+</table>
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr>
+    <td>Pose</td>
+    <td>Output</td>
+  </tr>
+  <tr>
+    <td><img src="asset/pose3.jpg" width="100%" /></td>
+    <td><img src="results/pose3.png" width="100%" /></td>
+  </tr>
+</table>
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr>
+    <td>Canny</td>
+    <td>Output</td>
+  </tr>
+  <tr>
+    <td><img src="asset/canny.jpg" width="100%" /></td>
+    <td><img src="results/canny.png" width="100%" /></td>
+  </tr>
+</table>
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr>
+    <td>HED</td>
+    <td>Output</td>
+  </tr>
+  <tr>
+    <td><img src="asset/hed.jpg" width="100%" /></td>
+    <td><img src="results/hed.png" width="100%" /></td>
+  </tr>
+</table>
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr>
+    <td>Depth</td>
+    <td>Output</td>
+  </tr>
+  <tr>
+    <td><img src="asset/depth.jpg" width="100%" /></td>
+    <td><img src="results/depth.png" width="100%" /></td>
+  </tr>
+</table>
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr>
+    <td>Depth</td>
+    <td>Output</td>
+  </tr>
+  <tr>
+    <td><img src="asset/gray.jpg" width="100%" /></td>
+    <td><img src="results/gray.png" width="100%" /></td>
+  </tr>
+</table>
+
+## Inference
+Go to the VideoX-Fun repository for more details.
+
+Please clone the VideoX-Fun repository and create the required directories:
+
+```sh
+# Clone the code
+git clone https://github.com/aigc-apps/VideoX-Fun.git
+
+# Enter VideoX-Fun's directory
+cd VideoX-Fun
+
+# Create model directories
+mkdir -p models/Diffusion_Transformer
+mkdir -p models/Personalized_Model
+```
+
+Then download the weights into models/Diffusion_Transformer and models/Personalized_Model.
+
+```
+📦 models/
+├── 📂 Diffusion_Transformer/
+│   └── 📂 Z-Image/
+├── 📂 Personalized_Model/
+│   ├── 📦 Z-Image-Fun-Controlnet-Union-2.1.safetensors
+│   └── 📦 Z-Image-Fun-Controlnet-Union-2.1-lite.safetensors
+```
+
+Then run the file `examples/z_image_fun/predict_t2i_control_2.1.py` and `examples/z_image_fun/predict_i2i_inpaint_2.1.py`.

Z-Image-Fun-Controlnet-Union-2.1-lite.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c3e2265d1c301b444e83d668a2d12416b0b6d91cd2d18bc11eec12a7fc335cb
+size 2016627488

Z-Image-Fun-Controlnet-Union-2.1.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2393b0c58c52a12134f6ffd96ff9b6ea3c80bb233665fb2c3b9aebcee71ae3e4
+size 6712485600