File size: 4,857 Bytes
7d88b5e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 | ---
library_name: videox_fun
license: other
license_name: flux-dev-non-commercial-license
license_link: https://huggingface.co/black-forest-labs/FLUX.2-dev/blob/main/LICENSE.txt
---
# Flux.2-dev-Fun-Controlnet-Union
[](https://github.com/aigc-apps/VideoX-Fun)
## Model Card
| Name | Description |
|--|--|
| FLUX.2-dev-Fun-Controlnet-Union-2602.safetensors | Compared to the previous version of the model, we have added Scribble and Gray controls. Similar to Z-Image-Turbo, the Flux2 model loses its CFG distillation capability after Control training, which is why the previous version performed poorly. Building upon the previous version, we trained on a better dataset and performed CFG distillation after training, resulting in superior performance. |
| FLUX.2-dev-Fun-Controlnet-Union.safetensors | ControlNet weights for Flux2. The model supports multiple control conditions such as Canny, HED, Depth, Pose and MLSD. |
# Model features
- This ControlNet is added on 4 double blocks.
- It supports multiple control conditions—including Canny, HED, Depth, Pose, MLSD, Scribble and Gray can be used like a standard ControlNet.
- Inpainting mode is also supported.
- You can adjust controlnet_conditioning_scale for stronger control and better detail preservation. For better stability, we highly recommend using a detailed prompt. The optimal range for controlnet_conditioning_scale is from 0.65 to 0.80.
- Although Flux.2‑dev supports certain image‑editing capabilities, its generation speed slows down when handling multiple images, and it sometimes produces similarity issues or fails to follow the control images. Compared with edit‑based methods, using ControlNet adheres more reliably to control instructions and makes it easier to apply multiple types of control.
# Results
<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
<tr>
<td>Pose + Ref</td>
<td>Output</td>
</tr>
<tr>
<td><img src="asset/pose.jpg" width="100%" /><img src="asset/ref.jpg" width="100%" /></td>
<td><img src="results/pose_ref.png" width="100%" /></td>
</tr>
</table>
<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
<tr>
<td>Pose</td>
<td>Output</td>
</tr>
<tr>
<td><img src="asset/pose.jpg" width="100%" /></td>
<td><img src="results/pose.png" width="100%" /></td>
</tr>
</table>
<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
<tr>
<td>Pose</td>
<td>Output</td>
</tr>
<tr>
<td><img src="asset/pose2.jpg" width="100%" /></td>
<td><img src="results/pose2.png" width="100%" /></td>
</tr>
</table>
<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
<tr>
<td>Canny</td>
<td>Output</td>
</tr>
<tr>
<td><img src="asset/canny.jpg" width="100%" /></td>
<td><img src="results/canny.png" width="100%" /></td>
</tr>
</table>
<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
<tr>
<td>HED</td>
<td>Output</td>
</tr>
<tr>
<td><img src="asset/hed.jpg" width="100%" /></td>
<td><img src="results/hed.png" width="100%" /></td>
</tr>
</table>
<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
<tr>
<td>Depth</td>
<td>Output</td>
</tr>
<tr>
<td><img src="asset/depth.jpg" width="100%" /></td>
<td><img src="results/depth.png" width="100%" /></td>
</tr>
</table>
<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
<tr>
<td>Gray</td>
<td>Output</td>
</tr>
<tr>
<td><img src="asset/gray.jpg" width="100%" /></td>
<td><img src="results/gray.png" width="100%" /></td>
</tr>
</table>
<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
<tr>
<td>Pose + Inpaint</td>
<td>Output</td>
</tr>
<tr>
<td><img src="asset/ref.jpg" width="100%" /><img src="asset/mask.jpg" width="100%" /><img src="asset/pose.jpg" width="100%" /></td>
<td><img src="results/pose_inpaint.png" width="100%" /></td>
</tr>
</table>
# Inference
Go to VideoX-Fun repository for more details.
Please git clone VideoX-Fun and mkdirs.
```sh
# clone code
git clone https://github.com/aigc-apps/VideoX-Fun.git
# enter VideoX-Fun's dir
cd VideoX-Fun
# download weights
mkdir models/Diffusion_Transformer
mkdir models/Personalized_Model
```
Then download weights to models/Diffusion_Transformer and models/Personalized_Model.
```
📦 models/
├── 📂 Diffusion_Transformer/
│ └── 📂 FLUX.2-dev/
├── 📂 Personalized_Model/
│ ├── 📦 FLUX.2-dev-Fun-Controlnet-Union-2602.safetensors
│ └── 📦 FLUX.2-dev-Fun-Controlnet-Union.safetensors
```
Then run the file `examples/flux2_fun/predict_t2i_control.py`. |