ussoewwin commited on
Commit
f38bc94
Β·
verified Β·
1 Parent(s): 55a5f05

docs: update ControlNet ConvRot INT8 model card

Browse files
Files changed (1) hide show
  1. README.md +134 -0
README.md CHANGED
@@ -1,3 +1,137 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ tags:
4
+ - controlnet
5
+ - text-to-image
6
+ - image-to-image
7
+ - inpainting
8
+ - qwen-image
9
+ - qwen-image-2512
10
+ - videox-fun
11
+ - quantized
12
+ - int8
13
+ - convrot
14
+ - comfyui
15
+ pipeline_tag: image-to-image
16
+ library_name: videox_fun
17
  ---
18
+
19
+ # ControlNet Models (ConvRot INT8)
20
+
21
+ <p align="center">
22
+ <img src="https://raw.githubusercontent.com/ussoewwin/Hybrid-Sensitivity-Weighted-Quantization/main/icon.png" width="128">
23
+ </p>
24
+
25
+ High-fidelity **ConvRot INT8** quantized weights for diffusion ControlNet models. This repository provides memory-efficient, production-grade INT8 quantized checkpoints maintaining high structural fidelity and multi-condition guidance.
26
+
27
+ ---
28
+
29
+ ## 🌟 Model Overview
30
+
31
+ This repository hosts the **ConvRot INT8** quantized edition of **`Qwen-Image-2512-Fun-Controlnet-Union-2602`**, originally developed and trained by **Alibaba PAI / VideoX-Fun**.
32
+
33
+ The **ConvRot INT8** format applies rotational transformation matrix optimizations to eliminate outlier activation skew and ensure minimal reconstruction error, reducing the model footprint to **~1.64 GB** while preserving multi-condition structural precision.
34
+
35
+ - **Base Model:** [alibaba-pai/Qwen-Image-2512-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/Qwen-Image-2512-Fun-Controlnet-Union)
36
+ - **Architecture:** Multi-condition Union ControlNet added across 5 layer blocks of Qwen-Image-2512.
37
+ - **Quantization:** Full ConvRot INT8 (weights stored as `int8_tensorwise` with rotation matrices).
38
+ - **License:** Apache-2.0
39
+
40
+ ---
41
+
42
+ ## πŸ“¦ Available Models
43
+
44
+ | Filename | Base Architecture | Supported Conditions | Quantization | Size | License |
45
+ | :--- | :--- | :--- | :--- | :--- | :--- |
46
+ | `Qwen-Image-2512-Fun-Controlnet-Union-2602_convrot_int8.safetensors` | Qwen-Image-2512 (5 Layer Blocks) | Canny, HED, Depth, Pose, MLSD, Scribble, Gray, Inpaint | ConvRot INT8 | ~1.64 GB | Apache-2.0 |
47
+
48
+ ---
49
+
50
+ ## πŸ› οΈ Supported Control Modalities & Features
51
+
52
+ 1. **Union Condition Architecture**:
53
+ Supports unified multi-condition control within a single checkpoint:
54
+ - **Pose** (OpenPose / DWPose keypoints)
55
+ - **Canny** (Edge detection)
56
+ - **HED** (Soft edge detection)
57
+ - **Depth** (Z-depth estimation)
58
+ - **MLSD** (Straight-line wireframe extraction)
59
+ - **Scribble** (Interactive doodle / sketch guidance)
60
+ - **Gray** (Grayscale / luminance guidance)
61
+ 2. **Inpainting Mode Support**:
62
+ Fully compatible with masked inpainting pipelines (`Pose + Inpaint`, `Depth + Inpaint`, etc.).
63
+ 3. **Multi-Resolution Conditioning**:
64
+ Extracting control maps in multi-resolution formats improves generalization across arbitrary target aspect ratios.
65
+
66
+ ---
67
+
68
+ ## βš™οΈ Recommended Inference Settings
69
+
70
+ - **`control_context_scale`**: Optimal range is **`0.70` – `0.95`**. Higher values yield stronger adherence to the control input, while lower values offer increased prompt stylization freedom.
71
+ - **Prompt Formulation**: Detailed natural language descriptions significantly enhance semantic stability and detail preservation.
72
+ - **Resolution**: Multi-scale aspect ratios matching the base Qwen-Image model resolution specifications.
73
+
74
+ ---
75
+
76
+ ## πŸš€ How to Use
77
+
78
+ ### 1. VideoX-Fun Framework
79
+
80
+ Clone the upstream [VideoX-Fun](https://github.com/aigc-apps/VideoX-Fun) repository:
81
+
82
+ ```bash
83
+ git clone https://github.com/aigc-apps/VideoX-Fun.git
84
+ cd VideoX-Fun
85
+
86
+ mkdir -p models/Diffusion_Transformer
87
+ mkdir -p models/Personalized_Model
88
+ ```
89
+
90
+ Place the quantized model under `models/Personalized_Model/`:
91
+
92
+ ```
93
+ models/
94
+ β”œβ”€β”€ Diffusion_Transformer/
95
+ β”‚ └── Qwen-Image-2512/
96
+ └── Personalized_Model/
97
+ └── Qwen-Image-2512-Fun-Controlnet-Union-2602_convrot_int8.safetensors
98
+ ```
99
+
100
+ Execute standard prediction pipelines:
101
+ ```bash
102
+ # Text-to-Image with ControlNet
103
+ python examples/qwenimage_fun/predict_t2i_control.py
104
+
105
+ # Image-to-Image with Inpainting + ControlNet
106
+ python examples/qwenimage_fun/predict_i2i_inpaint.py
107
+ ```
108
+
109
+ ### 2. ComfyUI
110
+
111
+ For ComfyUI workflows, use the dedicated loader node pack:
112
+ - **[ComfyUI-HSWQ-Loader-and-Tools](https://github.com/ussoewwin/ComfyUI-HSWQ-Loader-and-Tools)**
113
+ - **[ComfyUI-QwenImageLoraLoader](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader)**
114
+
115
+ ---
116
+
117
+ ## πŸ“œ Credits & Citation
118
+
119
+ ### Upstream Creators
120
+ Deep gratitude and acknowledgement to the **Alibaba PAI VideoX-Fun Team** for developing, training, and open-sourcing the Qwen-Image ControlNet series.
121
+ - **Original Model:** [alibaba-pai/Qwen-Image-2512-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/Qwen-Image-2512-Fun-Controlnet-Union)
122
+ - **Official Repository:** [aigc-apps/VideoX-Fun](https://github.com/aigc-apps/VideoX-Fun)
123
+
124
+ ```bibtex
125
+ @misc{videox_fun,
126
+ author = {VideoX-Fun Team},
127
+ title = {VideoX-Fun: A Flexible Video and Image Generation Toolset},
128
+ year = {2025},
129
+ publisher = {GitHub},
130
+ journal = {GitHub repository},
131
+ howpublished = {\url{https://github.com/aigc-apps/VideoX-Fun}}
132
+ }
133
+ ```
134
+
135
+ ---
136
+
137
+ **Disclaimer:** These model weights are released strictly for research, non-commercial, and optimization workflows under the Apache-2.0 License.