Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -25,70 +25,109 @@ pipeline_tag: image-to-image
|
|
| 25 |
|
| 26 |
# ControlNet Models (ConvRot INT8)
|
| 27 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
High-fidelity **ConvRot INT8** quantized weights for multi-condition and dedicated ControlNet models across diverse generative architectures (**Illustrious-XL / SDXL**, **SDXL 1.0**, **Z-Image-Turbo**, **Qwen-Image / Qwen-Image-2512**, and **FLUX.1-dev**).
|
| 29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
---
|
| 31 |
|
| 32 |
## ๐ Model Overview
|
| 33 |
|
| 34 |
-
This repository hosts high-quality **ConvRot INT8** quantized weights for all-in-one ControlNet Union and dedicated condition-specific ControlNet models.
|
| 35 |
|
| 36 |
- **CN-anytest4_illustrious2 (Variants A & B)**: Multi-purpose all-in-one Anytest v4 ControlNet models fine-tuned for **Illustrious-XL / SDXL** by **2vXpSwA7**, offering high-precision anime and illustration structure guiding.
|
| 37 |
- **controlnet-union-pro-max-sdxl-1.0**: The comprehensive all-in-one ControlNet Union model for **SDXL 1.0** by **xinsir**, supporting 10+ control conditions in a single compact file.
|
| 38 |
-
- **Z-Image-Turbo-Fun-Controlnet-Union-2.1-lite-
|
| 39 |
-
- **Z-Image-Turbo-Fun-Controlnet-Tile-2.1-lite-2601-8steps**: Dedicated high-resolution Tile and upscaling ControlNet model (lite architecture, 8-step distilled) for the **Z-Image-Turbo** architecture by **alibaba-pai**, optimized for super-resolution detail enhancement.
|
| 40 |
- **Qwen-Image-2512-Fun-Controlnet-Union-2602**: Multi-condition ControlNet Union model (5 layer blocks) for the **Qwen-Image-2512** architecture.
|
| 41 |
- **Qwen-Image-ControlNet-Inpainting**: Dedicated inpainting and editing ControlNet model for the **Qwen-Image** architecture.
|
| 42 |
-
- **Qwen-Image DiffSynth ControlNet Suite**: Dedicated single-condition ControlNet models for the **Qwen-Image** architecture developed by **DiffSynth-Studio**
|
| 43 |
-
- **`qwen_image_canny_diffsynth_controlnet_convrot_int8`**: High-precision Canny edge structural control.
|
| 44 |
-
- **`qwen_image_depth_diffsynth_controlnet_convrot_int8`**: Geometric spatial depth guidance.
|
| 45 |
-
- **`qwen_image_inpaint_diffsynth_controlnet_convrot_int8`**: Masked inpainting and localized semantic replacement.
|
| 46 |
- **FLUX.1-dev-ControlNet-Union-Pro-2.0**: Next-generation unified 7-in-1 ControlNet for the **FLUX.1-dev** architecture by **Shakker Labs**.
|
| 47 |
|
| 48 |
---
|
| 49 |
|
| 50 |
## ๐ฆ Available Models
|
| 51 |
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
|
| 55 |
-
|
|
| 56 |
-
| `
|
| 57 |
-
| `
|
| 58 |
-
| `
|
| 59 |
-
| `
|
| 60 |
-
| `Qwen-Image-
|
| 61 |
-
| `
|
| 62 |
-
| `
|
| 63 |
-
| `
|
| 64 |
-
| `
|
|
|
|
|
|
|
| 65 |
|
| 66 |
---
|
| 67 |
|
| 68 |
-
##
|
| 69 |
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
|
| 74 |
---
|
| 75 |
|
| 76 |
-
##
|
| 77 |
|
| 78 |
-
|
|
|
|
|
|
|
|
|
|
| 79 |
|
| 80 |
-
-
|
| 81 |
|
| 82 |
-
##
|
| 83 |
|
| 84 |
-
|
| 85 |
|
| 86 |
```bash
|
| 87 |
cd ComfyUI/custom_nodes
|
| 88 |
git clone https://github.com/ussoewwin/ComfyUI-HSWQ-Loader-and-Tools.git
|
| 89 |
```
|
| 90 |
|
| 91 |
-
|
|
|
|
|
|
|
|
|
|
| 92 |
|
| 93 |
---
|
| 94 |
|
|
|
|
| 25 |
|
| 26 |
# ControlNet Models (ConvRot INT8)
|
| 27 |
|
| 28 |
+
<p align="center">
|
| 29 |
+
<img src="https://raw.githubusercontent.com/ussoewwin/Hybrid-Sensitivity-Weighted-Quantization/main/icon.png" width="128">
|
| 30 |
+
</p>
|
| 31 |
+
|
| 32 |
High-fidelity **ConvRot INT8** quantized weights for multi-condition and dedicated ControlNet models across diverse generative architectures (**Illustrious-XL / SDXL**, **SDXL 1.0**, **Z-Image-Turbo**, **Qwen-Image / Qwen-Image-2512**, and **FLUX.1-dev**).
|
| 33 |
|
| 34 |
+
- **Technical Details & Quantization Framework:** [Hybrid-Sensitivity-Weighted-Quantization (HSWQ)](https://github.com/ussoewwin/Hybrid-Sensitivity-Weighted-Quantization)
|
| 35 |
+
- **ComfyUI Loader for ConvRot INT8:** [ComfyUI-HSWQ-Loader-and-Tools](https://github.com/ussoewwin/ComfyUI-HSWQ-Loader-and-Tools)
|
| 36 |
+
|
| 37 |
+
---
|
| 38 |
+
|
| 39 |
+
## โ ๏ธ Where each pack goes, and how it loads
|
| 40 |
+
|
| 41 |
+
**This repository mixes two runtime object types.** They use different loaders, and getting this wrong means the file silently fails to apply.
|
| 42 |
+
|
| 43 |
+
| group | the runtime object | the loader | the directory |
|
| 44 |
+
| :--- | :--- | :--- | :--- |
|
| 45 |
+
| **ControlNet** (SDXL anytest A/B, SDXL union-pro-max, Qwen-Image-2512 Union 2602, Qwen-Image Inpainting, FLUX.1-dev Union Pro 2.0) | a ControlNet | `ControlNetLoader` (stock) or the HSWQ ControlNet loader | `ComfyUI/models/controlnet/` |
|
| 46 |
+
| **Model patch** (Z-Image-Turbo Fun Controlnet Union/Tile, Qwen-Image DiffSynth canny/depth/inpaint) | a **model patch**, despite the "ControlNet" name | **`HSWQ Model Patch Loader`** (`HSWQModelPatchLoaderCustom`). The **stock `ModelPatchLoader` cannot read the INT8 layout** | `ComfyUI/models/model_patches/` |
|
| 47 |
+
|
| 48 |
+
The model patches are applied by the dedicated apply nodes (`ZImageFunControlnet`, `QwenImageDiffsynthControlnet`), not by `Apply ControlNet`.
|
| 49 |
+
|
| 50 |
---
|
| 51 |
|
| 52 |
## ๐ Model Overview
|
| 53 |
|
| 54 |
+
This repository hosts high-quality **ConvRot INT8** quantized weights for all-in-one ControlNet Union and dedicated condition-specific ControlNet models. Orthogonal Hadamard rotation is applied to each 2D weight before per-channel INT8 quantization, which suppresses activation outlier distortion and cuts the disk footprint roughly in half while preserving structural control fidelity:
|
| 55 |
|
| 56 |
- **CN-anytest4_illustrious2 (Variants A & B)**: Multi-purpose all-in-one Anytest v4 ControlNet models fine-tuned for **Illustrious-XL / SDXL** by **2vXpSwA7**, offering high-precision anime and illustration structure guiding.
|
| 57 |
- **controlnet-union-pro-max-sdxl-1.0**: The comprehensive all-in-one ControlNet Union model for **SDXL 1.0** by **xinsir**, supporting 10+ control conditions in a single compact file.
|
| 58 |
+
- **Z-Image-Turbo-Fun-Controlnet-Union-2.1-lite-2602-8steps**: Multi-condition ControlNet Union model (lite architecture, 8-step distilled) for the **Z-Image-Turbo** architecture by **alibaba-pai**, providing lightweight and fast structure control (Canny, Depth, Pose, Color, etc.). **Model patch.**
|
| 59 |
+
- **Z-Image-Turbo-Fun-Controlnet-Tile-2.1-lite-2601-8steps**: Dedicated high-resolution Tile and upscaling ControlNet model (lite architecture, 8-step distilled) for the **Z-Image-Turbo** architecture by **alibaba-pai**, optimized for super-resolution detail enhancement. **Model patch.**
|
| 60 |
- **Qwen-Image-2512-Fun-Controlnet-Union-2602**: Multi-condition ControlNet Union model (5 layer blocks) for the **Qwen-Image-2512** architecture.
|
| 61 |
- **Qwen-Image-ControlNet-Inpainting**: Dedicated inpainting and editing ControlNet model for the **Qwen-Image** architecture.
|
| 62 |
+
- **Qwen-Image DiffSynth ControlNet Suite** (canny / depth / inpaint): Dedicated single-condition ControlNet models for the **Qwen-Image** architecture developed by **DiffSynth-Studio**. **Model patches.**
|
|
|
|
|
|
|
|
|
|
| 63 |
- **FLUX.1-dev-ControlNet-Union-Pro-2.0**: Next-generation unified 7-in-1 ControlNet for the **FLUX.1-dev** architecture by **Shakker Labs**.
|
| 64 |
|
| 65 |
---
|
| 66 |
|
| 67 |
## ๐ฆ Available Models
|
| 68 |
|
| 69 |
+
Sizes are decimal (1 GB = 10\^9 bytes) and exact; the `SHA256` column is the pack's own hash, so a download can be verified against it.
|
| 70 |
+
|
| 71 |
+
| Filename | Base Architecture | Base Model / Author | Supported Conditions | Quantization | File Size | SHA256 | License |
|
| 72 |
+
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
|
| 73 |
+
| `CN-anytest4_illustrious2_A_hswq_convrot_int8.safetensors` | Illustrious-XL / SDXL | [2vXpSwA7/iroiro-lora (Anytest v4 Variant A)](https://huggingface.co/2vXpSwA7/iroiro-lora) | Multi-condition (Canny, Lineart, Depth, Pose, Structure) | ConvRot INT8 | 1425.1 MB | `be9ff80b2f4895684c7114afff12220394f838179b5c82eadd8ebc46f5d82c25` | Fair AI / OpenRAIL++-M |
|
| 74 |
+
| `CN-anytest4_illustrious2_B_hswq_convrot_int8.safetensors` | Illustrious-XL / SDXL | [2vXpSwA7/iroiro-lora (Anytest v4 Variant B)](https://huggingface.co/2vXpSwA7/iroiro-lora) | Multi-condition (Canny, Lineart, Depth, Pose, Structure) | ConvRot INT8 | 1423.5 MB | `7e34dc3cc3334afa2b9fe5461131a1f61e289a83b1a11413d6c687e94b9dde6e` | Fair AI / OpenRAIL++-M |
|
| 75 |
+
| `controlnet-union-pro-max-sdxl-1.0_hswq_convrot_int8.safetensors` | SDXL 1.0 | [xinsir/controlnet-union-sdxl-1.0](https://huggingface.co/xinsir/controlnet-union-sdxl-1.0) | OpenPose, Depth, Canny, Lineart, Anime Lineart, Scribble, Soft Edge, Normal, Segment, Tile, Inpaint | ConvRot INT8 | 1435.3 MB | `dcdf6f2037205c4a40b1f8863a91827ca3a5b55ec3261e07bb13d2dc855b47fb` | OpenRAIL++-M / Apache-2.0 |
|
| 76 |
+
| `FLUX.1-dev-ControlNet-Union-Pro-2.0_hswq_convrot_int8.safetensors` | FLUX.1-dev | [Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0) | Canny, Depth, Pose, Blur, Gray, Soft Edge, Low Quality | ConvRot INT8 | 2710.6 MB | `fbcba3c76a221fba461c83680f682d3ff7aa813ef554c42e3a5449b981090c46` | Other / Non-Commercial |
|
| 77 |
+
| `Qwen-Image-2512-Fun-Controlnet-Union-2602_hswq_convrot_int8.safetensors` | Qwen-Image-2512 | [alibaba-pai/Qwen-Image-2512-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/Qwen-Image-2512-Fun-Controlnet-Union) | Canny, HED, Depth, Pose, MLSD, Scribble, Gray, Inpaint | ConvRot INT8 | 1758.7 MB | `5aec39a53f583bade3b7db18f38ed5a01905a0610c2ad2f9f792802178596bc4` | Apache-2.0 |
|
| 78 |
+
| `Qwen-Image-ControlNet-Inpainting_hswq_convrot_int8.safetensors` | Qwen-Image | [alibaba-pai/Qwen-Image-ControlNet-Inpainting](https://huggingface.co/alibaba-pai/Qwen-Image-ControlNet-Inpainting) | Inpainting, Image Editing | ConvRot INT8 | 2120.3 MB | `38f0ab7495029517d4b0bb4187cedcbdfa090a29f0bb50dfa5733f5076a0c451` | Apache-2.0 |
|
| 79 |
+
| `qwen_image_canny_diffsynth_controlnet_hswq_convrot_int8.safetensors` | Qwen-Image | [modelscope/DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) | Canny Edge Detection & Structural Guidance | ConvRot INT8 | 1135.9 MB | `c2899255252b81fd64ef2fae0fbfd86e1496ba04cf10cc95fe45679120f1c996` | Apache-2.0 |
|
| 80 |
+
| `qwen_image_depth_diffsynth_controlnet_hswq_convrot_int8.safetensors` | Qwen-Image | [modelscope/DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) | Geometric Depth Map Guidance | ConvRot INT8 | 1135.9 MB | `7b82a9360169fa99524c03bcf913ce63713930174ad1fc332e553e5646bad44c` | Apache-2.0 |
|
| 81 |
+
| `qwen_image_inpaint_diffsynth_controlnet_hswq_convrot_int8.safetensors` | Qwen-Image | [modelscope/DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) | Masked Inpainting & Localized Editing | ConvRot INT8 | 1135.9 MB | `471b6affc7fcedafb9940fafd6354dd6eb5fac5705e647029c90598c407c82d3` | Apache-2.0 |
|
| 82 |
+
| `Z-Image-Turbo-Fun-Controlnet-Union-2.1-lite-2602-8steps_hswq_convrot_int8.safetensors` | Z-Image-Turbo | [alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1](https://huggingface.co/alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1) | Multi-condition (Canny, Depth, Pose, Color, Scribble, Inpaint) | ConvRot INT8 | 1009.7 MB | `1dd12b2d028b12df6b2a4f93856f8ab161c1edc05125e3b1c987a1694e99dba4` | Apache-2.0 |
|
| 83 |
+
| `Z-Image-Turbo-Fun-Controlnet-Tile-2.1-lite-2601-8steps_hswq_convrot_int8.safetensors` | Z-Image-Turbo | [alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1](https://huggingface.co/alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1) | Tile, Super-Resolution, Detail Enhancing | ConvRot INT8 | 1009.7 MB | `2998f83ef65903b4b6b22fb7ac270ed66f9a2443220563aa43db4188024763a8` | Apache-2.0 |
|
| 84 |
|
| 85 |
---
|
| 86 |
|
| 87 |
+
## ๐ Measured fidelity
|
| 88 |
|
| 89 |
+
Each pack's protection set was chosen by a **measured** aggregate contribution on the model's own output, over 8 seeds, and the acceptance was measured separately on a fixed 256-item set (seed 42) through the same loader the runtime uses. "Gate" is `cos.mean >= 0.99` and `cos.min >= 0.98`.
|
| 90 |
+
|
| 91 |
+
| Pack | the score (mean / min) | the gate | the note |
|
| 92 |
+
| :--- | :--- | :--- | :--- |
|
| 93 |
+
| `CN-anytest4_illustrious2_A` | 0.999626 (forward, relMSE 1.11e-03) | PASS | trajectory-impact selection, 36 protected |
|
| 94 |
+
| `CN-anytest4_illustrious2_B` | 0.992958 / 0.948279 (trajectory) | PASS | trajectory gate, 35 protected |
|
| 95 |
+
| `controlnet-union-pro-max-sdxl-1.0` | 0.999961 | PASS | curve flat, smallest satisfying pack taken (41 protected) |
|
| 96 |
+
| `Qwen-Image-2512-Fun-Controlnet-Union-2602` | 0.999158 / 0.998215 | PASS | floor case: all 77 layers INT8, no protection |
|
| 97 |
+
| `Qwen-Image-ControlNet-Inpainting` | 0.999545 / 0.998887 | PASS | floor case: all 95 layers INT8, no protection |
|
| 98 |
+
| `qwen_image_canny_diffsynth` | 0.9999924905 / 0.9999924096 | PASS | one protected layer (`img_in`), 598x above the runner-up |
|
| 99 |
+
| `qwen_image_depth_diffsynth` | 0.9999924109 / 0.9999923262 | PASS | one protected layer (`img_in`), 376x above the runner-up |
|
| 100 |
+
| `qwen_image_inpaint_diffsynth` | 0.9999921236 / 0.9999919542 | PASS | one protected layer (`img_in`), 147x above the runner-up |
|
| 101 |
+
| `Z-Image-Turbo-Fun-Controlnet-Union` | 0.999862 / 0.999491 (trajectory) | PASS | all 38 layers INT8, no protection |
|
| 102 |
+
| `Z-Image-Turbo-Fun-Controlnet-Tile` | 0.999956 / 0.999813 (trajectory) | PASS | all 38 layers INT8, no protection |
|
| 103 |
+
| `FLUX.1-dev-ControlNet-Union-Pro-2.0` | 0.999771 / 0.999488 | PASS | 12 protected; the all-INT8 state measured 0.999622 / 0.999020 |
|
| 104 |
+
|
| 105 |
+
The per-checkpoint reasoning and the full measured curves are in the HSWQ repository's `md/` guides.
|
| 106 |
|
| 107 |
---
|
| 108 |
|
| 109 |
+
## ๐ ๏ธ Key Features
|
| 110 |
|
| 111 |
+
- **All-in-One & Dedicated Condition Control**: Unified multi-condition architectures alongside specialized single-condition models (Canny, Depth, Tile, Inpainting) for fine-grained workflow flexibility.
|
| 112 |
+
- **ConvRot INT8 Precision**: Orthogonal Hadamard rotations redistribute channel-wise outlier spikes across the input dimension before per-channel INT8 quantization, preventing quantization error buildup in deep control layers.
|
| 113 |
+
- **Measured, not assumed, protection**: Where a layer's measured contribution is large it stays FP16 (kept tensors are stored FP16, not FP32). Several packs here need no protection at all; the measured curves say so.
|
| 114 |
+
- **Footprint**: roughly half the unquantized checkpoint, with the INT8 execution path (Tensor Core INT8 matmul) as the additional runtime gain.
|
| 115 |
|
| 116 |
+
---
|
| 117 |
|
| 118 |
+
## ๐ Usage in ComfyUI
|
| 119 |
|
| 120 |
+
Install the loader extension:
|
| 121 |
|
| 122 |
```bash
|
| 123 |
cd ComfyUI/custom_nodes
|
| 124 |
git clone https://github.com/ussoewwin/ComfyUI-HSWQ-Loader-and-Tools.git
|
| 125 |
```
|
| 126 |
|
| 127 |
+
Then place each file in the directory its type requires (see the table at the top):
|
| 128 |
+
|
| 129 |
+
- **ControlNet** files go in `ComfyUI/models/controlnet/` and load with `ControlNetLoader`.
|
| 130 |
+
- **Model patch** files go in `ComfyUI/models/model_patches/` and load with **`HSWQ Model Patch Loader`** (the stock `ModelPatchLoader` reads the INT8 tensors as plain weights), then apply with `ZImageFunControlnet` / `QwenImageDiffsynthControlnet`.
|
| 131 |
|
| 132 |
---
|
| 133 |
|