| --- |
| license: other |
| tags: |
| - clip |
| - text-encoder |
| - t5 |
| - openclip |
| - text-to-image |
| - image-to-image |
| - quantized |
| - int8 |
| - convrot |
| - comfyui |
| pipeline_tag: feature-extraction |
| --- |
| |
| # Text Encoder & CLIP Models (ConvRot INT8) |
|
|
| High-fidelity **Native ConvRot INT8** quantized weights for primary Text Encoders and Vision-Language CLIP models across state-of-the-art generative diffusion architectures (**SDXL**, **FLUX.1**, **SD 1.5 / SD 2.1**, **SD3 / SD3.5**, and **Next-Gen Diffusion** pipelines). |
|
|
| --- |
|
|
| ## 🌟 Model Overview |
|
|
| This repository hosts high-quality **Native ConvRot INT8** quantized weights for essential text encoders and CLIP vision-language backbones. By applying orthogonal Hadamard rotation ($W_{rot} = W \cdot H^T$) prior to per-channel symmetric INT8 quantization, activation outlier spikes in transformer feed-forward and projection layers are redistributed uniformly across channels. This eliminates quantization-induced semantic drift and cuts VRAM and storage footprint by ~50% while preserving precise prompt conditioning fidelity: |
| |
| - **CLIP-SAE-ViT-L-14-FP32**: Primary ViT-L/14 text encoder with high-precision FP32/SAE baseline representation, quantized to native ConvRot INT8 for ultra-fast, lightweight prompt encoding across SD 1.5, SDXL, and FLUX workflows. |
| - **CLIP-ViT-bigG-14-laion2B-39B-b160k**: The massive OpenCLIP ViT-bigG/14 text encoder (laion2B) for SDXL and multi-encoder generative pipelines, reducing disk and VRAM demands from 3.69 GB down to 1.85 GB. |
| - **flan_t5_xxl**: The 11B-parameter dense Flan-T5 XXL text encoder for next-generation text-to-image and multimodal architectures, compressed from ~22.4 GB down to ~11.28 GB to allow inference on consumer GPUs. |
| |
| --- |
| |
| ## 📦 Available Models |
| |
| | Filename | Base Architecture | Base Model / Upstream | Quantization | File Size | Baseline Size | VRAM Savings | |
| | :--- | :--- | :--- | :--- | :--- | :--- | :--- | |
| | `CLIP-SAE-ViT-L-14-FP32_native_convrot_int8.safetensors` | OpenAI CLIP ViT-L/14 | [openai/clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14) | ConvRot INT8 (`int8_tensorwise`) | **~0.43 GB** (412 MB) | ~0.93 GB | **~54% Reduction** | |
| | `CLIP-ViT-bigG-14-laion2B-39B-b160k_convrot_int8.safetensors` | OpenCLIP ViT-bigG/14 | [laion/CLIP-ViT-bigG-14-laion2B-39B-b160k](https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k) | ConvRot INT8 (`int8_tensorwise`) | **~1.85 GB** (1,765 MB) | ~3.69 GB | **~50% Reduction** | |
| | `flan_t5_xxl_convrot_int8.safetensors` | Google Flan-T5 XXL (11B) | [google/flan-t5-xxl](https://huggingface.co/google/flan-t5-xxl) | ConvRot INT8 (`int8_tensorwise`) | **~11.28 GB** (10.50 GB) | ~22.40 GB | **~50% Reduction** | |
|
|
| --- |
|
|
| ## 🛠️ Key Features |
|
|
| - **Orthogonal Hadamard Pre-Rotation (ConvRot):** Weight matrices are pre-rotated group-wise along the input feature dimension ($W_{rot} = W \cdot H^T$) using power-of-4 normalized Hadamard blocks (`groupsize=256`), eliminating catastrophic dynamic range compression caused by outlier channels. |
| - **Native ComfyUI Compatibility:** Encodes standard `comfy_quant` header metadata (`{"format": "int8_tensorwise", "convrot": true, "convrot_groupsize": 256}`) and per-channel scale tensors, loading automatically in ComfyUI with standard `CLIPLoader`, `DualCLIPLoader`, or `TripleCLIPLoader` without custom nodes. |
| - **Full Precision Preservation for Critical Weights:** Non-2D weights (token embeddings, position embeddings, layer normalizations, biases) are maintained in native floating-point precision to guarantee prompt parsing integrity and boundary consistency. |
| - **Hardware Acceleration:** Fully compatible with Tensor Core INT8 matrix multiplication across modern GPUs (NVIDIA Turing, Ampere, Ada Lovelace, Blackwell). |
|
|
| --- |
|
|
| ## 🚀 Usage in ComfyUI |
|
|
| ### Model Placement |
| Download the desired `.safetensors` files and place them into your ComfyUI text encoder directory: |
| ``` |
| ComfyUI/models/clip/ |
| ``` |
|
|
| ### Loading |
| Load directly via standard ComfyUI CLIP loader nodes: |
| - **Single CLIP:** `Load CLIP` / `CLIPLoader` $\rightarrow$ select `<model_name>_convrot_int8.safetensors`. |
| - **Dual CLIP (SDXL):** `DualCLIPLoader` $\rightarrow$ set `clip_name1` to `clip_l` (or `CLIP-SAE-ViT-L-14-FP32_native_convrot_int8.safetensors`) and `clip_name2` to `CLIP-ViT-bigG-14-laion2B-39B-b160k_convrot_int8.safetensors` (type `sdxl`). |
| - **Triple CLIP (SD3 / FLUX):** `TripleCLIPLoader` $\rightarrow$ assign CLIP-L, CLIP-G, and T5-XXL ConvRot INT8 weights accordingly. |
|
|
| ComfyUI automatically recognizes the `comfy_quant` stamp and dispatches optimized INT8 execution kernels seamlessly. |
|
|
| --- |
|
|
| ## 📜 Credits & License |
|
|
| - **OpenAI CLIP ViT-L/14:** [openai/clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14) (MIT License) |
| - **OpenCLIP ViT-bigG/14:** [LAION / OpenCLIP](https://github.com/mlfoundations/open_clip) (MIT / Apache-2.0 License) |
| - **Google Flan-T5 XXL:** [google/flan-t5-xxl](https://huggingface.co/google/flan-t5-xxl) (Apache-2.0 License) |
| - **Quantization Pipeline:** [HSWQ / ConvRot INT8 Suite](https://github.com/ussoewwin/Hybrid-Sensitivity-Weighted-Quantization) |
| - **Base Licenses:** Please adhere to the individual upstream licenses of the respective foundation models. |
|
|
