| --- |
| license: other |
| license_name: sam-license |
| license_link: https://github.com/facebookresearch/segment-anything/blob/main/LICENSE |
| base_model: |
| - facebook/sam3.1 |
| tags: |
| - sam |
| - sam3 |
| - sam3.1 |
| - segment-anything |
| - segmentation |
| - video-segmentation |
| - multiplex |
| - quantized |
| - int8 |
| - convrot |
| - comfyui |
| pipeline_tag: image-segmentation |
| --- |
| |
| # SAM 3.1 Multiplex (ConvRot INT8) |
|
|
| High-fidelity **Native ConvRot INT8** quantized weights for the **SAM 3.1 Multiplex** (Segment Anything Model 3.1) architecture. |
|
|
| --- |
|
|
| ## 🌟 Model Overview |
|
|
| This repository provides **Native ConvRot INT8** quantized weights for the **SAM 3.1 Multiplex** foundation model. By applying orthogonal Hadamard rotation before per-channel INT8 quantization, activation outliers are evenly distributed across matrix dimensions, drastically suppressing quantization noise while halving VRAM and storage footprint: |
|
|
| - **Architecture:** SAM 3.1 Multiplex (Vision Backbone + FPN Neck + Decoupled Memory Attention + 16-Object Multiplex Mask Decoder) |
| - **Base Checkpoint:** [Comfy-Org/sam3.1](https://huggingface.co/Comfy-Org/sam3.1) / [facebook/sam3.1](https://huggingface.co/facebook/sam3.1) (`sam3.1_multiplex_fp16.safetensors`) |
| - **Native ComfyUI Support:** Features native `int8_tensorwise` and `comfy_quant` metadata stamps, loading seamlessly with ComfyUI's standard `UNetLoader` / `DiffusionModelLoader` without requiring external custom quantization nodes. |
|
|
| --- |
|
|
| ## 📦 Model Details & Comparison |
|
|
| | Filename | Base Architecture | Precision / Format | File Size | Memory Footprint | Native ComfyUI Support | |
| | :--- | :--- | :--- | :--- | :--- | :--- | |
| | `sam3.1_multiplex_convrot_int8.safetensors` | SAM 3.1 Multiplex | **ConvRot INT8** (`int8_tensorwise`) | **~0.84 GB** (899 MB) | **~50% VRAM Reduction** | ✅ Supported natively (`UNetLoader`) | |
| | `sam3.1_multiplex_fp16.safetensors` (Reference) | SAM 3.1 Multiplex | FP16 | ~1.63 GB (1,746 MB) | Baseline | ✅ Supported natively | |
|
|
| ### Quantization Breakdown |
| - **ConvRot INT8 Layers:** 647 Linear projection layers (Hadamard rotation `groupsize=256` + per-out-channel scale) |
| - **Plain INT8 Fallback Layers:** 4 layers (non-power-of-4 channel dimensions safely quantized with row-wise scaling) |
| - **Preserved High-Precision Tensors:** 1,183 tensors (LayerNorm, Embedding, Biases, and 4D Conv2d weights preserved in full float precision for boundary and spatial accuracy) |
|
|
| --- |
|
|
| ## 🛠️ Key Highlights |
|
|
| - **Hadamard Orthogonal Rotation (ConvRot):** Pre-rotates weight matrices ($W_{rot} = W \cdot H^T$) along the input channel axis using normalized Sylvester Hadamard blocks, eliminating channel-wise outlier spikes and preventing mask degradation. |
| - **Full Multiplex Support:** Preserves the complete 16-object simultaneous tracking, interactive segmentation, and decoupled cross-attention capabilities of the SAM 3.1 architecture. |
| - **50% Disk & VRAM Savings:** Reduces weight storage from 1.63 GB down to 0.84 GB, enabling lightweight deployment on consumer hardware and multi-model video segmentation workflows. |
| |
| --- |
| |
| ## 🚀 Usage in ComfyUI |
| |
| ### Model Placement |
| Download `sam3.1_multiplex_convrot_int8.safetensors` and place it in your ComfyUI models directory: |
| ``` |
| ComfyUI/models/unet/sam3.1_multiplex_convrot_int8.safetensors |
| ``` |
|
|
| ### Loading |
| 1. Load the model using the standard **`UNetLoader`** (or **`Load Diffusion Model`**) node in ComfyUI. |
| 2. Connect the loaded model output directly into your SAM3 image / video segmentation workflow nodes. |
| 3. ComfyUI automatically detects the `comfy_quant` header metadata and executes optimized INT8 Tensor Core operations natively. |
|
|
| --- |
|
|
| ## 📜 Credits & License |
|
|
| - **Base Architecture & Model:** [Comfy-Org/sam3.1](https://huggingface.co/Comfy-Org/sam3.1) / [facebook/sam3.1](https://huggingface.co/facebook/sam3.1) (Meta SAM / Segment Anything Team) |
| - **Quantization Framework:** [HSWQ / ConvRot INT8 Pipeline](https://github.com/ussoewwin/Hybrid-Sensitivity-Weighted-Quantization) |
| - **License:** [SAM License (Meta SAM License)](https://github.com/facebookresearch/segment-anything/blob/main/LICENSE) |
|
|
