README.md

---
license: other
license_name: openmdw-1.1
license_link: https://openmdw.ai/license/1-1/
tags:
  - text-to-video
  - image-to-video
  - quantized
  - int8
  - int4
  - comfyui
  - cosmos3
---

# Cosmos3 quantized transformers

Weight-only quantized transformers of the NVIDIA Cosmos3 models, for use with
[ComfyUI-Cosmos3](https://github.com/RyukoMatoiFan/ComfyUI-Cosmos3). Each file is one quantized
transformer; take the VAE, tokenizer(s) and `config.json` from the matching official
`nvidia/Cosmos3-*` repo.

Two formats:

- **int8** — weight-only INT8, per-output-channel scale + a group-wise Hadamard rotation (ConvRot).
- **int4** — MLP weights INT4 (GPTQ-calibrated with a ConvRot Hadamard rotation, packed as AWQ
  W4A16); attention kept INT8; the excluded layers (below) stay bf16. Larger quantization error than
  int8 (at a fixed seed the output diverges from bf16 more than int8 does); about a third of the bf16
  size.

Both are **weight-only**: activations stay bf16, so this lowers memory/download size, not compute
speed. int4 runs at about bf16 speed — the per-forward dequantize and un-rotate add a little.

## Files and sizes

| Model | bf16 | int8 | int4 |
|-------|------|------|------|
| `Cosmos3-Nano` | 30 GB | **16.5 GB** | **12.4 GB** |
| `Cosmos3-Super` | 128 GB | **65.7 GB** | **46.8 GB** |
| `Cosmos3-Super-Image2Video` | 128 GB | **65.6 GB** | **46.7 GB** |
| `Cosmos3-Super-Image2Video-4Step` | 128 GB | **65.6 GB** | **46.7 GB** |
| `Cosmos3-Edge` | 6.7 GB | **3.9 GB** | **3.0 GB** |

File names: `Cosmos3-<name>-int8-convrot.safetensors` and `Cosmos3-<name>-int4-convrot.safetensors`.
int4 and int8 are provided for every model.

**Prompting note (Edge):** `Cosmos3-Edge` is trained on JSON-structured prompts and is less robust to
plain text than the larger Nano/Super. Plain text usually works, but on some detailed scenes (notably
reflective surfaces) it can produce flare/pulsation artifacts; wrapping the text as
`{"temporal_caption": "<your prompt>"}` avoids them. This is a base-model property, not a quantization
effect (it shows in bf16 too).

## Runtime footprint

With ComfyUI's dynamic VRAM the transformer is streamed from host RAM, so the GPU holds only the
activations. Measured at 832×480, 93 frames, at the minimum VRAM budget (maximum streaming):

| Model | Min VRAM | RAM (bf16 / int8 / int4) |
|-------|----------|---------------------------|
| `Cosmos3-Edge` | ≈6 GB | 14 / 7 / 7 GB |
| `Cosmos3-Nano` | ≈7 GB | 58 / 21 / 20 GB |
| `Cosmos3-Super` (t2v & i2v) | ≈8–9 GB | 240 / 67 / 63 GB |

Min VRAM is the activation floor (set by resolution × frame count, not the weight format). RAM is the
peak host memory — larger than the file on disk (staging + overhead), and bf16 peaks near twice the
weight size. RAM and VRAM trade off: giving the GPU more VRAM holds more weights on-card and lowers the
RAM figure. The Super-family int4 checkpoints fit a 64 GB host (≈63 GB); int8 needs a little more
(≈67 GB).

## Usage

1. Download the official `nvidia/Cosmos3-<name>` into `ComfyUI/models/cosmos3/<name>/`.
2. In its `transformer/` folder, delete the bf16 shards and `*.index.json`, then put the quantized
   file there renamed to `diffusion_pytorch_model.safetensors`. Keep the official `config.json`,
   `vae/`, `text_tokenizer/`, `sound_tokenizer/`.
3. Load with the Cosmos3 Loader (`weight_dtype = default`); the loader reads the format from the
   checkpoint metadata. Requires comfy-kitchen (int8 from ComfyUI >= 0.27) and the latest
   ComfyUI-Cosmos3 (int4 needs the ConvRot-aware loader).

## How these were quantized

Reproducible in method, not bit-for-bit (the calibration set and RNG vary per run). Both formats keep
the same **escape set in bf16**: `proj_in`, `proj_out`, `time_embedder`, `audio_proj`,
`modality_embed`, the embeddings, and all norms and biases.

### int8 — every linear

Weight-only symmetric INT8, per-output-channel scale (`weight_scale`, float32, `[out, 1]`), with a
group-wise Hadamard rotation (ConvRot, **group 256**) applied before quantization and undone at load.
comfy_quant tag `int8_tensorwise`, `convrot=true`. No calibration: at 8-bit the per-channel scale and
the rotation keep the round-to-nearest error small — error feedback (GPTQ) is only needed at int4.
Produced with
[convert_to_quant](https://github.com/silveroxides/convert_to_quant):

    ctq -i transformer_bf16.safetensors -o out_int8_convrot.safetensors \
        --int8 --scaling_mode row --simple --convrot --convrot-group-size 256 \
        --comfy_quant --save-quant-metadata --cosmos3 --device cuda --low-memory

### int4 — MLP → int4, attention → int8

Round-to-nearest INT4 — even with the ConvRot rotation — leaves visible artifacts on these models, so
the MLP path uses **GPTQ error compensation on real activations**. Steps:

1. **Capture activations.** Run genuine denoising at 832×480, 93 frames with the model's normal
   sampler (t2v / base-i2v: 35 steps, cfg 6, `uni_pc_bh2`; 4-step model: 4 steps, cfg 1, `euler`) over
   ≈4 prompts, with a forward pre-hook on every target linear. Keep a reservoir of up to **4096** rows
   per layer (random replacement beyond that). The understanding tower sees the text prefill; the
   generation tower sees every denoising step.
2. **Rotate (ConvRot).** Apply a Sylvester block-Hadamard — symmetric, orthonormal, **group 32** — to
   both the weight and the captured activations of each MLP linear.
3. **GPTQ.** Quantize the rotated weight to symmetric INT4 (codes −8…7, per-(row, group-32) scale).
   Hessian `H = XᵀX · 2/N` from the rotated activations; diagonal damping raised through
   `{0.01, 0.03, 0.1, 0.3, 1, 3} × mean(diag)` until the Cholesky factors; columns processed in blocks
   of **128** with per-column error feedback into the not-yet-quantized columns; plain round-to-nearest
   only if damping never succeeds.
4. **Attention → int8** (row-wise, `[out, 1]` scale). INT4 on attention produces visible artifacts.
5. **Pack** the INT4 codes into comfy-kitchen's **AWQ W4A16** layout; the loader un-rotates each group
   at dequant.

Layer counts follow the checkpoint's `config.json` — e.g. `Cosmos3-Super-Image2Video` packs 384 MLP
linears (INT4) + 512 attention linears (INT8).

**Execution.** Both formats run as an explicit dequantize-then-bf16 matmul (unpack the 4-bit weights
to bf16, un-rotate, multiply in bf16). This is weight-only regardless: there is no int4-weight ×
bf16-activation tensor-core op on any GPU (Hopper or Blackwell — Blackwell's FP4 cores are for NVFP4,
both operands 4-bit, not this W4A16 layout), so a W4A16 kernel would dequantize internally too, with no
compute speedup either way. We dequantize explicitly instead of calling comfy-kitchen's
`gemv_awq_w4a16`, which is non-deterministic (atomic accumulation jitters the video frame-to-frame) and
numerically off on these shapes.

## License

Derived from NVIDIA Cosmos3 checkpoints; the [OpenMDW-1.1](https://openmdw.ai/license/1-1/) license
applies (same as the upstream `nvidia/Cosmos3-*` models).