--- license: other license_name: openmdw-1.1 license_link: https://openmdw.ai/license/1-1/ base_model: nvidia/Cosmos3-Super library_name: diffusers pipeline_tag: image-to-video tags: - nvidia - cosmos3 - world-model - omnimodel - diffusion - text-to-image - text-to-video - image-to-video - quantized - modelopt - fp8 - blackwell --- # Cosmos3-Super — Weight-Only FP8 (NVIDIA ModelOpt) Weight-only quantization of the `Cosmos3OmniTransformer` from NVIDIA's [`nvidia/Cosmos3-Super`](https://huggingface.co/nvidia/Cosmos3-Super) — the 64B omnimodal Cosmos 3 world model (text-to-image, text-to-video, image-to-video, optional synchronized sound). Produced with [NVIDIA TensorRT Model Optimizer (ModelOpt)](https://github.com/NVIDIA/TensorRT-Model-Optimizer) on a single 96 GB workstation GPU, via a streaming method that never materializes the ~128 GB bf16 model (method scripts included). > **Only the transformer is quantized.** The VAEs and tokenizers are the original > bf16 components, bundled so the repo is self-contained. Loading requires the > bundled `load_cosmos3_modelopt.py` (see *How to use*). ## Variants & measured performance Measured on an RTX 6000 Pro Blackwell (96 GB), 1024×1024 single-frame render, 50 steps. Drop-in loading of these repos performs identically to the in-memory quantization path they were validated against. | Build | Bits (weights) | Repo size | Resident VRAM | s/it (1024² still) | | -------------------- | --------------------- | --------- | ------------- | ------------------ | | FP8 **(this repo)** | 8-bit (E4M3) | ~64 GB | ~67 GB (meas.)| **~1.2** | | [NVFP4 (sibling)](https://huggingface.co/prometheusAIR/Cosmos3-Super-nvfp4) | 4-bit (E2M1 + scales) | ~36 GB | ~43 GB (meas.) | ~4.6 | **Pick FP8 if it fits** — in this serving path it is both higher fidelity *and* ~4× faster, because FP8 dequant is a single cheap scale on a native float8 tensor, while NVFP4 dequant must unpack two 4-bit values per byte and apply two-level block scales in PyTorch. **Pick NVFP4 for footprint** (it brings the model into ~48 GB-card territory for stills). Note this is dequant-on-the-fly: quantization here buys **memory, not speed** — NVFP4's hardware FP4 tensor-core advantage only materializes in engines with FP4 GEMM kernels (TRT-LLM/vLLM territory), not in diffusers. Layers kept in **bf16** (not quantized): embeddings, norms, the reasoner head, in/out projections, time/modality adapters, audio adapter. The 64 transformer blocks' attention + MLP linears (incl. MoE experts) are quantized. ## Status - ✅ **Drop-in loading verified** end to end (load → render → performance parity with the in-memory method) on Blackwell (sm_120), **via the bundled loader**. - ✅ `modelopt_state.pth` is part of the checkpoint and is **required** — it restores the quantized module structure at load. Do not delete it. - ⚠️ The loader (`load_cosmos3_modelopt.py`) is **required**, not optional. The current diffusers/accelerate/modelopt combination cannot materialize a pre-quantized ModelOpt checkpoint unaided; the loader applies three small, source-verified workarounds (parameter materialization for packed weights, payload-dtype restoration for FP8, and weight-only quantizer enforcement) plus the validated bf16 dtype normalization. ModelOpt marks this path experimental; expect the loader to become unnecessary as upstream catches up. - ❌ **vLLM-Omni:** not a working path as of 0.22.0. This is an upstream roadmap gap, not a defect of this checkpoint: vLLM-Omni's ModelOpt integration is currently wired for LLMs only, and ModelOpt-quantized diffusion support is an open RFC ([#2709](https://github.com/vllm-project/vllm-omni/issues/2709), [#1959](https://github.com/vllm-project/vllm-omni/issues/1959)). - ❌ **ComfyUI:** no known node support for this ModelOpt layout (the NF4 build linked below has community nodes; this one does not). - Validated only on Blackwell. FP8 on Hopper/Ada is plausible but unverified here. ## How to use Requires a `diffusers` build with Cosmos 3 support (currently from source) plus `modelopt` and `accelerate`. Pin to the verified versions for guaranteed reproducibility (newer versions may also work, but this code path moves fast): ```bash pip install "git+https://github.com/huggingface/diffusers.git@2c7efb95349296cf6bcce981ea036275a82a94df" pip install accelerate "nvidia-modelopt==0.44.0" ``` ```python from load_cosmos3_modelopt import load_pipe # bundled in this repo from diffusers import UniPCMultistepScheduler pipe = load_pipe("prometheusAIR/Cosmos3-Super-fp8") # or a local path pipe.scheduler = UniPCMultistepScheduler.from_config( pipe.scheduler.config, flow_shift=3.0 # NVIDIA's text-to-image setting; use 5.0 for image-to-video ) # Single image -- pass parameters EXPLICITLY (see warning below): r = pipe("a weathered lighthouse on a cliff at golden hour, photoreal, 50mm", height=1024, width=1024, num_frames=1, num_inference_steps=50, guidance_scale=4.0) r.video[0].save("out.png") # .video is the list of PIL frames; [0] is the image # Video (~2 s): frame counts of the form 4n+1 map cleanly to the VAE's 4x # temporal compression; 24 fps is the native rate and conditions the model. r = pipe("The lighthouse beam sweeps slowly across the water. Static camera.", height=704, width=1280, num_frames=49, fps=24.0, num_inference_steps=35, guidance_scale=6.0) ``` These still-image settings (1024², 50 steps, guidance 4.0, `flow_shift=3.0`, `result.video[0]`) match NVIDIA's first-party Cosmos3 text-to-image reference. > ⚠️ **A bare `pipe(prompt)` call renders a 189-frame 720×1280 video** (~8 s at > 24 fps) — that is the pipeline's built-in default, not a still. It takes ~40× > the compute of a single frame and is the most common reason this model > "seems slow." Always pass `num_frames`/`height`/`width` explicitly. Cosmos 3 expects a dense structured-JSON prompt for best quality; plain prompts work but render softer. See NVIDIA's prompt-upsampling docs. **Reproducing from scratch:** `quantize_cosmos3_super_streaming.py` (included) streams the bf16 shards directly into compressed FP8/NVFP4 form (peak memory ≈ the compressed footprint, so a single 96 GB card suffices), and `repackage_for_hf.py` emits this repo's round-trippable layout via `save_pretrained` + `enable_huggingface_checkpointing()` — note that ModelOpt's `export_hf_checkpoint()` produces a *deployment* checkpoint that diffusers cannot round-trip; the `modelopt_state.pth` from `save_pretrained` is what makes drop-in loading possible. `serve_cosmos3_diffusers.py` is a small FastAPI server (text→image, image→video) around the same model. ## Known limitations / caveats - **The bundled loader is required** (see *Status*). FP8 additionally depends on its payload-dtype restoration: diffusers' loader casts floating params to `torch_dtype` when no hf_quantizer is present (flagged by a TODO in diffusers' own source), which would otherwise corrupt float8 payloads. - **QKV scale unification was skipped at export** (ModelOpt's fusion probe doesn't recognize this architecture); q/k/v keep independent scales. Harmless here; relevant only to engines that fuse QKV. - Render sharpness depends heavily on prompt density, scheduler settings, and guidance — tune these; they are not quantization loss. ## Guardrails Cosmos 3 ships an optional safety checker (`cosmos_guardrail`). The bundled loader passes `enable_safety_checker=False` for local single-user use. If you deploy this or publish generated media, install `cosmos-guardrail`, accept the gated [`nvidia/Cosmos-Guardrail1`](https://huggingface.co/nvidia/Cosmos-Guardrail1) model (released under its own NVIDIA Open Model License, separate from this repo's OpenMDW-1.1), and run with `load_pipe(..., enable_safety_checker=True)`. ## Provenance & License - **Derivative of:** [`nvidia/Cosmos3-Super`](https://huggingface.co/nvidia/Cosmos3-Super) (bf16). This repo modifies only the weight encoding of the transformer. - **Produced with:** NVIDIA TensorRT Model Optimizer + diffusers (from source). - **Exact versions used:** `diffusers 0.39.0.dev0` @ [`2c7efb9`](https://github.com/huggingface/diffusers/commit/2c7efb95349296cf6bcce981ea036275a82a94df), `nvidia-modelopt 0.44.0`, `accelerate 1.13.0`, `torch 2.12.0`, CUDA 13.3. - **License:** [OpenMDW-1.1](https://openmdw.ai/license/1-1/), inherited from the base model. This repo includes a copy of the agreement (`LICENSE`) and documents its origin above; the upstream repo ships no separate NOTICE file. OpenMDW-1.1 permits modification and redistribution and places no restrictions on generated outputs; you remain responsible for clearing any third-party rights embodied in the materials. ## Related repos - **Sibling NVFP4 build (smaller footprint, ~36 GB):** [`prometheusAIR/Cosmos3-Super-nvfp4`](https://huggingface.co/prometheusAIR/Cosmos3-Super-nvfp4) - **Original (bf16, source):** [`nvidia/Cosmos3-Super`](https://huggingface.co/nvidia/Cosmos3-Super) - **NF4 (bitsandbytes; broad GPU compatibility incl. Ampere/Ada; drop-in + ComfyUI nodes):** [`SanDiegoDude/Cosmos3-Super-nf4`](https://huggingface.co/SanDiegoDude/Cosmos3-Super-nf4) — a good choice if you are not on Blackwell-class hardware or want turnkey ComfyUI support.