README.md

---
library_name: diffusers
license: other
license_name: openmdw1.1-license
license_link: https://openmdw.ai/license/1-1/
pipeline_tag: image-to-video
tags:
- nvidia
- cosmos
- cosmos3
- vllm-omni
- video-generation
---

# **Cosmos 3: Omnimodal World Models for Physical AI**
**[Model Collection](https://huggingface.co/collections/nvidia/cosmos3)** | **[Code](https://github.com/nvidia/cosmos)** | **[Paper](https://huggingface.co/papers/2606.02800)** | **[Website](https://research.nvidia.com/labs/cosmos-lab/cosmos3/)**

[NVIDIA Cosmos™](https://github.com/nvidia/cosmos) is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments.

## Model Overview: Cosmos3-Super-Image2Video

**Cosmos3-Super-Image2Video** is a 64B parameter model designed for generating temporally coherent video sequences from a single input image and text instructions. It is part of the Cosmos 3 family, which uses a unified Mixture-of-Transformers (MoT) architecture to process and generate multimodal content.

## Sample Usage

Cosmos 3 is fully supported within the Hugging Face `diffusers` library.

### Installation

```bash
uv pip install \
  "diffusers @ git+https://github.com/huggingface/diffusers.git" \
  accelerate av cosmos_guardrail huggingface_hub imageio imageio-ffmpeg torch torchvision transformers
```

### Inference with Diffusers

```python
import json
import torch
from diffusers import Cosmos3OmniPipeline, UniPCMultistepScheduler
from diffusers.utils import export_to_video, load_image

pipe = Cosmos3OmniPipeline.from_pretrained(
    "nvidia/Cosmos3-Super-Image2Video",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    enable_safety_checker=True,
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=5.0)

image = load_image("assets/example_first_frame.png")

# JSON-format prompt (see the GitHub repository to build your own).
spec = json.load(open("assets/example_prompt.json"))
prompt = spec["prompt"]
negative_prompt = spec["negative_prompt"]

result = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    image=image,
    num_frames=189,
    height=480,
    width=832,
    fps=24.0,
    num_inference_steps=50,
    guidance_scale=6.0,
    add_resolution_template=False,
    add_duration_template=False,
)

export_to_video(result.video, "output.mp4", fps=24, quality=7, macro_block_size=1)
```

## Model Architecture

**Architecture Type:** Transformer
**Network Architecture:** Mixture-of-Transformers (MoT)

Cosmos3 is an omnimodal foundation model built on a Mixture-of-Transformers (MoT) architecture consisting of two complementary transformer towers: an autoregressive transformer for discrete token generation and a diffusion transformer for continuous multimodal generation.

## Limitations

Cosmos3 may produce imperfect outputs in challenging scenarios. Artifacts can include temporal inconsistency, unstable camera or object motion, imprecise physical interactions, and action-state drift. Because the model approximates physical laws without an explicit physics simulator, users may see disappearing objects or unrealistic collisions.

## License

NVIDIA Cosmos source code and models are released under the [OpenMDW-1.1](https://openmdw.ai/license/1-1/) License.