library_name: diffusers
license: other
license_name: openmdw1.1-license
license_link: https://openmdw.ai/license/1-1/
pipeline_tag: image-to-video
tags:
- nvidia
- cosmos
- cosmos3
- vllm-omni
- video-generation
Cosmos 3: Omnimodal World Models for Physical AI
Model Collection | Code | Paper | Website
NVIDIA Cosmos™ is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments.
Model Overview: Cosmos3-Super-Image2Video
Cosmos3-Super-Image2Video is a 64B parameter model designed for generating temporally coherent video sequences from a single input image and text instructions. It is part of the Cosmos 3 family, which uses a unified Mixture-of-Transformers (MoT) architecture to process and generate multimodal content.
Sample Usage
Cosmos 3 is fully supported within the Hugging Face diffusers library.
Installation
uv pip install \
"diffusers @ git+https://github.com/huggingface/diffusers.git" \
accelerate av cosmos_guardrail huggingface_hub imageio imageio-ffmpeg torch torchvision transformers
Inference with Diffusers
import json
import torch
from diffusers import Cosmos3OmniPipeline, UniPCMultistepScheduler
from diffusers.utils import export_to_video, load_image
pipe = Cosmos3OmniPipeline.from_pretrained(
"nvidia/Cosmos3-Super-Image2Video",
torch_dtype=torch.bfloat16,
device_map="cuda",
enable_safety_checker=True,
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=5.0)
image = load_image("assets/example_first_frame.png")
# JSON-format prompt (see the GitHub repository to build your own).
spec = json.load(open("assets/example_prompt.json"))
prompt = spec["prompt"]
negative_prompt = spec["negative_prompt"]
result = pipe(
prompt=prompt,
negative_prompt=negative_prompt,
image=image,
num_frames=189,
height=480,
width=832,
fps=24.0,
num_inference_steps=50,
guidance_scale=6.0,
add_resolution_template=False,
add_duration_template=False,
)
export_to_video(result.video, "output.mp4", fps=24, quality=7, macro_block_size=1)
Model Architecture
Architecture Type: Transformer Network Architecture: Mixture-of-Transformers (MoT)
Cosmos3 is an omnimodal foundation model built on a Mixture-of-Transformers (MoT) architecture consisting of two complementary transformer towers: an autoregressive transformer for discrete token generation and a diffusion transformer for continuous multimodal generation.
Limitations
Cosmos3 may produce imperfect outputs in challenging scenarios. Artifacts can include temporal inconsistency, unstable camera or object motion, imprecise physical interactions, and action-state drift. Because the model approximates physical laws without an explicit physics simulator, users may see disappearing objects or unrealistic collisions.
License
NVIDIA Cosmos source code and models are released under the OpenMDW-1.1 License.