README.md

---
library_name: diffusers
license: other
license_name: openmdw1.1-license
license_link: https://openmdw.ai/license/1-1/
pipeline_tag: text-to-image
tags:
- nvidia
- cosmos
- cosmos3
- vllm-omni
- image-generation
---

# **Cosmos 3: Omnimodal World Models for Physical AI**
**[Model Collection](https://huggingface.co/collections/nvidia/cosmos3)** | **[Code](https://github.com/nvidia/cosmos)** | **[Paper](https://huggingface.co/papers/2606.02800)** | **[Website](https://research.nvidia.com/labs/cosmos-lab/cosmos3/)**

[NVIDIA Cosmos™](https://github.com/nvidia/cosmos) is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments.

Cosmos 3 is a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences within a unified mixture-of-transformers architecture.

# Model Overview: Cosmos3-Super-Text2Image

## Description

Cosmos3-Super-Text2Image is a 64B parameter model designed to generate high-fidelity images from text input. It serves as a foundational building block for Physical AI applications spanning world understanding, simulation, and embodied policy learning.

- **Model Developer:** NVIDIA
- **Architecture:** Mixture-of-Transformers (MoT)
- **Trainable Parameters:** 64B
- **License:** [OpenMDW1.1](https://openmdw.ai/license/1-1/)

## Usage

### Diffusers

Cosmos3 is fully supported within the Hugging Face [Diffusers](https://huggingface.co/docs/diffusers/en/index) library.

```python
import json
import torch
from diffusers import Cosmos3OmniPipeline
from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler

# Cosmos 3 works best with structured, upsampled prompts.
# See https://github.com/nvidia/cosmos for prompt upsampling details.
json_prompt = {
    "comprehensive_t2i_caption": "A photorealistic studio photograph of two hands working a spinning cylinder of wet gray clay on a pottery wheel, fingers gently pinching and pulling the walls upward to form a narrow neck and rounded belly."
}

pipe = Cosmos3OmniPipeline.from_pretrained(
    "nvidia/Cosmos3-Super-Text2Image",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    enable_safety_checker=True,
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=3.0)

result = pipe(
    prompt=json.dumps(json_prompt),
    negative_prompt="",
    num_frames=1,
    height=1024,
    width=1024,
    num_inference_steps=50,
    guidance_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(1143),
)

result.video[0].save("cosmos3_t2i.png")
```

### vLLM-Omni

You can use the `vllm-omni` package for deploying an OpenAI-compatible API inference endpoint.

```bash
vllm serve nvidia/Cosmos3-Super-Text2Image \
  --omni \
  --host 0.0.0.0 \
  --port 8000 \
  --cfg-parallel-size 2 \
  --ulysses-degree 4 \
  --tensor-parallel-size 1 \
  --use-hsdp \
  --hsdp-shard-size 8 \
  --init-timeout 1800
```

## Input/Output Specifications

- **Input Type**: Text
- **Output Type**: Image (JPEG)
- **Input Size**: Up to 4096 tokens
- **Supported Resolutions**: 256p, 480p, and 720p resolution at aspect ratios 16:9, 4:3, 1:1, 3:4, 9:16.

## Limitations

Cosmos3 may produce imperfect outputs in challenging scenarios, such as imprecise physical interactions or object morphing. Because the model lacks an explicit physics simulator, physical laws are only approximated. Quality further degrades in out-of-distribution environments or safety-critical edge cases. Outputs should not be treated as physically accurate simulations or safety-certified decision-making tools.

## Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility. Users are responsible for ensuring safe integration of this model, including implementing guardrails and other safety mechanisms prior to deployment. Please ensure you have proper rights and permissions for all input content.