Cosmos
Diffusers
Safetensors
cosmos3_omni
nvidia
cosmos3
vllm
vllm-omni
sglang
sglang-diffusion
text, image, video, audio, and action generation
omnimodel
Instructions to use nvidia/Cosmos3-Super with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use nvidia/Cosmos3-Super with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Diffusers
How to use nvidia/Cosmos3-Super with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("nvidia/Cosmos3-Super", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Add pipeline tag, library name, and sample usage
Browse filesThis PR improves the model card for Cosmos3-Super by:
- Adding the `pipeline_tag: any-to-any` to reflect its omnimodal capabilities.
- Setting `library_name: diffusers` to enable the Hub's automated code snippets.
- Adding a sample usage section with a code snippet for the `diffusers` integration.
- Including a link to the paper page on Hugging Face.
README.md
CHANGED
|
@@ -1,22 +1,22 @@
|
|
| 1 |
---
|
|
|
|
| 2 |
license: other
|
| 3 |
license_name: openmdw1.1-license
|
| 4 |
-
license_link:
|
| 5 |
-
|
| 6 |
-
library_name: cosmos
|
| 7 |
tags:
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
---
|
| 17 |
|
| 18 |
# **Cosmos 3: Omnimodal World Models for Physical AI**
|
| 19 |
-
**[Model Collection](https://huggingface.co/collections/nvidia/cosmos3)** | **[Code](https://github.com/nvidia/cosmos)** | **[White Paper](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf)** | **[Website](https://research.nvidia.com/labs/cosmos-lab/cosmos3/)**
|
| 20 |
|
| 21 |
[NVIDIA Cosmos™](https://github.com/nvidia/cosmos) is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments, including industrial and factory-scale applications.
|
| 22 |
|
|
@@ -30,6 +30,37 @@ This model is ready for commercial and non-commercial use.
|
|
| 30 |
|
| 31 |
**Model Developer:** NVIDIA
|
| 32 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
### Model Versions
|
| 34 |
- Cosmos3-Nano:
|
| 35 |
- Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.
|
|
@@ -816,7 +847,7 @@ client = openai.OpenAI(
|
|
| 816 |
base_url="http://localhost:8000/v1",
|
| 817 |
)
|
| 818 |
|
| 819 |
-
response = client.chat.
|
| 820 |
model=client.models.list().data[0].id,
|
| 821 |
messages=[
|
| 822 |
{
|
|
@@ -941,4 +972,4 @@ Please make sure you have proper rights and permissions for all input image and
|
|
| 941 |
|
| 942 |
Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.
|
| 943 |
|
| 944 |
-
For more detailed information on ethical considerations for this model, please see the Model Card++ [Explainability](EXPLAINABILITY.md), [Bias](BIAS.md), [Safety & Security](SAFETY.md), and [Privacy](PRIVACY.md) subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
|
|
|
|
| 1 |
---
|
| 2 |
+
library_name: diffusers
|
| 3 |
license: other
|
| 4 |
license_name: openmdw1.1-license
|
| 5 |
+
license_link: https://openmdw.ai/license/1-1/
|
| 6 |
+
pipeline_tag: any-to-any
|
|
|
|
| 7 |
tags:
|
| 8 |
+
- nvidia
|
| 9 |
+
- cosmos
|
| 10 |
+
- cosmos3
|
| 11 |
+
- vllm
|
| 12 |
+
- vllm-omni
|
| 13 |
+
- diffusers
|
| 14 |
+
- text, image, video, audio, and action generation
|
| 15 |
+
- omnimodel
|
| 16 |
---
|
| 17 |
|
| 18 |
# **Cosmos 3: Omnimodal World Models for Physical AI**
|
| 19 |
+
**[Paper Page](https://huggingface.co/papers/2606.02800)** | **[Model Collection](https://huggingface.co/collections/nvidia/cosmos3)** | **[Code](https://github.com/nvidia/cosmos)** | **[White Paper](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf)** | **[Website](https://research.nvidia.com/labs/cosmos-lab/cosmos3/)**
|
| 20 |
|
| 21 |
[NVIDIA Cosmos™](https://github.com/nvidia/cosmos) is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments, including industrial and factory-scale applications.
|
| 22 |
|
|
|
|
| 30 |
|
| 31 |
**Model Developer:** NVIDIA
|
| 32 |
|
| 33 |
+
### Sample Usage
|
| 34 |
+
|
| 35 |
+
You can use the model with the [diffusers](https://github.com/huggingface/diffusers) library as shown below:
|
| 36 |
+
|
| 37 |
+
```python
|
| 38 |
+
import torch
|
| 39 |
+
from diffusers import Cosmos3OmniPipeline
|
| 40 |
+
from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
|
| 41 |
+
from diffusers.utils import export_to_video
|
| 42 |
+
|
| 43 |
+
pipe = Cosmos3OmniPipeline.from_pretrained(
|
| 44 |
+
"nvidia/Cosmos3-Super",
|
| 45 |
+
torch_dtype=torch.bfloat16,
|
| 46 |
+
device_map="cuda",
|
| 47 |
+
)
|
| 48 |
+
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=10.0)
|
| 49 |
+
|
| 50 |
+
result = pipe(
|
| 51 |
+
prompt="A mobile robot navigates a warehouse aisle and stops at a shelf.",
|
| 52 |
+
num_frames=189,
|
| 53 |
+
height=720,
|
| 54 |
+
width=1280,
|
| 55 |
+
fps=24,
|
| 56 |
+
num_inference_steps=35,
|
| 57 |
+
guidance_scale=6.0,
|
| 58 |
+
generator=torch.Generator(device="cuda").manual_seed(123),
|
| 59 |
+
)
|
| 60 |
+
|
| 61 |
+
export_to_video(result.video, "cosmos3_super_t2v.mp4", fps=24)
|
| 62 |
+
```
|
| 63 |
+
|
| 64 |
### Model Versions
|
| 65 |
- Cosmos3-Nano:
|
| 66 |
- Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.
|
|
|
|
| 847 |
base_url="http://localhost:8000/v1",
|
| 848 |
)
|
| 849 |
|
| 850 |
+
response = client.chat.completion.create(
|
| 851 |
model=client.models.list().data[0].id,
|
| 852 |
messages=[
|
| 853 |
{
|
|
|
|
| 972 |
|
| 973 |
Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.
|
| 974 |
|
| 975 |
+
For more detailed information on ethical considerations for this model, please see the Model Card++ [Explainability](EXPLAINABILITY.md), [Bias](BIAS.md), [Safety & Security](SAFETY.md), and [Privacy](PRIVACY.md) subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
|