Add robotics pipeline tag, library name, and paper link

#3
by nielsr HF Staff - opened
Files changed (2) hide show
  1. README.md +60 -341
  2. modular_model_index.json +0 -61
README.md CHANGED
@@ -1,47 +1,58 @@
1
  ---
 
2
  license: other
3
  license_name: openmdw1.1-license
4
- license_link: >-
5
- https://openmdw.ai/license/1-1/
6
- library_name: cosmos
7
  tags:
8
- - nvidia
9
- - cosmos
10
- - cosmos3
11
- - world action model
12
- - policy model
13
- countDownloads:
14
- - checkpoint.json
15
- - config.json
16
- - generation_config.json
17
- - model.safetensors.index.json
18
- - model_index.json
19
- - tokenizer.json
20
- - tokenizer_config.json
21
- - sound_tokenizer/config.json
22
- - sound_tokenizer/diffusion_pytorch_model.safetensors
23
- - text_tokenizer/tokenizer.json
24
- - text_tokenizer/tokenizer_config.json
25
- - transformer/config.json
26
- - transformer/diffusion_pytorch_model-00001-of-00007.safetensors
27
- - transformer/diffusion_pytorch_model-00002-of-00007.safetensors
28
- - transformer/diffusion_pytorch_model-00003-of-00007.safetensors
29
- - transformer/diffusion_pytorch_model-00004-of-00007.safetensors
30
- - transformer/diffusion_pytorch_model-00005-of-00007.safetensors
31
- - transformer/diffusion_pytorch_model-00006-of-00007.safetensors
32
- - transformer/diffusion_pytorch_model-00007-of-00007.safetensors
33
- - transformer/diffusion_pytorch_model.safetensors.index.json
34
- - vae/config.json
35
- - vae/diffusion_pytorch_model.safetensors
36
- - vision_encoder/config.json
37
- - vision_encoder/model.safetensors
38
  ---
39
 
40
  # **Cosmos 3: Omnimodal World Models for Physical AI**
41
- **[Model Collection](https://huggingface.co/collections/nvidia/cosmos3)** | **[Code](https://github.com/nvidia/cosmos)** | **[White Paper](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf)** | **[Website](https://research.nvidia.com/labs/cosmos-lab/cosmos3/)**
42
 
43
  [NVIDIA Cosmos™](https://github.com/nvidia/cosmos) is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments, including industrial and factory-scale applications.
44
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
45
  # Model Overview: Cosmos3-Nano-Policy-DROID
46
 
47
  ## Description
@@ -53,273 +64,38 @@ This model is ready for commercial and non-commercial use.
53
  **Model Developer:** NVIDIA
54
 
55
  ### Model Versions
56
- - Cosmos3-Nano:
57
- - Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.
58
-
59
- - Cosmos3-Super:
60
- - Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.
61
-
62
- - Cosmos3-Nano-Policy-DROID:
63
- - Given language instructions and visual observations from the DROID robot platform, generate robot action trajectories for manipulation and control tasks.
64
-
65
- - Cosmos3-Super-Image2Video:
66
- - Given one input image and text instructions, generate temporally coherent video sequences that are consistent with the provided visual content.
67
-
68
- - Cosmos3-Super-Text2Image:
69
- - Given text input, generate high-fidelity images that are consistent with the provided description.
70
 
71
  ### License
72
 
73
  This model is released under the [OpenMDW1.1](https://openmdw.ai/license/1-1/)
74
 
75
- ### Deployment Geography
76
-
77
- Global
78
-
79
- ### Use Case
80
-
81
- Physical AI: Encompassing robotics, autonomous vehicles (AV), and smart space environments, including industrial and factory-scale applications.
82
-
83
- ### Release Date
84
-
85
- Hugging Face 05/31/2026 via [https://huggingface.co/collections/nvidia/cosmos3](https://huggingface.co/collections/nvidia/cosmos3)
86
- GitHub 05/31/2026 via [https://github.com/nvidia/cosmos](https://github.com/nvidia/cosmos)
87
-
88
  ## Model Architecture
89
 
90
  **Architecture Type:** Transformer
91
-
92
  **Network Architecture:** Mixture-of-Transformers (MoT)
93
 
94
- Cosmos3 is an Omni-modal foundation model built on a Mixture-of-Transformers (MoT) architecture consisting of two complementary transformer towers: an autoregressive transformer for discrete token generation and a diffusion transformer for continuous multimodal generation. During inference, text is generated through standard next-token autoregressive decoding, while non-text modalities, such as images, video, audio, and actions, are synthesized through iterative denoising. This unified architecture enables Cosmos3 to model heterogeneous modalities within a single framework while preserving generation mechanisms best suited to each modality.
95
-
96
- **This model was developed based on:** [Cosmos Framework](https://github.com/nvidia/cosmos-framework)
97
-
98
- **Number of trainable model parameters:**
99
-
100
- - Cosmos3-Nano: 16B
101
- - Cosmos3-Super: 64B
102
- - Cosmos3-Nano-Policy-DROID: 16B
103
- - Cosmos3-Super-Image2Video: 64B
104
- - Cosmos3-Super-Text2Image: 64B
105
-
106
- ## Input/Output Specifications
107
-
108
- - **Generator Input**
109
- - **Input Type(s)**: Text, Image, Video (with audio or without audio), Action Trajectory
110
- - **Input Format(s)**:
111
- - Text: String
112
- - Image: jpg, png, jpeg, webp
113
- - Video (with or without audio): mp4
114
- - Action: json (1D list)
115
- - **Input Parameters**:
116
- - Text: One-dimensional (1D)
117
- - Image: Two-dimensional (2D)
118
- - Video: Three-dimensional (3D)
119
- - Audio: One-dimensional (1D)
120
- - Action trajectory: One-dimensional (1D)
121
- - **Other Properties Related to Input**:
122
- - For video inputs, we accept various resolutions, including 720p, 480p, and 256p.
123
- - When using input video with audio muxed into the video MP4 file, the audio should have 2 channels (stereo) and a 48 kHz sample rate.
124
- - Image and video inputs are RGB color (8 bits per channel, sRGB color space); grayscale inputs are not supported.
125
- - Action input is a per-frame sequence of robot/agent state or control values (e.g., joint positions, gripper state, camera pose). The full input is a 2D array shaped (T, D), where T is the number of frames and D is the embodiment-specific dimensionality listed below.
126
- - Input action is only supported for compatible embodiments, including general camera motion (9D), autonomous vehicle (9D), egocentric motion (57D), single Franka Panda arm with RobotiQ gripper (10D), dual Franka Panda arm with RobotiQ gripper (20D), Agibot (29D), UR (10D), Google robot (10D), WidowX 250 (10D), UMI (9D).
127
- - **Input Size and Length limits:**
128
- - **Text:** 4096 tokens
129
- - **Image:** 256p, 480p, and 720p resolution at one of these aspect ratios (16:9, 4:3, 1:1, 3:4, 9:16)
130
- - **Video:** 256p, 480p, and 720p resolution at one of these aspect ratios (16:9, 4:3, 1:1, 3:4, 9:16). Max number of frames = 5.
131
- - **Audio:** Max 0.5 second
132
- - **Action:** 16 – 400 video frames
133
- - **Generator Output**
134
- - **Output Type(s)**: Image, video, audio, action, text
135
- - **Output Format(s)**:
136
- - Image: JPG
137
- - Video: MP4
138
- - Audio: Advanced Audio Coding (AAC) stream (muxed within the MP4)
139
- - Action: 1D list (.json)
140
- - Text: string
141
- - **Output Parameters**:
142
- - Image: Two-dimensional (2D)
143
- - Video: Three-dimensional (3D)
144
- - Audio: One-dimensional (1D)
145
- - Action: One-dimensional (1D)
146
- - Text: One-dimensional (1D)
147
- - **Other Properties Related to Output**:
148
- - The generated video is an MP4 file, with the resolution, frame rate, and duration specified in the input. The generated audio is encoded in AAC format, muxed into the video MP4 file with 2 channels (stereo) and a 48 kHz sample rate.
149
- - Video generation supports durations from 5 to 400 frames, with 189 frames as the default generation duration.
150
- - The generated action is only supported for compatible embodiments, including general camera motion (9D), autonomous vehicle (9D), egocentric motion (57D), single Franka Panda arm with RobotiQ gripper (10D), dual Franka Panda arm with RobotiQ gripper (20D), Agibot (29D), UR (10D), Google robot (10D), WidowX 250 (10D), UMI (9D).
151
- - Audio: 48 kHz stereo AAC stream muxed into video mp4
152
- - Video: mp4 at the FPS specified in input
153
- - Image: JPEG
154
- - **Reasoner Input**
155
- - **Input Type(s)**: Text, Text+Image, Text+Video
156
- - **Input Format(s)**:
157
- - Text: String
158
- - Image: jpg, png, jpeg, webp
159
- - Video: mp4
160
- - **Input Parameters**:
161
- - Text: One-dimensional (1D)
162
- - Image: Two-dimensional (2D)
163
- - Video: Three-dimensional (3D)
164
- - **Other Properties Related to Input**:
165
- - Video inputs are recommended at a frame rate of 4 fps.
166
- - Long-context inputs supported up to 256K tokens.
167
- - **Input Size and Length limits:**
168
- - **Text:** Up to 256K tokens (context window).
169
- - **Image:** Standard input image formats; passed as file or URL.
170
- - **Video:** mp4 at the recommended 4 fps.
171
- - **Reasoner Output**
172
- - **Output Type(s)**: Text
173
- - **Output Format(s)**:
174
- - Text: string
175
- - **Output Parameters**:
176
- - Text: One-dimensional (1D)
177
- - **Other Properties Related to Output**:
178
- - Default `max_tokens=4096+` is recommended for reasoning outputs; longer outputs may be requested.
179
- - Reasoning outputs may include structured chain-of-thought, 2D/3D point localization, and bounding-box coordinates for vision-based tasks.
180
-
181
- The video content visualizes the input text description as a short animated scene, capturing key elements within the specified time constraints.
182
-
183
- Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
184
-
185
- ## Software Integration
186
-
187
- **Runtime Engine(s):**
188
-
189
- - [PyTorch](https://github.com/nvidia/cosmos3)
190
- - [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
191
- - [Hugging Face Diffusers](https://huggingface.co/docs/diffusers/en/index)
192
-
193
- **Supported Hardware Microarchitecture Compatibility:**
194
-
195
- - NVIDIA Ampere
196
- - NVIDIA Blackwell
197
- - NVIDIA Hopper
198
-
199
- **Operating System(s):**
200
-
201
- - Linux (We have not tested on other operating systems.)
202
-
203
- **Note:** Only BF16 precision is tested. Other precisions like FP4, FP8, and FP16 are not officially supported.
204
-
205
- The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
206
-
207
- ## Training, Testing, and Evaluation Datasets
208
-
209
- ### Dataset Overview
210
-
211
- - **Total Size:** 1.3B data points
212
- - **Total Number of Datasets:** 393 dataset entries
213
- - **Dataset partition:** Training [100%], Testing [N/A — evaluation benchmarks used separately], Validation [N/A — evaluation benchmarks used separately]
214
- - **Time period for training data collection:** 2024–2026
215
- - **Time period for testing data collection:** N/A (standard public benchmarks)
216
- - **Time period for validation data collection:** N/A (standard public benchmarks)
217
-
218
- Raw data from internal and external sources is transformed into training-ready data through multiple stages of curation, filtering, and quality review. Data acquisition spans diverse multimodal sources — robotics, autonomous driving, industrial environments, indoor and outdoor scenes, varied lighting and weather conditions, camera viewpoints, object categories, and human activities — to broaden coverage across Physical AI operating environments. Automated filtering pipelines remove corrupted, duplicate, low-quality, and restricted content. Metadata analysis, heuristic rules, and model-assisted classifiers are applied during preprocessing to flag anomalous distributions and low-diversity subsets. Human review supplements automated filtering for selected datasets, benchmark construction, and targeted quality analysis. Datasets are balanced across modalities and task categories — visual reasoning, text-to-image, text-to-video, image-to-video, audio generation, video transfer, action-conditioned generation, and action command generation — to reduce overrepresentation of narrow domains. Synthetic and simulation-based augmentation supplements coverage of rare physical interactions and edge-case scenarios. Deduplication and provenance tracking are applied across the corpus. The resulting processed data is converted into model-ready tokenized or encoded representations through modality-specific preprocessors before training begins.
219
-
220
- Training datasets passed through multiple layers of automated and manual safeguards designed to reduce the presence of harmful or policy-violating content across categories including weapons and weapons-related instructional content, criminal planning, child sexual abuse material (CSAM), non-consensual intimate imagery (NCII), sexual content involving minors, harassment, hate speech, profanity, threats and incitement to violence, self-harm or suicide-related content, and graphic violence. Data sources are reviewed for licensing compatibility, provenance, and alignment with internal data governance and safety policies before admission into training corpora. Automated filtering pipelines combine multiple detection strategies: hash-matching against known CSAM and NCII reference databases; classifier-based moderation models trained for explicit sexual content, hate speech, violence, weapons imagery, and other restricted categories; keyword and regex-based screening for criminal-planning, threats, and self-harm phrases in text data; metadata and provenance heuristics for source-level risk signals; and embedding-based anomaly detection to surface samples that fall outside expected distributions. Human review and targeted audits supplement automated filtering for selected datasets, benchmark construction, and safety-sensitive evaluation. For multimodal Physical AI data (robotics, autonomous driving, industrial scenes), additional filtering targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark evaluations and red-team testing are applied post-training to surface remaining safety gaps across world generation, reasoning, audio, and action tasks. No large-scale data-filtering process can guarantee complete removal of all harmful content; residual risks may remain, particularly in rare edge cases or open-world deployment settings. Ongoing monitoring and dataset review continue post-release.
221
-
222
- **Data Modality and Training Data Size**
223
-
224
- | Modality | Reasoning Data Sample Count | Generation Data Sample Count |
225
- | -------- | ------------------- | -------------------- |
226
- | Text | 22M | Not Applicable |
227
- | Image | 19M | 767M |
228
- | Video | 1M | 348M |
229
- | Audio | Not Applicable | 139M |
230
- | Action | Not Applicable | 8M |
231
-
232
- **Data Collection Method by dataset**
233
-
234
- - Hybrid: Automatic/Sensors, Synthetic, Automated
235
-
236
- **Labeling Method by dataset**
237
-
238
- - Hybrid: Human, Automated
239
-
240
- **Properties:** The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.
241
-
242
- ### Public Datasets
243
-
244
- | Dataset                                                             | Samples           |
245
- |---|---|
246
- | OpenImage | 1.2M |
247
- | Coyo700M | 100M |
248
- | YouTube Video | 340M |
249
- | UMI | 4.5M |
250
-
251
- ### Private Datasets
252
-
253
- | Dataset                                                             | Samples           |
254
- |---|---|
255
- | Egocentric | 7M |
256
- | Nexar | 0.6M |
257
- | AgiBot | 0.2M |
258
- | HOI | 0.3M |
259
-
260
- ### Synthetic Datasets
261
-
262
- | Dataset | Samples |
263
- |---|---|
264
- | synthetic images generated using HiDream-I1 | 15M |
265
- | synthetic images generated using Qwen-Image-2512 | 14M |
266
- | synthetic captions generated using Qwen3-VL | 1115M |
267
-
268
- ## Evaluation Datasets
269
-
270
- **Data Collection Method by dataset**
271
-
272
- - Hybrid: Automatic/Sensors, Synthetic, Automated
273
-
274
- **Labeling Method by dataset**
275
-
276
- - Hybrid: Human, Automated
277
-
278
- **Properties:** The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.
279
-
280
- ## Benchmarks
281
-
282
- Please see our [technical paper](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf) for detailed evaluations of the base model.
283
-
284
- ### Action
285
 
286
- #### RoboLab
287
 
288
- ![RoboLab task success rates across language specificity and difficulty levels](images/benchmark-action-2.png)
289
 
290
- #### RoboArena
291
-
292
- <p align="center"><img src="images/benchmark-roboarena.png" alt="RoboArena Policy Leaderboard — Cosmos3-Nano-Policy ranked #1"></p>
293
-
294
- ## Usage
295
-
296
- - See [Cosmos](https://github.com/nvidia/cosmos) for details.
297
-
298
- ### Quickstart
299
-
300
- Cosmos3-Nano-Policy-DROID is served by a policy **Server** that streams actions to a **Client** driving a simulated or real robot. This example uses [`RoboLab`](https://github.com/NVlabs/RoboLab), a simulation benchmark for task-generalist policies, as the client. Start the server first, then connect the client.
301
-
302
- #### Server
303
 
304
  First, clone [`cosmos-framework`](https://github.com/NVIDIA/cosmos-framework):
305
 
306
  ```bash
307
  git clone https://github.com/NVIDIA/cosmos-framework.git
308
  cd cosmos-framework
 
309
  ```
310
 
311
- Build the Docker image:
312
-
313
- ```bash
314
- docker build \
315
- -t cosmos-framework:latest \
316
- .
317
- ```
318
-
319
- Set your Hugging Face token and launch the container, which installs the dependencies:
320
 
321
  ```bash
322
- # Set your Hugging Face token (https://huggingface.co/settings/tokens):
323
  export HF_TOKEN=<your_hf_token>
324
 
325
  docker run \
@@ -333,82 +109,25 @@ docker run \
333
  -v /workspace/.venv \
334
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
335
  cosmos-framework:latest \
336
- bash -c '\
337
- uv sync \
338
- --all-extras \
339
- --group=cu130-train \
340
- --group=policy-server && \
341
- exec bash; \
342
- '
343
  ```
344
 
345
- Inside the container, start the policy server:
346
 
347
- ```
348
- python -m cosmos_framework.scripts.action_policy_server_robolab \
349
- --port 8000
350
- ```
351
-
352
- #### Client
353
-
354
- Clone [`RoboLab`](https://github.com/NVlabs/RoboLab):
355
 
356
  ```bash
357
  git clone https://github.com/NVlabs/RoboLab.git
358
  cd RoboLab
359
- ```
360
-
361
- Build the Docker image:
362
-
363
- ```bash
364
  ./docker/build_docker.sh latest
365
- ```
366
-
367
- Launch the container:
368
-
369
- ```bash
370
  ./docker/run_docker.sh latest
 
371
  ```
372
 
373
- Run a task against the policy server. This opens a viewer window for real-time visualization of the simulation:
374
-
375
- ```bash
376
- python policies/cosmos3/run.py \
377
- --task BananaInBowlTask
378
- ```
379
-
380
- To evaluate across multiple sub-environments in parallel in headless mode:
381
-
382
- ```bash
383
- python policies/cosmos3/run.py \
384
- --task BananaInBowlTask \
385
- --num-envs 10 \
386
- --headless
387
- ```
388
-
389
- Example output:
390
-
391
- <video controls width="864" height="480" src="
392
- https://huggingface.co/nvidia/Cosmos3-Nano-Policy-DROID/resolve/main/assets/Pick_up_the_banana_and_place_it_in_the_bowl_0_env0_viewport.mp4"></video>
393
-
394
  ## Limitations
395
 
396
- Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artifacts include temporal inconsistency, unstable camera or object motion, imprecise physical interactions, inaccurate audio-video synchronization, and action-state drift especially in long-horizon or high-resolution outputs. Reasoning may also be incorrect: object states, causal relationships, spatial geometry, temporal ordering, agent intent, and future outcomes can be misinferred, and complex or long-context inputs may yield hallucinated entities, inconsistent interpretations, or implausible predictions. Because the model lacks an explicit physics simulator, 3D geometry, 4D space-time evolution, object permanence, contact dynamics, and physical laws are only approximated — producing artifacts such as disappearing or morphing objects, unrealistic collisions, and physically implausible motions. Quality further degrades in out-of-distribution environments, safety-critical edge cases, and domains underrepresented in training.
397
-
398
- Cosmos3 outputs should not be treated as physically accurate simulation, reliable ground-truth reasoning, or safety-certified decision making. Applications involving robotics control, autonomous systems, scientific simulation, or safety-critical planning require additional validation, external constraints, system-level safety analysis, and domain-specific guardrails before deployment.
399
-
400
- ## Inference
401
-
402
- **Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM](https://github.com/vllm-project/vllm), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
403
-
404
- **Test Hardware:** H100
405
 
406
  ## Ethical Considerations
407
 
408
- NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
409
-
410
- Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
411
-
412
- Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.
413
-
414
- For more detailed information on ethical considerations for this model, please see the Model Card++ [Explainability](EXPLAINABILITY.md), [Bias](BIAS.md), [Safety & Security](SAFETY.md), and [Privacy](PRIVACY.md) subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
 
1
  ---
2
+ library_name: diffusers
3
  license: other
4
  license_name: openmdw1.1-license
5
+ license_link: https://openmdw.ai/license/1-1/
6
+ pipeline_tag: robotics
 
7
  tags:
8
+ - nvidia
9
+ - cosmos
10
+ - cosmos3
11
+ - world action model
12
+ - policy model
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
  ---
14
 
15
  # **Cosmos 3: Omnimodal World Models for Physical AI**
16
+ **[Model Collection](https://huggingface.co/collections/nvidia/cosmos3)** | **[Code](https://github.com/nvidia/cosmos)** | **[White Paper](https://huggingface.co/papers/2606.02800)** | **[Website](https://research.nvidia.com/labs/cosmos-lab/cosmos3/)**
17
 
18
  [NVIDIA Cosmos™](https://github.com/nvidia/cosmos) is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments, including industrial and factory-scale applications.
19
 
20
+ ## Sample Usage
21
+
22
+ This model family can be used with the Hugging Face [Diffusers](https://github.com/huggingface/diffusers) library. While this specific checkpoint is a robot policy, the base Cosmos 3 capabilities can be accessed as follows:
23
+
24
+ ```python
25
+ import torch
26
+ from diffusers import Cosmos3OmniPipeline
27
+ from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
28
+ from diffusers.utils import export_to_video
29
+
30
+ pipe = Cosmos3OmniPipeline.from_pretrained(
31
+ "nvidia/Cosmos3-Nano",
32
+ torch_dtype=torch.bfloat16,
33
+ device_map="cuda",
34
+ )
35
+ pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=10.0)
36
+
37
+ result = pipe(
38
+ prompt="A mobile robot navigates a warehouse aisle and stops at a shelf.",
39
+ negative_prompt="",
40
+ image=None,
41
+ num_frames=189,
42
+ height=720,
43
+ width=1280,
44
+ fps=24,
45
+ num_inference_steps=35,
46
+ guidance_scale=6.0,
47
+ enable_sound=False,
48
+ add_resolution_template=False,
49
+ add_duration_template=False,
50
+ generator=torch.Generator(device="cuda").manual_seed(1234),
51
+ )
52
+
53
+ export_to_video(result.video, "cosmos3_t2v.mp4", fps=24, macro_block_size=1)
54
+ ```
55
+
56
  # Model Overview: Cosmos3-Nano-Policy-DROID
57
 
58
  ## Description
 
64
  **Model Developer:** NVIDIA
65
 
66
  ### Model Versions
67
+ - **Cosmos3-Nano**: 16B parameters compact omnimodal world model.
68
+ - **Cosmos3-Super**: 64B parameters frontier-scale omnimodal world model.
69
+ - **Cosmos3-Nano-Policy-DROID**: Given language instructions and visual observations from the DROID robot platform, generate robot action trajectories for manipulation and control tasks.
 
 
 
 
 
 
 
 
 
 
 
70
 
71
  ### License
72
 
73
  This model is released under the [OpenMDW1.1](https://openmdw.ai/license/1-1/)
74
 
 
 
 
 
 
 
 
 
 
 
 
 
 
75
  ## Model Architecture
76
 
77
  **Architecture Type:** Transformer
 
78
  **Network Architecture:** Mixture-of-Transformers (MoT)
79
 
80
+ Cosmos3 is an Omni-modal foundation model built on a Mixture-of-Transformers (MoT) architecture consisting of two complementary transformer towers: an autoregressive transformer for discrete token generation and a diffusion transformer for continuous multimodal generation.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
 
82
+ ## Quickstart (Robotics Policy)
83
 
84
+ Cosmos3-Nano-Policy-DROID is served by a policy **Server** that streams actions to a **Client** driving a simulated or real robot. This example uses [`RoboLab`](https://github.com/NVlabs/RoboLab) as the client.
85
 
86
+ ### Server Setup
 
 
 
 
 
 
 
 
 
 
 
 
87
 
88
  First, clone [`cosmos-framework`](https://github.com/NVIDIA/cosmos-framework):
89
 
90
  ```bash
91
  git clone https://github.com/NVIDIA/cosmos-framework.git
92
  cd cosmos-framework
93
+ docker build -t cosmos-framework:latest .
94
  ```
95
 
96
+ Launch the container and start the server:
 
 
 
 
 
 
 
 
97
 
98
  ```bash
 
99
  export HF_TOKEN=<your_hf_token>
100
 
101
  docker run \
 
109
  -v /workspace/.venv \
110
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
111
  cosmos-framework:latest \
112
+ bash -c 'uv sync --all-extras --group=cu130-train --group=policy-server && python -m cosmos_framework.scripts.action_policy_server_robolab --port 8000'
 
 
 
 
 
 
113
  ```
114
 
115
+ ### Client Setup
116
 
117
+ Clone [`RoboLab`](https://github.com/NVlabs/RoboLab) and run a task:
 
 
 
 
 
 
 
118
 
119
  ```bash
120
  git clone https://github.com/NVlabs/RoboLab.git
121
  cd RoboLab
 
 
 
 
 
122
  ./docker/build_docker.sh latest
 
 
 
 
 
123
  ./docker/run_docker.sh latest
124
+ python policies/cosmos3/run.py --task BananaInBowlTask
125
  ```
126
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
127
  ## Limitations
128
 
129
+ Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artifacts include temporal inconsistency, unstable camera motion, and action-state drift—especially in long-horizon outputs. Because the model lacks an explicit physics simulator, physical laws are only approximated. Quality further degrades in out-of-distribution environments and safety-critical edge cases.
 
 
 
 
 
 
 
 
130
 
131
  ## Ethical Considerations
132
 
133
+ NVIDIA believes Trustworthy AI is a shared responsibility. Users are responsible for model inputs and outputs and ensuring safe integration of this model, including implementing guardrails and safety mechanisms prior to deployment. For more detailed information, please see the Model Card++ subcards.
 
 
 
 
 
 
modular_model_index.json DELETED
@@ -1,61 +0,0 @@
1
- {
2
- "_blocks_class_name": "Cosmos3OmniBlocks",
3
- "_class_name": "Cosmos3OmniModularPipeline",
4
- "_diffusers_version": "0.39.0.dev0",
5
- "text_tokenizer": [
6
- "transformers",
7
- "Qwen2TokenizerFast",
8
- {
9
- "pretrained_model_name_or_path": "nvidia/Cosmos3-Nano-Policy-DROID",
10
- "revision": null,
11
- "subfolder": "text_tokenizer",
12
- "type_hint": [
13
- "transformers",
14
- "Qwen2TokenizerFast"
15
- ],
16
- "variant": null
17
- }
18
- ],
19
- "vae": [
20
- "diffusers",
21
- "AutoencoderKLWan",
22
- {
23
- "pretrained_model_name_or_path": "nvidia/Cosmos3-Nano-Policy-DROID",
24
- "revision": null,
25
- "subfolder": "vae",
26
- "type_hint": [
27
- "diffusers",
28
- "AutoencoderKLWan"
29
- ],
30
- "variant": null
31
- }
32
- ],
33
- "transformer": [
34
- "diffusers",
35
- "Cosmos3OmniTransformer",
36
- {
37
- "pretrained_model_name_or_path": "nvidia/Cosmos3-Nano-Policy-DROID",
38
- "revision": null,
39
- "subfolder": "transformer",
40
- "type_hint": [
41
- "diffusers",
42
- "Cosmos3OmniTransformer"
43
- ],
44
- "variant": null
45
- }
46
- ],
47
- "scheduler": [
48
- "diffusers",
49
- "UniPCMultistepScheduler",
50
- {
51
- "pretrained_model_name_or_path": "nvidia/Cosmos3-Nano-Policy-DROID",
52
- "revision": null,
53
- "subfolder": "scheduler",
54
- "type_hint": [
55
- "diffusers",
56
- "UniPCMultistepScheduler"
57
- ],
58
- "variant": null
59
- }
60
- ]
61
- }