Cosmos
Diffusers
Safetensors
cosmos3_omni
nvidia
cosmos3
vllm
vllm-omni
sglang
sglang-diffusion
text, image, video, audio, and action generation
omnimodel
Instructions to use nvidia/Cosmos3-Nano with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use nvidia/Cosmos3-Nano with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Diffusers
How to use nvidia/Cosmos3-Nano with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("nvidia/Cosmos3-Nano", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
[HF team] add pipeline tag to readme or better discoverability
#15
by linoyts HF Staff - opened
- README.md +12 -78
- modular_model_index.json +0 -75
- sound_tokenizer/diffusion_pytorch_model.safetensors +2 -2
README.md
CHANGED
|
@@ -1,20 +1,18 @@
|
|
| 1 |
---
|
| 2 |
license: other
|
| 3 |
license_name: openmdw1.1-license
|
| 4 |
-
license_link:
|
| 5 |
-
https://openmdw.ai/license/1-1/
|
| 6 |
library_name: cosmos
|
| 7 |
tags:
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
- omnimodel
|
| 18 |
---
|
| 19 |
|
| 20 |
# **Cosmos 3: Omnimodal World Models for Physical AI**
|
|
@@ -169,7 +167,6 @@ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated sys
|
|
| 169 |
- [PyTorch](https://github.com/nvidia/cosmos3)
|
| 170 |
- [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
|
| 171 |
- [Hugging Face Diffusers](https://huggingface.co/docs/diffusers/en/index)
|
| 172 |
-
- [SGLang](https://github.com/sgl-project/sglang)
|
| 173 |
|
| 174 |
**Supported Hardware Microarchitecture Compatibility:**
|
| 175 |
|
|
@@ -919,69 +916,6 @@ Example output:
|
|
| 919 |
|
| 920 |
<video controls width="1280" height="720" src="https://huggingface.co/nvidia/Cosmos3-Nano/resolve/main/assets/example_t2v_diffusers_output.mp4"></video>
|
| 921 |
|
| 922 |
-
### SGLang
|
| 923 |
-
|
| 924 |
-
[SGLang Diffusion](https://docs.sglang.io/docs/sglang-diffusion/index) can serve `nvidia/Cosmos3-Nano` through OpenAI-compatible image and video generation endpoints. Install SGLang from the main branch with diffusion dependencies, then start a server:
|
| 925 |
-
|
| 926 |
-
```shell
|
| 927 |
-
git clone --branch main https://github.com/sgl-project/sglang.git
|
| 928 |
-
cd sglang
|
| 929 |
-
pip install -e "python[diffusion]"
|
| 930 |
-
pip install "cosmos-guardrail==0.3.1"
|
| 931 |
-
|
| 932 |
-
sglang serve --model-path nvidia/Cosmos3-Nano
|
| 933 |
-
```
|
| 934 |
-
|
| 935 |
-
Cosmos 3 support in SGLang Diffusion currently requires the SGLang main branch. Switch to a stable SGLang release once Cosmos 3 support is included there.
|
| 936 |
-
|
| 937 |
-
For a video-specialized checkpoint, use `Cosmos3-Super-Image2Video` with multiple GPUs:
|
| 938 |
-
|
| 939 |
-
```shell
|
| 940 |
-
sglang serve \
|
| 941 |
-
--model-path nvidia/Cosmos3-Super-Image2Video \
|
| 942 |
-
--num-gpus 4
|
| 943 |
-
```
|
| 944 |
-
|
| 945 |
-
Supported SGLang endpoints:
|
| 946 |
-
|
| 947 |
-
| Mode | Endpoint | Notes |
|
| 948 |
-
| --- | --- | --- |
|
| 949 |
-
| Text to image | `POST /v1/images/generations` | Returns base64 image data by default |
|
| 950 |
-
| Text to video | `POST /v1/videos` | Creates an async job; poll `GET /v1/videos/{id}` and download `/content` |
|
| 951 |
-
| Image to video | `POST /v1/videos` | Upload the conditioning image with `input_reference` |
|
| 952 |
-
|
| 953 |
-
Example text-to-video request:
|
| 954 |
-
|
| 955 |
-
```shell
|
| 956 |
-
job_id=$(curl -sS -X POST http://localhost:30000/v1/videos \
|
| 957 |
-
--form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
|
| 958 |
-
--form-string "negative_prompt=blurry, distorted, low quality" \
|
| 959 |
-
--form-string "size=1280x720" \
|
| 960 |
-
--form-string "num_frames=81" \
|
| 961 |
-
--form-string "fps=24" \
|
| 962 |
-
--form-string "num_inference_steps=35" \
|
| 963 |
-
--form-string "guidance_scale=4.0" \
|
| 964 |
-
--form-string "flow_shift=10.0" \
|
| 965 |
-
--form-string "seed=42" \
|
| 966 |
-
--form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
|
| 967 |
-
| python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
|
| 968 |
-
|
| 969 |
-
while true; do
|
| 970 |
-
status=$(curl -sS "http://localhost:30000/v1/videos/${job_id}" \
|
| 971 |
-
| python -c 'import json, sys; print(json.load(sys.stdin)["status"])')
|
| 972 |
-
[ "$status" = "completed" ] && break
|
| 973 |
-
[ "$status" = "failed" ] && exit 1
|
| 974 |
-
sleep 1
|
| 975 |
-
done
|
| 976 |
-
|
| 977 |
-
curl -sS -L "http://localhost:30000/v1/videos/${job_id}/content" \
|
| 978 |
-
-o cosmos3_t2v_output.mp4
|
| 979 |
-
```
|
| 980 |
-
|
| 981 |
-
SGLang accepts Cosmos 3 request options including `max_sequence_length`, `flow_shift`, `extra_params.guardrails`, `extra_params.use_resolution_template`, and `extra_params.use_duration_template`. Video-to-video, video-with-sound, and action generation are not supported by SGLang yet.
|
| 982 |
-
|
| 983 |
-
For complete serving instructions and request examples, see the [Cosmos3 SGLang cookbook](https://docs.sglang.io/cookbook/diffusion/Cosmos/Cosmos3).
|
| 984 |
-
|
| 985 |
## Limitations
|
| 986 |
|
| 987 |
Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artifacts include temporal inconsistency, unstable camera or object motion, imprecise physical interactions, inaccurate audio-video synchronization, and action-state drift — especially in long-horizon or high-resolution outputs. Reasoning may also be incorrect: object states, causal relationships, spatial geometry, temporal ordering, agent intent, and future outcomes can be misinferred, and complex or long-context inputs may yield hallucinated entities, inconsistent interpretations, or implausible predictions. Because the model lacks an explicit physics simulator, 3D geometry, 4D space-time evolution, object permanence, contact dynamics, and physical laws are only approximated — producing artifacts such as disappearing or morphing objects, unrealistic collisions, and physically implausible motions. Quality further degrades in out-of-distribution environments, safety-critical edge cases, and domains underrepresented in training.
|
|
@@ -990,7 +924,7 @@ Cosmos3 outputs should not be treated as physically accurate simulation, reliabl
|
|
| 990 |
|
| 991 |
## Inference
|
| 992 |
|
| 993 |
-
**Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM](https://github.com/vllm-project/vllm), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
|
| 994 |
|
| 995 |
**Test Hardware:** GB200 and H100
|
| 996 |
|
|
@@ -1002,4 +936,4 @@ Please make sure you have proper rights and permissions for all input image and
|
|
| 1002 |
|
| 1003 |
Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.
|
| 1004 |
|
| 1005 |
-
For more detailed information on ethical considerations for this model, please see the Model Card++ [Explainability](EXPLAINABILITY.md), [Bias](BIAS.md), [Safety & Security](SAFETY.md), and [Privacy](PRIVACY.md) subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
|
|
|
|
| 1 |
---
|
| 2 |
license: other
|
| 3 |
license_name: openmdw1.1-license
|
| 4 |
+
license_link: https://openmdw.ai/license/1-1/
|
|
|
|
| 5 |
library_name: cosmos
|
| 6 |
tags:
|
| 7 |
+
- nvidia
|
| 8 |
+
- cosmos
|
| 9 |
+
- cosmos3
|
| 10 |
+
- vllm
|
| 11 |
+
- vllm-omni
|
| 12 |
+
- diffusers
|
| 13 |
+
- text, image, video, audio, and action generation
|
| 14 |
+
- omnimodel
|
| 15 |
+
pipeline_tag: any-to-any
|
|
|
|
| 16 |
---
|
| 17 |
|
| 18 |
# **Cosmos 3: Omnimodal World Models for Physical AI**
|
|
|
|
| 167 |
- [PyTorch](https://github.com/nvidia/cosmos3)
|
| 168 |
- [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
|
| 169 |
- [Hugging Face Diffusers](https://huggingface.co/docs/diffusers/en/index)
|
|
|
|
| 170 |
|
| 171 |
**Supported Hardware Microarchitecture Compatibility:**
|
| 172 |
|
|
|
|
| 916 |
|
| 917 |
<video controls width="1280" height="720" src="https://huggingface.co/nvidia/Cosmos3-Nano/resolve/main/assets/example_t2v_diffusers_output.mp4"></video>
|
| 918 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 919 |
## Limitations
|
| 920 |
|
| 921 |
Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artifacts include temporal inconsistency, unstable camera or object motion, imprecise physical interactions, inaccurate audio-video synchronization, and action-state drift — especially in long-horizon or high-resolution outputs. Reasoning may also be incorrect: object states, causal relationships, spatial geometry, temporal ordering, agent intent, and future outcomes can be misinferred, and complex or long-context inputs may yield hallucinated entities, inconsistent interpretations, or implausible predictions. Because the model lacks an explicit physics simulator, 3D geometry, 4D space-time evolution, object permanence, contact dynamics, and physical laws are only approximated — producing artifacts such as disappearing or morphing objects, unrealistic collisions, and physically implausible motions. Quality further degrades in out-of-distribution environments, safety-critical edge cases, and domains underrepresented in training.
|
|
|
|
| 924 |
|
| 925 |
## Inference
|
| 926 |
|
| 927 |
+
**Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM](https://github.com/vllm-project/vllm), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
|
| 928 |
|
| 929 |
**Test Hardware:** GB200 and H100
|
| 930 |
|
|
|
|
| 936 |
|
| 937 |
Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.
|
| 938 |
|
| 939 |
+
For more detailed information on ethical considerations for this model, please see the Model Card++ [Explainability](EXPLAINABILITY.md), [Bias](BIAS.md), [Safety & Security](SAFETY.md), and [Privacy](PRIVACY.md) subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
|
modular_model_index.json
DELETED
|
@@ -1,75 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"_blocks_class_name": "Cosmos3OmniBlocks",
|
| 3 |
-
"_class_name": "Cosmos3OmniModularPipeline",
|
| 4 |
-
"_diffusers_version": "0.39.0.dev0",
|
| 5 |
-
"text_tokenizer": [
|
| 6 |
-
"transformers",
|
| 7 |
-
"Qwen2TokenizerFast",
|
| 8 |
-
{
|
| 9 |
-
"pretrained_model_name_or_path": "nvidia/Cosmos3-Nano",
|
| 10 |
-
"revision": null,
|
| 11 |
-
"subfolder": "text_tokenizer",
|
| 12 |
-
"type_hint": [
|
| 13 |
-
"transformers",
|
| 14 |
-
"Qwen2TokenizerFast"
|
| 15 |
-
],
|
| 16 |
-
"variant": null
|
| 17 |
-
}
|
| 18 |
-
],
|
| 19 |
-
"vae": [
|
| 20 |
-
"diffusers",
|
| 21 |
-
"AutoencoderKLWan",
|
| 22 |
-
{
|
| 23 |
-
"pretrained_model_name_or_path": "nvidia/Cosmos3-Nano",
|
| 24 |
-
"revision": null,
|
| 25 |
-
"subfolder": "vae",
|
| 26 |
-
"type_hint": [
|
| 27 |
-
"diffusers",
|
| 28 |
-
"AutoencoderKLWan"
|
| 29 |
-
],
|
| 30 |
-
"variant": null
|
| 31 |
-
}
|
| 32 |
-
],
|
| 33 |
-
"transformer": [
|
| 34 |
-
"diffusers",
|
| 35 |
-
"Cosmos3OmniTransformer",
|
| 36 |
-
{
|
| 37 |
-
"pretrained_model_name_or_path": "nvidia/Cosmos3-Nano",
|
| 38 |
-
"revision": null,
|
| 39 |
-
"subfolder": "transformer",
|
| 40 |
-
"type_hint": [
|
| 41 |
-
"diffusers",
|
| 42 |
-
"Cosmos3OmniTransformer"
|
| 43 |
-
],
|
| 44 |
-
"variant": null
|
| 45 |
-
}
|
| 46 |
-
],
|
| 47 |
-
"scheduler": [
|
| 48 |
-
"diffusers",
|
| 49 |
-
"UniPCMultistepScheduler",
|
| 50 |
-
{
|
| 51 |
-
"pretrained_model_name_or_path": "nvidia/Cosmos3-Nano",
|
| 52 |
-
"revision": null,
|
| 53 |
-
"subfolder": "scheduler",
|
| 54 |
-
"type_hint": [
|
| 55 |
-
"diffusers",
|
| 56 |
-
"UniPCMultistepScheduler"
|
| 57 |
-
],
|
| 58 |
-
"variant": null
|
| 59 |
-
}
|
| 60 |
-
],
|
| 61 |
-
"sound_tokenizer": [
|
| 62 |
-
"diffusers",
|
| 63 |
-
"Cosmos3AVAEAudioTokenizer",
|
| 64 |
-
{
|
| 65 |
-
"pretrained_model_name_or_path": "nvidia/Cosmos3-Nano",
|
| 66 |
-
"revision": null,
|
| 67 |
-
"subfolder": "sound_tokenizer",
|
| 68 |
-
"type_hint": [
|
| 69 |
-
"diffusers",
|
| 70 |
-
"Cosmos3AVAEAudioTokenizer"
|
| 71 |
-
],
|
| 72 |
-
"variant": null
|
| 73 |
-
}
|
| 74 |
-
]
|
| 75 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
sound_tokenizer/diffusion_pytorch_model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9d4c61cde38acfb0cad9048a140c3533750277a8462b19dc08450d9fe1ad9879
|
| 3 |
+
size 1892409600
|