Cosmos
Diffusers
Safetensors
cosmos3_omni
nvidia
cosmos3
vllm
vllm-omni
sglang
sglang-diffusion
text, image, video, audio, and action generation
omnimodel
Instructions to use nvidia/Cosmos3-Super with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use nvidia/Cosmos3-Super with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Diffusers
How to use nvidia/Cosmos3-Super with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("nvidia/Cosmos3-Super", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Add SGLang serving instructions
Browse filesAdd SGLang-Diffusion model card examples and tags for Cosmos3 serving.
README.md
CHANGED
|
@@ -992,6 +992,67 @@ Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artif
|
|
| 992 |
|
| 993 |
Cosmos3 outputs should not be treated as physically accurate simulation, reliable ground-truth reasoning, or safety-certified decision making. Applications involving robotics control, autonomous systems, scientific simulation, or safety-critical planning require additional validation, external constraints, system-level safety analysis, and domain-specific guardrails before deployment.
|
| 994 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 995 |
## Inference
|
| 996 |
|
| 997 |
**Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM](https://github.com/vllm-project/vllm), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
|
|
|
|
| 992 |
|
| 993 |
Cosmos3 outputs should not be treated as physically accurate simulation, reliable ground-truth reasoning, or safety-certified decision making. Applications involving robotics control, autonomous systems, scientific simulation, or safety-critical planning require additional validation, external constraints, system-level safety analysis, and domain-specific guardrails before deployment.
|
| 994 |
|
| 995 |
+
### SGLang
|
| 996 |
+
|
| 997 |
+
SGLang-Diffusion can serve `nvidia/Cosmos3-Super` through OpenAI-compatible image and video endpoints. Install SGLang from source with diffusion dependencies, then start the server:
|
| 998 |
+
|
| 999 |
+
```bash
|
| 1000 |
+
git clone https://github.com/sgl-project/sglang.git
|
| 1001 |
+
cd sglang
|
| 1002 |
+
pip install -e "python[diffusion]"
|
| 1003 |
+
pip install "cosmos-guardrail==0.3.1"
|
| 1004 |
+
|
| 1005 |
+
sglang serve \
|
| 1006 |
+
--model-path nvidia/Cosmos3-Super \
|
| 1007 |
+
--num-gpus 4
|
| 1008 |
+
```
|
| 1009 |
+
|
| 1010 |
+
For the video-specialized checkpoint:
|
| 1011 |
+
|
| 1012 |
+
```bash
|
| 1013 |
+
sglang serve \
|
| 1014 |
+
--model-path nvidia/Cosmos3-Super-Image2Video \
|
| 1015 |
+
--num-gpus 4
|
| 1016 |
+
```
|
| 1017 |
+
|
| 1018 |
+
Supported SGLang endpoints:
|
| 1019 |
+
|
| 1020 |
+
| Mode | Endpoint | Notes |
|
| 1021 |
+
| --- | --- | --- |
|
| 1022 |
+
| Text to image | `POST /v1/images/generations` | Returns base64 image data by default |
|
| 1023 |
+
| Text to video | `POST /v1/videos` | Creates an async job; poll `GET /v1/videos/{id}` and download `/content` |
|
| 1024 |
+
| Image to video | `POST /v1/videos` | Upload the conditioning image with `input_reference` |
|
| 1025 |
+
|
| 1026 |
+
Example text-to-video request:
|
| 1027 |
+
|
| 1028 |
+
```bash
|
| 1029 |
+
job_id=$(curl -sS -X POST http://localhost:30000/v1/videos \
|
| 1030 |
+
--form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
|
| 1031 |
+
--form-string "negative_prompt=blurry, distorted, low quality" \
|
| 1032 |
+
--form-string "size=1280x720" \
|
| 1033 |
+
--form-string "num_frames=81" \
|
| 1034 |
+
--form-string "fps=24" \
|
| 1035 |
+
--form-string "num_inference_steps=35" \
|
| 1036 |
+
--form-string "guidance_scale=4.0" \
|
| 1037 |
+
--form-string "flow_shift=10.0" \
|
| 1038 |
+
--form-string "seed=42" \
|
| 1039 |
+
--form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
|
| 1040 |
+
| python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
|
| 1041 |
+
|
| 1042 |
+
while true; do
|
| 1043 |
+
status=$(curl -sS "http://localhost:30000/v1/videos/${job_id}" \
|
| 1044 |
+
| python -c 'import json, sys; print(json.load(sys.stdin)["status"])')
|
| 1045 |
+
[ "$status" = "completed" ] && break
|
| 1046 |
+
[ "$status" = "failed" ] && exit 1
|
| 1047 |
+
sleep 1
|
| 1048 |
+
done
|
| 1049 |
+
|
| 1050 |
+
curl -sS -L "http://localhost:30000/v1/videos/${job_id}/content" \
|
| 1051 |
+
-o cosmos3_super_t2v_output.mp4
|
| 1052 |
+
```
|
| 1053 |
+
|
| 1054 |
+
Video-to-video, video-with-sound, and action generation are not supported by SGLang yet.
|
| 1055 |
+
|
| 1056 |
## Inference
|
| 1057 |
|
| 1058 |
**Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM](https://github.com/vllm-project/vllm), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
|