Instructions to use nvidia/Cosmos3-Super-Image2Video-4Step with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use nvidia/Cosmos3-Super-Image2Video-4Step with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Add Diffusers usage section for the distilled 4-step I2V pipeline
#22
by yzhautouskay - opened
- .gitattributes +1 -0
- README.md +66 -1
- assets/example_output_diffusers.mp4 +3 -0
.gitattributes
CHANGED
|
@@ -38,3 +38,4 @@ assets/example_first_frame.png filter=lfs diff=lfs merge=lfs -text
|
|
| 38 |
assets/example_output.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 39 |
images/aa-i2v-all-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
|
| 40 |
images/aa-i2v-open-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 38 |
assets/example_output.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 39 |
images/aa-i2v-all-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
|
| 40 |
images/aa-i2v-open-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
assets/example_output_diffusers.mp4 filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -196,6 +196,7 @@ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated sys
|
|
| 196 |
|
| 197 |
- [PyTorch](https://github.com/nvidia/cosmos3)
|
| 198 |
- [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
|
|
|
|
| 199 |
|
| 200 |
**Supported Hardware Microarchitecture Compatibility:**
|
| 201 |
|
|
@@ -490,6 +491,70 @@ Example output generated from `assets/example_first_frame.png`:
|
|
| 490 |
|
| 491 |
<video controls width="832" height="480" src="https://huggingface.co/nvidia/Cosmos3-Super-Image2Video-4Step/resolve/main/assets/example_output.mp4"></video>
|
| 492 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 493 |
### Inferencing with Custom Prompts
|
| 494 |
|
| 495 |
`Cosmos3-Super-Image2Video-4Step` supports dense natural-language prompts together with a first-frame image, and it also supports JSON-format upsampled prompts for improved quality. The recommended way to prepare an upsampled prompt is to use [Prepare Prompts with Prompt Upsampling](#usage-prepare-prompts-with-prompt-upsampling). The helper below is a simple proof-of-concept script for building a JSON-format prompt from a custom natural-language instruction and the input image. It requires an OpenAI-compatible VLM model like `claude-opus-4.7` and `gpt-5.5`.
|
|
@@ -512,7 +577,7 @@ Cosmos3 outputs should not be treated as physically accurate simulation, reliabl
|
|
| 512 |
|
| 513 |
## Inference
|
| 514 |
|
| 515 |
-
**Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
|
| 516 |
|
| 517 |
**Test Hardware:** RTX PRO 6000, H20, H100, H200, B200
|
| 518 |
|
|
|
|
| 196 |
|
| 197 |
- [PyTorch](https://github.com/nvidia/cosmos3)
|
| 198 |
- [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
|
| 199 |
+
- [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
|
| 200 |
|
| 201 |
**Supported Hardware Microarchitecture Compatibility:**
|
| 202 |
|
|
|
|
| 491 |
|
| 492 |
<video controls width="832" height="480" src="https://huggingface.co/nvidia/Cosmos3-Super-Image2Video-4Step/resolve/main/assets/example_output.mp4"></video>
|
| 493 |
|
| 494 |
+
## Usage: Run Inference with Diffusers
|
| 495 |
+
|
| 496 |
+
`Cosmos3-Super-Image2Video-4Step` is supported by the Hugging Face Diffusers **modular** pipeline. The distilled checkpoint is not compatible with `Cosmos3OmniPipeline` or `Cosmos3OmniModularPipeline`; it must be loaded with `Cosmos3DistilledModularPipeline`.
|
| 497 |
+
|
| 498 |
+
### Install
|
| 499 |
+
|
| 500 |
+
Install Diffusers from the main branch until a release containing `Cosmos3DistilledModularPipeline` is available:
|
| 501 |
+
|
| 502 |
+
```bash
|
| 503 |
+
uv venv --python 3.13 --seed --managed-python
|
| 504 |
+
source .venv/bin/activate
|
| 505 |
+
uv pip install \
|
| 506 |
+
"diffusers @ git+https://github.com/huggingface/diffusers.git" \
|
| 507 |
+
accelerate \
|
| 508 |
+
av \
|
| 509 |
+
cosmos_guardrail \
|
| 510 |
+
huggingface_hub \
|
| 511 |
+
imageio \
|
| 512 |
+
imageio-ffmpeg \
|
| 513 |
+
torch \
|
| 514 |
+
torchvision \
|
| 515 |
+
transformers
|
| 516 |
+
```
|
| 517 |
+
|
| 518 |
+
The safety checker requires access to the gated [nvidia/Cosmos-1.0-Guardrail](https://huggingface.co/nvidia/Cosmos-1.0-Guardrail) repository.
|
| 519 |
+
|
| 520 |
+
### Example: Image to Video Generation
|
| 521 |
+
|
| 522 |
+
```python
|
| 523 |
+
import json
|
| 524 |
+
|
| 525 |
+
import torch
|
| 526 |
+
from diffusers import Cosmos3DistilledModularPipeline
|
| 527 |
+
from diffusers.utils import export_to_video, load_image
|
| 528 |
+
|
| 529 |
+
json_prompt = json.load(open("assets/example_prompt.json"))
|
| 530 |
+
image = load_image("assets/example_first_frame.png")
|
| 531 |
+
|
| 532 |
+
pipe = Cosmos3DistilledModularPipeline.from_pretrained("nvidia/Cosmos3-Super-Image2Video-4Step")
|
| 533 |
+
pipe.load_components(torch_dtype=torch.bfloat16)
|
| 534 |
+
pipe.enable_safety_checker()
|
| 535 |
+
pipe.to("cuda")
|
| 536 |
+
|
| 537 |
+
videos = pipe(
|
| 538 |
+
prompt=json_prompt["prompt"],
|
| 539 |
+
image=image,
|
| 540 |
+
num_frames=189,
|
| 541 |
+
height=480,
|
| 542 |
+
width=832,
|
| 543 |
+
fps=24,
|
| 544 |
+
add_resolution_template=False,
|
| 545 |
+
add_duration_template=False,
|
| 546 |
+
output="videos",
|
| 547 |
+
)
|
| 548 |
+
export_to_video(videos, "cosmos3_i2v_4step_diffusers.mp4", fps=24, macro_block_size=1)
|
| 549 |
+
print("Saved video to cosmos3_i2v_4step_diffusers.mp4")
|
| 550 |
+
```
|
| 551 |
+
|
| 552 |
+
The 4-step schedule and the guidance scale come from the checkpoint: `num_inference_steps` is fixed to the length of the distilled sigma schedule and `guidance_scale` is forced to `1.0`, so passing any other value raises an error. `negative_prompt` is ignored, and unlike the base Cosmos3 models there is no scheduler to swap in and no `flow_shift` to set.
|
| 553 |
+
|
| 554 |
+
Example output generated from `assets/example_first_frame.png`:
|
| 555 |
+
|
| 556 |
+
<video controls width="832" height="480" src="https://huggingface.co/nvidia/Cosmos3-Super-Image2Video-4Step/resolve/main/assets/example_output_diffusers.mp4"></video>
|
| 557 |
+
|
| 558 |
### Inferencing with Custom Prompts
|
| 559 |
|
| 560 |
`Cosmos3-Super-Image2Video-4Step` supports dense natural-language prompts together with a first-frame image, and it also supports JSON-format upsampled prompts for improved quality. The recommended way to prepare an upsampled prompt is to use [Prepare Prompts with Prompt Upsampling](#usage-prepare-prompts-with-prompt-upsampling). The helper below is a simple proof-of-concept script for building a JSON-format prompt from a custom natural-language instruction and the input image. It requires an OpenAI-compatible VLM model like `claude-opus-4.7` and `gpt-5.5`.
|
|
|
|
| 577 |
|
| 578 |
## Inference
|
| 579 |
|
| 580 |
+
**Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
|
| 581 |
|
| 582 |
**Test Hardware:** RTX PRO 6000, H20, H100, H200, B200
|
| 583 |
|
assets/example_output_diffusers.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7337f6ae74b214bcca061c7de85e952e8cd8206d69755e8fb2fdb46784aa2641
|
| 3 |
+
size 1135801
|