Add Diffusers usage section for the distilled 4-step I2V pipeline

#22
by yzhautouskay - opened
.gitattributes CHANGED
@@ -38,3 +38,4 @@ assets/example_first_frame.png filter=lfs diff=lfs merge=lfs -text
38
  assets/example_output.mp4 filter=lfs diff=lfs merge=lfs -text
39
  images/aa-i2v-all-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
40
  images/aa-i2v-open-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
 
 
38
  assets/example_output.mp4 filter=lfs diff=lfs merge=lfs -text
39
  images/aa-i2v-all-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
40
  images/aa-i2v-open-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
41
+ assets/example_output_diffusers.mp4 filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -196,6 +196,7 @@ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated sys
196
 
197
  - [PyTorch](https://github.com/nvidia/cosmos3)
198
  - [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
 
199
 
200
  **Supported Hardware Microarchitecture Compatibility:**
201
 
@@ -490,6 +491,70 @@ Example output generated from `assets/example_first_frame.png`:
490
 
491
  <video controls width="832" height="480" src="https://huggingface.co/nvidia/Cosmos3-Super-Image2Video-4Step/resolve/main/assets/example_output.mp4"></video>
492
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
493
  ### Inferencing with Custom Prompts
494
 
495
  `Cosmos3-Super-Image2Video-4Step` supports dense natural-language prompts together with a first-frame image, and it also supports JSON-format upsampled prompts for improved quality. The recommended way to prepare an upsampled prompt is to use [Prepare Prompts with Prompt Upsampling](#usage-prepare-prompts-with-prompt-upsampling). The helper below is a simple proof-of-concept script for building a JSON-format prompt from a custom natural-language instruction and the input image. It requires an OpenAI-compatible VLM model like `claude-opus-4.7` and `gpt-5.5`.
@@ -512,7 +577,7 @@ Cosmos3 outputs should not be treated as physically accurate simulation, reliabl
512
 
513
  ## Inference
514
 
515
- **Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
516
 
517
  **Test Hardware:** RTX PRO 6000, H20, H100, H200, B200
518
 
 
196
 
197
  - [PyTorch](https://github.com/nvidia/cosmos3)
198
  - [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
199
+ - [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
200
 
201
  **Supported Hardware Microarchitecture Compatibility:**
202
 
 
491
 
492
  <video controls width="832" height="480" src="https://huggingface.co/nvidia/Cosmos3-Super-Image2Video-4Step/resolve/main/assets/example_output.mp4"></video>
493
 
494
+ ## Usage: Run Inference with Diffusers
495
+
496
+ `Cosmos3-Super-Image2Video-4Step` is supported by the Hugging Face Diffusers **modular** pipeline. The distilled checkpoint is not compatible with `Cosmos3OmniPipeline` or `Cosmos3OmniModularPipeline`; it must be loaded with `Cosmos3DistilledModularPipeline`.
497
+
498
+ ### Install
499
+
500
+ Install Diffusers from the main branch until a release containing `Cosmos3DistilledModularPipeline` is available:
501
+
502
+ ```bash
503
+ uv venv --python 3.13 --seed --managed-python
504
+ source .venv/bin/activate
505
+ uv pip install \
506
+ "diffusers @ git+https://github.com/huggingface/diffusers.git" \
507
+ accelerate \
508
+ av \
509
+ cosmos_guardrail \
510
+ huggingface_hub \
511
+ imageio \
512
+ imageio-ffmpeg \
513
+ torch \
514
+ torchvision \
515
+ transformers
516
+ ```
517
+
518
+ The safety checker requires access to the gated [nvidia/Cosmos-1.0-Guardrail](https://huggingface.co/nvidia/Cosmos-1.0-Guardrail) repository.
519
+
520
+ ### Example: Image to Video Generation
521
+
522
+ ```python
523
+ import json
524
+
525
+ import torch
526
+ from diffusers import Cosmos3DistilledModularPipeline
527
+ from diffusers.utils import export_to_video, load_image
528
+
529
+ json_prompt = json.load(open("assets/example_prompt.json"))
530
+ image = load_image("assets/example_first_frame.png")
531
+
532
+ pipe = Cosmos3DistilledModularPipeline.from_pretrained("nvidia/Cosmos3-Super-Image2Video-4Step")
533
+ pipe.load_components(torch_dtype=torch.bfloat16)
534
+ pipe.enable_safety_checker()
535
+ pipe.to("cuda")
536
+
537
+ videos = pipe(
538
+ prompt=json_prompt["prompt"],
539
+ image=image,
540
+ num_frames=189,
541
+ height=480,
542
+ width=832,
543
+ fps=24,
544
+ add_resolution_template=False,
545
+ add_duration_template=False,
546
+ output="videos",
547
+ )
548
+ export_to_video(videos, "cosmos3_i2v_4step_diffusers.mp4", fps=24, macro_block_size=1)
549
+ print("Saved video to cosmos3_i2v_4step_diffusers.mp4")
550
+ ```
551
+
552
+ The 4-step schedule and the guidance scale come from the checkpoint: `num_inference_steps` is fixed to the length of the distilled sigma schedule and `guidance_scale` is forced to `1.0`, so passing any other value raises an error. `negative_prompt` is ignored, and unlike the base Cosmos3 models there is no scheduler to swap in and no `flow_shift` to set.
553
+
554
+ Example output generated from `assets/example_first_frame.png`:
555
+
556
+ <video controls width="832" height="480" src="https://huggingface.co/nvidia/Cosmos3-Super-Image2Video-4Step/resolve/main/assets/example_output_diffusers.mp4"></video>
557
+
558
  ### Inferencing with Custom Prompts
559
 
560
  `Cosmos3-Super-Image2Video-4Step` supports dense natural-language prompts together with a first-frame image, and it also supports JSON-format upsampled prompts for improved quality. The recommended way to prepare an upsampled prompt is to use [Prepare Prompts with Prompt Upsampling](#usage-prepare-prompts-with-prompt-upsampling). The helper below is a simple proof-of-concept script for building a JSON-format prompt from a custom natural-language instruction and the input image. It requires an OpenAI-compatible VLM model like `claude-opus-4.7` and `gpt-5.5`.
 
577
 
578
  ## Inference
579
 
580
+ **Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
581
 
582
  **Test Hardware:** RTX PRO 6000, H20, H100, H200, B200
583
 
assets/example_output_diffusers.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7337f6ae74b214bcca061c7de85e952e8cd8206d69755e8fb2fdb46784aa2641
3
+ size 1135801