yzhautouskay commited on
Commit
2baa929
·
verified ·
1 Parent(s): fda1636

Add Diffusers usage section for the distilled 4-step I2V pipeline

Browse files

Documents Cosmos3DistilledModularPipeline with the same example prompt and first-frame image as the vLLM-Omni example, and adds the generated example output video.

.gitattributes CHANGED
@@ -38,3 +38,4 @@ assets/example_first_frame.png filter=lfs diff=lfs merge=lfs -text
38
  assets/example_output.mp4 filter=lfs diff=lfs merge=lfs -text
39
  images/aa-i2v-all-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
40
  images/aa-i2v-open-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
 
 
38
  assets/example_output.mp4 filter=lfs diff=lfs merge=lfs -text
39
  images/aa-i2v-all-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
40
  images/aa-i2v-open-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
41
+ assets/example_output_diffusers.mp4 filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -490,6 +490,77 @@ Example output generated from `assets/example_first_frame.png`:
490
 
491
  <video controls width="832" height="480" src="https://huggingface.co/nvidia/Cosmos3-Super-Image2Video-4Step/resolve/main/assets/example_output.mp4"></video>
492
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
493
  ### Inferencing with Custom Prompts
494
 
495
  `Cosmos3-Super-Image2Video-4Step` supports dense natural-language prompts together with a first-frame image, and it also supports JSON-format upsampled prompts for improved quality. The recommended way to prepare an upsampled prompt is to use [Prepare Prompts with Prompt Upsampling](#usage-prepare-prompts-with-prompt-upsampling). The helper below is a simple proof-of-concept script for building a JSON-format prompt from a custom natural-language instruction and the input image. It requires an OpenAI-compatible VLM model like `claude-opus-4.7` and `gpt-5.5`.
 
490
 
491
  <video controls width="832" height="480" src="https://huggingface.co/nvidia/Cosmos3-Super-Image2Video-4Step/resolve/main/assets/example_output.mp4"></video>
492
 
493
+ ## Usage: Run Inference with Diffusers
494
+
495
+ `Cosmos3-Super-Image2Video-4Step` is supported by the Hugging Face Diffusers **modular** pipeline, using the same JSON-format upsampled prompt and first-frame image as the vLLM-Omni example above. The distilled checkpoint is not compatible with `Cosmos3OmniPipeline` or `Cosmos3OmniModularPipeline`; it must be loaded with `Cosmos3DistilledModularPipeline`.
496
+
497
+ This path loads the full 64B model on a single device in BF16, so it needs a B200/GB200-class GPU. On H100/H200-class GPUs, use the multi-GPU vLLM-Omni serving configuration above instead.
498
+
499
+ ### Install
500
+
501
+ Install Diffusers from the main branch until a release containing `Cosmos3DistilledModularPipeline` is available:
502
+
503
+ ```bash
504
+ uv venv --python 3.13 --seed --managed-python
505
+ source .venv/bin/activate
506
+ uv pip install \
507
+ "diffusers @ git+https://github.com/huggingface/diffusers.git" \
508
+ accelerate \
509
+ av \
510
+ cosmos_guardrail \
511
+ huggingface_hub \
512
+ imageio \
513
+ imageio-ffmpeg \
514
+ torch \
515
+ torchvision \
516
+ transformers
517
+ ```
518
+
519
+ The safety checker requires access to the gated [nvidia/Cosmos-1.0-Guardrail](https://huggingface.co/nvidia/Cosmos-1.0-Guardrail) repository.
520
+
521
+ ### Example: Image to Video Generation
522
+
523
+ Run this from the downloaded repo root, using the same `assets/example_prompt.json` and `assets/example_first_frame.png` inputs as the vLLM-Omni request. The example generates at 480p (`832x480` at 16:9), the recommended setting for this distilled I2V checkpoint.
524
+
525
+ ```python
526
+ import json
527
+
528
+ import torch
529
+ from diffusers import Cosmos3DistilledModularPipeline
530
+ from diffusers.utils import export_to_video, load_image
531
+
532
+ # The prompt file holds the same positive and negative prompt fields the vLLM-Omni
533
+ # example uses. Only the positive prompt is passed here: classifier-free guidance is
534
+ # baked into the distilled weights, so the negative prompt has no effect.
535
+ json_prompt = json.load(open("assets/example_prompt.json"))
536
+ image = load_image("assets/example_first_frame.png")
537
+
538
+ pipe = Cosmos3DistilledModularPipeline.from_pretrained("nvidia/Cosmos3-Super-Image2Video-4Step")
539
+ pipe.load_components(torch_dtype=torch.bfloat16)
540
+ pipe.enable_safety_checker()
541
+ pipe.to("cuda")
542
+
543
+ videos = pipe(
544
+ prompt=json_prompt["prompt"],
545
+ image=image,
546
+ num_frames=189,
547
+ height=480,
548
+ width=832,
549
+ fps=24,
550
+ add_resolution_template=False,
551
+ add_duration_template=False,
552
+ output="videos",
553
+ )
554
+ export_to_video(videos, "cosmos3_i2v_4step_diffusers.mp4", fps=24, macro_block_size=1)
555
+ print("Saved video to cosmos3_i2v_4step_diffusers.mp4")
556
+ ```
557
+
558
+ The 4-step schedule and the guidance scale come from the checkpoint: `num_inference_steps` is fixed to the length of the distilled sigma schedule and `guidance_scale` is forced to `1.0`, so passing any other value raises an error. `negative_prompt` is ignored, and unlike the base Cosmos3 models there is no scheduler to swap in and no `flow_shift` to set.
559
+
560
+ Example output generated from `assets/example_first_frame.png`:
561
+
562
+ <video controls width="832" height="480" src="https://huggingface.co/nvidia/Cosmos3-Super-Image2Video-4Step/resolve/main/assets/example_output_diffusers.mp4"></video>
563
+
564
  ### Inferencing with Custom Prompts
565
 
566
  `Cosmos3-Super-Image2Video-4Step` supports dense natural-language prompts together with a first-frame image, and it also supports JSON-format upsampled prompts for improved quality. The recommended way to prepare an upsampled prompt is to use [Prepare Prompts with Prompt Upsampling](#usage-prepare-prompts-with-prompt-upsampling). The helper below is a simple proof-of-concept script for building a JSON-format prompt from a custom natural-language instruction and the input image. It requires an OpenAI-compatible VLM model like `claude-opus-4.7` and `gpt-5.5`.
assets/example_output_diffusers.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7337f6ae74b214bcca061c7de85e952e8cd8206d69755e8fb2fdb46784aa2641
3
+ size 1135801