| --- |
| library_name: diffusers |
| tags: |
| - modular-diffusers |
| - diffusers |
| - wan-vace |
| - text-to-image |
| --- |
| This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework. |
|
|
| **Pipeline Type**: Wan22VaceBlocks |
|
|
| **Description**: Modular pipeline for controllable video generation using Wan2.2 VACE. |
|
|
| This pipeline uses a 5-block architecture that can be customized and extended. |
|
|
| ## Example Usage |
|
|
| [TODO] |
|
|
| ## Pipeline Architecture |
|
|
| This modular pipeline is composed of the following blocks: |
|
|
| 1. **text_encoder** (`WanTextEncoderStep`) |
| - Text Encoder step that generate text_embeddings to guide the video generation |
| 2. **vace_encoder** (`WanVaceEncoderStep`) |
| - Vace Encoder step that preprocesses the control video, mask and reference images and encodes them into the conditioning latents used by the VACE control branch of the transformer |
| 3. **denoise** (`Wan22VaceCoreDenoiseStep`) |
| - denoise block that takes encoded text and vace conditioning latents and runs the denoising process. |
| 4. **trim_latents** (`WanVaceTrimReferenceLatentsStep`) |
| - Step that removes the prepended reference image frames from the denoised latents before decoding |
| 5. **decode** (`WanVaeDecoderStep`) |
| - Step that decodes the denoised latents into images |
| |
| ## Model Components |
| |
| 1. text_encoder (`UMT5EncoderModel`) |
| 2. tokenizer (`AutoTokenizer`) |
| 3. guider (`ClassifierFreeGuidance`) |
| 4. transformer (`WanVACETransformer3DModel`) |
| 5. vae (`AutoencoderKLWan`) |
| 6. video_processor (`VideoProcessor`) |
| 7. scheduler (`UniPCMultistepScheduler`) |
| 8. guider_2 (`ClassifierFreeGuidance`) |
| 9. transformer_2 (`WanVACETransformer3DModel`) |
| |
| ## Configuration Parameters |
| |
| boundary_ratio (default: 0.875): The boundary ratio to divide the denoising loop into high noise and low noise stages. |
|
|
| ## Input/Output Specification |
|
|
| **Inputs:** |
|
|
| - `prompt` (`None`, *optional*): No description provided |
| - `negative_prompt` (`None`, *optional*): No description provided |
| - `max_sequence_length` (`None`, *optional*, defaults to `512`): No description provided |
| - `video` (`list`, *optional*): The control video to condition the generation on. If not provided, an empty video is used. |
| - `mask` (`list`, *optional*): The mask that defines which video regions to condition on (black) and which to generate (white). Can only be passed if `video` is passed as well. |
| - `reference_images` (`Image | list`, *optional*): One or more reference images as extra conditioning for the generation. |
| - `conditioning_scale` (`float | list | Tensor`, *optional*, defaults to `1.0`): The conditioning scale applied in each control layer of the model. If a float, it is applied uniformly to all layers; a list or tensor must have the same length as the number of control layers. |
| - `height` (`None`, *optional*): No description provided |
| - `width` (`None`, *optional*): No description provided |
| - `num_frames` (`int`, *optional*, defaults to `81`): No description provided |
| - `generator` (`None`, *optional*): No description provided |
| - `num_videos_per_prompt` (`None`, *optional*, defaults to `1`): No description provided |
| - `num_inference_steps` (`None`, *optional*, defaults to `50`): No description provided |
| - `timesteps` (`None`, *optional*): No description provided |
| - `sigmas` (`None`, *optional*): No description provided |
| - `latents` (`Tensor | NoneType`, *optional*): No description provided |
| - `attention_kwargs` (`None`, *optional*): No description provided |
| - `output_type` (`str`, *optional*, defaults to `np`): The output type of the decoded videos |
|
|
| **Outputs:** |
|
|
| - `videos` (`list`): The generated videos. |
|
|
