---
library_name: diffusers
tags:
- modular-diffusers
- diffusers
- wan-vace
- text-to-image
---
This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.

**Pipeline Type**: Wan22VaceBlocks

**Description**: Modular pipeline for controllable video generation using Wan2.2 VACE.

This pipeline uses a 5-block architecture that can be customized and extended.

## Example Usage

[TODO]

## Pipeline Architecture

This modular pipeline is composed of the following blocks:

1. **text_encoder** (`WanTextEncoderStep`)
   - Text Encoder step that generate text_embeddings to guide the video generation
2. **vace_encoder** (`WanVaceEncoderStep`)
   - Vace Encoder step that preprocesses the control video, mask and reference images and encodes them into the conditioning latents used by the VACE control branch of the transformer
3. **denoise** (`Wan22VaceCoreDenoiseStep`)
   - denoise block that takes encoded text and vace conditioning latents and runs the denoising process.
4. **trim_latents** (`WanVaceTrimReferenceLatentsStep`)
   - Step that removes the prepended reference image frames from the denoised latents before decoding
5. **decode** (`WanVaeDecoderStep`)
   - Step that decodes the denoised latents into images 

## Model Components

1. text_encoder (`UMT5EncoderModel`)
2. tokenizer (`AutoTokenizer`)
3. guider (`ClassifierFreeGuidance`)
4. transformer (`WanVACETransformer3DModel`)
5. vae (`AutoencoderKLWan`)
6. video_processor (`VideoProcessor`)
7. scheduler (`UniPCMultistepScheduler`)
8. guider_2 (`ClassifierFreeGuidance`)
9. transformer_2 (`WanVACETransformer3DModel`) 

## Configuration Parameters

boundary_ratio (default: 0.875): The boundary ratio to divide the denoising loop into high noise and low noise stages.

## Input/Output Specification

**Inputs:**

- `prompt` (`None`, *optional*): No description provided
- `negative_prompt` (`None`, *optional*): No description provided
- `max_sequence_length` (`None`, *optional*, defaults to `512`): No description provided
- `video` (`list`, *optional*): The control video to condition the generation on. If not provided, an empty video is used.
- `mask` (`list`, *optional*): The mask that defines which video regions to condition on (black) and which to generate (white). Can only be passed if `video` is passed as well.
- `reference_images` (`Image | list`, *optional*): One or more reference images as extra conditioning for the generation.
- `conditioning_scale` (`float | list | Tensor`, *optional*, defaults to `1.0`): The conditioning scale applied in each control layer of the model. If a float, it is applied uniformly to all layers; a list or tensor must have the same length as the number of control layers.
- `height` (`None`, *optional*): No description provided
- `width` (`None`, *optional*): No description provided
- `num_frames` (`int`, *optional*, defaults to `81`): No description provided
- `generator` (`None`, *optional*): No description provided
- `num_videos_per_prompt` (`None`, *optional*, defaults to `1`): No description provided
- `num_inference_steps` (`None`, *optional*, defaults to `50`): No description provided
- `timesteps` (`None`, *optional*): No description provided
- `sigmas` (`None`, *optional*): No description provided
- `latents` (`Tensor | NoneType`, *optional*): No description provided
- `attention_kwargs` (`None`, *optional*): No description provided
- `output_type` (`str`, *optional*, defaults to `np`): The output type of the decoded videos

**Outputs:**

- `videos` (`list`): The generated videos.