File size: 3,562 Bytes
e12b699 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | ---
library_name: diffusers
tags:
- modular-diffusers
- diffusers
- wan-vace
- text-to-image
---
This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.
**Pipeline Type**: Wan22VaceBlocks
**Description**: Modular pipeline for controllable video generation using Wan2.2 VACE.
This pipeline uses a 5-block architecture that can be customized and extended.
## Example Usage
[TODO]
## Pipeline Architecture
This modular pipeline is composed of the following blocks:
1. **text_encoder** (`WanTextEncoderStep`)
- Text Encoder step that generate text_embeddings to guide the video generation
2. **vace_encoder** (`WanVaceEncoderStep`)
- Vace Encoder step that preprocesses the control video, mask and reference images and encodes them into the conditioning latents used by the VACE control branch of the transformer
3. **denoise** (`Wan22VaceCoreDenoiseStep`)
- denoise block that takes encoded text and vace conditioning latents and runs the denoising process.
4. **trim_latents** (`WanVaceTrimReferenceLatentsStep`)
- Step that removes the prepended reference image frames from the denoised latents before decoding
5. **decode** (`WanVaeDecoderStep`)
- Step that decodes the denoised latents into images
## Model Components
1. text_encoder (`UMT5EncoderModel`)
2. tokenizer (`AutoTokenizer`)
3. guider (`ClassifierFreeGuidance`)
4. transformer (`WanVACETransformer3DModel`)
5. vae (`AutoencoderKLWan`)
6. video_processor (`VideoProcessor`)
7. scheduler (`UniPCMultistepScheduler`)
8. guider_2 (`ClassifierFreeGuidance`)
9. transformer_2 (`WanVACETransformer3DModel`)
## Configuration Parameters
boundary_ratio (default: 0.875): The boundary ratio to divide the denoising loop into high noise and low noise stages.
## Input/Output Specification
**Inputs:**
- `prompt` (`None`, *optional*): No description provided
- `negative_prompt` (`None`, *optional*): No description provided
- `max_sequence_length` (`None`, *optional*, defaults to `512`): No description provided
- `video` (`list`, *optional*): The control video to condition the generation on. If not provided, an empty video is used.
- `mask` (`list`, *optional*): The mask that defines which video regions to condition on (black) and which to generate (white). Can only be passed if `video` is passed as well.
- `reference_images` (`Image | list`, *optional*): One or more reference images as extra conditioning for the generation.
- `conditioning_scale` (`float | list | Tensor`, *optional*, defaults to `1.0`): The conditioning scale applied in each control layer of the model. If a float, it is applied uniformly to all layers; a list or tensor must have the same length as the number of control layers.
- `height` (`None`, *optional*): No description provided
- `width` (`None`, *optional*): No description provided
- `num_frames` (`int`, *optional*, defaults to `81`): No description provided
- `generator` (`None`, *optional*): No description provided
- `num_videos_per_prompt` (`None`, *optional*, defaults to `1`): No description provided
- `num_inference_steps` (`None`, *optional*, defaults to `50`): No description provided
- `timesteps` (`None`, *optional*): No description provided
- `sigmas` (`None`, *optional*): No description provided
- `latents` (`Tensor | NoneType`, *optional*): No description provided
- `attention_kwargs` (`None`, *optional*): No description provided
- `output_type` (`str`, *optional*, defaults to `np`): The output type of the decoded videos
**Outputs:**
- `videos` (`list`): The generated videos.
|