README.md

---
library_name: diffusers
tags:
- modular-diffusers
- diffusers
- echo-wm-flash
- text-to-image
---
This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.

**Pipeline Type**: EchoWMFlashBlocks

**Description**: 

This pipeline uses a 5-block architecture that can be customized and extended.

## Example Usage

[TODO]

## Pipeline Architecture

This modular pipeline is composed of the following blocks:

1. **text** (`EchoWMFlashTextConditioningStep`)
2. **camera** (`EchoWMFlashCameraConditionStep`)
3. **image_encoder** (`EchoWMVaeEncoderStep`)
   - VAE encoder step that encodes the input `image` into normalized latents for image-to-video generation.
4. **denoise** (`EchoWMFlashDenoiseStep`)
   - Autoregressively denoise and commit each Echo-WM Flash video/audio chunk.
5. **decode** (`EchoWMDecoderStep`) 

## Model Components

1. text_encoder (`PreTrainedModel`)
2. tokenizer (`PreTrainedTokenizerBase`)
3. connectors (`LTX2TextConnectors`)
4. transformer (`EchoWMTransformer3DModel`)
5. vae (`AutoencoderKLLTX2Video`)
6. video_processor (`VideoProcessor`)
7. audio_vae (`AutoencoderKLLTX2Audio`)
8. scheduler (`FlowMatchEulerDiscreteScheduler`)
9. vocoder (`LTX2Vocoder`) 

## Input/Output Specification

**Inputs:**

- `prompt` (`str`): The prompt or prompts to guide image generation.
- `max_sequence_length` (`int`, *optional*, defaults to `1024`): Maximum sequence length for prompt encoding.
- `action` (`str`): WASD/IJKL action program.
- `height` (`int`, *optional*, defaults to `704`): The height in pixels of the generated image.
- `width` (`int`, *optional*, defaults to `1280`): The width in pixels of the generated image.
- `num_frames` (`int`, *optional*, defaults to `241`): Number of output video frames.
- `frame_rate` (`float`, *optional*, defaults to `24.0`): Output video frame rate.
- `translation_speed` (`float`, *optional*, defaults to `0.05`): Per-frame camera translation speed for W/A/S/D actions.
- `rotation_speed_deg` (`float`, *optional*, defaults to `0.4`): Per-frame camera yaw speed in degrees for J/L actions.
- `pitch_speed_deg` (`float`, *optional*, defaults to `0.2`): Per-frame camera pitch speed in degrees for I/K actions.
- `pitch_limit_deg` (`float`, *optional*, defaults to `40.0`): Maximum absolute camera pitch in degrees.
- `fov_deg` (`float`, *optional*, defaults to `70.0`): Horizontal camera field of view in degrees.
- `num_videos_per_prompt` (`int`, *optional*, defaults to `1`): The number of images to generate per prompt.
- `image` (`Image | list`): Reference image(s) for denoising. Can be a single image or list of images.
- `image_crf` (`int`, *optional*): H.264 CRF used to re-compress the conditioning `image` before VAE encode, matching the compression the model was trained against. `None` (default) resolves from the text-encoder generation (33 through LTX-2.3, 18 for LTX-2.5). Pass `0` to skip re-compression. Requires a `PIL.Image.Image` when re-compression runs.
- `generator` (`Generator`, *optional*): Torch generator for deterministic generation.
- `timesteps` (`list`, *optional*, defaults to `[1000, 750, 500, 250]`): Distilled denoising timestep IDs for each autoregressive chunk.
- `video_cache_size` (`int`, *optional*, defaults to `19`): Maximum number of latent video frames retained in the bounded KV cache.
- `video_sink_size` (`int`, *optional*, defaults to `7`): Number of leading latent video frames permanently retained as the cache sink.
- `output_type` (`str`, *optional*, defaults to `pil`): Output format: 'pil', 'np', 'pt'.
- `decode_timestep` (`None`, *optional*, defaults to `0.0`): The timestep at which the VAE decodes the final latents.
- `decode_noise_scale` (`None`, *optional*): Noise interpolation factor applied to the latents at the decode timestep.
- `vae_tiling` (`bool`, *optional*, defaults to `True`): Enable spatial and temporal VAE decoding tiles to reduce peak memory usage.
- `vae_tile_size` (`int`, *optional*, defaults to `512`): Spatial tile long-side size in pixels; the short side follows the video aspect ratio.
- `vae_tile_overlap` (`int`, *optional*, defaults to `64`): Spatial tile overlap in pixels.
- `vae_temporal_tile_size` (`int`, *optional*, defaults to `64`): Temporal tile size in sample frames, excluding the causal boundary frame.
- `vae_temporal_tile_overlap` (`int`, *optional*, defaults to `24`): Temporal tile overlap in sample frames.

**Outputs:**

- `videos` (`list`): The generated videos.
- `audio` (`Tensor`): The generated audio waveform.