Echo-Team/tiny-echo-wm-base-diffusers

This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.

Pipeline Type: EchoWMBlocks

Description:

This pipeline uses a 5-block architecture that can be customized and extended.

Example Usage

[TODO]

Pipeline Architecture

This modular pipeline is composed of the following blocks:

1. text (LTX2TextConditioningStep)
   • Text-conditioning stage for LTX-2.X: encodes the prompt(s), then runs the text connectors to produce the video/audio-branch connector embeddings the denoiser consumes. Outputs stay at one row per prompt -- the denoise stage expands them by num_videos_per_prompt -- so they can be reused across denoise runs.
2. camera (EchoWMCameraConditionStep)
3. image_encoder (EchoWMVaeEncoderStep)
   • VAE encoder step that encodes the input image into normalized latents for image-to-video generation.
4. denoise (EchoWMImage2VideoCoreDenoiseStep)
   • Denoise block (image-to-video) that expands the text conditioning by num_videos_per_prompt, adds image conditioning and runs the joint denoising loop.
5. decode (EchoWMDecoderStep)

Model Components

1. text_encoder (PreTrainedModel)
2. tokenizer (PreTrainedTokenizerBase)
3. connectors (LTX2TextConnectors)
4. transformer (EchoWMTransformer3DModel)
5. vae (AutoencoderKLLTX2Video)
6. video_processor (VideoProcessor)
7. scheduler (FlowMatchEulerDiscreteScheduler)
8. audio_vae (AutoencoderKLLTX2Audio)
9. guider (LTX2Guidance)
10. audio_guider (LTX2Guidance)
11. vocoder (LTX2Vocoder)

Input/Output Specification

Inputs:

• prompt (str): The prompt or prompts to guide image generation.
• negative_prompt (str, optional): The prompt or prompts not to guide the image generation.
• max_sequence_length (int, optional, defaults to 1024): Maximum sequence length for prompt encoding.
• action (str): WASD/IJKL action program.
• height (int, optional, defaults to 704): The height in pixels of the generated image.
• width (int, optional, defaults to 1280): The width in pixels of the generated image.
• num_frames (int, optional, defaults to 241): Number of output video frames.
• frame_rate (float, optional, defaults to 24.0): Output video frame rate.
• translation_speed (float, optional, defaults to 0.05): Per-frame camera translation speed for W/A/S/D actions.
• rotation_speed_deg (float, optional, defaults to 0.5): Per-frame camera yaw speed in degrees for J/L actions.
• pitch_speed_deg (float, optional, defaults to 0.2): Per-frame camera pitch speed in degrees for I/K actions.
• pitch_limit_deg (float, optional, defaults to 60.0): Maximum absolute camera pitch in degrees.
• fov_deg (float, optional, defaults to 70.0): Horizontal camera field of view in degrees.
• num_videos_per_prompt (int, optional, defaults to 1): The number of images to generate per prompt.
• image (Image | list): Reference image(s) for denoising. Can be a single image or list of images.
• image_crf (int, optional): H.264 CRF used to re-compress the conditioning image before VAE encode, matching the compression the model was trained against. None (default) resolves from the text-encoder generation (33 through LTX-2.3, 18 for LTX-2.5). Pass 0 to skip re-compression. Requires a PIL.Image.Image when re-compression runs.
• generator (Generator, optional): Torch generator for deterministic generation.
• num_inference_steps (int, optional, defaults to 30): The number of denoising steps.
• timesteps (Tensor, optional): Timesteps for the denoising process.
• sigmas (list, optional): Custom sigmas for the denoising process.
• latents (Tensor, optional): Pre-generated noisy latents for image generation.
• noise_scale (float, optional): Interpolation factor between random noise and any provided latents. None (default) resolves to 0.0, which keeps the provided latents.
• audio_latents (Tensor, optional): Optional pre-encoded audio latents; random noise is used when not provided.
• **denoiser_input_fields (None, optional): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc.
• use_cross_timestep (bool, optional, defaults to True): Whether to condition the transformer on a separate per-token cross timestep (LTX-2.3+).
• attention_kwargs (dict, optional): Additional kwargs for attention processors.
• output_type (str, optional, defaults to pil): Output format: 'pil', 'np', 'pt'.
• decode_timestep (None, optional, defaults to 0.0): The timestep at which the VAE decodes the final latents.
• decode_noise_scale (None, optional): Noise interpolation factor applied to the latents at the decode timestep.
• vae_tiling (bool, optional, defaults to True): Enable spatial and temporal VAE decoding tiles to reduce peak memory usage.
• vae_tile_size (int, optional, defaults to 512): Spatial tile long-side size in pixels; the short side follows the video aspect ratio.
• vae_tile_overlap (int, optional, defaults to 64): Spatial tile overlap in pixels.
• vae_temporal_tile_size (int, optional, defaults to 64): Temporal tile size in sample frames, excluding the causal boundary frame.
• vae_temporal_tile_overlap (int, optional, defaults to 24): Temporal tile overlap in sample frames.

Outputs:

• videos (list): The generated videos.
• audio (Tensor): The generated audio waveform.