diffusers/pr_15066/en/api/pipelines/edenlive.md

EdenLive

EdenLive generates video from a proxy video, a reference image, and a text prompt. EdenLivePipeline supports bidirectional generation; EdenLiveCausalPipeline generates a complete video in causal chunks and supports changing prompts between chunks.

Basic usage

Prepare an RGB reference image and a proxy video with at least 49 frames sampled at 24 fps. The image provides appearance, while the proxy provides motion and scene structure.

import torch
from diffusers import EdenLivePipeline
from diffusers.utils import export_to_video, load_image, load_video

pipe = EdenLivePipeline.from_pretrained(
    "VAST-AI/EdenLive-5B-Diffusers",
    torch_dtype={"default": torch.bfloat16, "proxy_encoder": torch.float32},
).to("cuda")

image = load_image("reference.png")
video = load_video("proxy-24fps.mp4")

frames = pipe(
    image=image,
    video=video,
    prompt="A character explores a snowy landscape.",
    num_frames=49,
).frames[0]
export_to_video(frames, "edenlive.mp4", fps=24)

Keep the proxy encoder in FP32 when loading the other models in BF16, as shown above. The transformer keeps its internal proxy projection in FP32 when loaded with from_pretrained.

Causal generation

Load the causal pipeline from its dedicated repository. Both model sizes default to four denoising steps per chunk.

from diffusers import EdenLiveCausalPipeline

causal_pipe = EdenLiveCausalPipeline.from_pretrained(
    "VAST-AI/EdenLive-5B-Causal-Diffusers",
    torch_dtype={"default": torch.bfloat16, "proxy_encoder": torch.float32},
).to("cuda")

Use prompt for fixed text, or chunk_prompts to change it during a single video. At the default chunk size, 49 frames require five prompts. Reusing the image and video above:

frames = causal_pipe(
    image=image,
    video=video,
    chunk_prompts=[
        "A character walks through snow.",
        "A character walks through snow.",
        "The character approaches a campfire.",
        "The character warms their hands beside the campfire.",
        "The character rests beside the campfire.",
    ],
    num_frames=49,
).frames[0]
export_to_video(frames, "chunk-prompts.mp4", fps=24)

chunk_prompts overrides a single prompt and cannot be combined with prompt_embeds. The negative prompt remains fixed. A list passed as prompt represents a batch of independent videos. Each pipeline call starts a new video.

LightTAE decoder

The full Wan VAE is the default. For faster decoding, pass the optional EdenLiveLightTAEDecoder to either pipeline:

from diffusers import EdenLiveLightTAEDecoder

pipe = EdenLivePipeline.from_pretrained(
    "VAST-AI/EdenLive-5B-Diffusers",
    decoder=EdenLiveLightTAEDecoder.from_pretrained(
        "VAST-AI/EdenLive-5B-Diffusers", subfolder="decoder", torch_dtype=torch.bfloat16
    ),
    torch_dtype={"default": torch.bfloat16, "proxy_encoder": torch.float32},
).to("cuda")

Notes

Checkpoint	Pipeline	Default height × width	Example fps	Steps
5B bidirectional	EdenLivePipeline	704 × 1280	24	48
14B bidirectional	EdenLivePipeline	480 × 832	16	30
5B causal	EdenLiveCausalPipeline	704 × 1280	24	4 per chunk
14B causal	EdenLiveCausalPipeline	480 × 832	16	4 per chunk

• Sample proxy frames at the desired output frame rate. num_frames must be $4k+1$ and fit within the input video; omitting it uses the largest valid prefix. Custom dimensions must be multiples of 32 for 5B or 16 for 14B.
• Guidance defaults to 3 for bidirectional generation and 1 for causal generation. Set guidance_scale above one to use a negative_prompt.
• For CPU offload, load without .to("cuda") and call enable_model_cpu_offload().
• See the example README for installation, video resampling, CPU offload, and checkpoint conversion.

EdenLivePipeline[[diffusers.EdenLivePipeline]]

diffusers.EdenLivePipeline[[diffusers.EdenLivePipeline]]

diffusers.EdenLivePipeline(vae: AutoencoderKLWan, text_encoder: UMT5EncoderModel, tokenizer: AutoTokenizer, transformer: EdenLiveTransformer3DModel, scheduler: diffusers.schedulers.scheduling_flow_match_euler_discrete.FlowMatchEulerDiscreteScheduler | diffusers.schedulers.scheduling_unipc_multistep.UniPCMultistepScheduler, proxy_encoder: DINOv3ViTModel, image_encoder: transformers.models.clip.modeling_clip.CLIPVisionModel | None = None, image_processor: transformers.models.clip.image_processing_pil_clip.CLIPImageProcessorPil | None = None, decoder: diffusers.models.autoencoders.autoencoder_edenlive.EdenLiveLightTAEDecoder | None = None, proxy_image_height: int = 480, proxy_image_width: int = 832)

Source

Parameters:

vae (AutoencoderKLWan) : Wan VAE for reference encoding and full-quality video decoding.

text_encoder (UMT5EncoderModel) : Text encoder.

tokenizer (AutoTokenizer) : Tokenizer corresponding to the text encoder.

transformer (EdenLiveTransformer3DModel) : EdenLive bidirectional video denoiser.

scheduler (FlowMatchEulerDiscreteScheduler or UniPCMultistepScheduler) : Scheduler corresponding to the transformer checkpoint.

proxy_encoder (DINOv3ViTModel) : Trained RGB proxy encoder.

image_encoder (CLIPVisionModel, optional) : Reference image encoder required by the 14B model.

image_processor (CLIPImageProcessor, optional) : Processor corresponding to the image encoder.

decoder (EdenLiveLightTAEDecoder, optional) : Optional faster decoder. When omitted, decode with the full Wan VAE.

proxy_image_height (int, defaults to 480) : Height of the proxy images passed to DINO.

proxy_image_width (int, defaults to 832) : Width of the proxy images passed to DINO.

Bidirectional video generation conditioned on a reference image, text, and an RGB proxy video.

call[[diffusers.EdenLivePipeline.call]]

__call__(prompt: str | list[str] | None = None, image = None, video = None, height: int | None = None, width: int | None = None, num_frames: int | None = None, num_inference_steps: int | None = None, guidance_scale: float | None = None, negative_prompt: str | list[str] | None = None, generator: typing.Union[torch.Generator, list[torch.Generator], NoneType] = None, latents: typing.Optional[torch.Tensor] = None, prompt_embeds: typing.Optional[torch.Tensor] = None, negative_prompt_embeds: typing.Optional[torch.Tensor] = None, max_sequence_length: int = 512, output_type: str = 'np', return_dict: bool = True, attention_kwargs: dict | None = None, callback_on_step_end: typing.Optional[typing.Callable] = None, callback_on_step_end_tensor_inputs: list[str] | None = None)

Source

Parameters:

prompt (str or list[str], optional) : Text describing the output video. Supply this or prompt_embeds.

image (PIL.Image.Image, np.ndarray, or torch.Tensor) : Photoreal first-frame reference. Tensor and NumPy RGB values are in [0, 1]. The image is resized with tensor bilinear interpolation and center-cropped to the output dimensions.

video (list[PIL.Image.Image], np.ndarray, or torch.Tensor) : Full-rate decoded RGB proxy video, already sampled at the desired output frame rate. Tensor inputs have shape (T, C, H, W) or (B, T, C, H, W); NumPy inputs use channels last. Floating-point values are in [0, 1]. Proxy frames are sampled at the VAE temporal stride exactly once.

height (int, optional) : Output height. Defaults to 704 for 5B or 480 for 14B.

width (int, optional) : Output width. Defaults to 1280 for 5B or 832 for 14B.

num_frames (int, optional) : Number of decoded output frames, of the form 4k + 1. When omitted, use the largest valid input prefix. The requested length cannot exceed the input video; missing frames are never repeated.

num_inference_steps (int, optional) : Denoising steps; defaults to 48 for 5B or 30 for 14B.

guidance_scale (float, optional) : Classifier-free guidance strength; defaults to 3. Values greater than one enable the negative prompt branch.

negative_prompt (str or list[str], optional) : Text describing content to discourage when classifier-free guidance is enabled.

generator (torch.Generator or list[torch.Generator], optional) : Random generator, or one per batch item. CPU generators are supported.

latents (torch.Tensor, optional) : Initial noise with shape (B, C, T_latent, H_latent, W_latent). When omitted, noise for the whole video is sampled before image conditioning. The fixed first frame still consumes its noise draw.

prompt_embeds (torch.Tensor, optional) : Precomputed positive text embeddings in place of prompt.

negative_prompt_embeds (torch.Tensor, optional) : Precomputed negative text embeddings in place of negative_prompt.

max_sequence_length (int, defaults to 512) : Token count for each text prompt, including zero padding.

output_type (str, defaults to "np") : Return NumPy videos ("np", shape (B, T, H, W, 3)), PyTorch videos ("pt", shape (B, T, 3, H, W)), nested PIL frame lists ("pil"), or normalized diffusion latents ("latent", shape (B, C, T_latent, H_latent, W_latent)). Decoded tensor and NumPy values are in [0, 1].

return_dict (bool, defaults to True) : Return EdenLivePipelineOutput; otherwise return a one-element tuple containing the frames.

attention_kwargs (dict, optional) : Transformer options. The scale entry sets the LoRA scale when adapters are loaded.

callback_on_step_end (Callable, optional) : Function called as callback(self, step_index, timestep, tensors) after each denoising step. Return a dictionary containing any replacement tensors.

callback_on_step_end_tensor_inputs (list[str], defaults to ["latents"]) : Tensors supplied to the callback, selected from latents, prompt_embeds, and negative_prompt_embeds.

Returns: EdenLivePipelineOutput or tuple

Generated frames, or latent videos when output_type="latent".

Generate a complete bidirectional video.

Examples:

>>> import torch
>>> from diffusers import EdenLivePipeline
>>> from diffusers.utils import export_to_video, load_image, load_video
>>> pipe = EdenLivePipeline.from_pretrained(
...     "VAST-AI/EdenLive-5B-Diffusers",
...     torch_dtype={"default": torch.bfloat16, "proxy_encoder": torch.float32},
... ).to("cuda")
>>> frames = pipe(
...     image=load_image("reference.png"), video=load_video("proxy.mp4"),
...     prompt="A character walks through snow.", num_frames=49,
... ).frames[0]
>>> export_to_video(frames, "edenlive.mp4", fps=24)

check_inputs[[diffusers.EdenLivePipeline.check_inputs]]

check_inputs(prompt, prompt_embeds, negative_prompt, negative_prompt_embeds, image, video, height, width, output_type, callback_inputs)

Source

Validate conditioning inputs, output dimensions, and callback tensor names.

decode_latents[[diffusers.EdenLivePipeline.decode_latents]]

decode_latents(latents, output_type)

Source

Decode normalized diffusion latents with the VAE or optional LightTAE.

encode_prompt[[diffusers.EdenLivePipeline.encode_prompt]]

encode_prompt(prompt = None, negative_prompt = None, do_classifier_free_guidance = True, prompt_embeds = None, negative_prompt_embeds = None, max_sequence_length = 512, device = None, dtype = None)

Source

Encode positive and optional negative text with UMT5, zeroing padded token embeddings.

prepare_conditions[[diffusers.EdenLivePipeline.prepare_conditions]]

prepare_conditions(image, proxy_video, height, width, num_frames, batch_size, device, dtype)

Source

Encode the reference image and sampled RGB proxy frames for the video transformer.

prepare_latents[[diffusers.EdenLivePipeline.prepare_latents]]

prepare_latents(batch_size, height, width, num_frames, dtype, device, generator = None, latents = None)

Source

Prepare full-video initial noise, including the fixed first frame's noise draw.

EdenLiveCausalPipeline[[diffusers.EdenLiveCausalPipeline]]

diffusers.EdenLiveCausalPipeline[[diffusers.EdenLiveCausalPipeline]]

diffusers.EdenLiveCausalPipeline(vae: AutoencoderKLWan, text_encoder: UMT5EncoderModel, tokenizer: AutoTokenizer, transformer: EdenLiveTransformer3DModel, scheduler: FlowMatchEulerDiscreteScheduler, proxy_encoder: DINOv3ViTModel, image_encoder: transformers.models.clip.modeling_clip.CLIPVisionModel | None = None, image_processor: transformers.models.clip.image_processing_pil_clip.CLIPImageProcessorPil | None = None, decoder: diffusers.models.autoencoders.autoencoder_edenlive.EdenLiveLightTAEDecoder | None = None, proxy_image_height: int = 480, proxy_image_width: int = 832, chunk_size: int = 3, max_context_frames: int = 18, sink_frames: int = 6)

Source

Parameters:

vae (AutoencoderKLWan) : Wan VAE for reference encoding and full-quality video decoding.

text_encoder (UMT5EncoderModel) : Text encoder.

tokenizer (AutoTokenizer) : Tokenizer corresponding to the text encoder.

transformer (EdenLiveTransformer3DModel) : EdenLive causal video denoiser.

scheduler (FlowMatchEulerDiscreteScheduler) : Scheduler corresponding to the transformer checkpoint.

proxy_encoder (DINOv3ViTModel) : Trained RGB proxy encoder.

image_encoder (CLIPVisionModel, optional) : Reference image encoder required by the 14B model.

image_processor (CLIPImageProcessor, optional) : Processor corresponding to the image encoder.

decoder (EdenLiveLightTAEDecoder, optional) : Optional faster decoder. When omitted, decode with the full Wan VAE.

proxy_image_height (int, defaults to 480) : Height of the proxy images passed to DINO.

proxy_image_width (int, defaults to 832) : Width of the proxy images passed to DINO.

chunk_size (int, defaults to 3) : Latent frames generated in each chunk.

max_context_frames (int, defaults to 18) : Maximum number of latent frames in the attention window, including the current chunk.

sink_frames (int, defaults to 6) : Initial latent frames retained when the attention window fills.

Causal video generation conditioned on a reference image, text, and an RGB proxy video.

call[[diffusers.EdenLiveCausalPipeline.call]]

__call__(prompt: str | list[str] | None = None, image = None, video = None, height: int | None = None, width: int | None = None, num_frames: int | None = None, num_inference_steps: int | None = None, guidance_scale: float | None = None, negative_prompt: str | list[str] | None = None, generator: typing.Union[torch.Generator, list[torch.Generator], NoneType] = None, latents: typing.Optional[torch.Tensor] = None, prompt_embeds: typing.Optional[torch.Tensor] = None, negative_prompt_embeds: typing.Optional[torch.Tensor] = None, chunk_prompts: list[str] | None = None, max_sequence_length: int = 512, output_type: str = 'np', return_dict: bool = True, attention_kwargs: dict | None = None, callback_on_step_end: typing.Optional[typing.Callable] = None, callback_on_step_end_tensor_inputs: list[str] | None = None)

Source

Parameters:

prompt (str or list[str], optional) : Text describing the output video. Supply this or prompt_embeds unless using chunk_prompts.

image (PIL.Image.Image, np.ndarray, or torch.Tensor) : Photoreal first-frame reference. Tensor and NumPy RGB values are in [0, 1]. The image is resized with tensor bilinear interpolation and center-cropped to the output dimensions.

video (list[PIL.Image.Image], np.ndarray, or torch.Tensor) : Full-rate decoded RGB proxy video, already sampled at the desired output frame rate. Tensor inputs have shape (T, C, H, W) or (B, T, C, H, W); NumPy inputs use channels last. Floating-point values are in [0, 1]. Proxy frames are sampled at the VAE temporal stride exactly once.

height (int, optional) : Output height. Defaults to 704 for 5B or 480 for 14B.

width (int, optional) : Output width. Defaults to 1280 for 5B or 832 for 14B.

num_frames (int, optional) : Number of decoded output frames, of the form 4k + 1. When omitted, use the largest valid input prefix. The requested length cannot exceed the input video; missing frames are never repeated.

num_inference_steps (int, optional) : Denoising steps per chunk. Defaults to four for the released causal checkpoints.

guidance_scale (float, optional) : Classifier-free guidance strength; defaults to 1. Values greater than one enable the negative prompt branch.

negative_prompt (str or list[str], optional) : Text describing content to discourage when classifier-free guidance is enabled.

generator (torch.Generator or list[torch.Generator], optional) : Random generator, or one per batch item. CPU generators are supported.

latents (torch.Tensor, optional) : Initial noise with shape (B, C, T_latent, H_latent, W_latent). When omitted, noise for the whole video is sampled before image conditioning. The fixed first frame still consumes its noise draw.

prompt_embeds (torch.Tensor, optional) : Precomputed positive text embeddings in place of prompt.

negative_prompt_embeds (torch.Tensor, optional) : Precomputed negative text embeddings in place of negative_prompt.

chunk_prompts (list[str], optional) : One positive prompt per latent chunk, including a short final chunk. Overrides a single prompt. Requires one video and cannot be combined with prompt_embeds. The negative prompt stays fixed. Text changes take effect at chunk boundaries while retaining previously generated video history.

max_sequence_length (int, defaults to 512) : Token count for each text prompt, including zero padding.

output_type (str, defaults to "np") : Return NumPy videos ("np", shape (B, T, H, W, 3)), PyTorch videos ("pt", shape (B, T, 3, H, W)), nested PIL frame lists ("pil"), or normalized diffusion latents ("latent", shape (B, C, T_latent, H_latent, W_latent)). Decoded tensor and NumPy values are in [0, 1].

return_dict (bool, defaults to True) : Return EdenLivePipelineOutput; otherwise return a one-element tuple containing the frames.

attention_kwargs (dict, optional) : Transformer options. The scale entry sets the LoRA scale when adapters are loaded.

callback_on_step_end (Callable, optional) : Function called as callback(self, step_index, timestep, tensors) after each denoising step. Return a dictionary containing any replacement tensors. In causal mode the tensors contain the current chunk and step indices count across all chunks.

callback_on_step_end_tensor_inputs (list[str], defaults to ["latents"]) : Tensors supplied to the callback, selected from latents, prompt_embeds, and negative_prompt_embeds.

Returns: EdenLivePipelineOutput or tuple

Generated frames, or latent videos when output_type="latent".

Generate a complete causal video.

Examples:

>>> import torch
>>> from diffusers import EdenLiveCausalPipeline
>>> from diffusers.utils import export_to_video, load_image, load_video
>>> pipe = EdenLiveCausalPipeline.from_pretrained(
...     "VAST-AI/EdenLive-5B-Causal-Diffusers",
...     torch_dtype={"default": torch.bfloat16, "proxy_encoder": torch.float32},
... ).to("cuda")
>>> frames = pipe(
...     image=load_image("reference.png"), video=load_video("proxy.mp4"),
...     prompt="A character walks through snow.", num_frames=49,
... ).frames[0]
>>> export_to_video(frames, "edenlive.mp4", fps=24)

check_inputs[[diffusers.EdenLiveCausalPipeline.check_inputs]]

check_inputs(prompt, prompt_embeds, negative_prompt, negative_prompt_embeds, image, video, height, width, output_type, callback_inputs)

Source

Validate conditioning inputs, output dimensions, and callback tensor names.

decode_latents[[diffusers.EdenLiveCausalPipeline.decode_latents]]

decode_latents(latents, output_type)

Source

Decode normalized diffusion latents with the VAE or optional LightTAE.

encode_prompt[[diffusers.EdenLiveCausalPipeline.encode_prompt]]

encode_prompt(prompt = None, negative_prompt = None, do_classifier_free_guidance = True, prompt_embeds = None, negative_prompt_embeds = None, max_sequence_length = 512, device = None, dtype = None)

Source

Encode positive and optional negative text with UMT5, zeroing padded token embeddings.

prepare_conditions[[diffusers.EdenLiveCausalPipeline.prepare_conditions]]

prepare_conditions(image, proxy_video, height, width, num_frames, batch_size, device, dtype)

Source

Encode the reference image and sampled RGB proxy frames for the video transformer.

prepare_latents[[diffusers.EdenLiveCausalPipeline.prepare_latents]]

prepare_latents(batch_size, height, width, num_frames, dtype, device, generator = None, latents = None)

Source

Prepare full-video initial noise, including the fixed first frame's noise draw.

EdenLiveLightTAEDecoder[[diffusers.EdenLiveLightTAEDecoder]]

diffusers.EdenLiveLightTAEDecoder[[diffusers.EdenLiveLightTAEDecoder]]

diffusers.EdenLiveLightTAEDecoder(latent_channels = 16, patch_size = 1, block_out_channels = (256, 128, 64, 64))

Source

Parameters:

latent_channels (int, optional, defaults to 16) : Latent channels: 16 for Wan 2.1 / EdenLive 14B, or 48 for Wan 2.2 / EdenLive 5B.

patch_size (int, optional, defaults to 1) : Output pixel shuffle factor: 1 for 14B or 2 for 5B.

block_out_channels (tuple[int], optional, defaults to (256, 128, 64, 64)) : Decoder widths at its four spatial resolutions.

Decoder-only LightTAE for EdenLive's Wan latents.

Input latents use the VAE's raw (unnormalized) scale and [batch, channels, frames, height, width] layout. Decoded RGB samples use the same layout, range [-1, 1], and (frames - 1) * 4 + 1 frames. Temporal state belongs to each decode call and is never retained on the model.

decode[[diffusers.EdenLiveLightTAEDecoder.decode]]

decode(latents: Tensor, return_dict: bool = True)

Source

Parameters:

latents (torch.Tensor) : Unnormalized VAE latents with shape (B, C, T_latent, H_latent, W_latent) on the decoder device and in its dtype. C must equal config.latent_channels, and at least one latent frame is required. Pipeline outputs with output_type="latent" must first be unnormalized with the VAE mean and standard deviation. The pipeline performs this conversion automatically when decoding frames.

return_dict (bool, defaults to True) : Return a DecoderOutput when true; otherwise return a one-element tuple containing the sample.

Returns: DecoderOutput or tuple[torch.Tensor]

RGB sample in (B, 3, T, H, W) layout and [-1, 1] range. The output has (T_latent - 1) * scale_factor_temporal + 1 frames and spatial dimensions multiplied by scale_factor_spatial. Internal chunks share temporal state, including a short final chunk.

Decode a complete video from raw VAE latents.

EdenLivePipelineOutput[[diffusers.EdenLivePipelineOutput]]

diffusers.EdenLivePipelineOutput[[diffusers.EdenLivePipelineOutput]]

diffusers.EdenLivePipelineOutput(frames: typing.Union[torch.Tensor, numpy.ndarray, list[list[PIL.Image.Image]]])

Source

Parameters:

frames (torch.Tensor, np.ndarray, or list[list[PIL.Image.Image]]) : Generated videos selected by output_type. "np" returns shape (B, T, H, W, 3) and "pt" returns (B, T, 3, H, W), both with floating-point RGB values in [0, 1]. "pil" returns one list of RGB images per video. "latent" bypasses decoding and returns normalized diffusion latents with shape (B, C, T_latent, H_latent, W_latent); these values are not bounded RGB pixels. B is the video batch size and T is the decoded frame count. The latent channel count C is 48 for 5B and 16 for 14B.

Output of EdenLivePipeline.