EdenLive / README.md

Update dit-14b to SGF-v6 step_000400 EMA; align README with the code repository

de40b28 verified 1 day ago

7.05 kB

license: other
license_name: edenlive-mixed
license_link: LICENSE
pipeline_tag: image-to-video
tags:
  - video-generation
  - diffusion
  - real-time

EdenLive

EdenLive: Real-Time Generative Rendering from Any Visual Reference

Project Page · Code · Models

Zehuan Huang^1,, Tianxing Xu^1,, Chunli Peng^1,, Cheng Zhang^1,, Yifei Song^1,†, Zeyi Zheng^1,†, Zhipeng Yu¹, Peng Wang^1,‡, Yanpei Cao^1,✉

* Equal contribution · † Intern at VAST · ‡ Project lead · ✉ Corresponding author

¹VAST · Project Eden

Overview

EdenLive renders a proxy or reference video stream, a target first frame, and a text prompt into refined or stylized video. References can come from 2D/3D scenes, existing engines, or recordings. Upstream code maintains interaction state; the model handles visual rendering.

Control is injected through a vision encoder: a post-trained DINOv3 encoder embeds each proxy frame, and those features condition the diffusion transformer.

The project page demonstrates over 24 FPS, minute-scale generation, and 480P/720P output. The 5B model supports single-GPU deployment at 720P.

This repository holds the model weights. Code, installation instructions, and inference/streaming usage live in the EdenLive repository.

Model weights

hf download VAST-AI/EdenLive --local-dir ./models/EdenLive

Everything loads from one directory, models/EdenLive under the code repository root by default (about 210 GB in total); nothing is pulled from the Hub at run time once downloaded.

models/EdenLive/
  dit-14b/            diffusion_pytorch_model.safetensors, config.json  14B few-step (chunk-AR) DiT
  dit-14b-bidir/      diffusion_pytorch_model.safetensors, config.json  14B bidirectional DiT
  dit-5b/             diffusion_pytorch_model.safetensors, config.json  5B few-step (chunk-AR) DiT
  dit-5b-bidir/       diffusion_pytorch_model.safetensors, config.json  5B bidirectional DiT
  vae-14b/            Wan2.1_VAE.pth, config.json
  vae-5b/             Wan2.2_VAE.pth, config.json
  decoder-14b/        lighttaew2_1.pth                    TAEHV decoder (real-time only)
  decoder-5b/         lighttaew2_2.pth                    TAEHV decoder (real-time only)
  scheduler-14b/      scheduler_config.json               offline only
  scheduler-5b/       scheduler_config.json               offline only
  umt5/               text_encoder/, tokenizer/           UMT5 text encoder
  clip/               image_encoder/, image_processor/    CLIP, 14B I2V only
  vision-encoder/     backbone/, projection.safetensors   proxy-DINO encoder
  flux-c2f/           Diffusers FLUX tree + c2f_lora.safetensors   workbench first frame (real-time only)

To load the weights from another directory:

python -m edenlive.scripts.infer --config ... model.pretrained_dir=/data/EdenLive

The real-time service reads the same tree through MODEL_ROOT in realtime/deploy.conf. See the main repository's README for how each component is wired into the offline and real-time pipelines.

Usage

Each DiT has a matching inference profile in the code repository:

Mode	Wan2.1-I2V-14B @ 480p	Wan2.2-TI2V-5B @ 720p
Bidirectional (`dit-*-bidir/`)	`wan21_14b_480p_bidir.yaml`	`wan22_5b_720p_bidir.yaml`
Few-step, chunk-AR (`dit-14b/`, `dit-5b/`)	`wan21_14b_480p_few_step.yaml`	`wan22_5b_720p_few_step.yaml`

Few-step inference (chunk-autoregressive sampling, 4 steps per chunk) on the bundled samples:

torchrun --nproc_per_node=2 -m edenlive.scripts.infer \
  --config edenlive/configs/wan21_14b_480p_few_step.yaml \
  runtime.sequence_parallel.size=2

python -m edenlive.scripts.infer \
  --config edenlive/configs/wan22_5b_720p_few_step.yaml

See the Quick start for bidirectional inference, the real-time service, and your own data.

Citation

@misc{edenlive2026,
  title  = {EdenLive: Real-Time Generative Rendering from Any Visual Reference},
  author = {Zehuan Huang and Tianxing Xu and Chunli Peng and Cheng Zhang and Yifei Song and Zeyi Zheng and Zhipeng Yu and Peng Wang and Yanpei Cao},
  year   = {2026},
  url    = {https://vast-eden.ai/?p=edenlive}
}

Acknowledgements

Built on Wan2.1 / Wan2.2, 🤗 Diffusers, DINOv3, FLUX.2 [klein] by Black Forest Labs, the LightX2V LightTAE decoders, and LAION CLIP ViT-H/14 trained with OpenCLIP.

License

The weights in this repository are covered by different licenses, per component:

Component	License	Notes
`dit-`, `vae-`, `umt5/`, `scheduler-*`	Apache 2.0	Derived from Wan2.1 / Wan2.2 (Apache 2.0)
`decoder-*`	Apache 2.0	LightTAE decoders from lightx2v/Autoencoders (Apache 2.0)
`clip/`	MIT	LAION CLIP ViT-H/14 trained with OpenCLIP; see NOTICE
`vision-encoder/`	DINOv3 License	DINOv3 by Meta and derivatives of it; see NOTICE
`flux-c2f/`	FLUX Non-Commercial License v2.1	FLUX.2 [klein] 9B by Black Forest Labs plus a LoRA by Vast AI; non-commercial, non-production use only; see NOTICE

flux-c2f/ is used only to render the real-time workbench's photoreal first frame; the offline and real-time video pipelines do not load it. Using the workbench therefore falls under the FLUX Non-Commercial License, while video generation does not depend on it. Public end-user or production deployment is not automatically permitted by being free of charge or described as a demo; check the license and obtain a separate authorization from Black Forest Labs where required.