--- license: other license_name: edenlive-mixed license_link: LICENSE pipeline_tag: image-to-video tags: - video-generation - diffusion - real-time ---
# EdenLive **EdenLive: Real-Time Generative Rendering from Any Visual Reference** [![Project Page](https://img.shields.io/badge/Project-Page-blue)](https://vast-eden.ai/?p=edenlive) [![Code](https://img.shields.io/badge/Code-GitHub-black)](https://github.com/VAST-AI-Research/EdenLive) [![License](https://img.shields.io/badge/License-Mixed%20(see%20below)-orange)](#license) [Project Page](https://vast-eden.ai/?p=edenlive) · [Code](https://github.com/VAST-AI-Research/EdenLive) · [Models](https://huggingface.co/VAST-AI/EdenLive) Zehuan Huang1,*, Tianxing Xu1,*, Chunli Peng1,*, Cheng Zhang1,*, Yifei Song1,†, Zeyi Zheng1,†, Zhipeng Yu1, Peng Wang1,‡, Yanpei Cao1,✉ \* Equal contribution · † Intern at VAST · ‡ Project lead · ✉ Corresponding author 1[VAST](https://www.tripo3d.ai/) · [Project Eden](https://vast-eden.ai/)
## Overview EdenLive renders a **proxy or reference video stream**, a **target first frame**, and a **text prompt** into refined or stylized video. References can come from 2D/3D scenes, existing engines, or recordings. Upstream code maintains interaction state; the model handles visual rendering. Control is injected through a **vision encoder**: a post-trained DINOv3 encoder embeds each proxy frame, and those features condition the diffusion transformer. The [project page](https://vast-eden.ai/?p=edenlive) demonstrates over 24 FPS, minute-scale generation, and 480P/720P output. The 5B model supports single-GPU deployment at 720P. This repository holds the model weights. Code, installation instructions, and inference/streaming usage live in the [EdenLive repository](https://github.com/VAST-AI-Research/EdenLive). ## Model weights ```bash hf download VAST-AI/EdenLive --local-dir ./models/EdenLive ``` Everything loads from one directory, `models/EdenLive` under the code repository root by default (about 210 GB in total); nothing is pulled from the Hub at run time once downloaded. ``` models/EdenLive/ dit-14b/ diffusion_pytorch_model.safetensors, config.json 14B few-step (chunk-AR) DiT dit-14b-bidir/ diffusion_pytorch_model.safetensors, config.json 14B bidirectional DiT dit-5b/ diffusion_pytorch_model.safetensors, config.json 5B few-step (chunk-AR) DiT dit-5b-bidir/ diffusion_pytorch_model.safetensors, config.json 5B bidirectional DiT vae-14b/ Wan2.1_VAE.pth, config.json vae-5b/ Wan2.2_VAE.pth, config.json decoder-14b/ lighttaew2_1.pth TAEHV decoder (real-time only) decoder-5b/ lighttaew2_2.pth TAEHV decoder (real-time only) scheduler-14b/ scheduler_config.json offline only scheduler-5b/ scheduler_config.json offline only umt5/ text_encoder/, tokenizer/ UMT5 text encoder clip/ image_encoder/, image_processor/ CLIP, 14B I2V only vision-encoder/ backbone/, projection.safetensors proxy-DINO encoder flux-c2f/ Diffusers FLUX tree + c2f_lora.safetensors workbench first frame (real-time only) ``` To load the weights from another directory: ```bash python -m edenlive.scripts.infer --config ... model.pretrained_dir=/data/EdenLive ``` The real-time service reads the same tree through `MODEL_ROOT` in [`realtime/deploy.conf`](https://github.com/VAST-AI-Research/EdenLive/blob/main/realtime/deploy.conf). See the [main repository's README](https://github.com/VAST-AI-Research/EdenLive#model-weights) for how each component is wired into the offline and real-time pipelines. ## Usage Each DiT has a matching inference profile in the code repository: | Mode | Wan2.1-I2V-14B @ 480p | Wan2.2-TI2V-5B @ 720p | | --- | --- | --- | | Bidirectional (`dit-*-bidir/`) | `wan21_14b_480p_bidir.yaml` | `wan22_5b_720p_bidir.yaml` | | Few-step, chunk-AR (`dit-14b/`, `dit-5b/`) | `wan21_14b_480p_few_step.yaml` | `wan22_5b_720p_few_step.yaml` | Few-step inference (chunk-autoregressive sampling, 4 steps per chunk) on the bundled samples: ```bash torchrun --nproc_per_node=2 -m edenlive.scripts.infer \ --config edenlive/configs/wan21_14b_480p_few_step.yaml \ runtime.sequence_parallel.size=2 ``` ```bash python -m edenlive.scripts.infer \ --config edenlive/configs/wan22_5b_720p_few_step.yaml ``` See the [Quick start](https://github.com/VAST-AI-Research/EdenLive#quick-start) for bidirectional inference, the real-time service, and your own data. ## Citation ```bibtex @misc{edenlive2026, title = {EdenLive: Real-Time Generative Rendering from Any Visual Reference}, author = {Zehuan Huang and Tianxing Xu and Chunli Peng and Cheng Zhang and Yifei Song and Zeyi Zheng and Zhipeng Yu and Peng Wang and Yanpei Cao}, year = {2026}, url = {https://vast-eden.ai/?p=edenlive} } ``` ## Acknowledgements Built on [Wan2.1 / Wan2.2](https://github.com/Wan-Video), 🤗 [Diffusers](https://github.com/huggingface/diffusers), [DINOv3](https://github.com/facebookresearch/dinov3), [FLUX.2 \[klein\]](https://bfl.ai/models/flux-2-klein) by Black Forest Labs, the [LightX2V](https://huggingface.co/lightx2v/Autoencoders) LightTAE decoders, and [LAION CLIP ViT-H/14](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) trained with [OpenCLIP](https://github.com/mlfoundations/open_clip). ## License The weights in this repository are covered by different licenses, per component: | Component | License | Notes | | --- | --- | --- | | `dit-*`, `vae-*`, `umt5/`, `scheduler-*` | [Apache 2.0](LICENSE) | Derived from Wan2.1 / Wan2.2 (Apache 2.0) | | `decoder-*` | [Apache 2.0](LICENSE) | LightTAE decoders from [lightx2v/Autoencoders](https://huggingface.co/lightx2v/Autoencoders) (Apache 2.0) | | `clip/` | [MIT](clip/LICENSE.md) | [LAION CLIP ViT-H/14](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) trained with OpenCLIP; see [NOTICE](clip/NOTICE.md) | | `vision-encoder/` | [DINOv3 License](vision-encoder/LICENSE.md) | DINOv3 by Meta and derivatives of it; see [NOTICE](vision-encoder/NOTICE.md) | | `flux-c2f/` | [FLUX Non-Commercial License v2.1](flux-c2f/LICENSE.md) | FLUX.2 \[klein\] 9B by Black Forest Labs plus a LoRA by Vast AI; **non-commercial, non-production use only**; see [NOTICE](flux-c2f/NOTICE.md) | `flux-c2f/` is used only to render the real-time workbench's photoreal first frame; the offline and real-time video pipelines do not load it. Using the workbench therefore falls under the FLUX Non-Commercial License, while video generation does not depend on it. Public end-user or production deployment is not automatically permitted by being free of charge or described as a demo; check the license and obtain a separate authorization from Black Forest Labs where required.