README.md

---
license: other
license_name: edenlive-mixed
license_link: LICENSE
pipeline_tag: image-to-video
tags:
  - video-generation
  - diffusion
  - real-time
---

<div align="center">

# EdenLive

**EdenLive: Real-Time Generative Rendering from Any Visual Reference**

[![Project Page](https://img.shields.io/badge/Project-Page-blue)](https://vast-eden.ai/?p=edenlive)
[![Code](https://img.shields.io/badge/Code-GitHub-black)](https://github.com/VAST-AI-Research/EdenLive)
[![License](https://img.shields.io/badge/License-Mixed%20(see%20below)-orange)](#license)

[Project Page](https://vast-eden.ai/?p=edenlive) · [Code](https://github.com/VAST-AI-Research/EdenLive) · [Models](https://huggingface.co/VAST-AI/EdenLive)

Zehuan Huang<sup>1,*</sup>, Tianxing Xu<sup>1,*</sup>, Chunli Peng<sup>1,*</sup>, Cheng Zhang<sup>1,*</sup>, Yifei Song<sup>1,†</sup>, Zeyi Zheng<sup>1,†</sup>, Zhipeng Yu<sup>1</sup>, Peng Wang<sup>1,‡</sup>, Yanpei Cao<sup>1,✉</sup>

\* Equal contribution · † Intern at VAST · ‡ Project lead · ✉ Corresponding author

<sup>1</sup>[VAST](https://www.tripo3d.ai/) · [Project Eden](https://vast-eden.ai/)

</div>

## Overview

EdenLive renders a **proxy or reference video stream**, a **target first frame**,
and a **text prompt** into refined or stylized video. References can come from
2D/3D scenes, existing engines, or recordings. Upstream code maintains interaction
state; the model handles visual rendering.

Control is injected through a **vision encoder**: a post-trained DINOv3 encoder embeds
each proxy frame, and those features condition the diffusion transformer.

The [project page](https://vast-eden.ai/?p=edenlive) demonstrates over 24 FPS,
minute-scale generation, and 480P/720P output. The 5B model supports
single-GPU deployment at 720P.

This repository holds the model weights. Code, installation instructions, and
inference/streaming usage live in the
[EdenLive repository](https://github.com/VAST-AI-Research/EdenLive).

## Model weights

```bash
hf download VAST-AI/EdenLive --local-dir ./models/EdenLive
```

Everything loads from one directory, `models/EdenLive` under the code repository root by
default (about 210 GB in total); nothing is pulled from the Hub at run time once
downloaded.

```
models/EdenLive/
  dit-14b/            diffusion_pytorch_model.safetensors, config.json  14B few-step (chunk-AR) DiT
  dit-14b-bidir/      diffusion_pytorch_model.safetensors, config.json  14B bidirectional DiT
  dit-5b/             diffusion_pytorch_model.safetensors, config.json  5B few-step (chunk-AR) DiT
  dit-5b-bidir/       diffusion_pytorch_model.safetensors, config.json  5B bidirectional DiT
  vae-14b/            Wan2.1_VAE.pth, config.json
  vae-5b/             Wan2.2_VAE.pth, config.json
  decoder-14b/        lighttaew2_1.pth                    TAEHV decoder (real-time only)
  decoder-5b/         lighttaew2_2.pth                    TAEHV decoder (real-time only)
  scheduler-14b/      scheduler_config.json               offline only
  scheduler-5b/       scheduler_config.json               offline only
  umt5/               text_encoder/, tokenizer/           UMT5 text encoder
  clip/               image_encoder/, image_processor/    CLIP, 14B I2V only
  vision-encoder/     backbone/, projection.safetensors   proxy-DINO encoder
  flux-c2f/           Diffusers FLUX tree + c2f_lora.safetensors   workbench first frame (real-time only)
```

To load the weights from another directory:

```bash
python -m edenlive.scripts.infer --config ... model.pretrained_dir=/data/EdenLive
```

The real-time service reads the same tree through `MODEL_ROOT` in
[`realtime/deploy.conf`](https://github.com/VAST-AI-Research/EdenLive/blob/main/realtime/deploy.conf).
See the [main repository's README](https://github.com/VAST-AI-Research/EdenLive#model-weights)
for how each component is wired into the offline and real-time pipelines.

## Usage

Each DiT has a matching inference profile in the code repository:

| Mode | Wan2.1-I2V-14B @ 480p | Wan2.2-TI2V-5B @ 720p |
| --- | --- | --- |
| Bidirectional (`dit-*-bidir/`) | `wan21_14b_480p_bidir.yaml` | `wan22_5b_720p_bidir.yaml` |
| Few-step, chunk-AR (`dit-14b/`, `dit-5b/`) | `wan21_14b_480p_few_step.yaml` | `wan22_5b_720p_few_step.yaml` |

Few-step inference (chunk-autoregressive sampling, 4 steps per chunk) on the bundled
samples:

```bash
torchrun --nproc_per_node=2 -m edenlive.scripts.infer \
  --config edenlive/configs/wan21_14b_480p_few_step.yaml \
  runtime.sequence_parallel.size=2
```

```bash
python -m edenlive.scripts.infer \
  --config edenlive/configs/wan22_5b_720p_few_step.yaml
```

See the [Quick start](https://github.com/VAST-AI-Research/EdenLive#quick-start) for
bidirectional inference, the real-time service, and your own data.

## Citation

```bibtex
@misc{edenlive2026,
  title  = {EdenLive: Real-Time Generative Rendering from Any Visual Reference},
  author = {Zehuan Huang and Tianxing Xu and Chunli Peng and Cheng Zhang and Yifei Song and Zeyi Zheng and Zhipeng Yu and Peng Wang and Yanpei Cao},
  year   = {2026},
  url    = {https://vast-eden.ai/?p=edenlive}
}
```

## Acknowledgements

Built on [Wan2.1 / Wan2.2](https://github.com/Wan-Video),
🤗 [Diffusers](https://github.com/huggingface/diffusers),
[DINOv3](https://github.com/facebookresearch/dinov3),
[FLUX.2 \[klein\]](https://bfl.ai/models/flux-2-klein) by Black Forest Labs, the
[LightX2V](https://huggingface.co/lightx2v/Autoencoders) LightTAE decoders, and
[LAION CLIP ViT-H/14](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K)
trained with [OpenCLIP](https://github.com/mlfoundations/open_clip).

## License

The weights in this repository are covered by different licenses, per component:

| Component | License | Notes |
| --- | --- | --- |
| `dit-*`, `vae-*`, `umt5/`, `scheduler-*` | [Apache 2.0](LICENSE) | Derived from Wan2.1 / Wan2.2 (Apache 2.0) |
| `decoder-*` | [Apache 2.0](LICENSE) | LightTAE decoders from [lightx2v/Autoencoders](https://huggingface.co/lightx2v/Autoencoders) (Apache 2.0) |
| `clip/` | [MIT](clip/LICENSE.md) | [LAION CLIP ViT-H/14](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) trained with OpenCLIP; see [NOTICE](clip/NOTICE.md) |
| `vision-encoder/` | [DINOv3 License](vision-encoder/LICENSE.md) | DINOv3 by Meta and derivatives of it; see [NOTICE](vision-encoder/NOTICE.md) |
| `flux-c2f/` | [FLUX Non-Commercial License v2.1](flux-c2f/LICENSE.md) | FLUX.2 \[klein\] 9B by Black Forest Labs plus a LoRA by Vast AI; **non-commercial, non-production use only**; see [NOTICE](flux-c2f/NOTICE.md) |

`flux-c2f/` is used only to render the real-time workbench's photoreal first frame; the
offline and real-time video pipelines do not load it. Using the workbench therefore falls
under the FLUX Non-Commercial License, while video generation does not depend on it.
Public end-user or production deployment is not automatically permitted by being free of
charge or described as a demo; check the license and obtain a separate authorization from
Black Forest Labs where required.