| | --- |
| | license: mit |
| | tags: |
| | - VAE |
| | - Video-Generation |
| | --- |
| | |
| | # Reducio-VAE Model Card |
| |
|
| | <!-- Provide a quick summary of what the model is/does. --> |
| | This model is a 3D VAE that encodes video into a compact latent space conditioned on a content frame. It compresses a video by a factor of \\(\frac{T}{4}\times\frac{H}{32}\times\frac{W}{32}\\), enabling 4096x downsampling. |
| | It is part of the [Reducio-DiT](https://arxiv.org/abs/2411.13552), which is a video generation method. Codebase available [here](https://github.com/microsoft/Reducio-VAE). |
| |
|
| |
|
| | ## Model Details |
| |
|
| | ### Model Sources |
| |
|
| | <!-- Provide the basic links for the model. --> |
| |
|
| | - **Repository:** [GitHub Repository](https://github.com/microsoft/Reducio-VAE) |
| | - **Paper:** [arXiv](https://arxiv.org/abs/2411.13552) |
| |
|
| | ## Uses |
| |
|
| | <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
| |
|
| | Common use scenario is described [here](https://github.com/microsoft/Reducio-VAE). |
| |
|
| | ### Direct Use |
| |
|
| | <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. --> |
| |
|
| | The model is typically used for supporting training a video diffusion model. After using this model to convert the data to the latent space, you can train your own diffusion model on the extremely compressed latent space. |
| |
|
| |
|
| | ## Results |
| |
|
| | <!-- This section describes the evaluation protocols and provides the results. --> |
| |
|
| |
|
| | ### Results |
| |
|
| | Metrics on 1K Pexels validation set and UCF-101: |
| |
|
| | |Method|Downsample Factor|\|z\||PSNR |SSIM |LPIPS |rFVD (Pexels)|rFVD (UCF-101)| |
| | |---------|---------------------|------------------|------------|--------------------|--------------|----------------|------------| |
| | |SD2.1-VAE|1\*8\*8|4|29.23|0.82|0.09|25.96|21.00| |
| | |SDXL-VAE|1\*8\*8|16|30.54|0.85|0.08|19.87|23.68| |
| | |OmniTokenizer|4\*8\*8|8|27.11|0.89|0.07|23.88|30.52| |
| | |OpenSora-1.2|4\*8\*8|16|30.72|0.85|0.11|60.88|67.52| |
| | |Cosmos Tokenizer|8\*8\*8|16|30.84|0.74|0.12|29.44|22.06| |
| | |Cosmos Tokenizer|8\*16\*16|16|28.14|0.65|0.18|77.87|119.37| |
| | |Reducio-VAE|4\*32\*32|16|35.88|0.94|0.05|17.88|65.17| |
| |
|
| |
|
| | ## Citation |
| |
|
| | **BibTeX:** |
| |
|
| | ``` |
| | @article{tian2024reducio, |
| | title={REDUCIO! Generating 1024*1024 Video within 16 Seconds using Extremely Compressed Motion Latents}, |
| | author={Tian, Rui and Dai, Qi and Bao, Jianmin and Qiu, Kai and Yang, Yifan and Luo, Chong and Wu, Zuxuan and Jiang, Yu-Gang}, |
| | journal={arXiv preprint arXiv:2411.13552}, |
| | year={2024} |
| | } |
| | ``` |
