README.md

---
pipeline_tag: video-to-video
tags:
  - video
  - video-reconstruction
  - video-generation
  - video-representation
  - v-rae
  - rae
  - pytorch
  - arxiv:2608.13556
datasets:
  - quchenyuan/UCF101-ZIP
---

<div align="center">
<h1>V-RAE: Rethinking Video Latent Spaces for Generation</h1>
<p>
  <a href="https://guominghui07.github.io/">Minghui Guo</a><sup>1</sup> &nbsp;&nbsp; <a href="https://sqwu.top/">Shengqiong Wu</a><sup>2</sup> &nbsp;&nbsp; <a href="https://haofei.vip/">Hao Fei</a><sup>2</sup>
</p>
<p>
  <sup>1</sup>National University of Singapore &nbsp;&nbsp; <sup>2</sup>University of Oxford
</p>
</div>

<p align="center">
  <a href="https://arxiv.org/abs/2608.13556"><img src="https://img.shields.io/badge/Paper-arXiv-b31b1b" alt="Paper"></a>
  <a href="https://v-rae.github.io/"><img src="https://img.shields.io/badge/Homepage-Project%20Page-blue" alt="Project Page"></a>
  <a href="https://github.com/V-RAE/V-RAE"><img src="https://img.shields.io/badge/Code-GitHub-black" alt="Code"></a>
</p>

V-RAE is a video representation autoencoder that builds compact generative
latents on top of frozen vision foundation model representations. It uses a
lightweight temporal pooling module to remove temporal redundancy while
preserving semantic structure, together with a video decoder for reconstructing
continuous motion.

![V-RAE method](https://raw.githubusercontent.com/V-RAE/V-RAE/main/assets/V-RAE.png)

![V-RAE overview](https://raw.githubusercontent.com/V-RAE/V-RAE/main/assets/V-RAE-overall.png)

## Released Models

This repository provides V-RAE checkpoints based on four frozen visual encoders:

| Checkpoint | Encoder |
| --- | --- |
| `vrae/vrae_dinov3.pt` | DINOv3 ViT-L/16 |
| `vrae/vrae_siglip2.pt` | SigLIP2 ViT-L/16 |
| `vrae/vrae_vjepa2.1.pt` | V-JEPA2.1 ViT-L/16 |
| `vrae/vrae_eupe.pt` | EUPE ViT-B/16 |

Class-conditional VideoDiT checkpoints and their matching latent statistics are
also provided for UCF101 and Kinetics-600:

- UCF101: V-JEPA2.1 and EUPE variants
- Kinetics-600: V-JEPA2.1 variant

## Quick Start

Install the official implementation:

```bash
git clone https://github.com/V-RAE/V-RAE.git
cd V-RAE

conda create -n vrae python=3.10 -y
conda activate vrae
conda install -c conda-forge ffmpeg -y

pip install uv
uv pip install -e .
```

Download the released checkpoints:

```bash
hf download Guomh0707/V-RAE-Models --local-dir ckpts
```

Download the matching frozen encoder by following the
[encoder instructions](https://github.com/V-RAE/V-RAE/blob/main/third_party/README.md#download-pre-trained-encoder-weights).

Place three input videos at `assets/sample1.mp4`, `assets/sample2.mp4`, and
`assets/sample3.mp4`, then run one of:

```bash
python sampling.py dino
python sampling.py siglip
python sampling.py vjepa
python sampling.py eupe
```

Reconstruction results are saved under `outputs/<variant>/`.

## Results

| Evaluation | Result |
| --- | ---: |
| Kinetics-600 reconstruction | **2.13 rFVD** |
| UCF101 class-conditional generation | **117.86 gFVD** |
| Kinetics-600 class-conditional generation | **19.16 gFVD** |


## Notes

- Each V-RAE checkpoint requires its corresponding frozen encoder weights.
- These are custom PyTorch checkpoints and require the official V-RAE codebase.
- This repository is intended primarily for research use.

## Citation

```bibtex
@article{guo2026vrae,
  title   = {V-RAE: Rethinking Video Latent Spaces for Generation},
  author  = {Guo, Minghui and Wu, Shengqiong and Fei, Hao},
  journal = {arXiv preprint arXiv:2608.13556},
  year    = {2026},
}
```