README.md

Sci-Fi: Symmetric Constraint for Frame Inbetweening

Liuhan Chen¹, Xiaodong Cun^2,*, Xiaoyu Li³, Xianyi He^1,4, Shenghai Yuan^1,4, Jie Chen¹, Ying Shan³, Li Yuan^1,*

¹Shenzhen Graduate School, Peking University     ²GVC Lab, Great Bay University     ³ARC Lab, Tencent PCG     ⁴Rabbitpre Intelligence

Method comparison

(a) In current I2V-DM-based methods, the end-frame constraint is weaker than the start-frame constraint due to the same injection mechanism but a smaller training scale, causing a distorted predicted path with collapsed content.
(b) Our Sci-Fi maintains start frame processing while enhancing end-frame constraint injection. This achieves symmetric start-end-frame constraints with small training, yielding a fine predicted path close to the real one with smoother inbetweening.

Some challenging examples of our Sci-Fi for frame inbetweening.

Start frame	End frame	Generated video

Deployment for personal use

1. Setup the repository and environment

git clone https://github.com/GVCLab/Sci-Fi.git
cd Sci-Fi
conda create -n Sci-Fi python==3.12
conda activate Sci-Fi
pip install -r requirements.txt

2. Download checkpoint

Download the CogVideoX-5B-I2V model (due to fine-tuning, the weights of the transformer denoiser are different from the original) and EF-Net. The weights are available at 🤗HuggingFace and 🤖ModelScope.

3. Launch the inference script!

The example input keyframe pairs are in examples/ folder, and the corresponding generated videos (720x480, 49 frames) are placed in outputs/ folder.
To interpolate, run:

bash Sci_Fi_frame_inbetweening.sh