README.md

---

license: mit
datasets:
- quanhaol/MagicData
base_model:
- quanhaol/Wan2.2-TI2V-5B-Turbo
- Wan-AI/Wan2.2-TI2V-5B
tags:
- image-to-video
- Trajectory-Control
- Fewstep-video-gen
---

<br>
<a href="https://arxiv.org/pdf/2603.12146"><img src="https://img.shields.io/static/v1?label=Paper&message=2603.12146&color=red&logo=arxiv"></a>
<a href="https://quanhaol.github.io/flashmotion-site/"><img src="https://img.shields.io/static/v1?label=Project&message=Page&color=green&logo=github-pages"></a>
<a href="https://huggingface.co/quanhaol/FlashMotion"><img src="https://img.shields.io/badge/🤗_HuggingFace-Model-ffbd45.svg" alt="HuggingFace"></a>
<a href="https://huggingface.co/datasets/quanhaol/FlashBench"><img src="https://img.shields.io/badge/🤗_HuggingFace-Benchmark-ffbd45.svg" alt="HuggingFace"></a>

> **FlashMotion: Few-Step Controllable Video Generation with Trajectory Guidance**
> <br>
> [Quanhao Li](https://github.com/quanhaol)<sup>1</sup>, [Zhen Xing](https://chenhsing.github.io/)<sup>1</sup>, [Rui Wang](https://scholar.google.com/citations?user=116smmsAAAAJ&hl=en)<sup>1</sup>, Haidong Cao<sup>1</sup>, [Qi Dai](https://daiqi1989.github.io/)<sup>2</sup>, Daoguo Dong<sup>1</sup> and [Zuxuan Wu](https://zxwu.azurewebsites.net/)<sup>1</sup>
> 
> <sup>1</sup> Fudan University; <sup>2</sup> Microsoft Research Asia

## 💡 Abstract

Recent advances in trajectory-controllable video generation have achieved remarkable progress. Previous methods mainly use adapter-based architectures for precise motion control along predefined trajectories.
However, all these methods rely on a multi-step denoising process, leading to substantial time redundancy and computational overhead.
While existing video distillation methods successfully distill multi-step generators into few-step, directly applying these approaches to trajectory-controllable video generation results in noticeable degradation in both video quality and trajectory accuracy.
To bridge this gap, we introduce **FlashMotion**, a novel training framework designed for few-step trajectory-controllable video generation.
We first train a trajectory adapter on a multi-step video generator for precise trajectory control.
Then, we distill the generator into a few-step version to accelerate video generation.
Finally, we finetune the adapter using a hybrid strategy that combines diffusion and adversarial objectives, aligning it with the few-step generator to produce high-quality, trajectory-accurate videos.
For evaluation, we introduce **FlashBench**, a benchmark for long-sequence trajectory-controllable video generation that measures both video quality and trajectory accuracy across varying numbers of foreground objects. 
Experiments on two adapter architectures show that FlashMotion surpasses existing video distillation methods and previous multi-step models in both visual quality and trajectory consistency.


## 📣 Updates
- `2026/03/13` 🔥🔥We released FlashMotion, including its training code, inference code, model weights and also the evaluation benchmark.
- `2026/02` 🔥🔥🔥 FlashMotion has been accepted by CVPR2026!

## 📑 Table of Contents

- [💡 Abstract](#-abstract)
- [📣 Updates](#-updates)
- [📑 Table of Contents](#-table-of-contents)
- [✅ TODO List](#-todo-list)
- [🐍 Installation](#-installation)
- [📦 Model Weights](#-model-weights)
  - [Folder Structure](#folder-structure)
  - [Download Links](#download-links)
- [⛽️ Dataset Prepare](#️-dataset-prepare)
- [🔄 Inference](#-inference)
  - [Scripts](#scripts)
- [🏎️ Train](#️-train)
  - [SlowAdapter Training](#slowadapter-training)
  - [FastGenerator Training](#fastgenerator-training)
  - [FastAdapter Training](#fastadapter-training)
- [🤝 Acknowledgements](#-acknowledgements)
- [📚 Contact](#-contact)

## ✅ TODO List

- [x] Release our inference code and model weights
- [x] Release our training code 
- [x] Release our evaluation benchmark

## 🐍 Installation

```bash
# Clone this repository.
git clone https://github.com/quanhaol/FlashMotion
cd FlashMotion

# Install requirements
conda create -n flashmotion python=3.10 -y
conda activate flashmotion
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
python setup.py develop
```

## 📦 Model Weights

### Folder Structure

```
FlashMotion
└── ckpts
    ├── FastGenerator
    │   ├── model.pt
    ├── SlowAdapter
    │   ├── ResNet
    │       └── model.pt
    │   ├── ControlNet
    │       └── model.pt
    ├── FastAdapter
    │   ├── ResNet
    │       └── model.pt
    │   ├── ControlNet
    │       └── model.pt
```

### Download Links

Please use the following commands to download the model weights

```bash
pip install "huggingface_hub[hf_transfer]"
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download quanhaol/FlashMotion --local-dir ckpts
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download Wan-AI/Wan2.2-TI2V-5B --local-dir wan_models/Wan2.2-TI2V-5B
```

## ⛽️ Dataset Prepare
All three training stages of FlashMotion uses [MagicData](https://huggingface.co/datasets/quanhaol/MagicData), an open-sourced dataset built for trajectory-controllable video generation.
Please follow [this README](https://huggingface.co/datasets/quanhaol/MagicData) to download and extract the data in a proper path on your machine.

The dataset structure can be organized as follows:
```
MagicData
├── videos
│   ├── videoid_1.mp4
│   ├── videoid_2.mp4
│   ├── ...
├── masks
│   ├── videoid_1
│   │   ├── annotated_frame_00000.png
│   │   ├── annotated_frame_00001.png
│   │   ├── ...
│   ├── videoid_2
│   │   ├── ...
├── boxs
│   ├── videoid_1
│   │   ├── annotated_frame_00000.png
│   │   ├── annotated_frame_00001.png
│   │   ├── ...
│   ├── videoid_2
│   │   ├── ...
├── MagicData.csv   # detailed information of each video
```

## 🔄 Inference
The Inference process requires around 42 GiB GPU memory to use the ResNet FastAdapter and 50GiB GPU memory to use the ControlNet FastAdapter, all tested on a single NVIDIA A100 GPU.

⚡️⚡️⚡️ It takes only 11 seconds for denoising a video using the ResNet Adapter, and around 24 seconds to denoise a video using the ControlNet Adapter.

### Scripts

We here provide demo scripts to run both types of trajectory adapter.
```bash
# Demo inference script of each adapter type
bash running_scripts/inference/i2v_control_fewstep_controlnet.sh
bash running_scripts/inference/i2v_control_fewstep_resnet.sh
```
We also provide sample input image and trajectory maps in `./assets`.

Feel free to replace the `--prompt`, `--image`, `--trajectory` with your customized input prompt, input image and input trajectory maps.
> **Note**: If you want to build your own trajectory maps, please refer to the box trajectory construction pipeline introduced in [MagicMotion](https://github.com/quanhaol/MagicMotion/tree/main/trajectory_construction#box-trajectory).

## 🏎️ Train

We here provide scripts for all three training stages of FlashMotion, including training the SlowAdapter, FastGenerator, and the FastAdapter.

### SlowAdapter Training
In this stage, we first train the SlowAdapter using the mask annotations in MagicData, and then finetune it using bounding box as the trajectory maps conditions.
```bash
# Demo training script of SlowAdapter
bash running_scripts/train/stage1_mask.sh
bash running_scripts/train/stage1_box.sh
```

### FastGenerator Training
In this stage, we distill the Wan2.2-TI2V-5B model into a 4-steps image-to-video generation model, named as the FastGenerator.
```bash
# Demo training script of FastGenerator
bash running_scripts/train/stage2.sh
```

### FastAdapter Training
In this stage, we trains the FastAdapter to fit with the FastGenerator and enable few-step trajectory controllable video generation.
```bash
# Demo training script of FastGenerator
bash running_scripts/train/stage3.sh
```

## 🤝 Acknowledgements

We would like to express our gratitude to the following open-source projects that have been instrumental in the development of our project:

- [Wan](https://github.com/Wan-Video/Wan2.2): An open sourced base video generation model.
- [Self-Forcing](https://github.com/guandeh17/Self-Forcing) and [Causvid](https://github.com/tianweiy/CausVid): Two frameworks that pioneer the field of distilling video generation methods.
- [MagicMotion](https://github.com/quanhaol/MagicMotion): An open source trajectory-controllable video generation framework.
- [Wan2.2-TI2V-5B-Turbo](https://github.com/quanhaol/Wan2.2-TI2V-5B-Turbo): An open source step distillation image-to-video generation framework that distill Wan2.2-5B-TI2V model into 4 steps.


Special thanks to the contributors of these libraries for their hard work and dedication!

## 📚 Contact

If you have any suggestions or find our work helpful, feel free to contact us

Email: liqh24@m.fudan.edu.cn

If you find our work useful, <b>please consider giving a star to this github repository and citing it</b>:

```bibtex
@misc{li2026flashmotionfewstepcontrollablevideo,
      title={FlashMotion: Few-Step Controllable Video Generation with Trajectory Guidance}, 
      author={Quanhao Li and Zhen Xing and Rui Wang and Haidong Cao and Qi Dai and Daoguo Dong and Zuxuan Wu},
      year={2026},
      eprint={2603.12146},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2603.12146}, 
}
```