| --- |
| base_model: |
| - stabilityai/stable-diffusion-2 |
| license: mit |
| library_name: diffusers |
| pipeline_tag: image-to-image |
| --- |
| |
| # StableMotion: Repurposing Diffusion-Based Image Priors for Motion Estimation |
| This is the official repo for paper [StableMotion: Repurposing Diffusion-Based Image Priors for Motion Estimation](https://www.arxiv.org/abs/2505.06668) |
|
|
| Official Code Repository: [GitHub - ivowang/StableMotion](https://github.com/ivowang/StableMotion) |
|
|
| ## Setup |
| 0. Clone the [code repo](https://github.com/ivowang/StableMotion). |
| 1. Create your environment from `requirements.txt`. |
| 2. Download [DIR-D](https://drive.google.com/file/d/1KR5DtekPJin3bmQPlTGP4wbM1zFR80ak/view?usp=sharing) and [RS-Real](https://huggingface.co/datasets/Yzl-code/RS-Diffusion/tree/main). Put them into `StableMotion_SIR` and `StableMotion_RSC` respectively. |
|
|
| ## StableMotion for Stitched Image Rectangling (SIR) |
| ### Inference |
| 0. Download the checkpoints of [StableMotion_SIR](https://huggingface.co/ivowang/StableMotion/tree/main/StableMotion_SIR) |
| 1. Run `cd StableMotion_SIR && sh sample.sh`. You may want to change this file to modify the inference configurations. |
| 2. Run `sh metrics.sh` to evaluate the results. |
|
|
| ### Training |
| 0. Replicate [RecDiffusion](https://github.com/lhaippp/RecDiffusion) to get pseudo labels, i.e., the flow labels generated by [RecDiffusion](https://github.com/lhaippp/RecDiffusion) on the training set of [DIR-D](https://drive.google.com/file/d/1KR5DtekPJin3bmQPlTGP4wbM1zFR80ak/view?usp=sharing). Put and rename it into `StableMotion_SIR/MDM_Flow`. |
| 1. Run `cd StableMotion_SIR && sh train.sh`. You may want to change this file to modify the training configurations. The default configuration requires approximately 80 GB of VRAM per card. |
|
|
| ## StableMotion for Rolling Shutter Correction (RSC) |
| ### Inference |
| 0. Download the checkpoints of [StableMotion_RSC](https://huggingface.co/ivowang/StableMotion/tree/main/StableMotion_RSC) |
| 1. Run `cd StableMotion_RSC && sh sample.sh`. You may want to change this file to modify the inference configurations. |
| 2. Run `sh metrics.sh` to evaluate the results. |
|
|
| ### Training |
| Run `cd StableMotion_RSC && sh train.sh`. You may want to change this file to modify the training configurations. The default configuration requires approximately 40 GB of VRAM per card. |
|
|
| ## GPT Rule-Based Evaluation |
| Each task folder has a `gpt_eval` subfolder with the script used in the paper to score results with a vision LLM (GPT) on a fixed rubric. `StableMotion_SIR/gpt_eval/score_rectangle.py` scores Stitched Image Rectangling (SIR) `(input, output)` pairs, and `StableMotion_RSC/gpt_eval/score_rolling_shutter.py` scores Rolling Shutter Correction (RSC) `[input | Yang | Ours]` triptychs. Both call an OpenAI-Responses-compatible API and emit per-pair scores plus an aggregate `summary.json` (mean/std/95% CI). To run, copy `provider.example.json` to `provider.json` in the relevant folder, add your endpoint/key, then e.g. `cd StableMotion_SIR/gpt_eval && python score_rectangle.py <input_dir> <result_dir>`. See each folder's `README.md` for the full rubric, flags, and outputs. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{wang2025stablemotion, |
| title={StableMotion: One-Step Motion Estimation with Diffusion Prior}, |
| author={Wang, Ziyi and Li, Haipeng and Sui, Lin and Zhou, Tianhao and Jiang, Hai and Nie, Lang and Liu, Shuaicheng}, |
| journal={arXiv preprint arXiv:2505.06668}, |
| year={2025} |
| } |
| ``` |
