Instructions to use ziyiwang/StableMotion with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use ziyiwang/StableMotion with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import load_image # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("ziyiwang/StableMotion", dtype=torch.bfloat16, device_map="cuda") prompt = "Turn this cat into a dog" input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png") image = pipe(image=input_image, prompt=prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Add pipeline tag, library name, and improve model card
#1
by nielsr HF Staff - opened
README.md
CHANGED
|
@@ -1,19 +1,24 @@
|
|
| 1 |
---
|
| 2 |
-
license: mit
|
| 3 |
base_model:
|
| 4 |
- stabilityai/stable-diffusion-2
|
|
|
|
|
|
|
|
|
|
| 5 |
---
|
|
|
|
| 6 |
# StableMotion: Repurposing Diffusion-Based Image Priors for Motion Estimation
|
| 7 |
This is the official repo for paper [StableMotion: Repurposing Diffusion-Based Image Priors for Motion Estimation](https://www.arxiv.org/abs/2505.06668)
|
| 8 |
|
|
|
|
|
|
|
| 9 |
## Setup
|
| 10 |
-
0. Clone the [code repo](https://github.com/
|
| 11 |
1. Create your environment from `requirements.txt`.
|
| 12 |
2. Download [DIR-D](https://drive.google.com/file/d/1KR5DtekPJin3bmQPlTGP4wbM1zFR80ak/view?usp=sharing) and [RS-Real](https://huggingface.co/datasets/Yzl-code/RS-Diffusion/tree/main). Put them into `StableMotion_SIR` and `StableMotion_RSC` respectively.
|
| 13 |
|
| 14 |
## StableMotion for Stitched Image Rectangling (SIR)
|
| 15 |
### Inference
|
| 16 |
-
0. Download the checkpoints of
|
| 17 |
1. Run `cd StableMotion_SIR && sh sample.sh`. You may want to change this file to modify the inference configurations.
|
| 18 |
2. Run `sh metrics.sh` to evaluate the results.
|
| 19 |
|
|
@@ -23,9 +28,23 @@ This is the official repo for paper [StableMotion: Repurposing Diffusion-Based I
|
|
| 23 |
|
| 24 |
## StableMotion for Rolling Shutter Correction (RSC)
|
| 25 |
### Inference
|
| 26 |
-
0. Download the checkpoints of
|
| 27 |
1. Run `cd StableMotion_RSC && sh sample.sh`. You may want to change this file to modify the inference configurations.
|
| 28 |
2. Run `sh metrics.sh` to evaluate the results.
|
| 29 |
|
| 30 |
### Training
|
| 31 |
-
Run `cd StableMotion_RSC && sh train.sh`. You may want to change this file to modify the training configurations. The default configuration requires approximately 40 GB of VRAM per card.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
|
|
|
| 2 |
base_model:
|
| 3 |
- stabilityai/stable-diffusion-2
|
| 4 |
+
license: mit
|
| 5 |
+
library_name: diffusers
|
| 6 |
+
pipeline_tag: image-to-image
|
| 7 |
---
|
| 8 |
+
|
| 9 |
# StableMotion: Repurposing Diffusion-Based Image Priors for Motion Estimation
|
| 10 |
This is the official repo for paper [StableMotion: Repurposing Diffusion-Based Image Priors for Motion Estimation](https://www.arxiv.org/abs/2505.06668)
|
| 11 |
|
| 12 |
+
Official Code Repository: [GitHub - ivowang/StableMotion](https://github.com/ivowang/StableMotion)
|
| 13 |
+
|
| 14 |
## Setup
|
| 15 |
+
0. Clone the [code repo](https://github.com/ivowang/StableMotion).
|
| 16 |
1. Create your environment from `requirements.txt`.
|
| 17 |
2. Download [DIR-D](https://drive.google.com/file/d/1KR5DtekPJin3bmQPlTGP4wbM1zFR80ak/view?usp=sharing) and [RS-Real](https://huggingface.co/datasets/Yzl-code/RS-Diffusion/tree/main). Put them into `StableMotion_SIR` and `StableMotion_RSC` respectively.
|
| 18 |
|
| 19 |
## StableMotion for Stitched Image Rectangling (SIR)
|
| 20 |
### Inference
|
| 21 |
+
0. Download the checkpoints of [StableMotion_SIR](https://huggingface.co/ziyiwhat/StableMotion/tree/main/StableMotion_SIR)
|
| 22 |
1. Run `cd StableMotion_SIR && sh sample.sh`. You may want to change this file to modify the inference configurations.
|
| 23 |
2. Run `sh metrics.sh` to evaluate the results.
|
| 24 |
|
|
|
|
| 28 |
|
| 29 |
## StableMotion for Rolling Shutter Correction (RSC)
|
| 30 |
### Inference
|
| 31 |
+
0. Download the checkpoints of [StableMotion_RSC](https://huggingface.co/ziyiwhat/StableMotion/tree/main/StableMotion_RSC)
|
| 32 |
1. Run `cd StableMotion_RSC && sh sample.sh`. You may want to change this file to modify the inference configurations.
|
| 33 |
2. Run `sh metrics.sh` to evaluate the results.
|
| 34 |
|
| 35 |
### Training
|
| 36 |
+
Run `cd StableMotion_RSC && sh train.sh`. You may want to change this file to modify the training configurations. The default configuration requires approximately 40 GB of VRAM per card.
|
| 37 |
+
|
| 38 |
+
## GPT Rule-Based Evaluation
|
| 39 |
+
Each task folder has a `gpt_eval` subfolder with the script used in the paper to score results with a vision LLM (GPT) on a fixed rubric. `StableMotion_SIR/gpt_eval/score_rectangle.py` scores Stitched Image Rectangling (SIR) `(input, output)` pairs, and `StableMotion_RSC/gpt_eval/score_rolling_shutter.py` scores Rolling Shutter Correction (RSC) `[input | Yang | Ours]` triptychs. Both call an OpenAI-Responses-compatible API and emit per-pair scores plus an aggregate `summary.json` (mean/std/95% CI). To run, copy `provider.example.json` to `provider.json` in the relevant folder, add your endpoint/key, then e.g. `cd StableMotion_SIR/gpt_eval && python score_rectangle.py <input_dir> <result_dir>`. See each folder's `README.md` for the full rubric, flags, and outputs.
|
| 40 |
+
|
| 41 |
+
## Citation
|
| 42 |
+
|
| 43 |
+
```bibtex
|
| 44 |
+
@article{wang2025stablemotion,
|
| 45 |
+
title={StableMotion: One-Step Motion Estimation with Diffusion Prior},
|
| 46 |
+
author={Wang, Ziyi and Li, Haipeng and Sui, Lin and Zhou, Tianhao and Jiang, Hai and Nie, Lang and Liu, Shuaicheng},
|
| 47 |
+
journal={arXiv preprint arXiv:2505.06668},
|
| 48 |
+
year={2025}
|
| 49 |
+
}
|
| 50 |
+
```
|