Update README.md
Browse files
README.md
CHANGED
|
@@ -26,7 +26,7 @@ Keywords: Video Inpainting, Video Editing, Video Generation
|
|
| 26 |
<p align="center">
|
| 27 |
<a href="https://yxbian23.github.io/project/video-painter">🌐Project Page</a> |
|
| 28 |
<a href="https://arxiv.org/abs/2503.05639">📜Arxiv</a> |
|
| 29 |
-
<a href="https://huggingface.co/
|
| 30 |
<a href="https://youtu.be/HYzNfsD3A0s">📹Video</a> |
|
| 31 |
<a href="https://huggingface.co/TencentARC/VideoPainter">🤗Hugging Face Model</a> |
|
| 32 |
</p>
|
|
@@ -53,14 +53,14 @@ Keywords: Video Inpainting, Video Editing, Video Generation
|
|
| 53 |
|
| 54 |
## 🔥 Update Log
|
| 55 |
- [2025/3/09] 📢 📢 [VideoPainter](https://huggingface.co/TencentARC/VideoPainter) are released, an efficient, any-length video inpainting & editing framework with plug-and-play context control.
|
| 56 |
-
- [2025/3/09] 📢 📢 [VPData and VPBench](https://huggingface.co/datasets/TencentARC/VPBench) are released, the largest video inpainting dataset with precise segmentation masks and dense video captions (>390K clips).
|
| 57 |
|
| 58 |
## TODO
|
| 59 |
|
| 60 |
- [x] Release trainig and inference code
|
| 61 |
- [x] Release evluation code
|
| 62 |
- [x] Release [VideoPainter checkpoints](https://huggingface.co/TencentARC/VideoPainter) (based on CogVideoX-5B)
|
| 63 |
-
- [x] Release [VPData and VPBench](https://huggingface.co/
|
| 64 |
- [x] Release gradio demo
|
| 65 |
- [ ] Data preprocessing code
|
| 66 |
## 🛠️ Method Overview
|
|
@@ -115,7 +115,7 @@ pip install -e .
|
|
| 115 |
|
| 116 |
**VPBench and VPData**
|
| 117 |
|
| 118 |
-
You can download the VPBench [here](https://huggingface.co/datasets/TencentARC/VPBench) (as well as the Davis we re-processed), which are used for training and testing the BrushNet. By downloading the data, you are agreeing to the terms and conditions of the license. The data structure should be like:
|
| 119 |
|
| 120 |
```
|
| 121 |
|-- data
|
|
@@ -170,7 +170,16 @@ unzip davis.zip
|
|
| 170 |
unzip video_inpainting.zip
|
| 171 |
```
|
| 172 |
|
| 173 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 174 |
|
| 175 |
```
|
| 176 |
cd data_utils
|
|
@@ -224,6 +233,11 @@ The ckpt structure should be like:
|
|
| 224 |
|-- transformer
|
| 225 |
|-- vae
|
| 226 |
|-- ...
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 227 |
```
|
| 228 |
|
| 229 |
|
|
|
|
| 26 |
<p align="center">
|
| 27 |
<a href="https://yxbian23.github.io/project/video-painter">🌐Project Page</a> |
|
| 28 |
<a href="https://arxiv.org/abs/2503.05639">📜Arxiv</a> |
|
| 29 |
+
<a href="https://huggingface.co/collections/TencentARC/videopainter-67cc49c6146a48a2ba93d159">🗄️Data</a> |
|
| 30 |
<a href="https://youtu.be/HYzNfsD3A0s">📹Video</a> |
|
| 31 |
<a href="https://huggingface.co/TencentARC/VideoPainter">🤗Hugging Face Model</a> |
|
| 32 |
</p>
|
|
|
|
| 53 |
|
| 54 |
## 🔥 Update Log
|
| 55 |
- [2025/3/09] 📢 📢 [VideoPainter](https://huggingface.co/TencentARC/VideoPainter) are released, an efficient, any-length video inpainting & editing framework with plug-and-play context control.
|
| 56 |
+
- [2025/3/09] 📢 📢 [VPData](https://huggingface.co/datasets/TencentARC/VPData) and [VPBench](https://huggingface.co/datasets/TencentARC/VPBench) are released, the largest video inpainting dataset with precise segmentation masks and dense video captions (>390K clips).
|
| 57 |
|
| 58 |
## TODO
|
| 59 |
|
| 60 |
- [x] Release trainig and inference code
|
| 61 |
- [x] Release evluation code
|
| 62 |
- [x] Release [VideoPainter checkpoints](https://huggingface.co/TencentARC/VideoPainter) (based on CogVideoX-5B)
|
| 63 |
+
- [x] Release [VPData and VPBench](https://huggingface.co/collections/TencentARC/videopainter-67cc49c6146a48a2ba93d159) for large-scale training and evaluation.
|
| 64 |
- [x] Release gradio demo
|
| 65 |
- [ ] Data preprocessing code
|
| 66 |
## 🛠️ Method Overview
|
|
|
|
| 115 |
|
| 116 |
**VPBench and VPData**
|
| 117 |
|
| 118 |
+
You can download the VPBench [here](https://huggingface.co/datasets/TencentARC/VPBench), and the VPData [here](https://huggingface.co/datasets/TencentARC/VPData) (as well as the Davis we re-processed), which are used for training and testing the BrushNet. By downloading the data, you are agreeing to the terms and conditions of the license. The data structure should be like:
|
| 119 |
|
| 120 |
```
|
| 121 |
|-- data
|
|
|
|
| 170 |
unzip video_inpainting.zip
|
| 171 |
```
|
| 172 |
|
| 173 |
+
You can download the VPData (only mask and text annotations due to the space limit), and put the dataset to the `data` folder by:
|
| 174 |
+
```
|
| 175 |
+
git lfs install
|
| 176 |
+
git clone https://huggingface.co/datasets/TencentARC/VPData
|
| 177 |
+
mv VPBench data
|
| 178 |
+
cd data
|
| 179 |
+
unzip video_inpainting.zip
|
| 180 |
+
```
|
| 181 |
+
|
| 182 |
+
Noted: *Due to the space limit, you need to run the following script to download the raw videos of the complete VPData. The format should be consistent with VPData/VPBench above (After download the VPData/VPBench, the script will automatically place the raw videos of VPData into the corresponding dataset directories that have been created by VPBench).*
|
| 183 |
|
| 184 |
```
|
| 185 |
cd data_utils
|
|
|
|
| 233 |
|-- transformer
|
| 234 |
|-- vae
|
| 235 |
|-- ...
|
| 236 |
+
|-- flux_inp
|
| 237 |
+
|-- scheduler
|
| 238 |
+
|-- transformer
|
| 239 |
+
|-- vae
|
| 240 |
+
|-- ...
|
| 241 |
```
|
| 242 |
|
| 243 |
|