Update README.md
Browse files
README.md
CHANGED
|
@@ -1,19 +1,112 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
| 4 |
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
frameworks:
|
| 4 |
+
- PyTorch
|
| 5 |
+
pipeline_tag: text-to-image
|
| 6 |
---
|
| 7 |
|
| 8 |
+
|
| 9 |
+
# AnimateDiff
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
# 目录
|
| 14 |
+
|
| 15 |
+
[英文](README_EN.md)
|
| 16 |
+
|
| 17 |
+
[链接](https://www.openeuler.org/zh/oEEP/?name=oEEP-0002%20oEEP%20%E6%A0%BC%E5%BC%8F%E4%B8%8E%E5%86%85%E5%AE%B9%E8%A7%84%E8%8C%83#oeep-%E5%86%85%E5%AE%B9%E8%A6%81%E6%B1%82)
|
| 18 |
+
|
| 19 |
+
- [概述](#概述)
|
| 20 |
+
- [准备训练环境](#准备训练环境)
|
| 21 |
+
- [模型训练](#模型训练)
|
| 22 |
+
- [模型推理](#模型推理)
|
| 23 |
+
- [版本说明](#版本说明)
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
## 概述
|
| 28 |
+
|
| 29 |
+
### 模型介绍
|
| 30 |
+
|
| 31 |
+
AnimateDiff提出了一个有效的框架，可将现有的大多数个性化文本到图像模型一次性制成动画，从而节省了针对特定模型进行微调的工作量。
|
| 32 |
+
|
| 33 |
+
本仓已经支持以下模型任务类型
|
| 34 |
+
|
| 35 |
+
| 模型 | 任务列表 | 是否支持 |
|
| 36 |
+
|:-----------:|:----:|:-----:|
|
| 37 |
+
| AnimateDiff | 训练 | ✔ |
|
| 38 |
+
| AnimateDiff | 推理 | ✔ |
|
| 39 |
+
|
| 40 |
+
- 参考实现：
|
| 41 |
+
|
| 42 |
+
```
|
| 43 |
+
url=https://github.com/guoyww/AnimateDiff.git
|
| 44 |
+
commit_id=cf80ddeb47b69cf0b16f225800de081d486d7f21
|
| 45 |
+
```
|
| 46 |
+
|
| 47 |
+
- 适配昇腾AI处理器的实现：
|
| 48 |
+
```shell
|
| 49 |
+
url=https://openmind.cn/models/Ascend-PyTorch/AnimateDiff
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
## 准备训练环境
|
| 53 |
+
|
| 54 |
+
### 创建python环境
|
| 55 |
+
|
| 56 |
+
- git clone 远程仓
|
| 57 |
+
```shell
|
| 58 |
+
git clone https://openmind.cn/Ascend-PyTorch/AnimateDiff.git
|
| 59 |
+
cd AnimateDiff
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
- 创建python环境并且安装python三方包
|
| 63 |
+
```shell
|
| 64 |
+
conda env create -f environment.yaml
|
| 65 |
+
conda activate animatediff
|
| 66 |
+
pip3 install torch==2.1.0+cpu --index-url https://download.pytorch.org/whl/cpu # For X86
|
| 67 |
+
pip3 install torch==2.1.0 # For Aarch64
|
| 68 |
+
pip3 install accelerate==0.28.0 diffusers==0.11.1 decorator==5.1.1 scipy==1.12.0 attrs==23.2.0 torchvision==0.16.0 transformers==4.25.1
|
| 69 |
+
```
|
| 70 |
+
- 环境准备指导
|
| 71 |
+
|
| 72 |
+
请参考《[Pytorch框架训练环境准备](https://www.hiascend.com/document/detail/zh/ModelZoo/pytorchframework/ptes)》。
|
| 73 |
+
|
| 74 |
+
**表 1** 昇腾软件版本支持表
|
| 75 |
+
|
| 76 |
+
| 软件类型 | 支持版本 |
|
| 77 |
+
|:-----------:|:--------:|
|
| 78 |
+
| FrameworkPTAdapter | 在研版本 |
|
| 79 |
+
| CANN | 在研版本 |
|
| 80 |
+
| 昇腾NPU固件 | 在研版本 |
|
| 81 |
+
| 昇腾NPU驱动 | 在研版本 |
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
### 准备数据集
|
| 85 |
+
|
| 86 |
+
- 需要自行下载WebVid10M数据集，分别将csv文件和2M_val文件夹传入训练脚本对应的csv_path和video_folder参数上面：
|
| 87 |
+
```
|
| 88 |
+
数据集结构
|
| 89 |
+
├── 2M_val
|
| 90 |
+
│ ├── 10003109.mp4
|
| 91 |
+
│ ├── 10023815.mp4
|
| 92 |
+
│ ├── 10024310.mp4
|
| 93 |
+
│ ├── 10042700.mp4
|
| 94 |
+
│ ├── 10052036.mp4
|
| 95 |
+
│ ├── 10052783.mp4
|
| 96 |
+
│ ├── 1005608956.mp4
|
| 97 |
+
└── results_2M_val.csv
|
| 98 |
+
```
|
| 99 |
+
数据来源可以参考 https://github.com/guoyww/AnimateDiff/blob/main/__assets__/docs/animatediff.md 中的数据准备章节。
|
| 100 |
+
### 准备预训练权重
|
| 101 |
+
|
| 102 |
+
- 需要准备2个模型权重:
|
| 103 |
+
```shell
|
| 104 |
+
runwayml/stable-diffusion-v1-5
|
| 105 |
+
openai/clip-vit-large-patch14
|
| 106 |
+
```
|
| 107 |
+
- 将stable-diffusion-v1-5 路径传入到configs/training/v1/image_finetune.yaml 的pretrained_model_path。
|
| 108 |
+
- openai/clip-vit-large-patch14需要放置到模型的根目录下面。
|
| 109 |
+
### 准备推理权重
|
| 110 |
+
- 如果想使用animatediff的推理功能需要下载下面提及到的模型，按照模型的名字对应放到models目录下面DreamBooth_LoRA、MotionLoRA、Motion_Module、SparseCtrl、StableDiffusion文件夹中。
|
| 111 |
+
模型获取可以参考 https://github.com/guoyww/AnimateDiff/blob/main/__assets__/docs/animatediff.md 文档。
|
| 112 |
+
<details>
|