Hugging Face's logo

360ZhiNao
/
FancyVideo

Image-to-Video
Diffusers
Safetensors
English
fancyvideo
video-generation
text-to-video
Model card Files
xet
 Community
360ZhiNao AoMa commited on
Commit
a134942
·
0 Parent(s):

Duplicate from qihoo360/FancyVideo

Browse files

Co-authored-by: AoMa <AoMa@users.noreply.huggingface.co>

Files changed (18) hide show
  1. .gitattributes +35 -0
  2. README.md +36 -0
  3. resources/models/CV-VAE/vae3d/config.json +34 -0
  4. resources/models/CV-VAE/vae3d/cv_vae.ckpt +3 -0
  5. resources/models/CV-VAE/vae3d/diffusion_pytorch_model.safetensors +3 -0
  6. resources/models/LongCLIP-L/longclip-L.pt +3 -0
  7. resources/models/fancyvideo_ckpts/vae_3d_125_frames/mp_rank_00_model_states.pt +3 -0
  8. resources/models/fancyvideo_ckpts/vae_3d_61_frames/mp_rank_00_model_states.pt +3 -0
  9. resources/models/fancyvideo_ckpts/video_backtracking/mp_rank_00_model_states.pt +3 -0
  10. resources/models/fancyvideo_ckpts/video_extending/mp_rank_00_model_states.pt +3 -0
  11. resources/models/res-adapter/resadapter_v1_sd1.5/diffusion_pytorch_model.safetensors +3 -0
  12. resources/models/res-adapter/resadapter_v1_sd1.5/pytorch_lora_weights.safetensors +3 -0
  13. resources/models/res-adapter/resadapter_v2_sd1.5/diffusion_pytorch_model.safetensors +3 -0
  14. resources/models/res-adapter/resadapter_v2_sd1.5/pytorch_lora_weights.safetensors +3 -0
  15. resources/models/sd_v1-5_base_models/pixarsRendermanInspo_mk1.safetensors +3 -0
  16. resources/models/sd_v1-5_base_models/realcartoon3d_v15.safetensors +3 -0
  17. resources/models/sd_v1-5_base_models/realisticVisionV60B1_v51VAE.safetensors +3 -0
  18. resources/models/sd_v1-5_base_models/toonyou_beta3.safetensors +3 -0
.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ tags:
6
+ - fancyvideo
7
+ - video-generation
8
+ - text-to-video
9
+ - image-to-video
10
+ ---
11
+ # FancyVideo
12
+ This repository is the official implementation of [FancyVideo](https://360cvgroup.github.io/FancyVideo/).
13
+
14
+ **[FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance](https://arxiv.org/abs/2408.08189)**
15
+ </br>
16
+ Jiasong Feng*, Ao Ma*, Jing Wang*, Bo Cheng, Xiaodan Liang, Dawei Leng†, Yuhui Yin(*Equal Contribution, ✝Corresponding Author)
17
+ </br>
18
+
19
+ ## We Are Hiring
20
+ We are seeking academic interns in the AIGC field. If interested, please send your resume to [maao@360.cn](mailto:maao@360.cn).
21
+
22
+ ## BibTeX
23
+ ```
24
+ @misc{feng2024fancyvideodynamicconsistentvideo,
25
+ title={FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance},
26
+ author={Jiasong Feng and Ao Ma and Jing Wang and Bo Cheng and Xiaodan Liang and Dawei Leng and Yuhui Yin},
27
+ year={2024},
28
+ eprint={2408.08189},
29
+ archivePrefix={arXiv},
30
+ primaryClass={cs.CV},
31
+ url={https://arxiv.org/abs/2408.08189},
32
+ }
33
+ ```
34
+
35
+ ## License
36
+ This project is licensed under the [Apache License (Version 2.0)](https://github.com/modelscope/modelscope/blob/master/LICENSE).
resources/models/CV-VAE/vae3d/config.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "CVVAEModel",
3
+ "_diffusers_version": "0.26.3",
4
+ "attn_resolutions": [],
5
+ "causal_decoder": false,
6
+ "causal_encoder": true,
7
+ "ch": 128,
8
+ "ch_mult": [
9
+ 1,
10
+ 2,
11
+ 4,
12
+ 4
13
+ ],
14
+ "decoder_attn_type": "spatial-temporal-xformer",
15
+ "double_z": true,
16
+ "dropout": 0.0,
17
+ "en_de_n_frames_a_time": 16,
18
+ "encoder_attn_type": "vanilla-xformers",
19
+ "half_3d": true,
20
+ "in_channels": 3,
21
+ "num_res_blocks": 2,
22
+ "num_video_frames": null,
23
+ "out_ch": 3,
24
+ "scaling_factor": 0.18215,
25
+ "force_upcast": true,
26
+ "reshape_x_dim_to_4": false,
27
+ "reshape_z_dim_to_4": false,
28
+ "spatial_n_compress": 8,
29
+ "tile_overlap_ratio": 0.2222,
30
+ "tile_spatial_size": 576,
31
+ "time_n_compress": 4,
32
+ "use_3d_conv": true,
33
+ "z_channels": 4
34
+ }
resources/models/CV-VAE/vae3d/cv_vae.ckpt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3a71060522dd74162ec3f734dd2a02c511179bca0960ff499d1c9fe544e6f712
3
+ size 904840525
resources/models/CV-VAE/vae3d/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:db6d128fc5ad7ba6e79c8c6f5870f785b4b48e4d7e9180ccff4f0141ddd34c24
3
+ size 729812244
resources/models/LongCLIP-L/longclip-L.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9d2c12ffcbf9099f95bfb9cf8f08f949b695e22fb0018e8895e088f4a88b6afd
3
+ size 1711923133
resources/models/fancyvideo_ckpts/vae_3d_125_frames/mp_rank_00_model_states.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:969216b028d7d5228cefa609b2132e28f0fd03571a6a5bb020c115f1577b43e9
3
+ size 4353119965
resources/models/fancyvideo_ckpts/vae_3d_61_frames/mp_rank_00_model_states.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:790c8c8ba84e46d3ca8aab2a1b7f5d8ef399c1cc0eee1ea421eb4aee99e423a7
3
+ size 4353119965
resources/models/fancyvideo_ckpts/video_backtracking/mp_rank_00_model_states.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:39f673e2727f0eaed574ec7245d6d0ccdb48d74712cd6fe304110d339d2be10e
3
+ size 4353119965
resources/models/fancyvideo_ckpts/video_extending/mp_rank_00_model_states.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f183d13d178aeeda6118fa2c7c6779ee4518a452cba221ec72dcaf9187cb073e
3
+ size 4353119965
resources/models/res-adapter/resadapter_v1_sd1.5/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fd162671c798970ca803048317bfcd23a7c201e3b90e5bc4eaaba147a26ea443
3
+ size 407928
resources/models/res-adapter/resadapter_v1_sd1.5/pytorch_lora_weights.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ab44862f8e732d3183004ca820d512aaf06a78576c2dd118f528d5902ed9682
3
+ size 1743012
resources/models/res-adapter/resadapter_v2_sd1.5/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:19b200eb2976055566af3e5c52c6f54937887c12e196a2e6e3eb97d0b578fafd
3
+ size 407928
resources/models/res-adapter/resadapter_v2_sd1.5/pytorch_lora_weights.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:17897742d0a561907a56d4da255ca293976888de35533b67beb5845da8c29cd3
3
+ size 1743012
resources/models/sd_v1-5_base_models/pixarsRendermanInspo_mk1.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2cc4e2ffb373d306ce361e8cd8896e68219e7613f1cfb2a7d6a8de55ac19ddce
3
+ size 4244290344
resources/models/sd_v1-5_base_models/realcartoon3d_v15.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:93f7411be9820c621e7243567e155750e2e77a1daaed1187bd28564e0ac5dc73
3
+ size 2132625894
resources/models/sd_v1-5_base_models/realisticVisionV60B1_v51VAE.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:15012c538f503ce2ebfc2c8547b268c75ccdaff7a281db55399940ff1d70e21d
3
+ size 2132625894
resources/models/sd_v1-5_base_models/toonyou_beta3.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:52768d2bc4e73af920df3466d0806d9a40727dab0869fc7aa2f3bc82214bb071
3
+ size 2132626252