leffff commited on
Commit
8d96562
Β·
verified Β·
1 Parent(s): fd14f6f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +139 -3
README.md CHANGED
@@ -1,3 +1,139 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ ---
4
+ <div align="center">
5
+ <picture>
6
+ <img src="assets/KANDINSKY_LOGO_1_BLACK.png">
7
+ </picture>
8
+ </div>
9
+
10
+ <div align="center">
11
+ <a href="https://habr.com/ru/companies/sberbank/articles/951800/">Habr</a> | <a href="https://kandinskylab.ai/">Project Page</a> | <a href="https://arxiv.org/abs/2511.14993">Technical Report</a> | <a href="https://github.com/kandinskylab/Kandinsky-5">Original Github</a> | <a href="https://huggingface.co/collections/kandinskylab/kandinsky-50-video-lite-diffusers"> πŸ€— Diffusers</a>
12
+ </div>
13
+
14
+ -----
15
+
16
+ <h1>Kandinsky 5.0 I2V Lite - Diffusers</h1>
17
+
18
+ This repository provides the πŸ€— Diffusers integration for Kandinsky 5.0 Lite - a lightweight video generation model (2B parameters) that ranks #1 among open-source models in its class.
19
+
20
+ ## Project Updates
21
+
22
+ - πŸ”₯ **2025/09/29**: We have open-sourced `Kandinsky 5.0 T2V Lite` a lite (2B parameters) version of `Kandinsky 5.0 Video` text-to-video generation model.
23
+ - πŸš€ **Diffusers Integration**: Now available with easy-to-use πŸ€— Diffusers pipeline!
24
+
25
+ ## Kandinsky 5.0 Lite
26
+
27
+ Kandinsky 5.0 T2V Lite is a lightweight video generation model (2B parameters) that ranks #1 among open-source models in its class. It outperforms larger Wan models (5B and 14B) and offers the best understanding of Russian concepts in the open-source ecosystem.
28
+
29
+ We provide 9 model variants, each optimized for different use cases:
30
+
31
+ * **SFT model** β€” delivers the highest generation quality
32
+ * **CFG-distilled** β€” runs 2Γ— faster
33
+ * **Diffusion-distilled** β€” enables low-latency generation with minimal quality loss (6Γ— faster)
34
+ * **Pretrain model** β€” designed for fine-tuning by researchers and enthusiasts
35
+
36
+
37
+ ## Basic Usage
38
+ ```python
39
+ import torch
40
+ from diffusers import Kandinsky5I2VPipeline
41
+ from diffusers.utils import export_to_video
42
+
43
+ # Load the pipeline
44
+ pipe = Kandinsky5I2VPipeline.from_pretrained(
45
+ "kandinskylab/Kandinsky-5.0-I2V-Lite-5s-Diffusers",
46
+ torch_dtype=torch.bfloat16
47
+ )
48
+ pipe = pipe.to("cuda")
49
+
50
+ image = load_image(
51
+ "https://frontofficesports.com/wp-content/uploads/2023/10/USATSI_19520555_168393969_lowres-scaled-e1697215176168.jpg?quality=100"
52
+ )
53
+
54
+ height = 480
55
+ width = 640
56
+ image = image.resize((width, height))
57
+
58
+ prompt = "A football player kicking a ball"
59
+ negative_prompt = "Static, 2D cartoon, cartoon, 2d animation, paintings, images, worst quality, low quality, ugly, deformed, walking backwards"
60
+
61
+ output = pipe(
62
+ image=image,
63
+ prompt=prompt,
64
+ negative_prompt=negative_prompt,
65
+ height=512,
66
+ width=768,
67
+ num_frames=121,
68
+ num_inference_steps=50,
69
+ guidance_scale=5.0,
70
+ ).frames[0]
71
+
72
+ ## Save the video
73
+ export_to_video(output, "output.mp4", fps=24, quality=9)
74
+ ```
75
+
76
+
77
+ ## Architecture
78
+ Latent diffusion pipeline with Flow Matching.
79
+
80
+ Diffusion Transformer (DiT) as the main generative backbone with cross-attention to text embeddings.
81
+
82
+ Qwen2.5-VL and CLIP provides text embeddings
83
+
84
+ HunyuanVideo 3D VAE encodes/decodes video into a latent space
85
+
86
+ DiT is the main generative module using cross-attention to condition on text
87
+
88
+ <div align="center">
89
+ <img width="1600" height="477" alt="Pipeline Architecture" src="https://github.com/user-attachments/assets/17fc2eb5-05e3-4591-9ec6-0f6e1ca397b3" />
90
+ </div>
91
+
92
+ <div align="center">
93
+ <img width="800" height="406" alt="Model Architecture" src="https://github.com/user-attachments/assets/f3006742-e261-4c39-b7dc-e39330be9a09" />
94
+ </div>
95
+
96
+ ## Examples
97
+
98
+ Kandinsky 5.0 T2V Lite SFT
99
+ <table border="0" style="width: 200; text-align: left; margin-top: 20px;"> <tr> <td> <video src="https://github.com/user-attachments/assets/bc38821b-f9f1-46db-885f-1f70464669eb" width=200 controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/9f64c940-4df8-4c51-bd81-a05de8e70fc3" width=200 controls autoplay loop></video> </td> <tr> <td> <video src="https://github.com/user-attachments/assets/77dd417f-e0bf-42bd-8d80-daffcd054add" width=200 controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/385a0076-f01c-4663-aa46-6ce50352b9ed" width=200 controls autoplay loop></video> </td> <tr> <td> <video src="https://github.com/user-attachments/assets/7c1bcb31-cc7d-4385-9a33-2b0cc28393dd" width=200 controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/990a8a0b-2df1-4bbc-b2e3-2859b6f1eea6" width=200 controls autoplay loop></video> </td> </tr> </table>
100
+ Kandinsky 5.0 T2V Lite Distill
101
+ <table border="0" style="width: 200; text-align: left; margin-top: 20px;"> <tr> <td> <video src="https://github.com/user-attachments/assets/861342f9-f576-4083-8a3b-94570a970d58" width=200 controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/302e4e7d-781d-4a58-9b10-8c473d469c4b" width=200 controls autoplay loop></video> </td> <tr> <td> <video src="https://github.com/user-attachments/assets/3e70175c-40e5-4aec-b506-38006fe91a76" width=200 controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/b7da85f7-8b62-4d46-9460-7f0e505de810" width=200 controls autoplay loop></video> </td> </table>
102
+ Results
103
+ Side-by-Side Evaluation
104
+ The evaluation is based on the expanded prompts from the Movie Gen benchmark.
105
+
106
+ <table border="0" style="width: 400; text-align: left; margin-top: 20px;"> <tr> <td> <img src="assets/sbs/kandinsky_5_video_lite_vs_sora.jpg" width=400 ></img> </td> <td> <img src="assets/sbs/kandinsky_5_video_lite_vs_wan_2.1_14B.jpg" width=400 ></img> </td> <tr> <td> <img src="assets/sbs/kandinsky_5_video_lite_vs_wan_2.2_5B.jpg" width=400 ></img> </td> <td> <img src="assets/sbs/kandinsky_5_video_lite_vs_wan_2.2_A14B.jpg" width=400 ></img> </td> <tr> <td> <img src="assets/sbs/kandinsky_5_video_lite_vs_wan_2.1_1.3B.jpg" width=400 ></img> </td> </table>
107
+ Distill Side-by-Side Evaluation
108
+ <table border="0" style="width: 400; text-align: left; margin-top: 20px;"> <tr> <td> <img src="assets/sbs/kandinsky_5_video_lite_5s_vs_kandinsky_5_video_lite_distill_5s.jpg" width=400 ></img> </td> <td> <img src="assets/sbs/kandinsky_5_video_lite_10s_vs_kandinsky_5_video_lite_distill_10s.jpg" width=400 ></img> </td> </table>
109
+ VBench Results
110
+ <div align="center"> <picture> <img src="assets/vbench.png"> </picture> </div>
111
+ Beta Testing
112
+ You can apply to participate in the beta testing of the Kandinsky Video Lite via the telegram bot.
113
+
114
+ ```bibtex
115
+ @misc{kandinsky2025,
116
+ author = {Alexey Letunovskiy, Maria Kovaleva, Ivan Kirillov, Lev Novitskiy, Denis Koposov,
117
+ Dmitrii Mikhailov, Anna Averchenkova, Andrey Shutkin, Julia Agafonova, Olga Kim,
118
+ Anastasiia Kargapoltseva, Nikita Kiselev, Vladimir Arkhipkin, Vladimir Korviakov,
119
+ Nikolai Gerasimenko, Denis Parkhomenko, Anna Dmitrienko, Anastasia Maltseva,
120
+ Kirill Chernyshev, Ilia Vasiliev, Viacheslav Vasilev, Vladimir Polovnikov,
121
+ Yury Kolabushin, Alexander Belykh, Mikhail Mamaev, Anastasia Aliaskina,
122
+ Tatiana Nikulina, Polina Gavrilova, Denis Dimitrov},
123
+ title = {Kandinsky 5.0: A family of diffusion models for Video & Image generation},
124
+ howpublished = {\url{https://github.com/kandinskylab/Kandinsky-5}},
125
+ year = 2025
126
+ }
127
+
128
+ @misc{mikhailov2025nablanablaneighborhoodadaptiveblocklevel,
129
+ title={$\nabla$NABLA: Neighborhood Adaptive Block-Level Attention},
130
+ author={Dmitrii Mikhailov and Aleksey Letunovskiy and Maria Kovaleva and Vladimir Arkhipkin
131
+ and Vladimir Korviakov and Vladimir Polovnikov and Viacheslav Vasilev
132
+ and Evelina Sidorova and Denis Dimitrov},
133
+ year={2025},
134
+ eprint={2507.13546},
135
+ archivePrefix={arXiv},
136
+ primaryClass={cs.CV},
137
+ url={https://arxiv.org/abs/2507.13546},
138
+ }
139
+ ```