Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,139 @@
|
|
| 1 |
-
---
|
| 2 |
-
license:
|
| 3 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
---
|
| 4 |
+
<div align="center">
|
| 5 |
+
<picture>
|
| 6 |
+
<img src="assets/KANDINSKY_LOGO_1_BLACK.png">
|
| 7 |
+
</picture>
|
| 8 |
+
</div>
|
| 9 |
+
|
| 10 |
+
<div align="center">
|
| 11 |
+
<a href="https://habr.com/ru/companies/sberbank/articles/951800/">Habr</a> | <a href="https://kandinskylab.ai/">Project Page</a> | <a href="https://arxiv.org/abs/2511.14993">Technical Report</a> | <a href="https://github.com/kandinskylab/Kandinsky-5">Original Github</a> | <a href="https://huggingface.co/collections/kandinskylab/kandinsky-50-video-lite-diffusers"> π€ Diffusers</a>
|
| 12 |
+
</div>
|
| 13 |
+
|
| 14 |
+
-----
|
| 15 |
+
|
| 16 |
+
<h1>Kandinsky 5.0 I2V Lite - Diffusers</h1>
|
| 17 |
+
|
| 18 |
+
This repository provides the π€ Diffusers integration for Kandinsky 5.0 Lite - a lightweight video generation model (2B parameters) that ranks #1 among open-source models in its class.
|
| 19 |
+
|
| 20 |
+
## Project Updates
|
| 21 |
+
|
| 22 |
+
- π₯ **2025/09/29**: We have open-sourced `Kandinsky 5.0 T2V Lite` a lite (2B parameters) version of `Kandinsky 5.0 Video` text-to-video generation model.
|
| 23 |
+
- π **Diffusers Integration**: Now available with easy-to-use π€ Diffusers pipeline!
|
| 24 |
+
|
| 25 |
+
## Kandinsky 5.0 Lite
|
| 26 |
+
|
| 27 |
+
Kandinsky 5.0 T2V Lite is a lightweight video generation model (2B parameters) that ranks #1 among open-source models in its class. It outperforms larger Wan models (5B and 14B) and offers the best understanding of Russian concepts in the open-source ecosystem.
|
| 28 |
+
|
| 29 |
+
We provide 9 model variants, each optimized for different use cases:
|
| 30 |
+
|
| 31 |
+
* **SFT model** β delivers the highest generation quality
|
| 32 |
+
* **CFG-distilled** β runs 2Γ faster
|
| 33 |
+
* **Diffusion-distilled** β enables low-latency generation with minimal quality loss (6Γ faster)
|
| 34 |
+
* **Pretrain model** β designed for fine-tuning by researchers and enthusiasts
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
## Basic Usage
|
| 38 |
+
```python
|
| 39 |
+
import torch
|
| 40 |
+
from diffusers import Kandinsky5I2VPipeline
|
| 41 |
+
from diffusers.utils import export_to_video
|
| 42 |
+
|
| 43 |
+
# Load the pipeline
|
| 44 |
+
pipe = Kandinsky5I2VPipeline.from_pretrained(
|
| 45 |
+
"kandinskylab/Kandinsky-5.0-I2V-Lite-5s-Diffusers",
|
| 46 |
+
torch_dtype=torch.bfloat16
|
| 47 |
+
)
|
| 48 |
+
pipe = pipe.to("cuda")
|
| 49 |
+
|
| 50 |
+
image = load_image(
|
| 51 |
+
"https://frontofficesports.com/wp-content/uploads/2023/10/USATSI_19520555_168393969_lowres-scaled-e1697215176168.jpg?quality=100"
|
| 52 |
+
)
|
| 53 |
+
|
| 54 |
+
height = 480
|
| 55 |
+
width = 640
|
| 56 |
+
image = image.resize((width, height))
|
| 57 |
+
|
| 58 |
+
prompt = "A football player kicking a ball"
|
| 59 |
+
negative_prompt = "Static, 2D cartoon, cartoon, 2d animation, paintings, images, worst quality, low quality, ugly, deformed, walking backwards"
|
| 60 |
+
|
| 61 |
+
output = pipe(
|
| 62 |
+
image=image,
|
| 63 |
+
prompt=prompt,
|
| 64 |
+
negative_prompt=negative_prompt,
|
| 65 |
+
height=512,
|
| 66 |
+
width=768,
|
| 67 |
+
num_frames=121,
|
| 68 |
+
num_inference_steps=50,
|
| 69 |
+
guidance_scale=5.0,
|
| 70 |
+
).frames[0]
|
| 71 |
+
|
| 72 |
+
## Save the video
|
| 73 |
+
export_to_video(output, "output.mp4", fps=24, quality=9)
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
## Architecture
|
| 78 |
+
Latent diffusion pipeline with Flow Matching.
|
| 79 |
+
|
| 80 |
+
Diffusion Transformer (DiT) as the main generative backbone with cross-attention to text embeddings.
|
| 81 |
+
|
| 82 |
+
Qwen2.5-VL and CLIP provides text embeddings
|
| 83 |
+
|
| 84 |
+
HunyuanVideo 3D VAE encodes/decodes video into a latent space
|
| 85 |
+
|
| 86 |
+
DiT is the main generative module using cross-attention to condition on text
|
| 87 |
+
|
| 88 |
+
<div align="center">
|
| 89 |
+
<img width="1600" height="477" alt="Pipeline Architecture" src="https://github.com/user-attachments/assets/17fc2eb5-05e3-4591-9ec6-0f6e1ca397b3" />
|
| 90 |
+
</div>
|
| 91 |
+
|
| 92 |
+
<div align="center">
|
| 93 |
+
<img width="800" height="406" alt="Model Architecture" src="https://github.com/user-attachments/assets/f3006742-e261-4c39-b7dc-e39330be9a09" />
|
| 94 |
+
</div>
|
| 95 |
+
|
| 96 |
+
## Examples
|
| 97 |
+
|
| 98 |
+
Kandinsky 5.0 T2V Lite SFT
|
| 99 |
+
<table border="0" style="width: 200; text-align: left; margin-top: 20px;"> <tr> <td> <video src="https://github.com/user-attachments/assets/bc38821b-f9f1-46db-885f-1f70464669eb" width=200 controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/9f64c940-4df8-4c51-bd81-a05de8e70fc3" width=200 controls autoplay loop></video> </td> <tr> <td> <video src="https://github.com/user-attachments/assets/77dd417f-e0bf-42bd-8d80-daffcd054add" width=200 controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/385a0076-f01c-4663-aa46-6ce50352b9ed" width=200 controls autoplay loop></video> </td> <tr> <td> <video src="https://github.com/user-attachments/assets/7c1bcb31-cc7d-4385-9a33-2b0cc28393dd" width=200 controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/990a8a0b-2df1-4bbc-b2e3-2859b6f1eea6" width=200 controls autoplay loop></video> </td> </tr> </table>
|
| 100 |
+
Kandinsky 5.0 T2V Lite Distill
|
| 101 |
+
<table border="0" style="width: 200; text-align: left; margin-top: 20px;"> <tr> <td> <video src="https://github.com/user-attachments/assets/861342f9-f576-4083-8a3b-94570a970d58" width=200 controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/302e4e7d-781d-4a58-9b10-8c473d469c4b" width=200 controls autoplay loop></video> </td> <tr> <td> <video src="https://github.com/user-attachments/assets/3e70175c-40e5-4aec-b506-38006fe91a76" width=200 controls autoplay loop></video> </td> <td> <video src="https://github.com/user-attachments/assets/b7da85f7-8b62-4d46-9460-7f0e505de810" width=200 controls autoplay loop></video> </td> </table>
|
| 102 |
+
Results
|
| 103 |
+
Side-by-Side Evaluation
|
| 104 |
+
The evaluation is based on the expanded prompts from the Movie Gen benchmark.
|
| 105 |
+
|
| 106 |
+
<table border="0" style="width: 400; text-align: left; margin-top: 20px;"> <tr> <td> <img src="assets/sbs/kandinsky_5_video_lite_vs_sora.jpg" width=400 ></img> </td> <td> <img src="assets/sbs/kandinsky_5_video_lite_vs_wan_2.1_14B.jpg" width=400 ></img> </td> <tr> <td> <img src="assets/sbs/kandinsky_5_video_lite_vs_wan_2.2_5B.jpg" width=400 ></img> </td> <td> <img src="assets/sbs/kandinsky_5_video_lite_vs_wan_2.2_A14B.jpg" width=400 ></img> </td> <tr> <td> <img src="assets/sbs/kandinsky_5_video_lite_vs_wan_2.1_1.3B.jpg" width=400 ></img> </td> </table>
|
| 107 |
+
Distill Side-by-Side Evaluation
|
| 108 |
+
<table border="0" style="width: 400; text-align: left; margin-top: 20px;"> <tr> <td> <img src="assets/sbs/kandinsky_5_video_lite_5s_vs_kandinsky_5_video_lite_distill_5s.jpg" width=400 ></img> </td> <td> <img src="assets/sbs/kandinsky_5_video_lite_10s_vs_kandinsky_5_video_lite_distill_10s.jpg" width=400 ></img> </td> </table>
|
| 109 |
+
VBench Results
|
| 110 |
+
<div align="center"> <picture> <img src="assets/vbench.png"> </picture> </div>
|
| 111 |
+
Beta Testing
|
| 112 |
+
You can apply to participate in the beta testing of the Kandinsky Video Lite via the telegram bot.
|
| 113 |
+
|
| 114 |
+
```bibtex
|
| 115 |
+
@misc{kandinsky2025,
|
| 116 |
+
author = {Alexey Letunovskiy, Maria Kovaleva, Ivan Kirillov, Lev Novitskiy, Denis Koposov,
|
| 117 |
+
Dmitrii Mikhailov, Anna Averchenkova, Andrey Shutkin, Julia Agafonova, Olga Kim,
|
| 118 |
+
Anastasiia Kargapoltseva, Nikita Kiselev, Vladimir Arkhipkin, Vladimir Korviakov,
|
| 119 |
+
Nikolai Gerasimenko, Denis Parkhomenko, Anna Dmitrienko, Anastasia Maltseva,
|
| 120 |
+
Kirill Chernyshev, Ilia Vasiliev, Viacheslav Vasilev, Vladimir Polovnikov,
|
| 121 |
+
Yury Kolabushin, Alexander Belykh, Mikhail Mamaev, Anastasia Aliaskina,
|
| 122 |
+
Tatiana Nikulina, Polina Gavrilova, Denis Dimitrov},
|
| 123 |
+
title = {Kandinsky 5.0: A family of diffusion models for Video & Image generation},
|
| 124 |
+
howpublished = {\url{https://github.com/kandinskylab/Kandinsky-5}},
|
| 125 |
+
year = 2025
|
| 126 |
+
}
|
| 127 |
+
|
| 128 |
+
@misc{mikhailov2025nablanablaneighborhoodadaptiveblocklevel,
|
| 129 |
+
title={$\nabla$NABLA: Neighborhood Adaptive Block-Level Attention},
|
| 130 |
+
author={Dmitrii Mikhailov and Aleksey Letunovskiy and Maria Kovaleva and Vladimir Arkhipkin
|
| 131 |
+
and Vladimir Korviakov and Vladimir Polovnikov and Viacheslav Vasilev
|
| 132 |
+
and Evelina Sidorova and Denis Dimitrov},
|
| 133 |
+
year={2025},
|
| 134 |
+
eprint={2507.13546},
|
| 135 |
+
archivePrefix={arXiv},
|
| 136 |
+
primaryClass={cs.CV},
|
| 137 |
+
url={https://arxiv.org/abs/2507.13546},
|
| 138 |
+
}
|
| 139 |
+
```
|