thanks to ai-forever ❤

.gitattributes

@@ -33,3 +33,19 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+__assets__/results/A[[:space:]]car[[:space:]]moving[[:space:]]on[[:space:]]the[[:space:]]road[[:space:]]from[[:space:]]the[[:space:]]sea[[:space:]]to[[:space:]]the[[:space:]]mountains.gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/A[[:space:]]red[[:space:]]car[[:space:]]drifting,[[:space:]]4k[[:space:]]video.gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/Craft[[:space:]]a[[:space:]]heartwarming[[:space:]]narrative[[:space:]]showcasing[[:space:]]the[[:space:]]bond[[:space:]]between[[:space:]]a[[:space:]]human[[:space:]]and[[:space:]]their[[:space:]]loyal[[:space:]]pet[[:space:]]companion..gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/Erupting[[:space:]]volcano_[[:space:]]raw[[:space:]]power,[[:space:]]molten[[:space:]]lava,[[:space:]]and[[:space:]]the[[:space:]]forces[[:space:]]of[[:space:]]the[[:space:]]Earth.gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/Evoke[[:space:]]the[[:space:]]sense[[:space:]]of[[:space:]]wonder[[:space:]]in[[:space:]]a[[:space:]]time-lapse[[:space:]]journey[[:space:]]through[[:space:]]changing[[:space:]]seasons..gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/Explore[[:space:]]the[[:space:]]fascinating[[:space:]]world[[:space:]]of[[:space:]]underwater[[:space:]]creatures[[:space:]]in[[:space:]]a[[:space:]]visually[[:space:]]stunning[[:space:]]sequence.gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/Majestic[[:space:]]humpback[[:space:]]whale[[:space:]]breaching_[[:space:]]power,[[:space:]]grace,[[:space:]]and[[:space:]]ocean[[:space:]]spectacle.gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/Majestic[[:space:]]waterfalls[[:space:]]in[[:space:]]a[[:space:]]lush[[:space:]]rainforest_[[:space:]]power,[[:space:]]mist,[[:space:]]and[[:space:]]biodiversity.gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/Polar[[:space:]]ice[[:space:]]caps_[[:space:]]the[[:space:]]pristine[[:space:]]wilderness[[:space:]]of[[:space:]]the[[:space:]]Arctic[[:space:]]and[[:space:]]Antarctic.gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/Rolling[[:space:]]waves[[:space:]]on[[:space:]]a[[:space:]]sandy[[:space:]]beach_[[:space:]]relaxation,[[:space:]]rhythm,[[:space:]]and[[:space:]]coastal[[:space:]]beauty.gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/Sloth[[:space:]]in[[:space:]]slow[[:space:]]motion_[[:space:]]deliberate[[:space:]]movements,[[:space:]]relaxation,[[:space:]]and[[:space:]]arboreal[[:space:]]life.gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/Time-lapse[[:space:]]of[[:space:]]a[[:space:]]flower[[:space:]]blooming_[[:space:]]growth,[[:space:]]beauty,[[:space:]]and[[:space:]]the[[:space:]]passage[[:space:]]of[[:space:]]time..gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/Wildlife[[:space:]]migration_[[:space:]]herds[[:space:]]on[[:space:]]the[[:space:]]move,[[:space:]]crossing[[:space:]]landscapes[[:space:]]in[[:space:]]harmony.gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/chemistry[[:space:]]laboratory,[[:space:]]chemical[[:space:]]explosion,[[:space:]]4k.gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/luminescent[[:space:]]jellyfish[[:space:]]swims[[:space:]]underwater,[[:space:]]neon,[[:space:]]4k.gif filter=lfs diff=lfs merge=lfs -text
+__assets__/results/white[[:space:]]ghost[[:space:]]flies[[:space:]]through[[:space:]]a[[:space:]]night[[:space:]]clearing,[[:space:]]4k.gif filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,133 @@
+---
+license: apache-2.0
+---
+# Kandinsky Video — a new text-to-video generation model 
+## SoTA quality among open-source solutions
+
+This repository is the official implementation of Kandinsky Video model
+
+
+Paper | [Project](https://ai-forever.github.io/kandinsky-video/) | ![Hugging Face Spaces](https://img.shields.io/badge/🤗-Huggingface-yello.svg) | [Telegram-bot](https://t.me/video_kandinsky_bot) | Habr post
+
+
+<p align="center">
+<img src="__assets__/title.JPG" width="800px"/>
+<br>
+<em>Kandinsky Video is a text-to-video generation model, which is based on the FusionFrames architecture, consisting of two main stages: keyframe generation and interpolation. Our approach for temporal conditioning allows us to generate videos with high-quality appearance, smoothness and dynamics.</em>
+</p>
+
+
+
+## Pipeline
+
+<p align="center">
+<img src="__assets__/pipeline.jpg" width="800px"/>
+<br>
+<em>The encoded text prompt enters the U-Net keyframe generation model with temporal layers or blocks, and then the sampled latent keyframes are  sent to the latent interpolation model in such a way as to predict three interpolation frames between two keyframes. A temporal MoVQ-GAN decoder is used to get the final video result.</em>
+</p>
+
+
+**Architecture details**
+
++ Text encoder (Flan-UL2) - 8.6B
++ Latent Diffusion U-Net3D - 4.0B
++ MoVQ encoder/decoder - 256M
+
+
+## How to use
+
+Check our jupyter notebooks with examples in `./examples` folder
+### 1. text2video
+
+```python
+from video_kandinsky3 import get_T2V_pipeline
+
+t2v_pipe = get_T2V_pipeline('cuda', fp16=True)
+
+pfps = 'medium' # ['low', 'medium', 'high']
+video = t2v_pipe(
+    'a red car is drifting on the mountain road, close view, fast movement',
+    width=640, height=384, fps=fps
+)
+```
+
+
+## Results
+
+
+<table class="center">
+<tr>
+  <td><img src="__assets__/results/A car moving on the road from the sea to the mountains.gif" raw=true></td>
+  <td><img src="__assets__/results/A red car drifting, 4k video.gif"></td>
+  <td><img src="__assets__/results/chemistry laboratory, chemical explosion, 4k.gif"></td>
+  <td><img src="__assets__/results/Erupting volcano_ raw power, molten lava, and the forces of the Earth.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"A car moving on the road from the sea to the mountains"</td>
+  <td width=25% align="center">"A red car drifting, 4k video"</td>
+  <td width=25% align="center">"Chemistry laboratory, chemical explosion, 4k"</td>
+  <td width=25% align="center">"Erupting volcano raw power, molten lava, and the forces of the Earth"</td>
+</tr>
+
+<tr>
+  <td><img src="__assets__/results/luminescent jellyfish swims underwater, neon, 4k.gif" raw=true></td>
+  <td><img src="__assets__/results/Majestic waterfalls in a lush rainforest_ power, mist, and biodiversity.gif"></td>
+  <td><img src="__assets__/results/white ghost flies through a night clearing, 4k.gif"></td>
+  <td><img src="__assets__/results/Wildlife migration_ herds on the move, crossing landscapes in harmony.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"Luminescent jellyfish swims underwater, neon, 4k"</td>
+  <td width=25% align="center">"Majestic waterfalls in a lush rainforest power, mist, and biodiversity"</td>
+  <td width=25% align="center">"White ghost flies through a night clearing, 4k"</td>
+  <td width=25% align="center">"Wildlife migration herds on the move, crossing landscapes in harmony"</td>
+</tr>
+
+<tr>
+  <td><img src="__assets__/results/Majestic humpback whale breaching_ power, grace, and ocean spectacle.gif" raw=true></td>
+  <td><img src="__assets__/results/Evoke the sense of wonder in a time-lapse journey through changing seasons..gif"></td>
+  <td><img src="__assets__/results/Explore the fascinating world of underwater creatures in a visually stunning sequence.gif"></td>
+  <td><img src="__assets__/results/Polar ice caps_ the pristine wilderness of the Arctic and Antarctic.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"Majestic humpback whale breaching power, grace, and ocean spectacle"</td>
+  <td width=25% align="center">"Evoke the sense of wonder in a time-lapse journey through changing seasons"</td>
+  <td width=25% align="center">"Explore the fascinating world of underwater creatures in a visually stunning sequence"</td>
+  <td width=25% align="center">"Polar ice caps the pristine wilderness of the Arctic and Antarctic"</td>
+</tr>
+
+
+<tr>
+  <td><img src="__assets__/results/Rolling waves on a sandy beach_ relaxation, rhythm, and coastal beauty.gif" raw=true></td>
+  <td><img src="__assets__/results/Sloth in slow motion_ deliberate movements, relaxation, and arboreal life.gif"></td>
+  <td><img src="__assets__/results/Time-lapse of a flower blooming_ growth, beauty, and the passage of time..gif"></td>
+  <td><img src="__assets__/results/Craft a heartwarming narrative showcasing the bond between a human and their loyal pet companion..gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"Rolling waves on a sandy beach relaxation, rhythm, and coastal beauty"</td>
+  <td width=25% align="center">"Sloth in slow motion deliberate movements, relaxation, and arboreal life"</td>
+  <td width=25% align="center">"Time-lapse of a flower blooming growth, beauty, and the passage of time"</td>
+  <td width=25% align="center">"Craft a heartwarming narrative showcasing the bond between a human and their loyal pet companion"</td>
+</tr>
+
+
+</table>
+
+
+# Authors
+
++ Vladimir Arkhipkin: [Github](https://github.com/oriBetelgeuse), [Google Scholar](https://scholar.google.com/citations?user=D-Ko0oAAAAAJ&hl=ru)
++ Zein Shaheen: [Github](https://github.com/zeinsh), [Google Scholar](https://scholar.google.ru/citations?user=bxlgMxMAAAAJ&hl=en)
++ Viacheslav Vasilev: [Github](https://github.com/vivasilev), [Google Scholar](https://scholar.google.com/citations?user=redAz-kAAAAJ&hl=ru&oi=sra)
++ Igor Pavlov: [Github](https://github.com/boomb0om)
++ Elizaveta Dakhova: [Github](https://github.com/LizaDakhova)
++ Anastasia Lysenko: [Github](https://github.com/LysenkoAnastasia)
++ Sergey Markov
++ Denis Dimitrov: [Github](https://github.com/denndimitrov), [Google Scholar](https://scholar.google.com/citations?user=3JSIJpYAAAAJ&hl=ru&oi=ao)
++ Andrey Kuznetsov: [Github](https://github.com/kuznetsoffandrey), [Google Scholar](https://scholar.google.com/citations?user=q0lIfCEAAAAJ&hl=ru)
+
+
+## BibTeX
+If you use our work in your research, please cite our publication:
+```
+TBD
+```

weights/kandinsky_video.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:75c8f416c05c67f231cd300961416a73e90f255288bff5dce6b80b0bf43da3a6
+size 16642806469

weights/kandinsky_video_interpolation.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc724de4187e29aeaccff8f5274cfbe325f23ecaf9fc7adbbdcc8d66ff8b2c83
+size 15490902029

weights/movq.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ad3a981f7cd5cd19ff85382319039b94be4c2c11760bf46079d96bb84ca98ef
+size 1082549221