Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -14,20 +14,23 @@ library_name: ltx-video
|
|
| 14 |
---
|
| 15 |
|
| 16 |
<p align="center">
|
| 17 |
-
<img src="assets/image.png" alt="
|
| 18 |
</p>
|
| 19 |
|
| 20 |
<div align="center">
|
| 21 |
|
| 22 |
-
<h1>
|
| 23 |
|
| 24 |
<p><strong>π¬ Pushing the Frontier of Long Video Generation</strong></p>
|
| 25 |
|
| 26 |
<p>Official model weights for <strong>minute-level multi-shot audio-video generation</strong> with a distilled DMD generator, paired cross-modal memory, and story-level consistency.</p>
|
| 27 |
|
|
|
|
|
|
|
| 28 |
<p>
|
| 29 |
-
<a href="
|
| 30 |
-
<a href="https://
|
|
|
|
| 31 |
<a href="#model-details"><b>𧬠Model</b></a> |
|
| 32 |
<a href="#usage"><b>π Usage</b></a> |
|
| 33 |
<a href="#results"><b>π Results</b></a> |
|
|
@@ -45,9 +48,9 @@ library_name: ltx-video
|
|
| 45 |
|
| 46 |
## Model Summary
|
| 47 |
|
| 48 |
-
**
|
| 49 |
|
| 50 |
-
|
| 51 |
|
| 52 |
This repository hosts the **released checkpoint**. Inference code is released separately β see the [Usage](#usage) section.
|
| 53 |
|
|
@@ -64,19 +67,23 @@ This repository hosts the **released checkpoint**. Inference code is released se
|
|
| 64 |
|
| 65 |
## Highlights
|
| 66 |
|
| 67 |
-
- ποΈ **Minute-level multi-shot stories**
|
| 68 |
-
- β‘ **DMD-distilled few-step inference**
|
| 69 |
-
- π **Joint audio-video generation**
|
| 70 |
-
- π§ **Paired cross-modal memory bank**
|
|
|
|
|
|
|
|
|
|
|
|
|
| 71 |
|
| 72 |
## Usage
|
| 73 |
|
| 74 |
-
Inference is run with the standalone **
|
| 75 |
|
| 76 |
### 1. Download the checkpoint
|
| 77 |
|
| 78 |
```bash
|
| 79 |
-
huggingface-cli download
|
| 80 |
--local-dir checkpoints
|
| 81 |
```
|
| 82 |
|
|
@@ -98,7 +105,7 @@ checkpoints/
|
|
| 98 |
### 2. Get the inference code
|
| 99 |
|
| 100 |
```bash
|
| 101 |
-
git clone https://github.com/
|
| 102 |
cd JoyAI-Echo
|
| 103 |
```
|
| 104 |
|
|
@@ -119,10 +126,7 @@ Inside each string, write these parts in order:
|
|
| 119 |
| **Style** | The overall visual and emotional aesthetic β e.g. realistic motorsport film language, cool daylight, restrained cinematic tension. |
|
| 120 |
| **Camera Movement** | The shot type and framing or movement β e.g. a stable close-up on the face, or a medium shot from the waist up. |
|
| 121 |
| **Background** | The setting and scene details behind the subject. |
|
| 122 |
-
| **Sound Effects & BGM** | The sounds in the scene and the background music β e.g. room tone, wind, footsteps and fabric, with a soft low music bed under the dialogue or
|
| 123 |
-
|
| 124 |
-
A more convenient prompt-writing workflow will be released as a **director agent** for everyone to use.
|
| 125 |
-
|
| 126 |
|
| 127 |
### 4. Run
|
| 128 |
|
|
@@ -147,14 +151,14 @@ python inference.py --num-frames 121 --video-height 480 --video-width 832
|
|
| 147 |
| Item | Value |
|
| 148 |
| --- | ---: |
|
| 149 |
| π¬ Long-form coherent story length | **5 min** |
|
| 150 |
-
| β‘
|
| 151 |
| π Benchmark stories | **100** |
|
| 152 |
| ποΈ Generated evaluation shots | **3,000** |
|
| 153 |
| π Frames per shot | **241 @ 25 fps** |
|
| 154 |
|
| 155 |
### Human Evaluation
|
| 156 |
|
| 157 |
-
GSB user study.
|
| 158 |
|
| 159 |
| Aspect (Long Video) | JoyAI-Echo | Tie | HappyOyster (Directing) |
|
| 160 |
| --- | ---: | ---: | ---: |
|
|
@@ -169,25 +173,36 @@ GSB user study. Values are the percentage of user preferences.
|
|
| 169 |
| Audio quality | 32.3% | 30.9% | 36.8% |
|
| 170 |
| Prompt following | 33.8% | 36.8% | 29.4% |
|
| 171 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 172 |
## Acknowledgements
|
| 173 |
|
| 174 |
-
We gratefully acknowledge
|
| 175 |
|
| 176 |
## Citation
|
| 177 |
|
| 178 |
-
If
|
| 179 |
|
| 180 |
```bibtex
|
| 181 |
@techreport{echo2026longvideo,
|
| 182 |
-
title = {
|
| 183 |
author = {{Echo Team @ Joy Future Academy, JD}},
|
| 184 |
institution = {Joy Future Academy, JD},
|
| 185 |
year = {2026},
|
| 186 |
-
month = {
|
| 187 |
-
url = {https://github.com/Echo-Team-Joy-Future-Academy-JD/JoyAI-Echo}
|
| 188 |
}
|
| 189 |
```
|
| 190 |
|
| 191 |
## License
|
| 192 |
|
| 193 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
---
|
| 15 |
|
| 16 |
<p align="center">
|
| 17 |
+
<img src="assets/image.png" alt="JoyAI-Echo generated video gallery" width="100%">
|
| 18 |
</p>
|
| 19 |
|
| 20 |
<div align="center">
|
| 21 |
|
| 22 |
+
<h1>JoyAI-Echo</h1>
|
| 23 |
|
| 24 |
<p><strong>π¬ Pushing the Frontier of Long Video Generation</strong></p>
|
| 25 |
|
| 26 |
<p>Official model weights for <strong>minute-level multi-shot audio-video generation</strong> with a distilled DMD generator, paired cross-modal memory, and story-level consistency.</p>
|
| 27 |
|
| 28 |
+
<p><strong>For academic research and non-commercial use only.</strong></p>
|
| 29 |
+
|
| 30 |
<p>
|
| 31 |
+
<a href="https://github.com/jd-opensource/JoyAI-Echo/blob/main/joyai-echo%20tech%20report.pdf"><b>π Paper</b></a> |
|
| 32 |
+
<a href="https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/"><b>π Project Page</b></a> |
|
| 33 |
+
<a href="https://github.com/jd-opensource/JoyAI-Echo"><b>π» Inference Code</b></a> |
|
| 34 |
<a href="#model-details"><b>𧬠Model</b></a> |
|
| 35 |
<a href="#usage"><b>π Usage</b></a> |
|
| 36 |
<a href="#results"><b>π Results</b></a> |
|
|
|
|
| 48 |
|
| 49 |
## Model Summary
|
| 50 |
|
| 51 |
+
**JoyAI-Echo** is a long-form, multi-shot, audio-video generation framework that breaks the barriers of error accumulation, weak temporal coherence, and prohibitive latency in long video generation. A cross-modal audio-visual memory bank preserves character appearance and voice timbre consistently over **five-minute** videos, while a post-training pipeline combining memory-based reinforcement learning with distribution matching distillation (DMD) delivers a **7.5Γ inference speedup** without sacrificing quality.
|
| 52 |
|
| 53 |
+
JoyAI-Echo decisively outperforms *HappyOyster* (directing mode) on long-form generation and even surpasses the short-video specialist *Wan 2.6* on human-centric tasks.
|
| 54 |
|
| 55 |
This repository hosts the **released checkpoint**. Inference code is released separately β see the [Usage](#usage) section.
|
| 56 |
|
|
|
|
| 67 |
|
| 68 |
## Highlights
|
| 69 |
|
| 70 |
+
- ποΈ **Minute-level multi-shot stories**: generate a sequence of coherent shots from one prompt JSON.
|
| 71 |
+
- β‘ **DMD-distilled few-step inference**: ~7.5Γ faster than the original pipeline.
|
| 72 |
+
- π **Joint audio-video generation**: one pipeline produces synchronized video and audio.
|
| 73 |
+
- π§ **Paired cross-modal memory bank**: conditions each new shot on prior visual identity and voice context for story-level consistency.
|
| 74 |
+
|
| 75 |
+
## Demo Gallery
|
| 76 |
+
|
| 77 |
+
Explore long-form and short-form JoyAI-Echo cases on the [Project Page](https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/). πΏ
|
| 78 |
|
| 79 |
## Usage
|
| 80 |
|
| 81 |
+
Inference is run with the standalone **JoyAI-Echo** inference repository.
|
| 82 |
|
| 83 |
### 1. Download the checkpoint
|
| 84 |
|
| 85 |
```bash
|
| 86 |
+
huggingface-cli download jdopensource/JoyAI-Echo \
|
| 87 |
--local-dir checkpoints
|
| 88 |
```
|
| 89 |
|
|
|
|
| 105 |
### 2. Get the inference code
|
| 106 |
|
| 107 |
```bash
|
| 108 |
+
git clone https://github.com/jd-opensource/JoyAI-Echo.git
|
| 109 |
cd JoyAI-Echo
|
| 110 |
```
|
| 111 |
|
|
|
|
| 126 |
| **Style** | The overall visual and emotional aesthetic β e.g. realistic motorsport film language, cool daylight, restrained cinematic tension. |
|
| 127 |
| **Camera Movement** | The shot type and framing or movement β e.g. a stable close-up on the face, or a medium shot from the waist up. |
|
| 128 |
| **Background** | The setting and scene details behind the subject. |
|
| 129 |
+
| **Sound Effects & BGM** | The sounds in the scene and the background music β e.g. room tone, wind, footsteps and fabric, with a soft low music bed under the dialogue or no background music. |
|
|
|
|
|
|
|
|
|
|
| 130 |
|
| 131 |
### 4. Run
|
| 132 |
|
|
|
|
| 151 |
| Item | Value |
|
| 152 |
| --- | ---: |
|
| 153 |
| π¬ Long-form coherent story length | **5 min** |
|
| 154 |
+
| β‘ Generation speedup over the original multi-step pipeline | **7.5Γ** |
|
| 155 |
| π Benchmark stories | **100** |
|
| 156 |
| ποΈ Generated evaluation shots | **3,000** |
|
| 157 |
| π Frames per shot | **241 @ 25 fps** |
|
| 158 |
|
| 159 |
### Human Evaluation
|
| 160 |
|
| 161 |
+
GSB user study on long- and short-video generation. The numbers denote the percentage of user preferences.
|
| 162 |
|
| 163 |
| Aspect (Long Video) | JoyAI-Echo | Tie | HappyOyster (Directing) |
|
| 164 |
| --- | ---: | ---: | ---: |
|
|
|
|
| 173 |
| Audio quality | 32.3% | 30.9% | 36.8% |
|
| 174 |
| Prompt following | 33.8% | 36.8% | 29.4% |
|
| 175 |
|
| 176 |
+
## Links
|
| 177 |
+
|
| 178 |
+
- Project page: [`https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/`](https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/)
|
| 179 |
+
- Inference code: [`https://github.com/jd-opensource/JoyAI-Echo`](https://github.com/jd-opensource/JoyAI-Echo)
|
| 180 |
+
- HuggingFace: [`https://huggingface.co/jdopensource/JoyAI-Echo`](https://huggingface.co/jdopensource/JoyAI-Echo)
|
| 181 |
+
|
| 182 |
## Acknowledgements
|
| 183 |
|
| 184 |
+
We gratefully acknowledge the open-source projects this work builds upon β in particular [LTX2.3](https://huggingface.co/Lightricks/LTX-2.3) for the base video generator and [Gemma](https://huggingface.co/google/gemma-3-12b-it) for the text encoder. Thanks to the broader research community whose contributions made this release possible.
|
| 185 |
|
| 186 |
## Citation
|
| 187 |
|
| 188 |
+
If JoyAI-Echo helps your research or products, please cite:
|
| 189 |
|
| 190 |
```bibtex
|
| 191 |
@techreport{echo2026longvideo,
|
| 192 |
+
title = {JoyAI-Echo: Pushing the Frontier of Long Video Generation},
|
| 193 |
author = {{Echo Team @ Joy Future Academy, JD}},
|
| 194 |
institution = {Joy Future Academy, JD},
|
| 195 |
year = {2026},
|
| 196 |
+
month = {May}
|
|
|
|
| 197 |
}
|
| 198 |
```
|
| 199 |
|
| 200 |
## License
|
| 201 |
|
| 202 |
+
This project is based on LTX-2 by Lightricks Ltd.
|
| 203 |
+
|
| 204 |
+
Portions of the original LTX-2 codebase have been modified by JD.com for academic and research purposes only.
|
| 205 |
+
This project is not intended for commercial use. For commercial use of LTX-2 or its derivatives, please contact Lightricks Ltd.
|
| 206 |
+
|
| 207 |
+
All original copyright, license, patent, trademark, and attribution notices from LTX-2 are retained.
|
| 208 |
+
This project remains subject to the LTX-2 Community License Agreement.
|