jahnsonblack commited on
Commit
7c3edb5
Β·
verified Β·
1 Parent(s): 9789b58

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +40 -25
README.md CHANGED
@@ -14,20 +14,23 @@ library_name: ltx-video
14
  ---
15
 
16
  <p align="center">
17
- <img src="assets/image.png" alt="Echo-LongVideo generated video gallery" width="100%">
18
  </p>
19
 
20
  <div align="center">
21
 
22
- <h1>Echo-LongVideo</h1>
23
 
24
  <p><strong>🎬 Pushing the Frontier of Long Video Generation</strong></p>
25
 
26
  <p>Official model weights for <strong>minute-level multi-shot audio-video generation</strong> with a distilled DMD generator, paired cross-modal memory, and story-level consistency.</p>
27
 
 
 
28
  <p>
29
- <a href="LongVideo_Tech_Report.pdf"><b>πŸ“„ Paper</b></a> |
30
- <a href="https://github.com/Echo-Team-Joy-Future-Academy-JD/JoyAI-Echo"><b>πŸ’» Inference Code</b></a> |
 
31
  <a href="#model-details"><b>🧬 Model</b></a> |
32
  <a href="#usage"><b>πŸš€ Usage</b></a> |
33
  <a href="#results"><b>πŸ“Š Results</b></a> |
@@ -45,9 +48,9 @@ library_name: ltx-video
45
 
46
  ## Model Summary
47
 
48
- **Echo-LongVideo** (a.k.a. **JoyAI-Echo**) is a long-form, multi-shot, audio-video generation model. A cross-modal audio-visual memory bank preserves character appearance and voice timbre consistently across up to **five-minute** videos, and a post-training pipeline combining memory-based reinforcement learning with distribution matching distillation (DMD) delivers a **7.5Γ— inference speedup** without sacrificing quality.
49
 
50
- In human evaluation, Echo-LongVideo decisively outperforms *HappyOyster* (directing mode) on long-form generation and surpasses the short-video specialist *Wan 2.6* on human-centric tasks.
51
 
52
  This repository hosts the **released checkpoint**. Inference code is released separately β€” see the [Usage](#usage) section.
53
 
@@ -64,19 +67,23 @@ This repository hosts the **released checkpoint**. Inference code is released se
64
 
65
  ## Highlights
66
 
67
- - 🎞️ **Minute-level multi-shot stories** from a single prompt JSON.
68
- - ⚑ **DMD-distilled few-step inference**, ~7.5Γ— faster than the original pipeline.
69
- - πŸ”Š **Joint audio-video generation** in a single pipeline.
70
- - 🧠 **Paired cross-modal memory bank** for story-level identity and voice consistency.
 
 
 
 
71
 
72
  ## Usage
73
 
74
- Inference is run with the standalone **Echo-LongVideo** inference repository.
75
 
76
  ### 1. Download the checkpoint
77
 
78
  ```bash
79
- huggingface-cli download <org>/Echo-LongVideo \
80
  --local-dir checkpoints
81
  ```
82
 
@@ -98,7 +105,7 @@ checkpoints/
98
  ### 2. Get the inference code
99
 
100
  ```bash
101
- git clone https://github.com/Echo-Team-Joy-Future-Academy-JD/JoyAI-Echo.git
102
  cd JoyAI-Echo
103
  ```
104
 
@@ -119,10 +126,7 @@ Inside each string, write these parts in order:
119
  | **Style** | The overall visual and emotional aesthetic β€” e.g. realistic motorsport film language, cool daylight, restrained cinematic tension. |
120
  | **Camera Movement** | The shot type and framing or movement β€” e.g. a stable close-up on the face, or a medium shot from the waist up. |
121
  | **Background** | The setting and scene details behind the subject. |
122
- | **Sound Effects & BGM** | The sounds in the scene and the background music β€” e.g. room tone, wind, footsteps and fabric, with a soft low music bed under the dialogue or nobackground music |
123
-
124
- A more convenient prompt-writing workflow will be released as a **director agent** for everyone to use.
125
-
126
 
127
  ### 4. Run
128
 
@@ -147,14 +151,14 @@ python inference.py --num-frames 121 --video-height 480 --video-width 832
147
  | Item | Value |
148
  | --- | ---: |
149
  | 🎬 Long-form coherent story length | **5 min** |
150
- | ⚑ Speedup over the original multi-step pipeline | **7.5Γ—** |
151
  | πŸ“š Benchmark stories | **100** |
152
  | 🎞️ Generated evaluation shots | **3,000** |
153
  | πŸ•’ Frames per shot | **241 @ 25 fps** |
154
 
155
  ### Human Evaluation
156
 
157
- GSB user study. Values are the percentage of user preferences.
158
 
159
  | Aspect (Long Video) | JoyAI-Echo | Tie | HappyOyster (Directing) |
160
  | --- | ---: | ---: | ---: |
@@ -169,25 +173,36 @@ GSB user study. Values are the percentage of user preferences.
169
  | Audio quality | 32.3% | 30.9% | 36.8% |
170
  | Prompt following | 33.8% | 36.8% | 29.4% |
171
 
 
 
 
 
 
 
172
  ## Acknowledgements
173
 
174
- We gratefully acknowledge [LTX-Video](https://github.com/Lightricks/LTX-Video) for the base video generator and [Gemma](https://huggingface.co/google/gemma-3-12b-it) for the text encoder, along with the broader open-source community.
175
 
176
  ## Citation
177
 
178
- If Echo-LongVideo helps your research or products, please cite:
179
 
180
  ```bibtex
181
  @techreport{echo2026longvideo,
182
- title = {Echo-LongVideo: Pushing the Frontier of Long Video Generation},
183
  author = {{Echo Team @ Joy Future Academy, JD}},
184
  institution = {Joy Future Academy, JD},
185
  year = {2026},
186
- month = {June},
187
- url = {https://github.com/Echo-Team-Joy-Future-Academy-JD/JoyAI-Echo}
188
  }
189
  ```
190
 
191
  ## License
192
 
193
- Released under the [LTX-2 Community License Agreement](https://github.com/Lightricks/LTX-2/blob/main/LICENSE). By downloading or using these weights, you agree to its terms. The bundled Gemma text encoder is governed by Google's separate Gemma license.
 
 
 
 
 
 
 
14
  ---
15
 
16
  <p align="center">
17
+ <img src="assets/image.png" alt="JoyAI-Echo generated video gallery" width="100%">
18
  </p>
19
 
20
  <div align="center">
21
 
22
+ <h1>JoyAI-Echo</h1>
23
 
24
  <p><strong>🎬 Pushing the Frontier of Long Video Generation</strong></p>
25
 
26
  <p>Official model weights for <strong>minute-level multi-shot audio-video generation</strong> with a distilled DMD generator, paired cross-modal memory, and story-level consistency.</p>
27
 
28
+ <p><strong>For academic research and non-commercial use only.</strong></p>
29
+
30
  <p>
31
+ <a href="https://github.com/jd-opensource/JoyAI-Echo/blob/main/joyai-echo%20tech%20report.pdf"><b>πŸ“„ Paper</b></a> |
32
+ <a href="https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/"><b>🌐 Project Page</b></a> |
33
+ <a href="https://github.com/jd-opensource/JoyAI-Echo"><b>πŸ’» Inference Code</b></a> |
34
  <a href="#model-details"><b>🧬 Model</b></a> |
35
  <a href="#usage"><b>πŸš€ Usage</b></a> |
36
  <a href="#results"><b>πŸ“Š Results</b></a> |
 
48
 
49
  ## Model Summary
50
 
51
+ **JoyAI-Echo** is a long-form, multi-shot, audio-video generation framework that breaks the barriers of error accumulation, weak temporal coherence, and prohibitive latency in long video generation. A cross-modal audio-visual memory bank preserves character appearance and voice timbre consistently over **five-minute** videos, while a post-training pipeline combining memory-based reinforcement learning with distribution matching distillation (DMD) delivers a **7.5Γ— inference speedup** without sacrificing quality.
52
 
53
+ JoyAI-Echo decisively outperforms *HappyOyster* (directing mode) on long-form generation and even surpasses the short-video specialist *Wan 2.6* on human-centric tasks.
54
 
55
  This repository hosts the **released checkpoint**. Inference code is released separately β€” see the [Usage](#usage) section.
56
 
 
67
 
68
  ## Highlights
69
 
70
+ - 🎞️ **Minute-level multi-shot stories**: generate a sequence of coherent shots from one prompt JSON.
71
+ - ⚑ **DMD-distilled few-step inference**: ~7.5Γ— faster than the original pipeline.
72
+ - πŸ”Š **Joint audio-video generation**: one pipeline produces synchronized video and audio.
73
+ - 🧠 **Paired cross-modal memory bank**: conditions each new shot on prior visual identity and voice context for story-level consistency.
74
+
75
+ ## Demo Gallery
76
+
77
+ Explore long-form and short-form JoyAI-Echo cases on the [Project Page](https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/). 🍿
78
 
79
  ## Usage
80
 
81
+ Inference is run with the standalone **JoyAI-Echo** inference repository.
82
 
83
  ### 1. Download the checkpoint
84
 
85
  ```bash
86
+ huggingface-cli download jdopensource/JoyAI-Echo \
87
  --local-dir checkpoints
88
  ```
89
 
 
105
  ### 2. Get the inference code
106
 
107
  ```bash
108
+ git clone https://github.com/jd-opensource/JoyAI-Echo.git
109
  cd JoyAI-Echo
110
  ```
111
 
 
126
  | **Style** | The overall visual and emotional aesthetic β€” e.g. realistic motorsport film language, cool daylight, restrained cinematic tension. |
127
  | **Camera Movement** | The shot type and framing or movement β€” e.g. a stable close-up on the face, or a medium shot from the waist up. |
128
  | **Background** | The setting and scene details behind the subject. |
129
+ | **Sound Effects & BGM** | The sounds in the scene and the background music β€” e.g. room tone, wind, footsteps and fabric, with a soft low music bed under the dialogue or no background music. |
 
 
 
130
 
131
  ### 4. Run
132
 
 
151
  | Item | Value |
152
  | --- | ---: |
153
  | 🎬 Long-form coherent story length | **5 min** |
154
+ | ⚑ Generation speedup over the original multi-step pipeline | **7.5Γ—** |
155
  | πŸ“š Benchmark stories | **100** |
156
  | 🎞️ Generated evaluation shots | **3,000** |
157
  | πŸ•’ Frames per shot | **241 @ 25 fps** |
158
 
159
  ### Human Evaluation
160
 
161
+ GSB user study on long- and short-video generation. The numbers denote the percentage of user preferences.
162
 
163
  | Aspect (Long Video) | JoyAI-Echo | Tie | HappyOyster (Directing) |
164
  | --- | ---: | ---: | ---: |
 
173
  | Audio quality | 32.3% | 30.9% | 36.8% |
174
  | Prompt following | 33.8% | 36.8% | 29.4% |
175
 
176
+ ## Links
177
+
178
+ - Project page: [`https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/`](https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/)
179
+ - Inference code: [`https://github.com/jd-opensource/JoyAI-Echo`](https://github.com/jd-opensource/JoyAI-Echo)
180
+ - HuggingFace: [`https://huggingface.co/jdopensource/JoyAI-Echo`](https://huggingface.co/jdopensource/JoyAI-Echo)
181
+
182
  ## Acknowledgements
183
 
184
+ We gratefully acknowledge the open-source projects this work builds upon β€” in particular [LTX2.3](https://huggingface.co/Lightricks/LTX-2.3) for the base video generator and [Gemma](https://huggingface.co/google/gemma-3-12b-it) for the text encoder. Thanks to the broader research community whose contributions made this release possible.
185
 
186
  ## Citation
187
 
188
+ If JoyAI-Echo helps your research or products, please cite:
189
 
190
  ```bibtex
191
  @techreport{echo2026longvideo,
192
+ title = {JoyAI-Echo: Pushing the Frontier of Long Video Generation},
193
  author = {{Echo Team @ Joy Future Academy, JD}},
194
  institution = {Joy Future Academy, JD},
195
  year = {2026},
196
+ month = {May}
 
197
  }
198
  ```
199
 
200
  ## License
201
 
202
+ This project is based on LTX-2 by Lightricks Ltd.
203
+
204
+ Portions of the original LTX-2 codebase have been modified by JD.com for academic and research purposes only.
205
+ This project is not intended for commercial use. For commercial use of LTX-2 or its derivatives, please contact Lightricks Ltd.
206
+
207
+ All original copyright, license, patent, trademark, and attribution notices from LTX-2 are retained.
208
+ This project remains subject to the LTX-2 Community License Agreement.