update prompts

README.md

@@ -12,256 +12,182 @@ tags:
   - dmd
 library_name: ltx-video
 ---
+
 <p align="center">
-  <img src="assets/image.png" alt="JoyAI-Echo generated video gallery" width="100%">
+  <img src="assets/image.png" alt="Echo-LongVideo generated video gallery" width="100%">
 </p>
 
 <div align="center">
 
-<h1>JoyAI-Echo</h1>
+<h1>Echo-LongVideo</h1>
 
 <p><strong>🎬 Pushing the Frontier of Long Video Generation</strong></p>
 
-<p>Standalone, inference-only release for <strong>minute-level multi-shot audio-video generation</strong> with a distilled DMD generator, paired cross-modal memory, and story-level consistency.</p>
-
-<p><strong>For academic research and non-commercial use only.</strong></p>
+<p>Official model weights for <strong>minute-level multi-shot audio-video generation</strong> with a distilled DMD generator, paired cross-modal memory, and story-level consistency.</p>
 
 <p>
-  <a href="https://github.com/jd-opensource/JoyAI-Echo/blob/main/joyai-echo%20tech%20report.pdf"><b>📄 Paper</b></a> |
-  <a href="https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/"><b>🌐 Project Page</b></a> |
-  <a href="#quickstart"><b>🚀 Quickstart</b></a> |
+  <a href="LongVideo_Tech_Report.pdf"><b>📄 Paper</b></a> |
+  <a href="https://github.com/Echo-Team-Joy-Future-Academy-JD/JoyAI-Echo"><b>💻 Inference Code</b></a> |
+  <a href="#model-details"><b>🧬 Model</b></a> |
+  <a href="#usage"><b>🚀 Usage</b></a> |
   <a href="#results"><b>📊 Results</b></a> |
   <a href="#citation"><b>📝 Citation</b></a>
 </p>
 
 <p>
-  <img src="https://img.shields.io/badge/Python-3.11-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python 3.11">
-  <img src="https://img.shields.io/badge/PyTorch-2.8-EE4C2C?style=flat-square&logo=pytorch&logoColor=white" alt="PyTorch 2.8">
-  <img src="https://img.shields.io/badge/CUDA-12.8-76B900?style=flat-square&logo=nvidia&logoColor=white" alt="CUDA 12.8">
-  <img src="https://img.shields.io/badge/Release-Inference--Only-black?style=flat-square" alt="Inference">
+  <img src="https://img.shields.io/badge/Task-Text--to--Video-blue?style=flat-square" alt="Text-to-Video">
+  <img src="https://img.shields.io/badge/Modality-Audio%2BVideo-purple?style=flat-square" alt="Audio + Video">
   <img src="https://img.shields.io/badge/Long%20Video-5%20min-d61f2c?style=flat-square" alt="5 minute long video">
+  <img src="https://img.shields.io/badge/Release-Model%20Weights-black?style=flat-square" alt="Model Weights">
 </p>
 
 </div>
 
-## Abstract
-
-Long video generation still suffers from error accumulation, weak temporal coherence, and prohibitive latency, limiting its applicability to interactive scenarios. We present **JoyAI-Echo**, a framework that breaks these barriers through four key advances.
-Central to its performance, a cross-modal audio-visual memory bank preserves character appearance and voice timbre consistently over five-minute videos, while a post-training pipeline combines memory-based reinforcement learning with distribution matching distillation for a **7.5× speedup** to substantially boost visual quality and alignment.
-Empowered by these two components, **JoyAI-Echo** decisively outperforms *HappyOyster* (directing mode) on long-form generation and even surpasses the short-video specialist *Wan 2.6* on human-centric tasks.
-Beyond raw generation quality, an interactive agent enables real-time user editing through conversational instructions, and a lightweight super-resolution module maintains high definition under streaming latency, further elevating the overall experience and delivering instantly editable, conversation-speed video creation.
-For the first time, **JoyAI-Echo** simultaneously achieves long-range cross-modal consistency, real-time inference for minute-long video, conversational interactivity, and high-resolution output — without compromise, inaugurating a new era of interactive video generation.
-Codes and weights will be open-sourced.
-
-## Highlights
-
-- 🎞️ **Minute-level multi-shot stories**: generate a sequence of coherent shots from one prompt JSON.
-- ⚡ **DMD-distilled few-step inference**: ~7.5x faster than the original pipeline.
-- 🔊 **Joint audio-video generation**: one pipeline produces synchronized video and audio.
-- 🧠 **Paired cross-modal memory bank**: conditions each new shot on prior visual identity and voice context for story-level consistency.
-
-## Current Release Scope
-
-JoyAI-Echo currently focuses on **text-to-video (T2V)** and **multi-shot long-video generation with paired audio-video memory**. The memory used in our official pipeline is built from generated T2V shots.
-
-Please note that **image-to-video (I2V)** is **not supported in the current release**.
-
-We are actively working on I2V support and plan to release it in a future version.
-
-## Demo Gallery
-
-Explore long-form and short-form JoyAI-Echo cases on the [Project Page](https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/). 🍿
-
-## Results
-
-### Reported Scale
-
-| Item | Value |
-| --- | ---: |
-| 🎬 Long-form coherent story length | **5 min** |
-| ⚡ Generation speedup over the original multi-step pipeline | **7.5x** |
-| 📚 Benchmark stories | **100** |
-| 🎞️ Generated evaluation shots | **3,000** |
-| 🕒 Frames per shot | **241 @ 25 fps** |
-
-### Human Evaluation
+## Model Summary
 
-GSB user study on long- and short-video generation. The numbers denote the percentage of user preferences.
+**Echo-LongVideo** (a.k.a. **JoyAI-Echo**) is a long-form, multi-shot, audio-video generation model. A cross-modal audio-visual memory bank preserves character appearance and voice timbre consistently across up to **five-minute** videos, and a post-training pipeline combining memory-based reinforcement learning with distribution matching distillation (DMD) delivers a **7.5× inference speedup** without sacrificing quality.
 
-| Aspect<br>(Long Video) | JoyAI-Echo | Tie | HappyOyster<br> (Directing) | 
-| --- | ---: | ---: | ---: | 
-| Visual aesthetics | **63.6%** | 8.8% | 27.6% | 
-| Audio quality | **81.7%** | 6.5% | 11.8% |
-| Prompt following | **80.6%** | 13.5% | 5.9% | 
-| IP consistency | **59.4%** | 12.9% | 27.7% |
+In human evaluation, Echo-LongVideo decisively outperforms *HappyOyster* (directing mode) on long-form generation and surpasses the short-video specialist *Wan 2.6* on human-centric tasks.
 
-| Aspect<br>(Short Video) | JoyAI-Echo | Tie | Wan 2.6 |
-| ---  | ---: | ---: | ---: |
-| Visual aesthetics | **58.8%** | 14.7% | 26.5% |
-| Audio quality  | 32.3% | 30.9% | 36.8% |
-| Prompt following | 33.8% | 36.8% | 29.4% |
+This repository hosts the **released checkpoint**. Inference code is released separately — see the [Usage](#usage) section.
 
+## Model Details
 
-## Quickstart
+- **Developed by:** Echo Team @ Joy Future Academy, JD
+- **Model type:** Text-to-(Audio+Video) diffusion transformer, DMD 8-step
+- **Modality:** Text → synchronized video + audio
+- **Backbone:** Built on top of [LTX-Video](https://github.com/Lightricks/LTX-Video)
+- **Text encoder:** [`google/gemma-3-12b-it`](https://huggingface.co/google/gemma-3-12b-it) (downloaded separately)
+- **Resolution / length (by default):** 1280 × 736, 241 frames @ 25 fps per shot
+- **Max story length:** up to 5 minutes (multi-shot)
+- **License:** LTX-2 Community License Agreement
 
-### 1. Clone
-
-Get the Repo at first! 
-
-```bash
-
-git clone https://github.com/jd-opensource/JoyAI-Echo.git
-cd JoyAI-Echo
-```
-
-### 2. Create the environment
+## Highlights
 
-The reference environment is **Python 3.11 + PyTorch 2.8 + CUDA 12.8**.
+- 🎞️ **Minute-level multi-shot stories** from a single prompt JSON.
+- ⚡ **DMD-distilled few-step inference**, ~7.5× faster than the original pipeline.
+- 🔊 **Joint audio-video generation** in a single pipeline.
+- 🧠 **Paired cross-modal memory bank** for story-level identity and voice consistency.
 
-With conda:
+## Usage
 
-```bash
-conda env create -f environment.yml
-conda activate echo-long
-```
+Inference is run with the standalone **Echo-LongVideo** inference repository.
 
-With `uv`:
+### 1. Download the checkpoint
 
 ```bash
-uv venv --python 3.11 .venv
-source .venv/bin/activate
-uv pip install --extra-index-url https://download.pytorch.org/whl/cu128 -r requirements.txt
+huggingface-cli download <org>/Echo-LongVideo \
+  --local-dir checkpoints
 ```
 
-[`ffmpeg`](https://ffmpeg.org/download.html) must be available on `PATH` for shot concatenation. The conda recipe includes it. If you use `uv`, install it with your system package manager:
+Also download the Gemma text encoder:
 
 ```bash
-sudo apt install ffmpeg
-# macOS:
-brew install ffmpeg
+huggingface-cli download google/gemma-3-12b-it \
+  --local-dir checkpoints/gemma-3-12b
 ```
 
-### 3. Download checkpoint
-
-Download the JoyAI-Echo release checkpoint and Gemma text encoder:
-
-| File | Description | Size | Link |
-| --- | --- | --- | --- |
-| `echo-longvideo-release.safetensors` | Full model (transformer + VAE + vocoder) | ~46 GB |[`JoyAI-Echo`](https://huggingface.co/jdopensource/JoyAI-Echo)  |
-| `gemma-3-12b/` | Instruction-tuned model (text encoder) | ~24 GB | [`gemma-3-12b-it`](https://huggingface.co/google/gemma-3-12b-it) |
-
-Place them under `checkpoints/`:
+Expected layout:
 
 ```text
 checkpoints/
-+-- echo-longvideo-release.safetensors
-`-- gemma-3-12b/
+├── echo-longvideo-release.safetensors
+└── gemma-3-12b/
 ```
 
-### 4. Write a story prompt
-
-Create a JSON file under `prompts/`. 
-
-Each string is one complete shot description. A single prompt creates a single shot. Multiple prompts create a multi-shot story conditioned through the paired audio-video memory bank.
-
-### 5. Run inference
+### 2. Get the inference code
 
 ```bash
-python inference.py
+git clone https://github.com/Echo-Team-Joy-Future-Academy-JD/JoyAI-Echo.git
+cd JoyAI-Echo
 ```
 
-This loads the model once and processes all prompt files under `prompts/`.
+Environment: **Python 3.11 + PyTorch 2.8 + CUDA 12.8** (see the inference repo's `environment.yml` / `requirements.txt`).
 
-> 💡 **Note**: The inference pipeline is optimized to run on lower-VRAM
-> GPUs. Peak GPU usage is around **46–50 GB**, at the cost of slightly
-> longer per-shot inference time.
+### 3. Write a story prompt
 
-Outputs are written to:
+**Enhance your prompt first.** We provide prompt enhancers — system prompts that expand a short story or idea into well-formed shot prompts: **`prompts/long_story_writer_system_prompt.md`** for long, multi-shot video, and **`prompts/short_story_writer_system_prompt.md`** for single-shot short video. We **strongly recommend** running your input through the matching enhancer before inference; un-enhanced prompts tend to produce noticeably weaker results.
 
-```text
-inference_result/outputs/<prompt-name>/inference_<timestamp>/
-```
-
-## Configuration
+Create a JSON file under `prompts/`. Each file is a single object with a `prompts` list, where **every string is one complete shot**. A single string produces one shot; multiple strings produce a multi-shot story, with each new shot conditioned on the previous ones through the paired audio-video memory bank.
 
-All inference parameters are managed in `configs/inference.yaml`. The file is organized into sections:
+Inside each string, write these parts in order:
 
-| Section | Contents |
+| Part | What to describe |
 | --- | --- |
-| `paths` | Checkpoint path, prompts directory, output root |
-| `video` | Resolution, frame count, FPS, seed |
-| `denoising` | Step list and sigma schedule |
-| `memory` | Memory bank size, save mode, LoRA settings |
-| `audio_memory` | Audio window, mel-spectrogram params |
-| `inference` | Device, dtype, grad scale |
-
-### Override via CLI
+| **Roles & Subjects** | Describe the appearance of all visible people, including age, build, hair, face, wardrobe, and speaking voice timbre when applicable. |
+| **Action & Dialogue** | What the subject does and speaks. |
+| **Style** | The overall visual and emotional aesthetic — e.g. realistic motorsport film language, cool daylight, restrained cinematic tension. |
+| **Camera Movement** | The shot type and framing or movement — e.g. a stable close-up on the face, or a medium shot from the waist up. |
+| **Background** | The setting and scene details behind the subject. |
+| **Sound Effects & BGM** | The sounds in the scene and the background music — e.g. room tone, wind, footsteps and fabric, with a soft low music bed under the dialogue or nobackground music |
 
-Any YAML parameter can be overridden from the command line:
+A more convenient prompt-writing workflow will be released as a **director agent** for everyone to use.
 
-```bash
-python inference.py --seed 42 --num-frames 121 --video-height 480 --video-width 832
-```
 
-Use a custom config file:
+### 4. Run
 
 ```bash
-python inference.py --config configs/my_experiment.yaml
+python inference.py
 ```
 
-The Python entrypoint exposes the full configuration surface:
-
-```bash
-python inference.py --help
-```
+Outputs land in `inference_result/outputs/<prompt-name>/inference_<timestamp>/`.
 
 ## Hardware
 
-Peak GPU usage is around **46–50 GB** for the default **25 fps x 241 frames x 1280 x 736** setting, so a single H100/A100-class (80 GB) or 48 GB GPU is sufficient.
-
-For smaller GPUs, reduce resolution/frames:
+Peak GPU memory is **~46–50 GB** at the default 1280 × 736 × 241 frame setting — a single H100/A100 (80 GB) or 48 GB GPU is sufficient. For smaller GPUs, lower resolution or frame count:
 
 ```bash
 python inference.py --num-frames 121 --video-height 480 --video-width 832
 ```
 
-## TODO List
+## Results
 
-- [x] Release inference code
-- [x] Release model checkpoints
-- [x] Add prompt examples
-- [ ] Release Director Agent 
+### Reported Scale
 
-## Links
+| Item | Value |
+| --- | ---: |
+| 🎬 Long-form coherent story length | **5 min** |
+| ⚡ Speedup over the original multi-step pipeline | **7.5×** |
+| 📚 Benchmark stories | **100** |
+| 🎞️ Generated evaluation shots | **3,000** |
+| 🕒 Frames per shot | **241 @ 25 fps** |
 
-- Project page: [`https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/`](https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/)
-- Repository: [`https://github.com/jd-opensource/JoyAI-Echo`](https://github.com/jd-opensource/JoyAI-Echo)
-- huggingface: [`https://huggingface.co/jdopensource/JoyAI-Echo`](https://huggingface.co/jdopensource/JoyAI-Echo)
+### Human Evaluation
+
+GSB user study. Values are the percentage of user preferences.
+
+| Aspect (Long Video) | JoyAI-Echo | Tie | HappyOyster (Directing) |
+| --- | ---: | ---: | ---: |
+| Visual aesthetics | **63.6%** | 8.8% | 27.6% |
+| Audio quality | **81.7%** | 6.5% | 11.8% |
+| Prompt following | **80.6%** | 13.5% | 5.9% |
+| IP consistency | **59.4%** | 12.9% | 27.7% |
+
+| Aspect (Short Video) | JoyAI-Echo | Tie | Wan 2.6 |
+| --- | ---: | ---: | ---: |
+| Visual aesthetics | **58.8%** | 14.7% | 26.5% |
+| Audio quality | 32.3% | 30.9% | 36.8% |
+| Prompt following | 33.8% | 36.8% | 29.4% |
 
 ## Acknowledgements
 
-We gratefully acknowledge the open-source projects this work builds upon — in particular [LTX2.3](https://huggingface.co/Lightricks/LTX-2.3) for the base video generator and [Gemma](https://huggingface.co/google/gemma-3-12b-it) for the text encoder. Thanks to the broader research community whose contributions made this release possible.
+We gratefully acknowledge [LTX-Video](https://github.com/Lightricks/LTX-Video) for the base video generator and [Gemma](https://huggingface.co/google/gemma-3-12b-it) for the text encoder, along with the broader open-source community.
 
 ## Citation
 
-If JoyAI-Echo helps your research or products, please cite:
+If Echo-LongVideo helps your research or products, please cite:
 
 ```bibtex
 @techreport{echo2026longvideo,
-  title        = {JoyAI-Echo: Pushing the Frontier of Long Video Generation},
+  title        = {Echo-LongVideo: Pushing the Frontier of Long Video Generation},
   author       = {{Echo Team @ Joy Future Academy, JD}},
   institution  = {Joy Future Academy, JD},
   year         = {2026},
-  month        = {May}
+  month        = {June},
+  url          = {https://github.com/Echo-Team-Joy-Future-Academy-JD/JoyAI-Echo}
 }
 ```
 
 ## License
 
-This project is based on LTX-2 by Lightricks Ltd.
-
-Portions of the original LTX-2 codebase have been modified by JD.com for academic and research purposes only. 
-This project is not intended for commercial use. For commercial use of LTX-2 or its derivatives, please contact Lightricks Ltd.
-
-All original copyright, license, patent, trademark, and attribution notices from LTX-2 are retained. 
-This project remains subject to the LTX-2 Community License Agreement.
+Released under the [LTX-2 Community License Agreement](https://github.com/Lightricks/LTX-2/blob/main/LICENSE). By downloading or using these weights, you agree to its terms. The bundled Gemma text encoder is governed by Google's separate Gemma license.