README.md

---
license: apache-2.0
language:
- en
- zh
pipeline_tag: video-to-video
---
<h1 align="center">JoyAI-Video-Edit<br><sub><sup>Real-Time Open-Ended Video Editing with Autoregressive Diffusion</sup></sub></h1>

<div align="center">

[![Arxiv](https://img.shields.io/badge/Report-PDF-red)](https://arxiv.org/abs/2608.03974)
[![Project](https://img.shields.io/badge/Project-JoyAI--Video--Edit-333399)](https://github.com/jd-opensource/JoyAI-Video-Edit)
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Checkpoint-JoyAI--Video--Edit-yellow)](https://huggingface.co/jdopensource/JoyAI-Video-Edit)&#160;
[![ModelScope](https://img.shields.io/badge/%F0%9F%A4%96%20ModelScope-JoyAI--Video--Edit-624aff)](https://modelscope.cn/models/jd-opensource/JoyAI-Video-Edit)&#160;
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)


</div>

## 🐶 JoyAI-Video-Edit

JoyAI-Video-Edit is a real-time, instruction-guided video editing system for open-ended video streams. Given a live camera stream or uploaded video and a natural-language edit instruction, it edits frames causally as they arrive, without waiting for the full video, requiring a predefined video length, or revisiting future frames. In our deployment benchmark, the full end-to-end pipeline reaches 30.19 FPS at 720x1280, pushing video editing from offline batch processing toward interactive streaming generation.

The system combines an MLLM-based condition encoder, a causal video VAE, and a 16B-parameter multimodal diffusion transformer. It is trained and deployed as an autoregressive diffusion editor, then accelerated with aligned autoregressive distribution matching distillation, long-horizon optimization, bounded KV-state inference, and deployment-oriented scheduling to sustain high-throughput 720p editing while reducing train-inference mismatch and accumulated temporal drift.

## 💎 Highlights

- **Real-time open-ended editing.** Edits live or uploaded videos as frames arrive, without requiring the full sequence upfront.
- **Diverse instruction control.** Supports subject edits, local edits, background changes, style transfer, motion changes, and reference-guided editing.
- **Autoregressive diffusion design.** Combines an MLLM condition encoder, causal video VAE, and MMDiT backbone for streaming video editing.
- **High-throughput 720p deployment.** Reaches 30.19 FPS end-to-end throughput at 720x1280 with bounded KV-state inference and stable per-chunk compute.

## 🚀 Quick Start

### 1. Install

```bash
conda create -n joyai-video-edit python=3.10 -y
conda activate joyai-video-edit
python -m pip install -r requirements.txt
```

### 2. Prepare Checkpoints

Download the released weights from the Hugging Face link above. MiMo-VL and the ONNX detector files are external runtime dependencies; see [`DEPLOYMENT.md`](DEPLOYMENT.md) for deployment details.

### 3. Launch

```bash
cd deploy
bash run_server.sh
```

Then open:

```text
http://localhost:8080
```

For remote machines, bind the server to `0.0.0.0` and open the selected port, or use SSH port forwarding.


## 📦 Model Download

Download the released JoyAI-Video-Edit weights from [Hugging Face](https://huggingface.co/jdopensource/JoyAI-Video-Edit), then place them under:

```text
deploy/deps/checkpoints/JoyAI-Video-Edit/
|-- dit/
|   `-- joyai_video_edit_dit_0804.pth
`-- vae/
    |-- config.json
    `-- diffusion_pytorch_model.safetensors
```

<a id="quick-start"></a>


## License Agreement

JoyAI-Video-Edit is licensed under Apache 2.0.