README.md

---
license: apache-2.0
pipeline_tag: video-to-video
tags:
- video-editing
- compositional-editing
- instruction-guided
- diffusion
- lora
- wan2.1
- qwen3-vl
base_model:
- Wan-AI/Wan2.1-T2V-14B
---

# CoinVE-Edit

<a href="https://arxiv.org/abs/2608.17566"><img src="https://img.shields.io/static/v1?label=Tech%20Report&message=Arxiv&color=red"></a> &ensp;
<a href="https://github.com/coinve200k/CoinVE-200K/tree/main/CoinVE-Edit"><img src="https://img.shields.io/static/v1?label=GitHub&message=CoinVE-Edit&color=blue"></a> &ensp;

---

## ✨ Highlights

- **Compositional Multi-Instruction Editing**: CoinVE-Edit processes 2–5 editing instructions in a single forward pass, applying each edit to its designated region simultaneously.
- **Region-Aware Mask Guidance**: Per-instruction mask injection via a lightweight mask head ensures each edit is confined to the correct spatial region, enabling precise compositional editing.
- **Built on Wan2.1-T2V-14B and Qwen3-VL-8B**: Leveraging a powerful video DiT and MLLM encoder for high-quality region-aware compositional instruction video editing.


## 🌍 Introduction

**CoinVE-Edit** is a video editing model trained on the [CoinVE-200K](https://huggingface.co/datasets/FireCRT/CoinVE-200K) dataset, designed to support **compositional instruction-guided video editing**. Unlike prior models that process a single instruction at a time, CoinVE-Edit can handle **multiple editing instructions simultaneously**, applying each edit to its designated region while maintaining overall video coherence.

**Key capabilities:**
- **Multi-Instruction Editing**: Process 2–5 editing instructions in a single forward pass.
- **Region-Aware Editing**: Per-instruction mask guidance ensures each edit is confined to the correct spatial region.
- **Compositional Operations**: Supports Replace, Add, Remove, and Background Change operations in any combination.
- **High-Quality Output**: Trained on 200K+ high-quality video-edit pairs with rigorous data filtering.

CoinVE-Edit is built on a video DiT architecture (Wan2.1-T2V-14B) with a Qwen3-VL-8B MLLM encoder, leveraging a residual-attention module and a lightweight mask head to inject region-aware guidance for compositional editing.


## 🏗️ Architecture

CoinVE-Edit extends the Wan2.1-T2V-14B video DiT with the following components:

| Component | Description |
|-----------|-------------|
| **Video DiT** | Wan2.1-T2V-14B as the backbone diffusion transformer (LoRA-tuned, rank 128). |
| **MLLM Encoder** | Qwen3-VL-8B-Instruct provides multimodal understanding of instructions and visual context (LoRA-tuned, rank 256). |
| **Mask Head** | A lightweight transformer head that predicts per-instruction spatial masks from MLLM visual tokens. |
| **Residual Attention** | A residual-attention module that injects per-instruction mask guidance into the DiT's attention layers. |

During inference, the MLLM encodes each instruction together with the source video, the mask head predicts a region mask per instruction, and the residual-attention module steers the DiT to apply each edit only within its masked region.


## 📥 Base Models

CoinVE-Edit is trained on top of two base models that must be downloaded before inference:

| Base Model | HuggingFace | Description |
|------------|-------------|-------------|
| Wan2.1-T2V-14B | [Wan-AI/Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B/tree/main) | Video diffusion transformer (DiT) + VAE |
| Qwen3-VL-8B-Instruct | [Qwen/Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) | Multimodal LLM encoder |

This repository contains the CoinVE-Edit checkpoint (`.safetensors`) including the DiT LoRA weights, MLLM LoRA weights, learned image/video query embeddings, connector, VAE condition encoder, and the mask head weights.


## 🔧 Installation

### Environment Requirements

- Python 3.10+
- CUDA 12.8
- PyTorch 2.8+
- FlashAttention-3 (`flash_attn_interface`, v3.0.0b1) for the video DiT

### Full Environment Setup

```bash
# Create conda environment
conda create -n coinve python=3.10 -y
conda activate coinve

# Install PyTorch 2.8.0 with CUDA 12.8
pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128

# Install dependencies
pip install -e .
pip install transformers accelerate wandb

# Install FlashAttention-3 (used by the Wan2.1 video DiT)
# Prebuilt wheel for CUDA 12.9 + PyTorch 2.8.0 (aarch64):
pip install https://github.com/windreamer/flash-attention3-wheels/releases/download/2026.01.26-f6c4937/flash_attn_3-3.0.0b1+20260126.cu129torch280cxx11abitrue.438325-cp39-abi3-linux_aarch64.whl
```


## 🚀 Inference

### Single-Video Editing

Edit one video with one or more instructions on a single GPU.

**Step 1.** Open `infer_coinve_single.sh` and set the base model paths at the top of the script:

```bash
LOCAL_MODEL_PATH="/path/to/base_models"                                              # root dir containing Wan-AI/Wan2.1-T2V-14B/
MLLM_MODEL="/path/to/Qwen3-VL-8B-Instruct"                                           # Qwen3-VL-8B-Instruct checkpoint dir
COMPOSITE_CKPT="/path/to/FireCRT/CoinVE-Edit/coinve_edit_composite_vllm256_dit128.safetensors"  # CoinVE-Edit checkpoint
```

`LOCAL_MODEL_PATH` is the root directory under which the Wan2.1-T2V-14B checkpoint lives as `<LOCAL_MODEL_PATH>/Wan-AI/Wan2.1-T2V-14B/` (download from [Wan-AI/Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B)). `MLLM_MODEL` points directly at the Qwen3-VL-8B-Instruct checkpoint directory (download from [Qwen/Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)). `COMPOSITE_CKPT` points to the CoinVE-Edit checkpoint (download from [FireCRT/CoinVE-Edit](https://huggingface.co/FireCRT/CoinVE-Edit)).

**Step 2.** Run:

```bash
bash infer_coinve_single.sh
```

Or call the Python script directly:

```bash
python infer_coinve_single.py \
  --src_video ./demo_data/source_video.mp4 \
  --prompts "Replace the car with a red truck." "Add a dog on the sidewalk." \
  --composite_checkpoint /path/to/FireCRT/CoinVE-Edit/coinve_edit_composite_vllm256_dit128.safetensors \
  --local_model_path /path/to/base_models \
  --mllm_model /path/to/Qwen3-VL-8B-Instruct \
  --output_dir ./output/single/ \
  --eval_max_pixels 921600 \
  --eval_max_frame 49 \
  --num_inference_steps 50 \
  --seed 0 \
  --show_progress \
  --save_instruction_masks
```

The script saves a side-by-side video `[source | edited]` by default. Pass `--no_side_by_side` to save only the edited video. Use `--save_instruction_masks` to additionally save per-instruction mask overlay videos.

#### Performance

Single-GPU inference cost on **NVIDIA H200** (720p, 49 frames, 50 diffusion steps):

| Metric | Value |
|--------|-------|
| GPU memory | 53.7 GB |
| Wall time | 279 s |

### Batch Inference on CoinVE-Bench

Run multi-GPU inference over the [CoinVE-Bench](https://huggingface.co/datasets/FireCRT/CoinVE-Bench) checklist JSON (361 cases).

**Step 1.** Open `infer_coinve_bench.sh` and set these paths at the top of the script:

```bash
EVAL_PATH="/path/to/CoinVE-Bench/checklist_json/coinve-bench-361-checklist.json"
DATA_ROOT="/path/to/CoinVE-Bench"               # root dir for resolving src_videos/xxx.mp4
COMPOSITE_CKPT="/path/to/FireCRT/CoinVE-Edit/coinve_edit_composite_vllm256_dit128.safetensors"
LOCAL_MODEL_PATH="/path/to/base_models"          # root dir containing Wan-AI/Wan2.1-T2V-14B/
MLLM_MODEL="/path/to/Qwen3-VL-8B-Instruct"       # Qwen3-VL-8B-Instruct checkpoint dir
```

**Step 2.** Run:

```bash
bash infer_coinve_bench.sh
```

This launches an `accelerate` DDP job across 4 GPUs (configurable via `CUDA_VISIBLE_DEVICES` and `NUM_GPUS` in the script).

Outputs are written to `${OUTPUT_BASE_DIR}/videos/`:

- Triptych mp4 per sample (`<idx:04d>_K<n>_<prompt>_<src_stem>.mp4`)
- Standalone edited videos at `tgt_videos/<case_id>.mp4` — already in the `{id}.mp4` naming convention expected by the [CoinVE-Bench](https://huggingface.co/datasets/FireCRT/CoinVE-Bench) evaluator.



## 📊 Results

### CoinVE-Bench

Compositional-Instruction Video Editing Comparisons on [CoinVE-Bench](https://huggingface.co/datasets/FireCRT/CoinVE-Bench). All metrics are scored by Gemini 3.6 Flash.

**Metric groups:** SA / SPA / EP → Edit. Acc. · AN / SC / MN → Phys. Natural. · CP → Seman. Pres.

| Model | SA | SPA | EP | AN | SC | MN | CP |
|---|---:|---:|---:|---:|---:|---:|:---:|
| Seedance 2.0 | 85.34 | 87.71 | 88.08 | **93.19** | **95.84** | 92.87 | **93.91** |
| Kling O3 | 86.91 | 80.93 | 89.06 | 92.55 | 90.30 | 93.91 | 84.51 |
| VACE | 3.98 | 17.15 | 6.50 | 26.69 | 13.82 | 15.21 | 87.83 |
| Ditto | 34.69 | 36.41 | 40.85 | 35.96 | 47.79 | 38.48 | 51.98 |
| VINO | 83.63 | 66.75 | 89.06 | 78.09 | 82.34 | 85.91 | 61.70 |
| OmniWeaving | 59.67 | 55.94 | 61.11 | 54.49 | 66.03 | 65.10 | 75.09 |
| KiWiEdit | 76.50 | 69.92 | 80.28 | 78.37 | 78.50 | 80.76 | 70.31 |
| SAMA | 75.58 | 73.35 | 79.63 | 83.43 | 83.88 | 88.14 | 90.08 |
| **CoinVE-Edit** | **87.97** | **89.45** | **89.60** | 91.85 | 91.17 | **95.30** | 90.83 |



## 📜 Citation

If you find CoinVE-Edit useful for your research, please cite our work:

```bibtex
@article{coinve200k,
  title={CoinVE-200K: A Large-Scale High-Quality Dataset for Compositional Instruction-Guided Video Editing},
  author={Long, Fuchen and Wang, Cong and Gao, Zitao and Zhong, Wenhao and Cheng, Yu and Hou, Xiaolu and Li, Yan and Cao, Xiao and Sun, Xinlong and Chen, Xi and Liu, Yu},
  journal={arXiv preprint arXiv:2608.17566},
  year={2026}
}
```


## ✉️ Contact

For any questions, issues, or collaborations, please feel free to contact longfc.ustc@gmail.com.


## 💖 Acknowledgement

Our model is built upon [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio), [Wan2.1](https://github.com/Wan-Video/Wan2.1), and inspired by [Kiwi-Edit](https://github.com/showlab/Kiwi-Edit). Thanks to the contributors of all these great projects!