license: apache-2.0
pipeline_tag: video-to-video
tags:
- video-editing
- compositional-editing
- instruction-guided
- diffusion
- lora
- wan2.1
- qwen3-vl
base_model:
- Wan-AI/Wan2.1-T2V-14B
CoinVE-Edit
✨ Highlights
- Compositional Multi-Instruction Editing: CoinVE-Edit processes 2–5 editing instructions in a single forward pass, applying each edit to its designated region simultaneously.
- Region-Aware Mask Guidance: Per-instruction mask injection via a lightweight mask head ensures each edit is confined to the correct spatial region, enabling precise compositional editing.
- Built on Wan2.1-T2V-14B and Qwen3-VL-8B: Leveraging a powerful video DiT and MLLM encoder for high-quality region-aware compositional instruction video editing.
🌍 Introduction
CoinVE-Edit is a video editing model trained on the CoinVE-200K dataset, designed to support compositional instruction-guided video editing. Unlike prior models that process a single instruction at a time, CoinVE-Edit can handle multiple editing instructions simultaneously, applying each edit to its designated region while maintaining overall video coherence.
Key capabilities:
- Multi-Instruction Editing: Process 2–5 editing instructions in a single forward pass.
- Region-Aware Editing: Per-instruction mask guidance ensures each edit is confined to the correct spatial region.
- Compositional Operations: Supports Replace, Add, Remove, and Background Change operations in any combination.
- High-Quality Output: Trained on 200K+ high-quality video-edit pairs with rigorous data filtering.
CoinVE-Edit is built on a video DiT architecture (Wan2.1-T2V-14B) with a Qwen3-VL-8B MLLM encoder, leveraging a residual-attention module and a lightweight mask head to inject region-aware guidance for compositional editing.
🏗️ Architecture
CoinVE-Edit extends the Wan2.1-T2V-14B video DiT with the following components:
| Component | Description |
|---|---|
| Video DiT | Wan2.1-T2V-14B as the backbone diffusion transformer (LoRA-tuned, rank 128). |
| MLLM Encoder | Qwen3-VL-8B-Instruct provides multimodal understanding of instructions and visual context (LoRA-tuned, rank 256). |
| Mask Head | A lightweight transformer head that predicts per-instruction spatial masks from MLLM visual tokens. |
| Residual Attention | A residual-attention module that injects per-instruction mask guidance into the DiT's attention layers. |
During inference, the MLLM encodes each instruction together with the source video, the mask head predicts a region mask per instruction, and the residual-attention module steers the DiT to apply each edit only within its masked region.
📥 Base Models
CoinVE-Edit is trained on top of two base models that must be downloaded before inference:
| Base Model | HuggingFace | Description |
|---|---|---|
| Wan2.1-T2V-14B | Wan-AI/Wan2.1-T2V-14B | Video diffusion transformer (DiT) + VAE |
| Qwen3-VL-8B-Instruct | Qwen/Qwen3-VL-8B-Instruct | Multimodal LLM encoder |
This repository contains the CoinVE-Edit checkpoint (.safetensors) including the DiT LoRA weights, MLLM LoRA weights, learned image/video query embeddings, connector, VAE condition encoder, and the mask head weights.
🔧 Installation
Environment Requirements
- Python 3.10+
- CUDA 12.8
- PyTorch 2.8+
- FlashAttention-3 (
flash_attn_interface, v3.0.0b1) for the video DiT
Full Environment Setup
# Create conda environment
conda create -n coinve python=3.10 -y
conda activate coinve
# Install PyTorch 2.8.0 with CUDA 12.8
pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128
# Install dependencies
pip install -e .
pip install transformers accelerate wandb
# Install FlashAttention-3 (used by the Wan2.1 video DiT)
# Prebuilt wheel for CUDA 12.9 + PyTorch 2.8.0 (aarch64):
pip install https://github.com/windreamer/flash-attention3-wheels/releases/download/2026.01.26-f6c4937/flash_attn_3-3.0.0b1+20260126.cu129torch280cxx11abitrue.438325-cp39-abi3-linux_aarch64.whl
🚀 Inference
Single-Video Editing
Edit one video with one or more instructions on a single GPU.
Step 1. Open infer_coinve_single.sh and set the base model paths at the top of the script:
LOCAL_MODEL_PATH="/path/to/base_models" # root dir containing Wan-AI/Wan2.1-T2V-14B/
MLLM_MODEL="/path/to/Qwen3-VL-8B-Instruct" # Qwen3-VL-8B-Instruct checkpoint dir
COMPOSITE_CKPT="/path/to/FireCRT/CoinVE-Edit/coinve_edit_composite_vllm256_dit128.safetensors" # CoinVE-Edit checkpoint
LOCAL_MODEL_PATH is the root directory under which the Wan2.1-T2V-14B checkpoint lives as <LOCAL_MODEL_PATH>/Wan-AI/Wan2.1-T2V-14B/ (download from Wan-AI/Wan2.1-T2V-14B). MLLM_MODEL points directly at the Qwen3-VL-8B-Instruct checkpoint directory (download from Qwen/Qwen3-VL-8B-Instruct). COMPOSITE_CKPT points to the CoinVE-Edit checkpoint (download from FireCRT/CoinVE-Edit).
Step 2. Run:
bash infer_coinve_single.sh
Or call the Python script directly:
python infer_coinve_single.py \
--src_video ./demo_data/source_video.mp4 \
--prompts "Replace the car with a red truck." "Add a dog on the sidewalk." \
--composite_checkpoint /path/to/FireCRT/CoinVE-Edit/coinve_edit_composite_vllm256_dit128.safetensors \
--local_model_path /path/to/base_models \
--mllm_model /path/to/Qwen3-VL-8B-Instruct \
--output_dir ./output/single/ \
--eval_max_pixels 921600 \
--eval_max_frame 49 \
--num_inference_steps 50 \
--seed 0 \
--show_progress \
--save_instruction_masks
The script saves a side-by-side video [source | edited] by default. Pass --no_side_by_side to save only the edited video. Use --save_instruction_masks to additionally save per-instruction mask overlay videos.
Performance
Single-GPU inference cost on NVIDIA H200 (720p, 49 frames, 50 diffusion steps):
| Metric | Value |
|---|---|
| GPU memory | 53.7 GB |
| Wall time | 279 s |
Batch Inference on CoinVE-Bench
Run multi-GPU inference over the CoinVE-Bench checklist JSON (361 cases).
Step 1. Open infer_coinve_bench.sh and set these paths at the top of the script:
EVAL_PATH="/path/to/CoinVE-Bench/checklist_json/coinve-bench-361-checklist.json"
DATA_ROOT="/path/to/CoinVE-Bench" # root dir for resolving src_videos/xxx.mp4
COMPOSITE_CKPT="/path/to/FireCRT/CoinVE-Edit/coinve_edit_composite_vllm256_dit128.safetensors"
LOCAL_MODEL_PATH="/path/to/base_models" # root dir containing Wan-AI/Wan2.1-T2V-14B/
MLLM_MODEL="/path/to/Qwen3-VL-8B-Instruct" # Qwen3-VL-8B-Instruct checkpoint dir
Step 2. Run:
bash infer_coinve_bench.sh
This launches an accelerate DDP job across 4 GPUs (configurable via CUDA_VISIBLE_DEVICES and NUM_GPUS in the script).
Outputs are written to ${OUTPUT_BASE_DIR}/videos/:
- Triptych mp4 per sample (
<idx:04d>_K<n>_<prompt>_<src_stem>.mp4) - Standalone edited videos at
tgt_videos/<case_id>.mp4— already in the{id}.mp4naming convention expected by the CoinVE-Bench evaluator.
📊 Results
CoinVE-Bench
Compositional-Instruction Video Editing Comparisons on CoinVE-Bench. All metrics are scored by Gemini 3.6 Flash.
Metric groups: SA / SPA / EP → Edit. Acc. · AN / SC / MN → Phys. Natural. · CP → Seman. Pres.
| Model | SA | SPA | EP | AN | SC | MN | CP |
|---|---|---|---|---|---|---|---|
| Seedance 2.0 | 85.34 | 87.71 | 88.08 | 93.19 | 95.84 | 92.87 | 93.91 |
| Kling O3 | 86.91 | 80.93 | 89.06 | 92.55 | 90.30 | 93.91 | 84.51 |
| VACE | 3.98 | 17.15 | 6.50 | 26.69 | 13.82 | 15.21 | 87.83 |
| Ditto | 34.69 | 36.41 | 40.85 | 35.96 | 47.79 | 38.48 | 51.98 |
| VINO | 83.63 | 66.75 | 89.06 | 78.09 | 82.34 | 85.91 | 61.70 |
| OmniWeaving | 59.67 | 55.94 | 61.11 | 54.49 | 66.03 | 65.10 | 75.09 |
| KiWiEdit | 76.50 | 69.92 | 80.28 | 78.37 | 78.50 | 80.76 | 70.31 |
| SAMA | 75.58 | 73.35 | 79.63 | 83.43 | 83.88 | 88.14 | 90.08 |
| CoinVE-Edit | 87.97 | 89.45 | 89.60 | 91.85 | 91.17 | 95.30 | 90.83 |
📜 Citation
If you find CoinVE-Edit useful for your research, please cite our work:
@article{coinve200k,
title={CoinVE-200K: A Large-Scale High-Quality Dataset for Compositional Instruction-Guided Video Editing},
author={Long, Fuchen and Wang, Cong and Gao, Zitao and Zhong, Wenhao and Cheng, Yu and Hou, Xiaolu and Li, Yan and Cao, Xiao and Sun, Xinlong and Chen, Xi and Liu, Yu},
journal={arXiv preprint arXiv:2608.17566},
year={2026}
}
✉️ Contact
For any questions, issues, or collaborations, please feel free to contact longfc.ustc@gmail.com.
💖 Acknowledgement
Our model is built upon DiffSynth-Studio, Wan2.1, and inspired by Kiwi-Edit. Thanks to the contributors of all these great projects!