Update README

95ebdfd verified about 2 months ago

22 kB

language:
  - en
tags:
  - video-generation
  - video-editing
  - multi-modal
  - diffusion
base_model:
  - Qwen/Qwen3-VL-8B-Instruct
  - Wan-AI/Wan2.2-TI2V-5B

LoomVideo: Unifying Multimodal Inputs into
Video Generation and Editing

Peking University · Alibaba Group

🔥 News

[2026-06-02] We release the codebase and model weights of LoomVideo!
[2026-06-02] We release the project page of LoomVideo!

📌 TL;DR

The Problem: Existing unified video generation & editing models are massive (13B+) and rely on token concatenation for source conditioning — doubling sequence length and quadrupling attention cost.

The Method: We present LoomVideo, a compact 5B-parameter unified architecture built on MLLM + DiT that introduces three key designs:

Deepstack Injection — extracts features from every MLLM layer and injects them into corresponding DiT layers via cross-attention, enabling rich multi-granular semantic guidance.
Scale-and-Add Conditioning — a zero-overhead approach that scales the clean source latent by the current timestep and directly adds it to the noised target, completely bypassing token concatenation.
Negative Temporal RoPE — assigns negative temporal indices to reference images, seamlessly integrating multi-image conditions without architectural modification.

The Result: Our 5B model achieves state-of-the-art or highly competitive performance across comprehensive benchmarks, with at least 5.41× inference speedup over models of similar capabilities — demonstrating that efficiency and quality can coexist.

🎯 Supported Tasks

LoomVideo supports four unified video generation and editing tasks within a single model:

Task	Input	Output	Description
Text-to-Video	Text 📝	Video 🎬	Generate a video from a text prompt
Instruction Editing	Video 🎬 + Text 📝	Video 🎬	Edit a video following text instructions
Instruction-Image Editing	Video 🎬 + Image 🖼 + Text 📝	Video 🎬	Edit a video with a reference image as guidance
Multi-Image-to-Video	Images 🖼 + Text 📝	Video 🎬	Compose multiple reference images into a coherent video

🎬 Text-to-Video

Prompt: Snow rocky mountains peaks canyon. Snow blanketed rocky mountains surround and shadow deep canyons. The canyons twist and bend through the high elevated mountain peaks.

Prompt: Vampire makeup face of beautiful girl, red contact lenses.

✂️ Instruction Editing

→

Prompt: Apply the Impressionist aesthetic to this video, ensuring seamless temporal consistency across all frames. The result should emulate the fluid brushstroke techniques and atmospheric focus of 19th-century Impressionist art, with each frame retaining the original motion, character actions, and camera movements.

→

Prompt: Replace the tree with a golden-leaved tree that shimmers softly, ensuring it maintains the same position and pose within the video scene.

🖼️ Instruction-Image Editing

→

Prompt: Replace the green t-shirt of the man with the suit in the image.

→

Prompt: Replace the background with a Chinese ink painting, featuring a large golden mountain peak rising above swirling clouds, ensuring it appears in the same position and pose within the video scene.

🎞️ Multi-Image-to-Video

→

Prompt: The girl (@Image 2), wearing the denim jacket (@Image 3), black inner top, and black shorts, wearing sunglasses and carrying the handbag, walks down the street (@Image 1). Then, the girl (@Image 2) stops walking and turns her head to look to one side, followed by the girl (@Image 2) crossing her arms over her chest and striking a confident pose.

→

Prompt: The man wearing a Polo shirt (@Image 2), black casual pants, white sneakers, sunglasses, and a watch, striding forward on the lawn (@Image 1) with one hand in his pocket.

🔧 Preparation

🎬 Inference

LoomVideo provides a unified inference script that supports four generation tasks through a single entry point. Each task is selected via the --task flag.

1. Text-to-Video / Text-to-Image (`t2v`)

Generate a video from a text description. Default resolution is 480×832 at 81 frames. When --num_frames is set to 1, the pipeline automatically switches to image generation mode and saves the output as a .jpg file.

Required: --prompt

NUM_GPUS=1

accelerate launch --num_processes=${NUM_GPUS} \
    scripts/inference/generate.py \
    --config_path configs/inference/generation.yaml \
    --ckpt_path checkpoints/LoomVideo \
    --task t2v \
    --prompt "Your prompt here" \
    --height 480 \
    --width 832 \
    --num_frames 97 \
    --num_inference_steps 50 \
    --seed 0 \
    --output_path outputs/t2v.mp4

2. Instruction Editing (`edit`)

Edit an existing image or video based on a text instruction. The source can be either an image file (.jpg, .png, etc.) or a video file (.mp4). Resolution and frame count are automatically inferred from the source when not specified.

Required: --prompt --source_video_path

NUM_GPUS=1

accelerate launch --num_processes=${NUM_GPUS} \
    scripts/inference/generate.py \
    --config_path configs/inference/generation.yaml \
    --ckpt_path checkpoints/LoomVideo \
    --task edit \
    --prompt "Your editing instruction here" \
    --source_video_path /path/to/source_video.mp4 \
    --num_inference_steps 50 \
    --seed 0 \
    --output_path outputs/edit.mp4

3. Instruction-Image Editing (`ref_edit`)

Edit a source video with guidance from one or more reference images along with a text instruction.

Required: --prompt --source_video_path --ref_image_paths

NUM_GPUS=1

accelerate launch --num_processes=${NUM_GPUS} \
    scripts/inference/generate.py \
    --config_path configs/inference/generation.yaml \
    --ckpt_path checkpoints/LoomVideo \
    --task ref_edit \
    --prompt "Your editing instruction" \
    --source_video_path /path/to/source_video.mp4 \
    --ref_image_paths /path/to/ref1.jpg /path/to/ref2.jpg \
    --num_inference_steps 50 \
    --seed 0 \
    --output_path outputs/ref_edit.mp4

4. Multi-Image-to-Video (`mi2v`)

Generate a video conditioned on multiple reference images and a text prompt. We recommend using @Image N in the prompt to reference specific input images.

Required: --prompt --ref_image_paths

NUM_GPUS=1

accelerate launch --num_processes=${NUM_GPUS} \
    scripts/inference/generate.py \
    --config_path configs/inference/generation.yaml \
    --ckpt_path checkpoints/LoomVideo \
    --task mi2v \
    --prompt "Your prompt here" \
    --ref_image_paths /path/to/img1.jpg /path/to/img2.jpg /path/to/img3.jpg \
    --num_frames 97 \
    --num_inference_steps 50 \
    --seed 0 \
    --output_path outputs/mi2v.mp4

Additional Arguments

The following arguments can be appended to any task command for further customization:

Generation Control

Argument	Type	Default	Description
`--num_inference_steps`	int	`50`	Number of denoising steps.
`--guidance_scale`	float	`5.0` / `2.5`	Text CFG scale. `5.0` for t2v/mi2v, `2.5` for edit/ref_edit.
`--guidance_scale_visual`	float	`1.5`	Visual CFG scale for source/reference conditioning.
`--negative_prompt`	str	(from config)	Negative prompt for quality improvement.
`--seed`	int	`0`	Random seed. Set to `-1` for random generation.

Resolution & Frames

Argument	Type	Default	Description
`--height`	int	auto	Output height. `480` for t2v; inferred from source for edit.
`--width`	int	auto	Output width. `832` for t2v; inferred from source for edit.
`--num_frames`	int	auto	Output frames. `81` for t2v/mi2v; inferred for edit.
`--fps`	int	`24`	Output video FPS.

📦 Data Preparation

Since our training relies heavily on proprietary datasets, we are unable to release the original data directly. However, we provide a flexible data organization framework that makes it easy to plug in your own data or publicly available datasets.

Open-Source Datasets

Below are the open-source datasets used in our training. You can download them or substitute with your own data:

Category	Dataset
Video Generation	Koala-36M, OpenVid-1M
Image Editing	CrispEdit-2M, OmniGen-2-Edit, GPT-Image-Edit-1.5M, NHR-Edit, Pico-Banana, ShareGPT-4o-Image
Video Editing	KIWI-Edit
Video Ref Editing / MI2V	RefVIE, Phantom-Data

Organize Data as Single JSON Files

Each data sample should be stored as an individual JSON file, placed in a single directory (e.g., single_jsons/), and named sequentially starting from 0.json:

your_dataset/
└── single_jsons/
    ├── 0.json
    ├── 1.json
    ├── 2.json
    ├── ...

JSON Format for Each Task

Each task type expects a specific set of keys in its JSON file. Below are the templates — fill in according to your data:

Text-to-Video (process_t2v_data):

{
    "text": "A caption describing the video content.",
    "path": "relative/path/to/video.mp4"
}

Text-to-Image (process_t2i_data):

{
    "caption": "A caption describing the image content.",
    "image_path": "relative/path/to/image.jpg"
}

Video Editing (process_video_edit_data):

{
    "source_video_path": "relative/path/to/source_video.mp4",
    "instruction": "The editing instruction.",
    "target_video_path": "relative/path/to/target_video.mp4"
}

Image Editing (process_image_edit_data):

{
    "source_image_path": "relative/path/to/source_image.jpg",
    "instruction": "The editing instruction.",
    "target_image_path": "relative/path/to/target_image.jpg"
}

Multi-Image-to-Video (process_t2v_data_withref):

{
    "instruction": "A prompt describing the video to generate with reference images.",
    "reference_image_paths": [
        "relative/path/to/ref1.jpg",
        "relative/path/to/ref2.jpg"
    ],
    "target_video_path": "relative/path/to/target_video.mp4"
}

Reference-Guided Video Editing (process_video_edit_data_withref):

{
    "source_video_path": "relative/path/to/source_video.mp4",
    "reference_image_paths": [
        "relative/path/to/ref1.jpg"
    ],
    "instruction": "The editing instruction with reference guidance.",
    "target_video_path": "relative/path/to/target_video.mp4"
}

💡 All paths in JSON files are relative to the data_root specified in the dataset config.

Custom Process Functions (Optional)

You may also organize your JSON files in any format you prefer, as long as you implement a corresponding process_* function. We provide several reference implementations in src/dataset/processors.py. Each process function takes (dataset_info, data_info) and returns a list of segments describing the data flow. See the existing functions for examples.

Dataset Config

Create a YAML config file to register your datasets. See configs/dataset/train_demo.yaml as a reference. The config is organized into train, val, and eval sections, each containing dataset entries with the following arguments:

Argument	Description
`task_weight`	Controls the sampling probability of this task group relative to others during training.
`process_func_name`	Name of the processing function in `src/dataset/processors.py` that parses each JSON sample.
`data_root`	Base directory for resolving relative paths in JSON files.
`data_json_dir`	Directory containing the JSON files (`0.json`, `1.json`, ...).
`num_samples`	Total number of samples in the directory.
`sample_weight`	Sampling weight of this dataset within its task group.

🏋️ Training

Training Config

The training behavior is fully controlled by a YAML config file (e.g., configs/train/stage3.yaml).

Key arguments:

Argument	Description
`log_dir`	Directory for saving logs, checkpoints, and generated samples.
`dataset_config_path`	Path to the dataset config YAML file.
`train_steps`	Total number of training iterations.
`checkpointing_interval`	Save a checkpoint every N steps.
`validation_interval`	Run validation every N steps.
`evaluation_interval`	Run evaluation benchmarks every N steps.

Model settings:

Argument	Description
`model.trainable_modules.gen_model`	Which modules to train. `"all"` trains the full generation model.
`model.gradient_checkpointing`	Enable gradient checkpointing to reduce GPU memory usage.
`model.und.pretrained_model_path`	Path to the pretrained understanding backbone.
`model.gen.pretrained_model_path`	Path to the pretrained generation backbone.
`model.pretrained_ckpt_path`	(Optional) Load weights from a previous training stage for continued training.

Data settings:

Argument	Description
`data.train.resolution_buckets`	List of resolution buckets for dynamic batching.
`data.train.num_frames`	Number of frames per training sample.
`data.train.fps`	Video FPS for frame sampling.
`data.train.all_dropout_rate`	Probability of dropping all conditions (for unconditional training).
`data.train.text_dropout_rate`	Probability of dropping text condition (for classifier-free guidance).

Launch Training

Once the data and configs are ready, you can simply start training with:

NUM_GPUS=8

accelerate launch --num_processes=${NUM_GPUS} \
    -m scripts.train.train \
    --config_path path/to/your/config.yaml

💡 All training outputs — including checkpoints, EMA weights, logs, and generated samples — are saved under the log_dir directory specified in the config.

📊 Evaluation

Environment Setup

Step 1: Prepare Benchmark Data

We evaluate on the following benchmarks. Download each dataset and organize it into the same single JSON format used for training data (see Data Preparation):

Benchmark	Category	Samples
GenEval	Image Generation	553
ImgEdit-Bench	Image Editing	737
VBench	Video Generation	165
OpenVE-Bench	Video Editing	431
RefVIE-Bench	Reference Video Editing	120
Intelligent-VBench-MI2V	Multi-Image-to-Video	320
Intelligent-VBench-TIV2V	Text-Image-Video-to-Video	210

💡 For Intelligent-VBench, we split the original benchmark into two subsets based on task type — MI2V and TIV2V. Their JSON files should be placed in separate directories.

After downloading, update the data_root and data_json_dir paths in configs/dataset/benchmarks.yaml to point to your local directories.

Step 2: Install Evaluation Dependencies

VBench:

mkdir -p libs && cd libs
git clone https://github.com/Vchitect/VBench.git

Add the following to libs/VBench/vbench/__init__.py:

import sys, os
local_lib_path = os.path.abspath("libs/VBench")
if local_lib_path not in sys.path:
    sys.path.append(local_lib_path)

If you encounter a NumPy 2.0 compatibility error (np.sctypes was removed), modify lines 45–47 of [YOUR_PYTHON_LIBS]/imgaug/imgaug.py:

# Replace:
# NP_FLOAT_TYPES = set(np.sctypes["float"])
# NP_INT_TYPES = set(np.sctypes["int"])
# NP_UINT_TYPES = set(np.sctypes["uint"])

# With:
NP_FLOAT_TYPES = {np.float16, np.float32, np.float64, np.longdouble}
NP_INT_TYPES = {np.int8, np.int16, np.int32, np.int64, np.longlong}
NP_UINT_TYPES = {np.uint8, np.uint16, np.uint32, np.uint64, np.ulonglong}

To save disk space, remove unnecessary files:

rm -rf libs/VBench/VBench-2.0 libs/VBench/.git libs/VBench/asset libs/VBench/vbench2_beta_trustworthiness

GenEval:

cd libs
git clone https://github.com/djghosh13/geneval.git
cd geneval
./evaluation/download_models.sh "../../checkpoints/"

cd ..
pip install mmcv-full
git clone https://github.com/open-mmlab/mmdetection.git
cd mmdetection && git checkout 2.x
pip install -v -e . --no-build-isolation

The GenEval model paths are configured in configs/evaluation/evaluation.yaml under model.evaluation.geneval:

model:
  evaluation:
    geneval:
      model_path: checkpoints/evaluation/mask2former_swin-s-p4-w7-224_lsj_8x2_50e_coco.pth
      model_config_path: libs/mmdetection/configs/mask2former/mask2former_swin-s-p4-w7-224_lsj_8x2_50e_coco.py
      clip_path: checkpoints/evaluation/ViT-L-14.pt

Step 3: Configure API Keys

Some benchmarks (OpenVE-Bench, RefVIE-Bench, ImgEdit-Bench, Intelligent-VBench) require LLM API calls for metric computation. Configure your API keys in configs/evaluation/evaluation.yaml under model.evaluation:

model:
  evaluation:
    # For OpenVE-Bench, RefVIE-Bench, Intelligent-VBench
    gemini:
      api_key: "YOUR_GEMINI_API_KEY"
      base_url: "YOUR_GEMINI_BASE_URL"
      model: "gemini-2.5-pro-06-17"
    # For ImgEdit-Bench
    openai:
      api_key: "YOUR_OPENAI_API_KEY"
      base_url: "YOUR_OPENAI_BASE_URL"
      model: "gpt-4.1"

Run Evaluation

Once the environment is set up, you can simply run evaluation with:

NUM_GPUS=8

accelerate launch --num_processes=${NUM_GPUS} \
    -m scripts.evaluation.evaluate \
    --config configs/evaluation/evaluation.yaml \
    --checkpoint_dir checkpoints/LoomVideo \
    --generation_configs configs/dataset/benchmarks.yaml \
    --output_dir results/evaluation \
    --calculate_metrics

📧 Contact

Jianzong Wu (吴健宗): jzwu@stu.pku.edu.cn

📄 Citation

TODO