README.md

metadata

license: mit
pipeline_tag: image-to-video
library_name: diffusers

MAGREF: Masked Guidance for Any-Reference Video Generation with Subject Disentanglement

       

MAGREF is a unified and effective framework for any-reference video generation, tackling challenges such as identity inconsistency, entanglement among multiple reference subjects, and copy-paste artifacts. This approach incorporates masked guidance and a subject disentanglement mechanism, enabling flexible video synthesis conditioned on diverse reference images and textual prompts.

Paper Abstract

We tackle the task of any-reference video generation, which aims to synthesize videos conditioned on arbitrary types and combinations of reference subjects, together with textual prompts. This task faces persistent challenges, including identity inconsistency, entanglement among multiple reference subjects, and copy-paste artifacts. To address these issues, we introduce MAGREF, a unified and effective framework for any-reference video generation. Our approach incorporates masked guidance and a subject disentanglement mechanism, enabling flexible synthesis conditioned on diverse reference images and textual prompts. Specifically, masked guidance employs a region-aware masking mechanism combined with pixel-wise channel concatenation to preserve appearance features of multiple subjects along the channel dimension. This design preserves identity consistency and maintains the capabilities of the pre-trained backbone, without requiring any architectural changes. To mitigate subject confusion, we introduce a subject disentanglement mechanism which injects the semantic values of each subject derived from the text condition into its corresponding visual region. Additionally, we establish a four-stage data pipeline to construct diverse training pairs, effectively alleviating copy-paste artifacts. Extensive experiments on a comprehensive benchmark demonstrate that MAGREF consistently outperforms existing state-of-the-art approaches, paving the way for scalable, controllable, and high-fidelity any-reference video synthesis.

Teaser

🔥 News

• [2025.10.10] 🔥 Our Research Paper of MAGREF is now available. The Project Page of MAGREF is created.
• [2025.06.20] 🙏 Thanks to Kijai for developing the ComfyUI nodes for MAGREF and FP8-quantized Hugging Face mode! Feel free to try them out and add MAGREF to your workflow.
• [2025.06.18] 🔥 In progress. We are actively collecting and processing more diverse datasets and scaling up training with increased computational resources to further improve resolution, temporal consistency, and generation quality. Stay turned！
• [2025.06.16] 🔥 MAGREF is coming! The inference codes and checkpoint have been released.

🎥 Demo

https://github.com/user-attachments/assets/ea8f7195-4ffc-4866-b210-f66bac993b7a

📑 Todo List

• Inference codes of MAGREF-480P
• Checkpoint of MAGREF-480P
• Checkpoint of MAGREF-14B Pro
• Training codes of MAGREF

✨ Community Works

ComfyUI

Thanks for Kijai develop the ComfyUI nodes for MAGREF: https://github.com/kijai/ComfyUI-WanVideoWrapper

FP8 quant Huggingface Mode: https://huggingface.co/Kijai/WanVideo_comfy/blob/main/Wan2_1-Wan-I2V-MAGREF-14B_fp8_e4m3fn.safetensors

Guideline

Guideline by Benji: https://www.youtube.com/watch?v=rwnh2Nnqje4

⚙️ Requirements and Installation

We recommend the requirements as follows.

Environment

# 0. Clone the repo
git clone https://github.com/MAGREF-Video/MAGREF.git
cd MAGREF

# 1. Create conda environment
conda create -n magref python=3.11.2
conda activate magref

# 2. Install PyTorch and other dependencies
# CUDA 12.1
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
# CUDA 12.4
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124

# 3. Install pip dependencies
pip install -r requirements.txt

# 4. (Optional) Install xfuser for multiple GPUs inference
pip install "xfuser>=0.4.1"

Download MAGREF Checkpoint

# if you are in china mainland, run this first: export HF_ENDPOINT=https://hf-mirror.com
# pip install -U "huggingface_hub[cli]"
huggingface-cli download MAGREF-Video/MAGREF --local-dir ./ckpts/magref

🤗 Quick Start

• Single-GPU inference
  Tested on a single NVIDIA H100 GPU. The inference consumes around 70 GB of VRAM, so an 80 GB GPU is recommended.

# way 1
bash infer_single_gpu.sh

# way 2
python generate.py \
    --ckpt_dir ./ckpts/magref \
    --save_dir ./samples \
    --prompt_path ./assets/single_id.txt \

• Multi-GPU inference

# way 1
bash infer_multi_gpu.sh

# way 2
torchrun --nproc_per_node=8 generate.py \
    --dit_fsdp --t5_fsdp --ulysses_size 8 \
    --ckpt_dir ./ckpts/magref \
    --save_dir ./samples \
    --prompt_path ./assets/multi_id.txt \

💡Note:

• To achieve the best generation results, we recommend that you describe the visual content of the reference image as accurately as possible when writing text prompt.
• When the generated video is unsatisfactory, the most straightforward solution is to try changing the --base_seed and modifying the description in the prompt.

👍 Acknowledgement

• This project wouldn't be possible without the following open-sourced repositories: Wan2.1, VACE, Phantom, SkyReels-A2, HunyuanCustom, ConsisID, Concat-ID

📧 Ethics Concerns

The images used in these demos are sourced from public domains or generated by models, and are intended solely to showcase the capabilities of this research. If you have any concerns, please contact us at dengyufan10@stu.pku.edu.cn, and we will promptly remove them.

✏️ Citation

If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil:.

BibTeX

@article{deng2025magref,
  title={MAGREF: Masked Guidance for Any-Reference Video Generation},
  author={Deng, Yufan and Guo, Xun and Yin, Yuanyang and Fang, Jacob Zhiyuan and Yang, Yiding and Wang, Yizhi and Yuan, Shenghai and Wang, Angtian and Liu, Bo and Huang, Haibin and others},
  journal={arXiv preprint arXiv:2505.23742},
  year={2025}
}