Add model card and usage documentation

3421113 verified 5 days ago

10.9 kB

	---
	license: apache-2.0
	pipeline_tag: video-to-video
	tags:
	- audio-video
	- audio-video-editing
	- instruction-guided-editing
	- video-editing
	- audio-editing
	- multimodal
	datasets:
	- suimu/InsAVE-80K
	---

	<div align="center">

	<h1>InstructAV2AV: Instruction-Guided Audio-Video Joint Editing</h1>

	<a href="https://arxiv.org/abs/2605.18467"><img src="https://img.shields.io/badge/arXiv-Paper-b31b1b.svg"></a>
	<a href="https://hjzheng.net/projects/InstructAV2AV/"><img src="https://img.shields.io/badge/Project-Page-green.svg"></a>
	<a href="https://github.com/suimuc/InstructAV2AV"><img src="https://img.shields.io/badge/GitHub-Code-black.svg"></a>
	<a href="https://huggingface.co/datasets/suimu/InsAVE-80K"><img src="https://img.shields.io/badge/%F0%9F%A4%97-Dataset-orange.svg"></a>

	Haojie Zheng<sup>1, 2</sup>, Yixin Yang<sup>2</sup>, Siqi Yang<sup>2</sup>, Shuchen Weng<sup>1, 2</sup>, Boxin Shi<sup>2</sup><br>
	<sup>*</sup>Corresponding authors<br>
	<sup>1</sup>BAAI, <sup>2</sup>Peking University

	</div>

	## Overview

	InstructAV2AV is an instruction-guided framework for joint audio-video editing. Given a source video with audio and a natural-language instruction, the model edits the visual and audio content together while preserving the unedited properties of the source as much as possible.

	This repository contains six task-specific InstructAV2AV checkpoints. They share the same architecture and inference pipeline, but are fine-tuned for different editing objectives, including general editing, content insertion and removal, visual identity preservation, voice preservation, and joint identity-and-voice preservation.

	The inference and training code is available in the [InstructAV2AV GitHub repository](https://github.com/suimuc/InstructAV2AV). The corresponding training dataset is available as [InsAVE-80K](https://huggingface.co/datasets/suimu/InsAVE-80K).

	## Video Demo

	<table align="center" border="0" style="width: 100%; text-align: center;">
	<tr>
	<td>
	<video align="center" src="https://github.com/user-attachments/assets/35cdb693-fa2c-40e7-a641-4b87753a5f0b" autoplay loop controls></video>
	</td>
	</tr>
	<tr align="center">
	<td><em>For the best experience, please enable audio.</em></td>
	</tr>
	</table>

	## Checkpoints

	\| Checkpoint \| Editing type \| Description \| Example instruction \|
	\| --- \| --- \| --- \| --- \|
	\| `general.safetensors` \| General editing \| Edits appearance, scenes, actions, speech, and sound. \| `Make the horse dark brown with a white saddle.` \|
	\| `insertion.safetensors` \| Content insertion \| Adds an object or other content to the source video. \| `Add a dark vintage sedan driving from the right to the left.` \|
	\| `removal.safetensors` \| Content removal \| Removes an object and its associated audiovisual content. \| `Remove the chipmunk standing on the stone surface among the peanuts.` \|
	\| `clone_id.safetensors` \| Identity cloning \| Preserves a person's visual identity while editing other visual or audio attributes. \| `Keep the person's appearance, change the timbre to a man, and change the spoken words to <S>I understand, but I think we need to consider.<E>.` \|
	\| `clone_voice.safetensors` \| Voice cloning \| Preserves the speaker's timbre while editing the video or spoken content. \| `Keep the timbre, change the person's appearance, and change the spoken words to <S>I came here to tell you that you should go.<E>.` \|
	\| `clone_id_voice.safetensors` \| Identity and voice cloning \| Preserves both the person's visual identity and the speaker's timbre. \| `Keep the person's identity and voice, and change the spoken words to <S>This is more than just art, it's a statement.<E>.` \|

	Select the checkpoint that most closely matches the requested edit and pass its path through `--finetune-path`.

	## Installation

	The checkpoints must be used with the InstructAV2AV codebase.

	```bash
	git clone https://github.com/suimuc/InstructAV2AV.git
	cd InstructAV2AV

	conda create -n instructav2av python=3.10 -y
	conda activate instructav2av

	pip install torch==2.6.0 torchvision torchaudio
	pip install -r requirements.txt
	pip install flash_attn --no-build-isolation
	pip install -e .
	```

	The provided installation instructions target CUDA 12.1.

	## Download

	Download all six InstructAV2AV checkpoints:

	```bash
	huggingface-cli download suimu/InstructAV2AV \
	--local-dir ckpts/InstructAV2AV
	```

	The inference pipeline also requires the Wan2.2 text encoder and video VAE, together with the MMAudio audio VAE and vocoder:

	```bash
	# Text encoder and video VAE
	huggingface-cli download Wan-AI/Wan2.2-TI2V-5B \
	--include \
	"google/*" \
	"models_t5_umt5-xxl-enc-bf16.pth" \
	"Wan2.2_VAE.pth" \
	--local-dir ckpts/Wan2.2-TI2V-5B

	# Audio VAE and vocoder
	huggingface-cli download hkchengrex/MMAudio \
	--include \
	"ext_weights/best_netG.pt" \
	"ext_weights/v1-16.pth" \
	--local-dir ckpts/MMAudio
	```

	The resulting checkpoint layout should include:

	```text
	ckpts/
	├── InstructAV2AV/
	│ ├── clone_id.safetensors
	│ ├── clone_id_voice.safetensors
	│ ├── clone_voice.safetensors
	│ ├── general.safetensors
	│ ├── insertion.safetensors
	│ └── removal.safetensors
	├── MMAudio/
	│ └── ext_weights/
	│ ├── best_netG.pt
	│ └── v1-16.pth
	└── Wan2.2-TI2V-5B/
	├── google/
	├── models_t5_umt5-xxl-enc-bf16.pth
	└── Wan2.2_VAE.pth
	```

	## Inference

	The unified command-line entry point is `scripts/edit.py`. The source video must contain an audio track unless a separate audio file is supplied with `--source-audio`.

	### Single-video editing

	```bash
	python scripts/edit.py \
	--source-video assets/input.mp4 \
	--instruction "Keep the person's identity and change the spoken words to <S>This is more than just art, it's a statement.<E>." \
	--finetune-path ckpts/InstructAV2AV/clone_id_voice.safetensors \
	--output outputs/edited.mp4
	```

	To use a separate audio track:

	```bash
	python scripts/edit.py \
	--source-video assets/input.mp4 \
	--source-audio assets/input.wav \
	--instruction "Make the horse dark brown with a white saddle." \
	--finetune-path ckpts/InstructAV2AV/general.safetensors \
	--output outputs/edited.mp4
	```

	### Prompt enhancement

	When specifying new spoken content, place the target utterance between `<S>` and `<E>` markers:

	```text
	Change the spoken words to <S>This is the new sentence.<E>.
	```

	For better editing quality and text-to-video alignment, append a detailed description of the desired final video immediately after the editing command. The enhanced prompt can describe the subject's appearance, clothing, actions, expressions, scene, and target speech. For example:

	```bash
	python scripts/edit.py \
	--source-video assets/input1.mp4 \
	--instruction "Change the man into a young woman with brown hair, wearing a gray blazer over a light pink top and a necklace with a heart-shaped pendant, and saying, <S>I really think we should give it another chance.<E>. A young woman with long, wavy brown hair and fair skin is engaged in a conversation. She is wearing a gray blazer over a light pink top and has a necklace with a heart-shaped pendant. Her facial expressions change throughout the sequence, showing a range of emotions that suggest she is either explaining something earnestly or reacting to a conversation, and says <S>I really think we should give it another chance.<E>. The setting appears to be indoors, with a dimly lit, blurred background that suggests a social environment, possibly a bar or restaurant. The focus remains on the woman's face, capturing her reactions and engagement in the dialogue." \
	--finetune-path ckpts/InstructAV2AV/general.safetensors \
	--output outputs/edited.mp4
	```

	### Gradio demo

	After downloading all six checkpoints and the dependency weights, launch the interactive demo with:

	```bash
	python scripts/demo.py --share
	```

	## Default Inference Configuration

	The released configuration uses 121 frames at 704 × 1280 resolution, 16 kHz audio, 50 sampling steps, and bfloat16 inference. CUDA is required. See `ovi/configs/inference/inference_av_edit.yaml` in the code repository for all options, including guidance scales, seed, sequence parallelism, and CPU offloading.

	## InsAVE-80K Dataset

	The training dataset is available at [suimu/InsAVE-80K](https://huggingface.co/datasets/suimu/InsAVE-80K).

	## Training

	The unified training entry point is `scripts/train.py`, with configuration in `ovi/configs/train/train_av_edit.yaml`. The trainer accepts CSV, JSON, or JSONL manifests with the following canonical fields:

	\| Field \| Description \|
	\| --- \| --- \|
	\| `source_video` \| Original video \|
	\| `source_audio` \| Original audio \|
	\| `target_video` \| Edited video \|
	\| `target_audio` \| Edited audio \|
	\| `instruction` \| Editing instruction \|

	Before training, set the checkpoint directory, initialization checkpoint, and dataset manifest in the training configuration:

	```yaml
	ckpt_dir: ./ckpts
	finetune_path: ./ckpts/InstructAV2AV/general.safetensors

	dataset:
	metadata_path: ./data/InsAVE-80K/path/to/manifest.csv
	```

	The manifest and initialization checkpoint can also be overridden from the command line:

	```bash
	accelerate launch \
	--config_file ovi/configs/train/accelerate_config.yaml \
	scripts/train.py \
	--config-file ovi/configs/train/train_av_edit.yaml \
	--data-manifest data/InsAVE-80K/path/to/manifest.csv \
	--finetune-path ckpts/InstructAV2AV/general.safetensors \
	--output-dir outputs/train_av_edit
	```

	The provided Accelerate configuration uses DeepSpeed ZeRO-2 and is configured for eight processes. Change `num_processes` in `ovi/configs/train/accelerate_config.yaml` to match the available GPUs.

	The training objective is selected from the modality flags:

	\| `has_video` \| `has_audio` \| Training objective \|
	\| ---: \| ---: \| --- \|
	\| `true` \| `true` \| Joint audio-video editing \|
	\| `true` \| `false` \| Video-only editing \|
	\| `false` \| `true` \| Audio-only editing \|



	## Related Resources

	- [Source code](https://github.com/suimuc/InstructAV2AV)
	- [Project page](https://hjzheng.net/projects/InstructAV2AV/)
	- [InsAVE-80K dataset](https://huggingface.co/datasets/suimu/InsAVE-80K)
	- [Wan2.2](https://github.com/Wan-Video/Wan2.2)
	- [MMAudio](https://github.com/hkchengrex/MMAudio)
	- [Ovi](https://github.com/character-ai/Ovi)

	## Citation

	If you find InstructAV2AV or InsAVE-80K useful in your research, please cite:

	```bibtex
	@article{instructav2av2026,
	title={InstructAV2AV: Instruction-Guided Audio-Video Joint Editing},
	author={Zheng, Haojie and Yang, Yixin and Yang, Siqi and Weng, Shuchen and Shi, Boxin},
	journal={arXiv preprint arXiv:2605.18467},
	year={2026}
	}
	```

	## License

	The project is released under the Apache License 2.0. Users must also comply with the licenses and terms of the underlying pretrained models and any input media or datasets they use.