VidDub

VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing

Paper-selected full-context two-step checkpoint.

• Paper
• Project page
• Code
• VidDub-Bench

News

• 2026-11-03: Training code released.
• 2026-11-03: Model weights released.
• 2026-11-02: Paper released on arXiv.

Quick Start

Requirements: Linux, an NVIDIA CUDA GPU, Python 3.10+, Git, and FFmpeg.

1. Install VidDub

git clone https://github.com/LAW1223/VidDub
cd VidDub
pip install -r requirements.txt

2. Download shared base dependencies

The model repository contains the trained VidDub components only. Download the shared Wan VAE/T5 and Whisper encoder once:

mkdir -p weights
huggingface-cli download Wan-AI/Wan2.1-VACE-1.3B \
  --local-dir weights/Wan2.1-VACE-1.3B
huggingface-cli download openai/whisper-small \
  --local-dir weights/whisper-small

3. Run full-video dubbing

The source video must contain a detectable, mostly front-facing face and be at least 81 frames long. The target audio is automatically resampled to 16 kHz.

python scripts/infer.py \
  --model AIPeanutman/VidDub \
  --weight-root ./weights \
  --video input.mp4 \
  --audio target.wav \
  --output output.mp4

--model accepts either a Hugging Face repo ID or a local model directory. To download this release for offline use:

huggingface-cli download AIPeanutman/VidDub \
  --local-dir weights/VidDub

The full-context sampling step count is fixed to 2 by the model config. Inference audio CFG is fixed to 1. The output keeps the input frame rate and resolution; generation is performed with the paper's overlapping-window protocol and pasted back into the source video. Exactly 81 frames use one full-context call; longer videos use 81-frame windows with stride 64 and 17 shared frames. Each window has its own seed, overlaps are linearly blended, and lower-face color harmonization is enabled.

Useful options

--mode auto       One call for 81 frames; bounded windows for longer videos
--overlap 17      Paper protocol: 17 shared frames, stride 64
--no-color-harmonization
                  Disable lower-face Lab correction (ablation)
--feather 15      Mouth-region paste-back feather radius
--seed 0          Sampling seed
--device cuda     Inference device
--sft-steps 30    SFT only: override denoising steps

Benchmark

Download the frozen 256-source same/cross benchmark and validate it with the code repository:

huggingface-cli download AIPeanutman/VidDub-Bench \
  --repo-type dataset \
  --local-dir data/VidDub-Bench
python scripts/validate_benchmark.py --root data/VidDub-Bench

Use scripts/run_benchmark.py to run all fixed same- and cross-audio conditions while loading this model only once. See the benchmark guide for the exact protocol and resumable execution options.

Model

• Mode: full-context 2-step DMD
• Video rate: 24 FPS
• Audio rate: 16 kHz mono
• Window: 81 frames
• Long-video stride / overlap: 64 / 17 frames
• Inference audio CFG: 1
• Sampling steps: 2
• Base dependency: Wan2.1-VACE-1.3B
• Audio dependency: Whisper-small

The repository stores the trained DiT, VACE and VidDub audio adapter. The base VAE, T5/tokenizer and Whisper checkpoint are not duplicated here. The released trained components contain 2,440,958,285 parameters in total; “1.3B” refers only to the Wan backbone class and is not the complete inference parameter count.

Files

• dit/diffusion_pytorch_model.safetensors: BF16 Wan DiT
• vace/diffusion_pytorch_model.safetensors: BF16 VACE
• audio_adapter/model.safetensors: FP32 VidDub audio adapter
• config.json: public architecture and sampling configuration
• checksums.json: SHA256 and tensor metadata
• LICENSE: VidDub Individual Commercial and Research License 1.0

Limitations

The current full-video implementation is offline/windowed rather than a persistent real-time streaming server. Large head motion, profile faces, occlusions, multiple faces and shots shorter than 81 frames are not the primary operating regime.

Responsible use

Use only with media and identities for which you have permission. Do not use for impersonation, fraud, harassment, non-consensual sexual content, or deceptive political/media manipulation. Generated or modified media should be clearly disclosed where appropriate.

License

This release uses the VidDub Individual Commercial and Research License. Individuals may use VidDub personally or commercially, including monetizing completed outputs. Non-commercial academic research is permitted. Company use, SaaS, APIs, hosted inference, web applications, bots, platform integrations, and other managed or remote services require separate written permission. Third-party terms apply independently.

Citation

@article{liu2026viddub,
  title   = {VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing},
  author  = {Liu, Yexin and Lin, Yunlong and Wang, Zehan and He, Guangxin and
             Xu, Xinli and Zha, Jiajun and Liu, Yaofu and Wang, Yueze and
             Ping, Bowen and Li, Yuanming and Wang, Amie and Ghanem, Bernard and
             Li, Bing and Yang, Harry},
  journal = {arXiv preprint arXiv:2611.03404},
  year    = {2026}
}