TBDub: Production-Oriented Visual Dubbing

High-quality lip sync. Fast, multilingual, and robust.

Give an existing video a new voice. TBDub synchronizes the speaker's lips with new speech while preserving their appearance, motion, and background.

Input: source video + driving audio → Output: lip-synced video.

Code · Teacher · Student — all open source under Apache-2.0.

See TBDub in action · 46-second demo

One voice track, four faces — multilingual lip sync with the two-step Student Model. Watch examples in seven languages across microphone and on-screen text occlusions, large head turns, and rapid head motion. The video opens with the original clips, then shows the generated results. Turn on sound to hear the language changes.

Open the video · Run the Student Model

V1.1 — Inference and deployment update

V1.1 reduces checkpoint download sizes and adds an optional lower-memory mode to the original V1.0 release. It uses the same trained Teacher and Student models; the paper's quality and H20 speed results remain the original evaluation.

• Smaller downloads: each variant needs one complete BF16 DiT checkpoint of about 12.59 GB, plus the shared auxiliary files listed below. Download only the Teacher or Student you plan to use.
• Optional lower GPU memory use: add --cpu-offload if inference runs out of GPU memory. In this mode, inactive models stay in system RAM. Measured process peaks were 19.49 GiB for Teacher and 15.01 GiB for Student at 512×512 and 376 output frames on an RTX PRO 5000 72GB. See the measured workload and tradeoffs below; these are separate from the paper's H20 speed benchmark.

V1.1 release notes · Project update and memory measurements

Why TBDub?

• 🏆 Leading perceptual quality. Highest mean opinion scores in all three dimensions among the open-source methods in our study: 3.85 lip sync, 3.80 visual quality, and 3.78 identity preservation. Student leads the first two; Teacher leads identity. See results.
• ⚡ Fast, two-step generation. 7.13 FPS on a single NVIDIA H20 at 512 × 512, or 13.93× the Teacher's core generation speed. See benchmark.
• 🌍 Multilingual lip sync. Lip sync across languages, with examples including English, Chinese, Japanese, Korean, and Russian. See lip sync and identity preservation in multilingual reconstruction. Watch examples.
• 🛡️ Robust in challenging scenes. Stable lip sync and appearance through large head turns, hands and microphones over the mouth, and rapid motion. Head turns · Occlusions.

MOS: 38 TalkVid clips, 114 ratings per method, on a 0–5 scale. Speed: VAE encode to decode, excluding preprocessing, audio encoding, and file output. Full evaluation protocol.

Watch comparisons · Get the code · Download models

Model Variants

File	Description
tbdub_teacher.safetensors	Complete BF16 Teacher for standard 30-step inference
tbdub_student.safetensors	Complete BF16 Student for two-step inference
null_prompt_emb.pt	Null text embedding used during inference

Choose one variant. Each checkpoint contains the complete DiT model and uses the same runtime parameter values as V1.0. Both variants also need the shared VAE, HuBERT, fixed prompt embedding, and Face Landmarker files listed below. For either variant, the complete required model download is about 15.27 GB: 12.59 GB for its DiT checkpoint plus about 2.68 GB of shared auxiliary files. The other variant's checkpoint is unnecessary. Sizes here use decimal GB and exclude Python packages and CUDA libraries.

For standard 30-step inference, download tbdub_teacher.safetensors. Its 12.59 GB DiT download replaces the 22.87 GB required by the V1.0 setup.

Exact size: 12,591,523,040 bytes. Teacher SHA-256: e73483dfde3b960d3d22f85e356c378067ccd3cff81f3b668d92167b8de605d0.

For two-step inference, download tbdub_student.safetensors. Its BF16 file is 12,591,523,048 bytes (about 12.59 GB), compared with the 37.77 GB DiT download documented for V1.0. All 1,245 tensors match the original Student cast to BF16, the dtype already used by inference. The storage change reduces download size; the optional CPU offload described below reduces GPU memory use.

SHA-256: 4e86900f9744b3650bfe58451ae89b3afba758233a3edf5947516268c1ea11e6.

The manifest lists each variant's single checkpoint, its checksum, and the shared auxiliary dependencies.

Installation and Inference

The inference code and complete installation instructions are available in the GitHub repository:

• GitHub: https://github.com/TaoLiveAIGC/TBDub

Clone the repository:

git clone https://github.com/TaoLiveAIGC/TBDub.git
cd TBDub

Use Linux with an NVIDIA GPU, Python 3.10 (validated environment), and ffmpeg on PATH. Install a CUDA-compatible PyTorch/torchvision build in your environment, then install the project dependencies:

pip install -r requirements.txt

Keep only the pinned opencv-contrib-python package; do not install opencv-python alongside it because both provide cv2.

Download the configuration manifest together with the model variant you plan to use. The current GitHub inference entry point reads checkpoints/config.json, validates its runtime compatibility, and uses its selected variant and model paths as defaults. Explicit command-line arguments override these defaults. Downloading the manifest also provides a model-level download query file.

Teacher (30 steps):

hf download TaoLiveAIGC/TBDub \
  config.json null_prompt_emb.pt \
  tbdub_teacher.safetensors \
  --local-dir checkpoints

Student (2 steps):

hf download TaoLiveAIGC/TBDub \
  config.json null_prompt_emb.pt \
  tbdub_student.safetensors \
  --local-dir checkpoints

Both variants also need the shared VAE and HuBERT files. Download only these files from their upstream repositories:

hf download KlingTeam/X-Dub Wan2.2_VAE.safetensors --local-dir checkpoints
hf download facebook/hubert-large-ll60k \
  config.json preprocessor_config.json pytorch_model.bin \
  --local-dir checkpoints/hubert-large-ll60k

Download the Face Landmarker bundle required for full-frame videos:

mkdir -p checkpoints
curl -fL https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task \
  -o checkpoints/face_landmarker.task
echo '64184e229b263107bc2b804c6625db1341ff2bb731874b0bcc2fe6544e0bc9ff  checkpoints/face_landmarker.task' | sha256sum -c -

Run Teacher inference:

bash infer.sh path/to/source.mp4 path/to/driving.wav results

Run Student inference with the current GitHub code:

python inference.py \
  --video path/to/source.mp4 --audio path/to/driving.wav \
  --inference-mode student \
  --dit-checkpoint checkpoints/tbdub_student.safetensors \
  --num-student-steps 2 --sigma-shift 1.0 --motion-from-latents \
  --seed 42 --output-dir results

To store all weights elsewhere, use --checkpoint-dir /path/to/models or set TBDUB_CHECKPOINT_DIR=/path/to/models for infer.sh. The directory must contain the downloaded config.json and the same relative model layout. See the configuration rules.

Use the explicit download list for your chosen variant; the other variant's checkpoint is unnecessary.

Both commands keep models resident on the GPU by default to prioritize speed. If inference runs out of GPU memory, rerun the Python inference command with --cpu-offload. This is a manual choice: the program does not detect GPU capacity, switch modes automatically, or retry an out-of-memory failure.

With --cpu-offload, inactive DiT/VAE weights stay in system RAM, and HuBERT moves to the GPU only for audio feature extraction. Computation keeps the same precision and sampling settings. This mode uses more system RAM and adds CPU/GPU transfer time. For example, rerun Teacher inference with:

python inference.py --video path/to/source.mp4 --audio path/to/driving.wav \
  --inference-mode teacher --cpu-offload --output-dir results

For 376 frames at 512×512 on an RTX PRO 5000 72GB, the CPU-offload runs completed under a 22 GiB PyTorch allocator limit, with process GPU peaks of 19.49 GiB (Teacher) and 15.01 GiB (Student). Decoded video and audio matched the resident-mode outputs exactly. Teacher's three simultaneous CFG branches share weights but need more intermediate activations than Student's single branch, even though their model sizes are equal. Overall peaks depend on the active stage and input length. See the memory, host-RAM and timing details; these are workstation measurements, not a 4090 hardware validation or a guarantee that arbitrarily long inputs fit in 24 GiB. The measured host-process RAM peaks were 21.9 GiB for Teacher and 21.3 GiB for Student, plus system overhead.

Already aligned face videos can use --cropped-input to skip face preprocessing. See the GitHub installation and inference instructions for download sources, MediaPipe model checksums, and command-line options.

Intended Use

TBDub is intended for research and authorized applications involving visual dubbing, video localization, digital humans, and audio-driven facial animation.

Users are responsible for obtaining appropriate consent and ensuring that generated content complies with applicable laws, licenses, and platform policies.

Limitations

The visual quality of the generated video remains dependent on the quality of the source video. Although TBDub can improve the apparent clarity of the lips and teeth relative to the input, it cannot recover high-fidelity facial details from severely low-resolution videos. Additional face restoration or face super-resolution may therefore be required for such inputs.

The current quantitative evaluation is primarily based on self-driven reconstruction, where each source video is reconstructed using its original audio. These metrics may not fully reflect performance in cross-driven settings with mismatched or cross-identity audio. Moreover, automatic metrics do not always align with human perception; quantitative results should therefore be considered together with qualitative video comparisons.

Authors

Bihan Li, Xinyang Li, Zeran Xu, Meiguang Jin, and Junfeng Ma
TaoLiveAIGC Team, Taobao & Tmall Group, Alibaba Group

License

The released model weights and code are provided under the Apache License 2.0 unless otherwise specified. Third-party dependencies and checkpoints remain subject to their respective licenses.

Citation

@misc{li2026tbdubproductionorientedvisualdubbing,
  title={TBDub: Production-Oriented Visual Dubbing},
  author={Bihan Li and Xinyang Li and Zeran Xu and Meiguang Jin and Junfeng Ma},
  year={2026},
  eprint={2609.06144},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2609.06144},
}