README.md

metadata

title: LeapTalk
emoji: 🗣️
colorFrom: pink
colorTo: yellow
sdk: gradio
sdk_version: 6.23.1
app_file: app.py
short_description: One-step audio-driven talking head video generation
python_version: '3.12'
startup_duration_timeout: 45m
pinned: false
license: apache-2.0
models:
  - z-rx/leaptalk
  - Soul-AILab/SoulX-FlashHead-1_3B
  - facebook/wav2vec2-base-960h

LeapTalk — real-time talking-head generation

Demo for z-rx/leaptalk: give it a portrait photo and a speech clip, and it generates a lip-synced talking-head video.

LeapTalk reformulates talking-head generation as a data-to-data Brownian-bridge transport (Bridge Forcing), which lets the distilled model synthesize each video chunk with a single network evaluation (1 NFE) while keeping identity and style stable over long autoregressive rollouts.

• Paper: LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation
• Project page: https://zhangrongxiang.github.io/leaptalk-page/
• Code: https://github.com/zhangrongxiang/LeapTalk

Pipeline

This Space reproduces the authors' reference streaming pipeline (inference.py --model_type pro --lite --num_inference_steps 1 --audio_encode_mode stream):

component	weights
base video DiT	Soul-AILab/SoulX-FlashHead-1_3B (Model_Pro)
LeapTalk LoRA (merged) + audio projector	z-rx/leaptalk
audio encoder	facebook/wav2vec2-base-960h
Lite TAE video autoencoder	taew2_1.pth from z-rx/leaptalk
sampler	ViBT Brownian-bridge scheduler (1 step / chunk)

Video is produced chunk-by-chunk (33 frames per chunk, 5 overlapping motion frames) at 512x512, 25 fps, with the SoulX-style VAE round-trip history update and reference-colour correction.

Notes

• Runs on ZeroGPU; the multi-GPU USP (xFuser sequence-parallel) path from the reference implementation is disabled since the Space is single-GPU.
• Example assets come from the LeapTalk and SoulX-FlashHead repositories (Apache-2.0); the neutral narration clip is public-domain audiobook narration from LibriSpeech (LibriVox, CC0 / public domain). The audio clips were trimmed to a few seconds.