README.md

---
title: LeapTalk
emoji: 🗣️
colorFrom: pink
colorTo: yellow
sdk: gradio
sdk_version: 6.23.1
app_file: app.py
short_description: One-step audio-driven talking head video generation
python_version: '3.12'
startup_duration_timeout: 45m
pinned: false
license: apache-2.0
models:
- z-rx/leaptalk
- Soul-AILab/SoulX-FlashHead-1_3B
- facebook/wav2vec2-base-960h
---

# LeapTalk — real-time talking-head generation

Demo for [**z-rx/leaptalk**](https://huggingface.co/z-rx/leaptalk): give it a portrait photo
and a speech clip, and it generates a lip-synced talking-head video.

LeapTalk reformulates talking-head generation as a *data-to-data* Brownian-bridge transport
(**Bridge Forcing**), which lets the distilled model synthesize each video chunk with a single
network evaluation (1 NFE) while keeping identity and style stable over long autoregressive
rollouts.

- Paper: [LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation](https://huggingface.co/papers/2608.00079)
- Project page: https://zhangrongxiang.github.io/leaptalk-page/
- Code: https://github.com/zhangrongxiang/LeapTalk

## Pipeline

This Space reproduces the authors' reference streaming pipeline
(`inference.py --model_type pro --lite --num_inference_steps 1 --audio_encode_mode stream`):

| component | weights |
|---|---|
| base video DiT | `Soul-AILab/SoulX-FlashHead-1_3B` (`Model_Pro`) |
| LeapTalk LoRA (merged) + audio projector | `z-rx/leaptalk` |
| audio encoder | `facebook/wav2vec2-base-960h` |
| Lite TAE video autoencoder | `taew2_1.pth` from `z-rx/leaptalk` |
| sampler | ViBT Brownian-bridge scheduler (1 step / chunk) |

Video is produced chunk-by-chunk (33 frames per chunk, 5 overlapping motion frames) at
512x512, 25 fps, with the SoulX-style VAE round-trip history update and reference-colour
correction.

## Notes

- Runs on ZeroGPU; the multi-GPU USP (xFuser sequence-parallel) path from the reference
  implementation is disabled since the Space is single-GPU.
- Example assets come from the [LeapTalk](https://github.com/zhangrongxiang/LeapTalk) and
  [SoulX-FlashHead](https://github.com/Soul-AILab/SoulX-FlashHead) repositories (Apache-2.0);
  the neutral narration clip is public-domain audiobook narration from
  [LibriSpeech](https://www.openslr.org/12) (LibriVox, CC0 / public domain). The audio
  clips were trimmed to a few seconds.