Buckets:
Advanced LipSync Model
A modern, high-quality lip synchronization model using single-step latent inpainting.
Architecture
- Audio Encoder: Frozen Whisper-tiny for robust speech features
- Image Encoder/Decoder: Frozen Stable Diffusion 1.5 VAE for latent space encoding
- Fusion Network: Trainable lightweight UNet with cross-attention
- Sync Discriminator: Custom SyncNet for lip-sync quality
Features
Video + Audio Lip-Sync
Upload a video and a new audio track — the model re-renders lip movements to match the audio.
Photo + Audio Lip-Sync (NEW)
Upload a single portrait photo and an audio file (speech or song) to generate a full talking-head video with:
- Accurate lip movements matching the audio
- Subtle natural head motion driven by audio energy
- Periodic blinking for realism
- Feathered face blending onto the original photo
python -m src.inference.photo_to_video \
--photo portrait.jpg \
--audio speech.wav \
--checkpoint checkpoints/best_model.pt \
--output talking_head.mp4
Quick Start
# Create environment
python -m venv venv
venv\Scripts\activate # Windows
# Install dependencies
pip install -e ".[dev]"
# Download and preprocess dataset
python scripts/download_hdtf.py --output data/raw
python scripts/preprocess.py --input data/raw --output data/processed
python scripts/extract_features.py --input data/processed --output data/cache
# Train SyncNet first
python -m src.training.train_syncnet --config configs/syncnet.yaml
# Train main model
python -m src.training.train_model --config configs/train.yaml
Distributed Training (Multi-GPU, RunPod)
For faster training on multi-GPU cloud instances:
# Train SyncNet (distributed, 4 GPUs, resume from checkpoint)
torchrun --nproc_per_node=4 \
-m src.training.train_syncnet_distributed \
--config configs/syncnet_distributed.yaml \
--resume checkpoints/syncnet_best.pt
# Train main model (distributed, 4 GPUs)
torchrun --nproc_per_node=4 \
-m src.training.train_model_distributed \
--config configs/train_distributed.yaml
# Run inference (video mode)
python -m src.inference.predict --video input.mp4 --audio audio.wav --output result.mp4
# Run inference (photo mode)
python -m src.inference.photo_to_video --photo face.jpg --audio speech.wav --output result.mp4
Web Application
Start the backend and open the React frontend for a drag-and-drop UI:
# Backend
uvicorn app.backend.main:app --host 0.0.0.0 --port 8000
# Frontend (dev mode)
cd app/frontend && npm run dev
The web UI supports both Video + Audio and Photo + Audio modes via a mode toggle.
Project Structure
├── configs/ # Training configs (YAML)
├── data/ # Dataset storage
├── docs/ # Model improvement plan & docs
├── scripts/ # Data preparation scripts
├── src/ # Source code
│ ├── models/ # Model architectures (UNet, SyncNet, LipSyncModel)
│ ├── data/ # Dataset & transforms
│ ├── training/ # Training loops
│ └── inference/ # Inference pipelines (video & photo modes)
├── app/ # Full-stack application
│ ├── backend/ # FastAPI server (REST + WebSocket)
│ └── frontend/ # React + Vite UI (mode selection, upload, preview)
├── guide/ # Learning guide
└── tests/ # Tests
Documentation
- Learning Guide — Deep-dive into how the model works, every design choice explained
- Distributed Training Guide — Multi-GPU training on RunPod with DDP
- Model Improvement Plan — Roadmap for scaling to higher quality (datasets, architecture, training)
- Implementation Plan — Original architecture design and planning
Requirements
- Python 3.10+
- NVIDIA GPU with 16GB+ VRAM (RTX 5060 Ti or similar)
- CUDA 12.x
- ffmpeg (for audio processing)
Xet Storage Details
- Size:
- 4.24 kB
- Xet hash:
- 45229c8015c354f1e26863ad03daa647eb36dcfa6da4fc20730ab3f5165a5792
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.