Buckets:
196 GB
40,392 files
Updated 8 months ago
Ctrl+K
| Name | Size | Uploaded | Xet hash |
|---|---|---|---|
| checkpoints | 7 items | ||
| configs | 10 items | ||
| data | 40,332 items | ||
| models | 1 items | ||
| scripts | 19 items | ||
| src | 18 items | ||
| tests | 2 items | ||
| PLAN.md | 5.34 kB xet | 5174c964 | |
| README.md | 4.24 kB xet | 45229c80 | |
| pyproject.toml | 1.63 kB xet | afdba86e |
Advanced LipSync Model
A modern, high-quality lip synchronization model using single-step latent inpainting.
Architecture
- Audio Encoder: Frozen Whisper-tiny for robust speech features
- Image Encoder/Decoder: Frozen Stable Diffusion 1.5 VAE for latent space encoding
- Fusion Network: Trainable lightweight UNet with cross-attention
- Sync Discriminator: Custom SyncNet for lip-sync quality
Features
Video + Audio Lip-Sync
Upload a video and a new audio track — the model re-renders lip movements to match the audio.
Photo + Audio Lip-Sync (NEW)
Upload a single portrait photo and an audio file (speech or song) to generate a full talking-head video with:
- Accurate lip movements matching the audio
- Subtle natural head motion driven by audio energy
- Periodic blinking for realism
- Feathered face blending onto the original photo
python -m src.inference.photo_to_video \
--photo portrait.jpg \
--audio speech.wav \
--checkpoint checkpoints/best_model.pt \
--output talking_head.mp4
Quick Start
# Create environment
python -m venv venv
venv\Scripts\activate # Windows
# Install dependencies
pip install -e ".[dev]"
# Download and preprocess dataset
python scripts/download_hdtf.py --output data/raw
python scripts/preprocess.py --input data/raw --output data/processed
python scripts/extract_features.py --input data/processed --output data/cache
# Train SyncNet first
python -m src.training.train_syncnet --config configs/syncnet.yaml
# Train main model
python -m src.training.train_model --config configs/train.yaml
Distributed Training (Multi-GPU, RunPod)
For faster training on multi-GPU cloud instances:
# Train SyncNet (distributed, 4 GPUs, resume from checkpoint)
torchrun --nproc_per_node=4 \
-m src.training.train_syncnet_distributed \
--config configs/syncnet_distributed.yaml \
--resume checkpoints/syncnet_best.pt
# Train main model (distributed, 4 GPUs)
torchrun --nproc_per_node=4 \
-m src.training.train_model_distributed \
--config configs/train_distributed.yaml
# Run inference (video mode)
python -m src.inference.predict --video input.mp4 --audio audio.wav --output result.mp4
# Run inference (photo mode)
python -m src.inference.photo_to_video --photo face.jpg --audio speech.wav --output result.mp4
Web Application
Start the backend and open the React frontend for a drag-and-drop UI:
# Backend
uvicorn app.backend.main:app --host 0.0.0.0 --port 8000
# Frontend (dev mode)
cd app/frontend && npm run dev
The web UI supports both Video + Audio and Photo + Audio modes via a mode toggle.
Project Structure
├── configs/ # Training configs (YAML)
├── data/ # Dataset storage
├── docs/ # Model improvement plan & docs
├── scripts/ # Data preparation scripts
├── src/ # Source code
│ ├── models/ # Model architectures (UNet, SyncNet, LipSyncModel)
│ ├── data/ # Dataset & transforms
│ ├── training/ # Training loops
│ └── inference/ # Inference pipelines (video & photo modes)
├── app/ # Full-stack application
│ ├── backend/ # FastAPI server (REST + WebSocket)
│ └── frontend/ # React + Vite UI (mode selection, upload, preview)
├── guide/ # Learning guide
└── tests/ # Tests
Documentation
- Learning Guide — Deep-dive into how the model works, every design choice explained
- Distributed Training Guide — Multi-GPU training on RunPod with DDP
- Model Improvement Plan — Roadmap for scaling to higher quality (datasets, architecture, training)
- Implementation Plan — Original architecture design and planning
Requirements
- Python 3.10+
- NVIDIA GPU with 16GB+ VRAM (RTX 5060 Ti or similar)
- CUDA 12.x
- ffmpeg (for audio processing)
- Total size
- 196 GB
- Files
- 40,392
- Last updated
- Apr 11
- Pre-warmed CDN
- US EU US EU