196 GB
40,392 files
Updated 8 months ago
Name
Size
checkpoints
configs
data
models
scripts
src
tests
PLAN.md5.34 kB
xet
README.md4.24 kB
xet
pyproject.toml1.63 kB
xet
README.md

Advanced LipSync Model

A modern, high-quality lip synchronization model using single-step latent inpainting.

Architecture

  • Audio Encoder: Frozen Whisper-tiny for robust speech features
  • Image Encoder/Decoder: Frozen Stable Diffusion 1.5 VAE for latent space encoding
  • Fusion Network: Trainable lightweight UNet with cross-attention
  • Sync Discriminator: Custom SyncNet for lip-sync quality

Features

Video + Audio Lip-Sync

Upload a video and a new audio track — the model re-renders lip movements to match the audio.

Photo + Audio Lip-Sync (NEW)

Upload a single portrait photo and an audio file (speech or song) to generate a full talking-head video with:

  • Accurate lip movements matching the audio
  • Subtle natural head motion driven by audio energy
  • Periodic blinking for realism
  • Feathered face blending onto the original photo
python -m src.inference.photo_to_video \
    --photo portrait.jpg \
    --audio speech.wav \
    --checkpoint checkpoints/best_model.pt \
    --output talking_head.mp4

Quick Start

# Create environment
python -m venv venv
venv\Scripts\activate  # Windows

# Install dependencies
pip install -e ".[dev]"

# Download and preprocess dataset
python scripts/download_hdtf.py --output data/raw
python scripts/preprocess.py --input data/raw --output data/processed
python scripts/extract_features.py --input data/processed --output data/cache

# Train SyncNet first
python -m src.training.train_syncnet --config configs/syncnet.yaml

# Train main model
python -m src.training.train_model --config configs/train.yaml

Distributed Training (Multi-GPU, RunPod)

For faster training on multi-GPU cloud instances:

# Train SyncNet (distributed, 4 GPUs, resume from checkpoint)
torchrun --nproc_per_node=4 \
    -m src.training.train_syncnet_distributed \
    --config configs/syncnet_distributed.yaml \
    --resume checkpoints/syncnet_best.pt

# Train main model (distributed, 4 GPUs)
torchrun --nproc_per_node=4 \
    -m src.training.train_model_distributed \
    --config configs/train_distributed.yaml

# Run inference (video mode)
python -m src.inference.predict --video input.mp4 --audio audio.wav --output result.mp4

# Run inference (photo mode)
python -m src.inference.photo_to_video --photo face.jpg --audio speech.wav --output result.mp4

Web Application

Start the backend and open the React frontend for a drag-and-drop UI:

# Backend
uvicorn app.backend.main:app --host 0.0.0.0 --port 8000

# Frontend (dev mode)
cd app/frontend && npm run dev

The web UI supports both Video + Audio and Photo + Audio modes via a mode toggle.

Project Structure

├── configs/           # Training configs (YAML)
├── data/              # Dataset storage
├── docs/              # Model improvement plan & docs
├── scripts/           # Data preparation scripts
├── src/               # Source code
│   ├── models/        # Model architectures (UNet, SyncNet, LipSyncModel)
│   ├── data/          # Dataset & transforms
│   ├── training/      # Training loops
│   └── inference/     # Inference pipelines (video & photo modes)
├── app/               # Full-stack application
│   ├── backend/       # FastAPI server (REST + WebSocket)
│   └── frontend/      # React + Vite UI (mode selection, upload, preview)
├── guide/             # Learning guide
└── tests/             # Tests

Documentation

Requirements

  • Python 3.10+
  • NVIDIA GPU with 16GB+ VRAM (RTX 5060 Ti or similar)
  • CUDA 12.x
  • ffmpeg (for audio processing)
Total size
196 GB
Files
40,392
Last updated
Apr 11
Pre-warmed CDN
US EU US EU

Contributors