---
title: SyncAI
emoji: 🎵
colorFrom: indigo
colorTo: pink
sdk: gradio
sdk_version: "6.8.0"
app_file: app.py
pinned: false
short_description: AI Music Ads Generator
---

# SyncAI — AI Music Video Generator

Generate beat-synced music video ads from a song clip. Upload ~15 seconds of audio, pick a visual style, and SyncAI produces a fully assembled vertical video with AI-generated visuals cut to the beat.

## How It Works

```
Song (audio file)
  ├─► Stem Separation (LALAL.AI) → Vocals + Drums
  ├─► Lyrics Extraction (WhisperX) → Word-level timestamps
  ├─► Beat Detection (madmom RNN + DBN) → Beat timestamps + drop detection
  ├─► Segmentation → Lyrics mapped to beat intervals
  ├─► Prompt Generation (Claude Sonnet 4.6) → Image + video motion prompts
  ├─► Image Generation (SDXL + Hyper-SD + style LoRA) → 768x1344 images
  ├─► Image-to-Video (Wan 2.1 14B) → Animated clips
  └─► Assembly (FFmpeg) → Beat-synced video with lyrics overlay
```

## Visual Styles

Each style applies a different LoRA to SDXL and sets a unique scene world for the LLM prompt generator. The Sunset Coastal Drive LoRA was custom-trained for this project; the others are community LoRAs from HuggingFace Hub:

| Style | LoRA | Setting |
|-------|------|---------|
| **Sunset Coastal Drive** | Custom-trained (`samuelsattler/warm-sunset-lora`) | Car cruising a coastal highway at golden hour |
| **Rainy City Night** | Film grain (`artificialguybr/filmgrain-redmond`) | Walking rain-soaked city streets after dark |
| **Cyberpunk** | Cyberpunk 2077 (`jbilcke-hf/sdxl-cyberpunk-2077`) | Neon-drenched futuristic megacity at night |
| **Watercolour Harbour** | Watercolor (`ostris/watercolor_style_lora_sdxl`) | Coastal fishing village during a storm |

## Assembly Features

- **Dynamic pacing**: 4-beat cuts before the drop, 2-beat cuts after for energy
- **Clip shuffling**: Each clip used twice (first/second half) in randomised order for visual variety
- **Ken Burns**: Alternating zoom in/out on every cut
- **Lyrics overlay**: Word-level timing with gap closing
- **Cover art overlay**: Album art + Spotify badge appear from the drop onwards
- **Reshuffle**: Re-run assembly with a new random clip order without regenerating

## Tech Stack

| Component | Tool |
|-----------|------|
| Stem separation | LALAL.AI API (Andromeda) |
| Lyrics (ASR) | WhisperX (large-v2 + wav2vec2) |
| Beat detection | madmom (RNN + DBN) |
| Prompt generation | Claude Sonnet 4.6 (Anthropic API) |
| Image generation | SDXL + Hyper-SD 8-step + style LoRA |
| Image-to-video | Wan 2.1 14B (ZeroGPU with FP8) |
| Video assembly | FFmpeg |
| UI | Gradio |