README.md

metadata

title: NAVA Audio-Video Generator
emoji: 🎬
colorFrom: purple
colorTo: blue
sdk: gradio
sdk_version: 6.16.0
app_file: app.py
models:
  - ernie-research/NAVA
  - Qwen/Qwen3-4B-Instruct-2507
  - Qwen/Qwen3-VL-4B-Instruct
short_description: Native AV alignment — joint video + audio generation
license: apache-2.0

NAVA on ZeroGPU

Project Page · arXiv · GitHub · Models

NAVA is a Native Audio-Visual Alignment framework for joint audio + video generation: text → synchronized 24fps video + stereo audio in one shot, image-to-AV with reference-voice timbre control included. This Space runs the FP8 checkpoint (~13 GB) on a single H200 via ZeroGPU.

How it works

1. (Optional) Upload a first-frame image — the aspect ratio dropdown auto-detects portrait/landscape/square.
2. Type a short prompt — Chinese or English, brief is fine.
3. Click "Rewrite Prompt" — Qwen3-VL captions the image (if any), composes with your prompt, and Qwen3 rewrites it into the long, dense Chinese caption NAVA was trained on. Auto-retries on <S>...<E> mismatch.
4. Click "Generate" — diffusion runs in FP8 (~5 minutes per 4s clip on H200).

Speech tags

Wrap any spoken line in <S>...<E>. The rewriter preserves these verbatim (auto-retried until the count matches your input). Add reference WAVs to bind voice identity to each speech span.

Hardware

ZeroGPU H200 (80 GB). Each Generate call is allocated up to 5 minutes of GPU time. If the queue is busy, you'll wait. The model + Qwen3 weights stay resident in CPU RAM between calls; only the active stage moves to GPU.