AURA-5B: AUdio-dRiven streaming Avatar
AURA turns a single portrait image and a speech audio stream into a talking-head video, chunk by chunk, in real time. It is a 4-step causal student distilled from the 50-step bidirectional TalkVerse teacher. It uses a rolling KV cache and needs only the current chunk's audio, with no multi-second look-ahead.
Project page · Paper · Code · AURA dataset
Nathaniel Cohen1,2, Nicolas Dufour1, Amélie Royer1, Alasdair Newson2, Patrick Pérez3* 1Kyutai · 2Sorbonne Université, ISIR · 3Google DeepMind · *work done while at Kyutai BMVC 2026
Samples
Generated by AURA-5B from the same speech with three different reference images and prompts. More samples are on the project page.
Highlights
- 9.7 fps end-to-end at 720p on a single H100 GPU (paper, optimized pipeline): 19.7 fps for the DiT and 22.7 fps for the VAE
- 17× faster than the teacher end-to-end, and 30× faster on the diffusion transformer alone
- 4 denoising steps per chunk, where the teacher uses 50
- Constant memory streaming: the rolling cache keeps the first chunk as an identity anchor plus the most recent chunk, so 60 s+ streams are possible
Model details
| Base model | TalkVerse-S2V-5B (Snap Inc.), itself built on Wan2.2-TI2V-5B |
| Teacher | TalkVerse-S2V-5B (50-step, bidirectional) |
| Format | Each checkpoint comes in two forms. *_merged: standalone DiT, fp32 safetensors (5.7B parameters, 22.8 GB). *_lora_only: the raw training checkpoint, LoRA (rank 128, alpha 128) + trained modules, to apply on the TalkVerse 5B base |
| Attention | Block-wise causal over chunks of 4 latent frames (≈0.64 s) |
| Audio encoder | wav2vec2-large-xlsr-53-english |
| Inference | 4 denoising steps / chunk, rolling KV cache |
Files
LoRA-only and merged checkpoints. Every training stage fine-tunes LoRA adapters (rank 128, alpha 128) plus a few non-LoRA modules (audio and output modules). We release each checkpoint in both forms: *_lora_only is the raw training checkpoint (LoRA updates and trained modules only, to apply on top of the TalkVerse-S2V-5B weights and, for stage 2, the stage-1 LoRA), and *_merged is a standalone model with all LoRA updates merged in.
| File | Description |
|---|---|
stage2_dmd/step-5000_merged.safetensors |
Final AURA-5B model (the paper checkpoint): stage 2, DMD, with the stage-1 and DMD LoRAs merged into the TalkVerse 5B weights. Use this one. |
stage2_dmd/step-5000_lora_only.safetensors |
Same, LoRA-only form. Apply on TalkVerse 5B → stage1_ode/step-121000_lora_only → this |
stage1_ode/step-121000_merged.safetensors |
Stage-1 ODE-distilled student (before DMD), standalone |
stage1_ode/step-121000_lora_only.safetensors |
Same, LoRA-only form (apply on TalkVerse 5B → this) |
{stage2_dmd,stage1_ode}/config.json |
DiT config (WanModel, ti2v) |
{stage2_dmd,stage1_ode}/launch_config.json |
Training arguments, launch command, seed and code commit |
{stage2_dmd,stage1_ode}/step-*.metrics.jsonl |
Training metrics (loss, val/critic losses, LR…) for that checkpoint, one JSON record per logged step up to the checkpoint step; resumed launches stitched together |
Training
Three changes turn the teacher into a causal student: block-wise causal attention over 4-latent-frame chunks, no motion-frame conditioning, and per-chunk mixed diffusion timesteps.
- Stage 1: ODE distillation. The student regresses with an MSE loss onto teacher ODE trajectories (5 timesteps, 30K synthetic clips generated by TalkVerse; the released AURA dataset holds the equivalent trajectories for AURA-1.3B).
- Stage 2: DMD. Distribution Matching Distillation with backward simulation trains on the student's own rollout states and closes the remaining quality gap with the teacher.
Both stages fine-tune LoRA adapters (rank 128) on 64 H100 GPUs. The exact hyperparameters are in */launch_config.json.
Usage
git clone https://github.com/kyutai-labs/aura && cd aura
uv sync
uv run generate.py --model aura-5b --image portrait.jpg --audio speech.wav --output out.mp4
The script downloads everything it needs from the Hub: stage2_dmd/step-5000_merged.safetensors from this repo (the full DiT, no LoRA to apply), the T5 text encoder and VAE from Wan2.2-TI2V-5B, and wav2vec2-large-xlsr-53-english for audio features.
It samples 4 steps per chunk (timesteps 1000, 745, 498, 255) without classifier-free guidance, at 25 fps, with a rolling KV cache (anchor chunk + latest chunk). Other options: --prompt, --num_chunks, --seed, --compile.
With --compile, this reference code runs at about 8 fps on one H100; the 9.7 fps reported in the paper come from an optimized pipeline.
The other checkpoints (stage1_ode/*, *_lora_only) are released for research and are not loaded by the reference code.
Evaluation
| Benchmark | FID ↓ | FVD ↓ | JEDi ↓ | CSIM ↑ | Sync-C ↑ | Sync-D ↓ | WE ↓ |
|---|---|---|---|---|---|---|---|
| HALLO3 test (100 clips, 10 s) | 56.9 | 195.5 | 0.313 | 0.79 | 4.30 | 9.87 | 0.0032 |
| CelebV-HQ test (150 clips) | 54.0 | 208.1 | 0.320 | 0.72 | 4.94 | 10.0 | 0.0043 |
In a user study (400 pairwise comparisons from 27 raters on HALLO3 test), AURA-5B is preferred over Hallo3 on all four axes (visual quality, lip-sync, reference fidelity, naturalness) and is on par with or above its 50-step teacher. See the paper for the full tables and ablations.
Limitations
- One-step sampling degrades sharply. Use 4 steps.
- Not real time: 9.7 fps end-to-end at 720p on one H100, below 25 Hz playback. For real-time streaming, use AURA-1.3B.
- Lip-sync is behind the strongest multi-step baselines (Sync-C 4.30 on HALLO3 test, vs. 5.06 for Hallo3 and 5.42 for Wan-S2V).
- Trained and evaluated on talking-head videos driven by English speech; other languages, non-speech audio and non-portrait framings are untested.
License
AURA-5B starts from the TalkVerse-S2V-5B checkpoint released by Snap Inc., and every file here (merged and LoRA-only) is derived from it. The weights are therefore released under the same Snap Inc. Non-Commercial License as TalkVerse (non-commercial, research use only). Its full text, as published in the TalkVerse repository, is reproduced in LICENSE, as the license requires for any redistribution. The T5 text encoder and VAE (Wan2.2-TI2V-5B, Apache 2.0) and wav2vec2 audio encoder you download separately keep their own licenses.
Responsible use
The model generates realistic videos of real-looking people speaking arbitrary audio. Do not use it to impersonate people, create non-consensual content or produce disinformation. Only animate portraits for which you have the subject's consent, and disclose that videos are AI-generated.
Contact
For questions, contact Nathaniel Cohen (nathaniel.cohen@kyutai.org) or Amélie Royer (amelie@kyutai.org).
Citation
@inproceedings{cohen2026aura,
title = {{AURA}: {AU}dio-d{R}iven streaming {A}vatar},
author = {Cohen, Nathaniel and Dufour, Nicolas and Royer, Am{\'e}lie
and Newson, Alasdair and P{\'e}rez, Patrick},
booktitle = {British Machine Vision Conference (BMVC)},
year = {2026},
}
Model tree for kyutai/AURA-5B
Base model
snap-research/talkverse-s2v-5b