README.md

---
license: other
license_name: ltx-2-community-license
license_link: https://huggingface.co/Lightricks/LTX-2/blob/main/LICENSE
language:
- en
tags:
- audio
- text-to-audio
- audio-generation
- speech
- reference-conditioning
- multi-speaker
- flow-matching
- diffusion
- ltx
library_name: ltx-audio
pipeline_tag: text-to-audio
---

# ScenA — Reference-Driven Multi-Speaker Audio Scene Generation

**ScenA** generates multi-speaker audio *scenes* (dialogue, conversation) from a text
prompt, conditioned on one or more **reference-audio** clips that set the speakers'
voices. It is an audio-only, reference-conditioned text-to-audio flow-matching DiT
built on the LTX-2 architecture (~4B parameters, 48 layers).

- 📄 **Paper:** [arXiv:2606.19325](https://arxiv.org/abs/2606.19325)
- 🌐 **Project page:** https://finmickey.github.io/scena/
- 💻 **Code:** https://github.com/finmickey/scena
- 🏢 Lightricks · Tel Aviv University

## Files

| file | size | contents |
|---|---|---|
| `scena.safetensors` | ~8.2 GB | the ScenA transformer + Gemma text-projection weights |
| `audio_vae.safetensors` | ~365 MB | the audio VAE (encoder+decoder) + vocoder (bundled so you don't need the full LTX-2 model) |

You also need the **Gemma-3-12B-IT** text encoder (`google/gemma-3-12b-it`, gated on HF).

## Quickstart

```bash
git clone https://github.com/finmickey/scena
cd scena
pip install -e .   # or: uv sync

# download the checkpoint + audio VAE bundle
huggingface-cli download mifinkelson/scena scena.safetensors audio_vae.safetensors --local-dir ./checkpoints
```

Two example reference voices ship with the repo under `examples/references/` — the same two voices
drive both examples below, so this runs out of the box:

```python
from ltx_pipelines.t2aud_ref_cond import T2AudRefCondPipeline

pipe = T2AudRefCondPipeline(
    checkpoint_path="checkpoints/scena.safetensors",
    audio_vae_path="checkpoints/audio_vae.safetensors",
    gemma_root="/path/to/google/gemma-3-12b-it",
)

refs = ["examples/references/reference_1.wav", "examples/references/reference_2.wav"]

# A simple two-speaker dialogue
pipe(
    prompt='The speaker from reference 1 says: "The taxi drivers are on strike again." The speaker from reference 2 says: "What for?" The speaker from reference 1 says: "They want the government to reduce the price of the gasoline." The speaker from reference 2 says: "It is really a hot potato."',
    ref_audio_paths=refs,
    duration=7.0,
    seed=1,
).save("dialogue.wav")

# Farm at sunrise — the same two voices in a scene with sound effects
pipe(
    prompt='A farm at sunrise: a rooster crows. Chickens cluck softly throughout. The speaker from reference 1 says with a yawn: "Way too early for this." The speaker from reference 2 chuckles: "Welcome to country life." The rooster crows again.',
    ref_audio_paths=refs,
    duration=8.0,
    seed=1,
).save("farm.wav")
```

### Prompting guide
- Refer to speakers as **"the speaker from reference 1"**, **"reference 2"**, … matching the
  order of `ref_audio_paths`.
- Put the spoken words **in quotes**; describe sound effects / ambience in plain prose
  (e.g. *"rain drums steadily on a tin roof"*, *"a roaring stadium crowd cheers"*).
- Reference clips: clean single-speaker speech, up to ~20 s each — the more the better (any sample rate; mono or stereo).
- You don't have to follow a strict turn-taking pattern; see the [demo page](https://finmickey.github.io/scena/) for more varied examples.
- 60 inference steps, guidance ~7, and output durations up to ~20 s work best (the model was trained on ≤20 s scenes).

## Citation

```bibtex
@article{finkelson2026scena,
  title   = {Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors},
  author  = {Finkelson, Michael and Segal, Daniel and Richardson, Eitan and Armon, Shahar and Goldring, Nani and Panet, Poriya and Zabari, Nir and Brazowski, Benjamin and Patashnik, Or and HaCohen, Yoav},
  journal = {arXiv preprint arXiv:2606.19325},
  year    = {2026}
}
```

## License
Released under the [LTX-2 Community License](https://huggingface.co/Lightricks/LTX-2/blob/main/LICENSE).