Duplicate from mifinkelson/scena

Co-authored-by: Michael Finkelson <mifinkelson@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,105 @@
+---
+license: other
+license_name: ltx-2-community-license
+license_link: https://huggingface.co/Lightricks/LTX-2/blob/main/LICENSE
+language:
+- en
+tags:
+- audio
+- text-to-audio
+- audio-generation
+- speech
+- reference-conditioning
+- multi-speaker
+- flow-matching
+- diffusion
+- ltx
+library_name: ltx-audio
+pipeline_tag: text-to-audio
+---
+
+# ScenA — Reference-Driven Multi-Speaker Audio Scene Generation
+
+**ScenA** generates multi-speaker audio *scenes* (dialogue, conversation) from a text
+prompt, conditioned on one or more **reference-audio** clips that set the speakers'
+voices. It is an audio-only, reference-conditioned text-to-audio flow-matching DiT
+built on the LTX-2 architecture (~4B parameters, 48 layers).
+
+- 📄 **Paper:** [arXiv:2606.19325](https://arxiv.org/abs/2606.19325)
+- 🌐 **Project page:** https://finmickey.github.io/scena/
+- 💻 **Code:** https://github.com/finmickey/scena
+- 🏢 Lightricks · Tel Aviv University
+
+## Files
+
+| file | size | contents |
+|---|---|---|
+| `scena.safetensors` | ~8.2 GB | the ScenA transformer + Gemma text-projection weights |
+| `audio_vae.safetensors` | ~365 MB | the audio VAE (encoder+decoder) + vocoder (bundled so you don't need the full LTX-2 model) |
+
+You also need the **Gemma-3-12B-IT** text encoder (`google/gemma-3-12b-it`, gated on HF).
+
+## Quickstart
+
+```bash
+git clone https://github.com/finmickey/scena
+cd scena
+pip install -e .   # or: uv sync
+
+# download the checkpoint + audio VAE bundle
+huggingface-cli download mifinkelson/scena scena.safetensors audio_vae.safetensors --local-dir ./checkpoints
+```
+
+Two example reference voices ship with the repo under `examples/references/` — the same two voices
+drive both examples below, so this runs out of the box:
+
+```python
+from ltx_pipelines.t2aud_ref_cond import T2AudRefCondPipeline
+
+pipe = T2AudRefCondPipeline(
+    checkpoint_path="checkpoints/scena.safetensors",
+    audio_vae_path="checkpoints/audio_vae.safetensors",
+    gemma_root="/path/to/google/gemma-3-12b-it",
+)
+
+refs = ["examples/references/reference_1.wav", "examples/references/reference_2.wav"]
+
+# A simple two-speaker dialogue
+pipe(
+    prompt='The speaker from reference 1 says: "The taxi drivers are on strike again." The speaker from reference 2 says: "What for?" The speaker from reference 1 says: "They want the government to reduce the price of the gasoline." The speaker from reference 2 says: "It is really a hot potato."',
+    ref_audio_paths=refs,
+    duration=7.0,
+    seed=1,
+).save("dialogue.wav")
+
+# Farm at sunrise — the same two voices in a scene with sound effects
+pipe(
+    prompt='A farm at sunrise: a rooster crows. Chickens cluck softly throughout. The speaker from reference 1 says with a yawn: "Way too early for this." The speaker from reference 2 chuckles: "Welcome to country life." The rooster crows again.',
+    ref_audio_paths=refs,
+    duration=8.0,
+    seed=1,
+).save("farm.wav")
+```
+
+### Prompting guide
+- Refer to speakers as **"the speaker from reference 1"**, **"reference 2"**, … matching the
+  order of `ref_audio_paths`.
+- Put the spoken words **in quotes**; describe sound effects / ambience in plain prose
+  (e.g. *"rain drums steadily on a tin roof"*, *"a roaring stadium crowd cheers"*).
+- Reference clips: clean single-speaker speech, up to ~20 s each — the more the better (any sample rate; mono or stereo).
+- You don't have to follow a strict turn-taking pattern; see the [demo page](https://finmickey.github.io/scena/) for more varied examples.
+- 60 inference steps, guidance ~7, and output durations up to ~20 s work best (the model was trained on ≤20 s scenes).
+
+## Citation
+
+```bibtex
+@article{finkelson2026scena,
+  title   = {Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors},
+  author  = {Finkelson, Michael and Segal, Daniel and Richardson, Eitan and Armon, Shahar and Goldring, Nani and Panet, Poriya and Zabari, Nir and Brazowski, Benjamin and Patashnik, Or and HaCohen, Yoav},
+  journal = {arXiv preprint arXiv:2606.19325},
+  year    = {2026}
+}
+```
+
+## License
+Released under the [LTX-2 Community License](https://huggingface.co/Lightricks/LTX-2/blob/main/LICENSE).

audio_vae.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1f536f192420e1abd936b5ff10ea0ef7204058db2e1e79a865ab36e06b29847
+size 364831364

scena.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3cb54ca58aa61eae5823cca01563463dd096b8fa39352f9a02f7232dfabadd35
+size 8153188488