Text-to-Audio
LTX.io
English
ltx-audio
audio
audio-generation
speech
reference-conditioning
multi-speaker
flow-matching
diffusion
Instructions to use mifinkelson/scena with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LTX.io
How to use mifinkelson/scena with LTX.io:
# Install the LTX-2 pipelines git clone https://github.com/Lightricks/LTX-2.git cd LTX-2 uv sync --frozen
# Download the weights from this repo, plus the Gemma text encoder hf download mifinkelson/scena --local-dir models/scena hf download google/gemma-3-12b-it-qat-q4_0-unquantized --local-dir models/gemma-3-12b
# Fast pipeline (distilled model, no distilled LoRA needed) uv run python -m ltx_pipelines.distilled \ --distilled-checkpoint-path models/scena/<distilled-checkpoint>.safetensors \ --spatial-upsampler-path models/scena/<spatial-upsampler>.safetensors \ --gemma-root models/gemma-3-12b \ --prompt "A beautiful sunset over the ocean" \ --output-path output.mp4 # For image-to-video, add: --image path/to/image.jpg 0 0.8# HQ pipeline (two-stage, higher quality) uv run python -m ltx_pipelines.ti2vid_two_stages_hq \ --checkpoint-path models/scena/<checkpoint>.safetensors \ --distilled-lora models/scena/<distilled-lora>.safetensors 0.8 \ --spatial-upsampler-path models/scena/<spatial-upsampler>.safetensors \ --gemma-root models/gemma-3-12b \ --prompt "A beautiful sunset over the ocean" \ --output-path output.mp4 # For image-to-video, add: --image path/to/image.jpg 0 0.8 - Notebooks
- Google Colab
- Kaggle
Update model card examples
Browse files
README.md
CHANGED
|
@@ -50,8 +50,8 @@ pip install -e . # or: uv sync
|
|
| 50 |
huggingface-cli download mifinkelson/scena scena.safetensors audio_vae.safetensors --local-dir ./checkpoints
|
| 51 |
```
|
| 52 |
|
| 53 |
-
Two example reference voices
|
| 54 |
-
|
| 55 |
|
| 56 |
```python
|
| 57 |
from ltx_pipelines.t2aud_ref_cond import T2AudRefCondPipeline
|
|
@@ -64,18 +64,20 @@ pipe = T2AudRefCondPipeline(
|
|
| 64 |
|
| 65 |
refs = ["examples/references/reference_1.wav", "examples/references/reference_2.wav"]
|
| 66 |
|
| 67 |
-
#
|
| 68 |
pipe(
|
| 69 |
-
prompt='The speaker from reference 1
|
| 70 |
ref_audio_paths=refs,
|
| 71 |
-
duration=
|
| 72 |
-
|
|
|
|
| 73 |
|
| 74 |
-
# Farm at sunrise — same two voices
|
| 75 |
pipe(
|
| 76 |
prompt='A farm at sunrise: a rooster crows. Chickens cluck softly throughout. The speaker from reference 1 says with a yawn: "Way too early for this." The speaker from reference 2 chuckles: "Welcome to country life." The rooster crows again.',
|
| 77 |
ref_audio_paths=refs,
|
| 78 |
-
duration=
|
|
|
|
| 79 |
).save("farm.wav")
|
| 80 |
```
|
| 81 |
|
|
@@ -83,7 +85,7 @@ pipe(
|
|
| 83 |
- Refer to speakers as **"the speaker from reference 1"**, **"reference 2"**, … matching the
|
| 84 |
order of `ref_audio_paths`.
|
| 85 |
- Put the spoken words **in quotes**; describe sound effects / ambience in plain prose
|
| 86 |
-
(e.g. *"
|
| 87 |
- Reference clips: clean single-speaker speech, up to ~20 s each — the more the better (any sample rate; mono or stereo).
|
| 88 |
- You don't have to follow a strict turn-taking pattern; see the [demo page](https://finmickey.github.io/scena/) for more varied examples.
|
| 89 |
- 60 inference steps, guidance ~7, and output durations up to ~20 s work best (the model was trained on ≤20 s scenes).
|
|
|
|
| 50 |
huggingface-cli download mifinkelson/scena scena.safetensors audio_vae.safetensors --local-dir ./checkpoints
|
| 51 |
```
|
| 52 |
|
| 53 |
+
Two example reference voices ship with the repo under `examples/references/` — the same two voices
|
| 54 |
+
drive both examples below, so this runs out of the box:
|
| 55 |
|
| 56 |
```python
|
| 57 |
from ltx_pipelines.t2aud_ref_cond import T2AudRefCondPipeline
|
|
|
|
| 64 |
|
| 65 |
refs = ["examples/references/reference_1.wav", "examples/references/reference_2.wav"]
|
| 66 |
|
| 67 |
+
# A simple two-speaker dialogue
|
| 68 |
pipe(
|
| 69 |
+
prompt='The speaker from reference 1 says: "The taxi drivers are on strike again." The speaker from reference 2 says: "What for?" The speaker from reference 1 says: "They want the government to reduce the price of the gasoline." The speaker from reference 2 says: "It is really a hot potato."',
|
| 70 |
ref_audio_paths=refs,
|
| 71 |
+
duration=7.0,
|
| 72 |
+
seed=1,
|
| 73 |
+
).save("dialogue.wav")
|
| 74 |
|
| 75 |
+
# Farm at sunrise — the same two voices in a scene with sound effects
|
| 76 |
pipe(
|
| 77 |
prompt='A farm at sunrise: a rooster crows. Chickens cluck softly throughout. The speaker from reference 1 says with a yawn: "Way too early for this." The speaker from reference 2 chuckles: "Welcome to country life." The rooster crows again.',
|
| 78 |
ref_audio_paths=refs,
|
| 79 |
+
duration=8.0,
|
| 80 |
+
seed=1,
|
| 81 |
).save("farm.wav")
|
| 82 |
```
|
| 83 |
|
|
|
|
| 85 |
- Refer to speakers as **"the speaker from reference 1"**, **"reference 2"**, … matching the
|
| 86 |
order of `ref_audio_paths`.
|
| 87 |
- Put the spoken words **in quotes**; describe sound effects / ambience in plain prose
|
| 88 |
+
(e.g. *"rain drums steadily on a tin roof"*, *"a roaring stadium crowd cheers"*).
|
| 89 |
- Reference clips: clean single-speaker speech, up to ~20 s each — the more the better (any sample rate; mono or stereo).
|
| 90 |
- You don't have to follow a strict turn-taking pattern; see the [demo page](https://finmickey.github.io/scena/) for more varied examples.
|
| 91 |
- 60 inference steps, guidance ~7, and output durations up to ~20 s work best (the model was trained on ≤20 s scenes).
|