Text-to-Audio
LTX.io
English
ltx-audio
audio
audio-generation
speech
reference-conditioning
multi-speaker
flow-matching
diffusion
Instructions to use mifinkelson/scena with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LTX.io
How to use mifinkelson/scena with LTX.io:
# Install the LTX-2 pipelines git clone https://github.com/Lightricks/LTX-2.git cd LTX-2 uv sync --frozen
# Download the weights from this repo, plus the Gemma text encoder hf download mifinkelson/scena --local-dir models/scena hf download google/gemma-3-12b-it-qat-q4_0-unquantized --local-dir models/gemma-3-12b
# Fast pipeline (distilled model, no distilled LoRA needed) uv run python -m ltx_pipelines.distilled \ --distilled-checkpoint-path models/scena/<distilled-checkpoint>.safetensors \ --spatial-upsampler-path models/scena/<spatial-upsampler>.safetensors \ --gemma-root models/gemma-3-12b \ --prompt "A beautiful sunset over the ocean" \ --output-path output.mp4 # For image-to-video, add: --image path/to/image.jpg 0 0.8# HQ pipeline (two-stage, higher quality) uv run python -m ltx_pipelines.ti2vid_two_stages_hq \ --checkpoint-path models/scena/<checkpoint>.safetensors \ --distilled-lora models/scena/<distilled-lora>.safetensors 0.8 \ --spatial-upsampler-path models/scena/<spatial-upsampler>.safetensors \ --gemma-root models/gemma-3-12b \ --prompt "A beautiful sunset over the ocean" \ --output-path output.mp4 # For image-to-video, add: --image path/to/image.jpg 0 0.8 - Notebooks
- Google Colab
- Kaggle
Update model card
Browse files
README.md
CHANGED
|
@@ -64,18 +64,18 @@ pipe = T2AudRefCondPipeline(
|
|
| 64 |
|
| 65 |
refs = ["examples/references/reference_1.wav", "examples/references/reference_2.wav"]
|
| 66 |
|
| 67 |
-
# Game-show buzzer
|
| 68 |
pipe(
|
| 69 |
prompt='The speaker from reference 1 fires: "Capital of France?" The speaker from reference 2 instantly: "Paris!" The speaker from reference 1: "Square root of eighty-one?" The speaker from reference 2: "Nine!" The speaker from reference 1: "Year of moon landing?" The speaker from reference 2: "Sixty-nine!" A loud, harsh game-show buzzer rings out — a sustained electronic BZZZZT.',
|
| 70 |
ref_audio_paths=refs,
|
| 71 |
-
duration=8.
|
| 72 |
).save("buzzer.wav")
|
| 73 |
|
| 74 |
-
# Farm at sunrise
|
| 75 |
pipe(
|
| 76 |
prompt='A farm at sunrise: a rooster crows. Chickens cluck softly throughout. The speaker from reference 1 says with a yawn: "Way too early for this." The speaker from reference 2 chuckles: "Welcome to country life." The rooster crows again.',
|
| 77 |
ref_audio_paths=refs,
|
| 78 |
-
duration=6.
|
| 79 |
).save("farm.wav")
|
| 80 |
```
|
| 81 |
|
|
@@ -84,8 +84,9 @@ pipe(
|
|
| 84 |
order of `ref_audio_paths`.
|
| 85 |
- Put the spoken words **in quotes**; describe sound effects / ambience in plain prose
|
| 86 |
(e.g. *"a loud, harsh game-show buzzer rings out"*, *"a roaring stadium crowd cheers"*).
|
| 87 |
-
- Reference clips
|
| 88 |
-
-
|
|
|
|
| 89 |
|
| 90 |
## Citation
|
| 91 |
|
|
|
|
| 64 |
|
| 65 |
refs = ["examples/references/reference_1.wav", "examples/references/reference_2.wav"]
|
| 66 |
|
| 67 |
+
# Game-show buzzer
|
| 68 |
pipe(
|
| 69 |
prompt='The speaker from reference 1 fires: "Capital of France?" The speaker from reference 2 instantly: "Paris!" The speaker from reference 1: "Square root of eighty-one?" The speaker from reference 2: "Nine!" The speaker from reference 1: "Year of moon landing?" The speaker from reference 2: "Sixty-nine!" A loud, harsh game-show buzzer rings out — a sustained electronic BZZZZT.',
|
| 70 |
ref_audio_paths=refs,
|
| 71 |
+
duration=8.5,
|
| 72 |
).save("buzzer.wav")
|
| 73 |
|
| 74 |
+
# Farm at sunrise — same two voices, a different scene
|
| 75 |
pipe(
|
| 76 |
prompt='A farm at sunrise: a rooster crows. Chickens cluck softly throughout. The speaker from reference 1 says with a yawn: "Way too early for this." The speaker from reference 2 chuckles: "Welcome to country life." The rooster crows again.',
|
| 77 |
ref_audio_paths=refs,
|
| 78 |
+
duration=6.5,
|
| 79 |
).save("farm.wav")
|
| 80 |
```
|
| 81 |
|
|
|
|
| 84 |
order of `ref_audio_paths`.
|
| 85 |
- Put the spoken words **in quotes**; describe sound effects / ambience in plain prose
|
| 86 |
(e.g. *"a loud, harsh game-show buzzer rings out"*, *"a roaring stadium crowd cheers"*).
|
| 87 |
+
- Reference clips: clean single-speaker speech, up to ~20 s each — the more the better (any sample rate; mono or stereo).
|
| 88 |
+
- You don't have to follow a strict turn-taking pattern; see the [demo page](https://finmickey.github.io/scena/) for more varied examples.
|
| 89 |
+
- 60 inference steps, guidance ~7, and output durations up to ~20 s work best (the model was trained on ≤20 s scenes).
|
| 90 |
|
| 91 |
## Citation
|
| 92 |
|