ResembleAI
/

Dramabox

@@ -22,132 +22,162 @@ base_model_relation: finetune
 > **Built on [LTX-2](https://github.com/Lightricks/LTX-2) by Lightricks.**
 > Dramabox is **Resemble AI's** expressive TTS, trained on top of the LTX-2.3 audio branch under the LTX-2 Community License. Huge thanks to the Lightricks team for open-sourcing the base.
-Dramabox generates expressive, emotionally rich speech from scene descriptions with optional voice cloning. It is an IC-LoRA fine-tune of the LTX-2.3 audio-only branch — a 3.3B Diffusion Transformer with flow matching, conditioned on Gemma 3 12B text embeddings.
-## Audio Samples
-### Regal Queen - Cold Fury to Venomous Whisper
-**Prompt:** A regal woman speaks with cold fury in a measured, low voice. She sighs deeply, "I have told you a thousand times, and yet here we are again." Her voice sharpens with rising anger, "Do you honestly think I enjoy repeating myself?! Do you?!" She lets out a cold, mocking laugh, "Hahaha, how utterly pathetic you are." She drops to a venomous whisper, leaning close, "Now get out of my sight before I do something we will both regret."
-<audio controls src="https://huggingface.co/ResembleAI/Dramabox/resolve/main/samples/01_queen_sighs_rage.wav"></audio>
-### Catgirl - Uncontrollable Giggling
-**Prompt:** A playful girl speaks in a bright, singsong voice, already mid-giggle, "Hehehe, oh my gosh you should see your face right now, it is priceless!" She gasps for air between giggles, "Oh my, hehe, oh my, I cannot stop laughing!" She tries to compose herself with a long sigh, "Ahhhhh okay okay okay, I will stop, I promise I will stop." She leans in and whispers conspiratorially, "But seriously though, between you and me," then immediately loses it again, "Haha, no I, hehehe, I just cannot! You are way too funny, haha!" She snorts mid-laugh, "Pfft, oh no no no, that was so embarrassing, pretend you did not hear that!"
-<audio controls src="https://huggingface.co/ResembleAI/Dramabox/resolve/main/samples/04_catgirl_giggles_snort.wav"></audio>
-### Action Hero - Panting Triumph
-**Prompt:** A muscular man speaks with a thick accent, panting heavily, completely out of breath, "Hah... hah... we made it, we actually made it." He coughs roughly, "Ugh, that was the hardest fight of my entire life, I swear." He groans and clutches his side, "Argh, my ribs, I think something is broken." But then a grin spreads and he laughs heartily despite the pain, "Hahaha! But we WON! Can you believe it? We actually won!" He takes a deep, shuddering breath, "I told you, heh, I told you we would make it. Ahhh, it is finally over."
-<audio controls src="https://huggingface.co/ResembleAI/Dramabox/resolve/main/samples/06_arnie_panting_triumph.wav"></audio>
-### Villain - Sinister Laugh
-**Prompt:** A deep-voiced villain speaks with theatrical menace, chuckling softly at first, "Heheheh. Hahahahahahaha! Oh, forgive me, forgive me." He catches his breath with a sinister grin, He clears his throat. "It is just SO amusing when they struggle, is it not?" His voice drips with contempt, "I expected more from you, truly I did. How disappointing." He leans in close and whispers with vicious intensity, "But fear not, my dear. The REAL entertainment has only just begun." He chuckles one last time, "Heheheh."
-<audio controls src="https://huggingface.co/ResembleAI/Dramabox/resolve/main/samples/09_villain_sinister_laugh.wav"></audio>
-### Talk Show Host - Wheezing Laughter
-**Prompt:** A talk show host speaks with animated enthusiasm. He gasps with exaggerated shock, "No! You did NOT just say that, tell me you did not just say that!" He bursts into uncontrollable laughter, "HAHAHA! Oh my god, oh my god!" He wheezes, barely getting words out, "I cannot, I literally cannot breathe right now!" He wipes his eyes, sniffling, "Oh that is so good, that is really genuinely good." He sighs happily, "Ahhh okay okay, let me compose myself, I am a professional." He takes one breath then immediately cracks up again, "Pfft hehehe, no I absolutely cannot, I am so sorry everybody!" He claps, "Folks, THIS, this right here, is why I love my job!"
-<audio controls src="https://huggingface.co/ResembleAI/Dramabox/resolve/main/samples/13_conan_wheezing_laughter.wav"></audio>
----
-## Model Description
-Dramabox is a prompt-driven TTS model where **the text prompt controls everything** - speaker identity, emotion, delivery style, laughs, sighs, pauses, and transitions. With voice cloning, a 10-second reference clip conditions the model to reproduce the speaker's timbre and characteristics.
-### Key Features
-- **Prompt-driven expressiveness** - laughs, sighs, whispers, shouts, emotional transitions all controlled by the scene description
-- **Voice cloning** from 10s reference audio
-- **English** speech synthesis
-- **Fast inference** - ~2.5s per generation with warm server on H100
-### Architecture
-| Component | Details |
-|-----------|---------|
-| **Transformer** | 3.3B parameter DiT, 48 layers, flow matching (30-step Euler) |
-| **Text Encoder** | Gemma 3 12B (q4 quantized) + learned embeddings processor |
-| **Audio VAE** | Encodes/decodes 48kHz audio via mel spectrogram latents |
-| **Voice Cloning** | Reference audio tokens appended to target with asymmetric attention mask |
-## Files
-| File | Size | Description |
-|------|------|-------------|
-| `dramabox-dit-v1.safetensors` | 6.6 GB | DiT transformer (voice cloning weights merged) |
-| `dramabox-audio-components.safetensors` | 2.7 GB | Audio VAE encoder/decoder + vocoder + text projection |
-| `assets/silence_latent_frame.pt` | 1.5 KB | VAE-encoded silence frame |
-| `config.json` | - | Model configuration |
-**Additional requirement**: [unsloth/gemma-3-12b-it-bnb-4bit](https://huggingface.co/unsloth/gemma-3-12b-it-bnb-4bit) (text encoder, pre-quantized 4-bit, auto-downloaded)
-## Quick Start
-```python
-from inference_server import TTSServer
-# Models auto-download from HuggingFace
-server = TTSServer(device="cuda", bnb_4bit=True)
-# Text-to-speech
-server.generate_to_file(
-    prompt='A woman speaks warmly, "Hello, how are you today?" She laughs, "Hahaha, it is so good to see you!"',
-    output="output.wav",
-)
-# Voice cloning
-server.generate_to_file(
-    prompt='A woman speaks warmly, "Hello, how are you today?"',
-    output="cloned.wav",
-    voice_ref="reference.wav",  # 10+ seconds of target voice
-)
-```
-## Prompt Format
-The prompt is a scene description that controls how the model speaks:
-```
-<speaker description>, "<dialogue>" <action direction> "<more dialogue>"
-```
-### What Works Inside Quotes (model produces actual sounds)
-- Laughs: `"Hahaha"` `"Hehehe"` (always as one word, never separated)
-- Sounds: `"Mmmmm"` `"Ugh"` `"Argh"` `"Ahhh"` `"Hmm"`
-### What Goes Outside Quotes (stage directions)
-- `She sighs deeply.` `He gulps nervously.` `A long pause.`
-- `Her voice cracks.` `He clears his throat.` `She scoffs.`
-### Never Inside Quotes (model speaks them literally)
-- Ahem, Pfft, Sigh, Gasp, Cough
-## Inference Settings
-| Parameter | Default | Notes |
-|-----------|---------|-------|
-| cfg_scale | 2.5 | Text adherence (lower = more natural) |
-| stg_scale | 1.5 | Skip-token guidance |
-| rescale | 0.0 | No rescaling |
-| modality | 1.0 | No modality guidance |
-| duration_multiplier | 1.1 | 10% extra breathing room |
-| steps | 30 | Euler flow matching |
-## VRAM Requirements
-| Setup | VRAM | Speed |
-|-------|------|-------|
-| Warm server (recommended) | **~24 GB** | **~2.5s** |
-| Cold inference (per-call loading) | ~8 GB peak | ~30s |
 ## License & acknowledgement
-Dramabox is a Resemble AI fine-tune of [LTX-2](https://github.com/Lightricks/LTX-2). Distributed under the LTX-2 Community License Agreement — see [`LICENSE`](https://huggingface.co/ResembleAI/Dramabox/blob/main/LICENSE). Thanks again to Lightricks for releasing the base model.

 > **Built on [LTX-2](https://github.com/Lightricks/LTX-2) by Lightricks.**
 > Dramabox is **Resemble AI's** expressive TTS, trained on top of the LTX-2.3 audio branch under the LTX-2 Community License. Huge thanks to the Lightricks team for open-sourcing the base.
+Dramabox is a prompt-driven TTS where **the prompt itself controls everything** — speaker identity, emotion, delivery, laughs, sighs, breaths, pauses, transitions. An optional 10-second voice reference clones the target timbre. It is an IC-LoRA fine-tune of the **LTX-2.3 3.3B audio-only** model (Diffusion Transformer + flow matching), conditioned on Gemma 3 12B text embeddings.
+| | |
+|---|---|
+| 🤗 Model | [`ResembleAI/Dramabox`](https://huggingface.co/ResembleAI/Dramabox) |
+| 🎭 Demo Space | [`ResembleAI/Dramabox`](https://huggingface.co/spaces/ResembleAI/Dramabox) (ZeroGPU) |
+| 🎚️ Sample audio | [`ResembleAI/Dramabox-samples`](https://huggingface.co/datasets/ResembleAI/Dramabox-samples) |
+| 🏗️ Base model | [`Lightricks/LTX-2`](https://huggingface.co/Lightricks/LTX-2) |
+| 📜 License | LTX-2 Community License — see [LICENSE](https://huggingface.co/ResembleAI/Dramabox/blob/main/LICENSE) |
+## Quick start
+### Python (warm server — recommended, ~2.5 s / generation)
+```python
+from src.inference_server import TTSServer
+server = TTSServer(device="cuda")              # downloads weights on first run
+server.generate_to_file(
+    prompt='A woman speaks warmly, "Hello, how are you today?" '
+           'She laughs, "Hahaha, it is so good to see you!"',
+    output="output.wav",
+    voice_ref="reference.wav",                  # optional, 10+ seconds of target voice
+    cfg_scale=2.5,
+    stg_scale=1.5,
+    duration_multiplier=1.1,
+    seed=42,
+)
+```
+### CLI
+```bash
+python src/inference.py \
+    --prompt 'A woman speaks warmly, "Hello, how are you today?"' \
+    --voice-sample reference.wav \
+    --output output.wav \
+    --cfg-scale 2.5 --stg-scale 1.5
+```
+For **long-form** generation (>20 s music or multi-section scenes), set the explicit target duration:
+```bash
+python src/inference.py --gen-duration 30 --prompt '...'   # CLI
+# or in Python:
+server.generate_to_file(prompt=..., gen_duration=30.0)
+```
+## Inference parameters
+Every knob exposed by the warm server, the CLI, and the Gradio Space sliders:
+| Parameter | Default | What it does |
+|---|---|---|
+| `prompt` | — | The scene description. Dialogue inside `"double quotes"`, stage directions outside. See "Prompt format" below. |
+| `voice_ref` (`--voice-sample`) | `None` | Optional 10+ s audio clip whose timbre the model clones. Without it, the model picks a voice that fits the description. |
+| `cfg_scale` | 2.5 | Classifier-free guidance — how strictly the output follows the prompt. Lower = more natural, higher = more text-faithful but more dramatic. Auto-rescaled internally to prevent clipping at high cfg (see *Auto rescale* below). |
+| `stg_scale` | 1.5 | Skip-token guidance — applied through the perturbed transformer block path (block 29). Increases expressive emphasis without saturating like cfg. |
+| `duration_multiplier` (`--duration-multiplier`) | 1.1 | Multiplier on the auto-estimated speech length (10 % breathing-room headroom). Only used when `gen_duration` (or `--gen-duration`) is 0. |
+| `gen_duration` (`--gen-duration`, "Target duration" slider) | 0 (auto) | Explicit output duration in seconds. Set to 20–60 s for music or long scenes. Overrides the prompt-based estimate when > 0. |
+| `ref_duration` (`--ref-duration`, "Reference duration" slider) | 10.0 | How many seconds of the voice reference the model conditions on (3–30 s). Longer ref → richer timbre capture, shorter ref → faster encode. |
+| `seed` | 42 | Reproducibility. |
+| `rescale_scale` (`--rescale-scale`) | `"auto"` | Latent-side CFG std-rescale. The default is a cfg-aware schedule (0 below cfg=2, ramping to 1.0 by cfg=10) that keeps the output peak below 0 dBFS at every cfg. Pass any float in [0, 1] to override or 0 to disable. |
+| `watermark` (`--no-watermark` to disable) | `True` | Apply [Resemble Perth](https://github.com/resemble-ai/Perth) imperceptible neural watermark to the output. Survives MP3/AAC, common edits; ≈ 100 % detection accuracy. |
+### Auto rescale (CFG safety)
+CFG amplifies the latent (`pred = cond + (cfg-1)·(cond - uncond)`). With no compensation, outputs hard-clip at `cfg ≥ 3`. Dramabox automatically applies a CFG-aware std-rescale schedule:
+| cfg | auto rescale | output peak |
+|---|---|---|
+| ≤ 2 | 0.0 (disabled) | safely below 0 dBFS |
+| 3 | 0.6 | ~−1.8 dBFS |
+| 4–8 | 0.8 | ~−1 to −3 dBFS |
+| 9 | 0.9 | ~−2.7 dBFS |
+| 10 | 1.0 | ~−4.4 dBFS |
+No clipping at any CFG, no manual tuning needed. Pass `rescale_scale=<float>` to override.
+### End-of-clip silence patch (long-form safety)
+The base LTX-2.3 DiT was trained on audio ≤ ~20 s and learned a strong end-of-clip silence prior at the next patchifier-aligned latent boundary (frame 513 ≈ 20.4 s). Dramabox automatically interpolates frames 512–513 from their neighbours before VAE decode whenever the output crosses 20.5 s — eliminating the ~30 ms silence dip that would otherwise show up in long generations. No flag, no override needed.
+## Prompt format
+```
+<speaker description>, "<dialogue>" <action direction> "<more dialogue>"
+```
+**Inside double quotes** — the model speaks these literally:
+- Dialogue: `"Hello, how are you?"`
+- Phonetic vocalisations (one word, no separators): `"Hahaha"`, `"Hehehe"`, `"Mmmmm"`, `"Ugh"`, `"Argh"`, `"Hmm"`
+**Outside quotes** — stage directions interpreted as performance cues, never spoken:
+- `She sighs deeply.` · `He clears his throat.` · `A long pause.` · `Her voice cracks.` · `He gulps nervously.`
+**Avoid inside quotes** (the model will speak the word literally): `Sigh`, `Gasp`, `Cough`, `Ahem`, `Pfft`.
+## Sample outputs
+Each clip below was generated by passing the prompt + voice ref to the warm server with default settings (cfg = 2.5, stg = 1.5).
+### Regal Queen — Cold Fury to Venomous Whisper
+> A regal woman speaks with cold fury in a measured, low voice. She sighs deeply, "I have told you a thousand times, and yet here we are again." Her voice sharpens with rising anger, "Do you honestly think I enjoy repeating myself?! Do you?!" She lets out a cold, mocking laugh, "Hahaha, how utterly pathetic you are." She drops to a venomous whisper, leaning close, "Now get out of my sight before I do something we will both regret."
+<audio controls src="https://huggingface.co/datasets/ResembleAI/Dramabox-samples/resolve/main/01_queen_sighs_rage.wav"></audio>
+### Catgirl — Uncontrollable Giggling
+> A playful girl speaks in a bright, singsong voice, already mid-giggle, "Hehehe, oh my gosh you should see your face right now, it is priceless!" She gasps for air between giggles, "Oh my, hehe, oh my, I cannot stop laughing!" She tries to compose herself with a long sigh, "Ahhhhh okay okay okay, I will stop, I promise I will stop." She leans in and whispers conspiratorially, "But seriously though, between you and me," then immediately loses it again, "Haha, no I, hehehe, I just cannot! You are way too funny, haha!" She snorts mid-laugh, "Pfft, oh no no no, that was so embarrassing, pretend you did not hear that!"
+<audio controls src="https://huggingface.co/datasets/ResembleAI/Dramabox-samples/resolve/main/04_catgirl_giggles_snort.wav"></audio>
+### Action Hero — Panting Triumph
+> A muscular man speaks with a thick accent, panting heavily, completely out of breath, "Hah... hah... we made it, we actually made it." He coughs roughly, "Ugh, that was the hardest fight of my entire life, I swear." He groans and clutches his side, "Argh, my ribs, I think something is broken." But then a grin spreads and he laughs heartily despite the pain, "Hahaha! But we WON! Can you believe it? We actually won!" He takes a deep, shuddering breath, "I told you, heh, I told you we would make it. Ahhh, it is finally over."
+<audio controls src="https://huggingface.co/datasets/ResembleAI/Dramabox-samples/resolve/main/06_arnie_panting_triumph.wav"></audio>
+### Villain — Sinister Laugh
+> A deep-voiced villain speaks with theatrical menace, chuckling softly at first, "Heh heh heh, ha ha ha ha ha! Oh, forgive me, forgive me." He catches his breath with a sinister grin, He clears his throat. "It is just SO amusing when they struggle, is it not?" His voice drips with contempt, "I expected more from you, truly I did. How disappointing." He leans in close and whispers with vicious intensity, "But fear not, my dear. The REAL entertainment has only just begun." He chuckles one last time, "Heh heh heh."
+<audio controls src="https://huggingface.co/datasets/ResembleAI/Dramabox-samples/resolve/main/09_villain_sinister_laugh.wav"></audio>
+### Talk Show Host — Wheezing Laughter
+> A talk show host speaks with animated enthusiasm. He gasps with exaggerated shock, "No! You did NOT just say that, tell me you did not just say that!" He bursts into uncontrollable laughter, "HAHAHA! Oh my god, oh my god!" He wheezes, barely getting words out, "I cannot, I literally cannot breathe right now!" He wipes his eyes, sniffling, "Oh that is so good, that is really genuinely good." He sighs happily, "Ahhh okay okay, let me compose myself, I am a professional." He takes one breath then immediately cracks up again, "Pfft hehehe, no I absolutely cannot, I am so sorry everybody!" He claps, "Folks, THIS, this right here, is why I love my job!"
+<audio controls src="https://huggingface.co/datasets/ResembleAI/Dramabox-samples/resolve/main/13_conan_wheezing_laughter.wav"></audio>
+## Files
+| File | Size | Contents |
+|---|---|---|
+| `dramabox-dit-v1.safetensors` | 6.6 GB | Audio-only DiT (LoRA already merged into base) |
+| `dramabox-audio-components.safetensors` | 1.9 GB | Audio embeddings connector + audio text projection + audio VAE + vocoder |
+| [`unsloth/gemma-3-12b-it-bnb-4bit`](https://huggingface.co/unsloth/gemma-3-12b-it-bnb-4bit) | ~8 GB | Text encoder (auto-downloaded on first run) |
+**VRAM**: ~24 GB peak, warm server. **Speed**: ~2.5 s / generation on H100 once warm.
+## Watermarking
+Every output of `inference.py` and `TTSServer.generate_to_file` is automatically watermarked with [Resemble Perth](https://github.com/resemble-ai/Perth) — an imperceptible neural watermark that survives MP3 compression, audio editing, and common manipulations while maintaining nearly 100 % detection accuracy.
+```python
+import perth, librosa
+wav, sr = librosa.load("output.wav", sr=None, mono=True)
+detector = perth.PerthImplicitWatermarker()
+print(detector.get_watermark(wav, sample_rate=sr))   # ≈ 1.0 for our outputs
+```
+Pass `--no-watermark` (CLI) or `watermark=False` (Python) to disable for debugging.
 ## License & acknowledgement
+Dramabox is a Resemble AI fine-tune of [LTX-2](https://github.com/Lightricks/LTX-2). Distributed under the LTX-2 Community License Agreement — see [LICENSE](https://huggingface.co/ResembleAI/Dramabox/blob/main/LICENSE). Thanks again to Lightricks for releasing the base model.