| --- |
| license: mit |
| language: |
| - en |
| library_name: chatterbox-flash |
| pipeline_tag: text-to-speech |
| tags: |
| - text-to-speech |
| - zero-shot-tts |
| - block-diffusion |
| - speech-synthesis |
| --- |
| <img width="800" alt="Resemble_AI_Chatterbox_Flash" src="https://cdn-uploads.huggingface.co/production/uploads/68af35bc5928acf3617150db/lCegiC8uo_eA3RHOVC6go.png" /> |
|
|
| <h1 style="font-size: 32px">Chatterbox-Flash</h1> |
|
|
| <div style="display: flex; align-items: center; gap: 12px"> |
| <a href="https://huggingface.co/spaces/ResembleAI/chatterbox-flash-demo"> |
| <img src="https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm.svg" alt="Open in HF" /> |
| </a> |
| </div> |
| <div style="display: flex; align-items: center; gap: 8px;"> |
| <span style="font-style: italic;white-space: pre-wrap">Made with ❤️ by</span> |
| <img width="100" alt="resemble-logo-horizontal" src="https://github.com/user-attachments/assets/35cf756b-3506-4943-9c72-c05ddfa4e525" /> |
| </div> |
| |
| Chatterbox-Flash is a **block-diffusion zero-shot TTS model** that extends the [Chatterbox-TTS](https://github.com/resemble-ai/chatterbox) pipeline with a parallel masked decoder while preserving streaming generation. It unmasks multiple speech tokens **in parallel within each block** while keeping **native block-by-block streaming**, delivering autoregressive-class quality at a fraction of the latency. |
|
|
| # Streaming & Speed Highlights |
| - **Native block-by-block streaming** — emits audio as each block is committed, no full-sequence wait. |
| - **~9× real-time** synthesis at the default config (`D = 16`, `α = 0.5`), up to **~13× real-time** at `D = 32`, `α = 0.75`. |
| - **Time-to-first-packet from 103 ms**, on par with streaming AR systems. |
| - **RTF as low as 0.076** — substantially faster than autoregressive streaming TTS at the same scale. |
| - **Early-decoding schedule** adaptively ends denoising early, cutting average steps per block by **~20%** at negligible quality cost. |
| - Built on a **FlashInfer paged KV cache + CUDA-graph** inference engine for low per-step overhead. |
|
|
| The released weights cover the four checkpoints needed at inference time: |
|
|
| | File | What it is | |
| | --------------------------- | ------------------------------------------------------------------ | |
| | `t3_flash.safetensors` | Block-diffusion T3 decoder (Llama-520M + 1 extra `[MASK]` token). | |
| | `s3gen.safetensors` | Flow-matching S3Gen vocoder. | |
| | `ve.safetensors` | GE2E voice encoder (taken verbatim from `ResembleAI/chatterbox`). | |
| | `tokenizer.json` | English BPE tokenizer (taken verbatim from `ResembleAI/chatterbox`).| |
|
|
| # Quick start |
| ```bash |
| pip install chatterbox-flash |
| ``` |
| ```python |
| import torchaudio as ta |
| from chatterbox_flash import ChatterboxFlashTTS |
| tts = ChatterboxFlashTTS.from_pretrained("ResembleAI/chatterbox-flash", device="cuda") |
| wav = tts.generate( |
| "Hello, world.", |
| audio_prompt_path="reference.wav", |
| ) |
| ta.save("out.wav", wav.unsqueeze(0).cpu(), tts.sr) |
| ``` |
|
|
| # Inference defaults (paper configuration) |
| - Block size `D = 16` |
| - Maximum `K = 10` denoising steps per block |
| - Sampling temperature `0.2` |
| - `shift` outlier schedule with `tau = 0.5` |
| - CFG with `w = 1.0`, `pmi_cfg` combination |
| - FlashInfer paged KV cache + CUDA graph capture |
|
|
| # Streaming Efficiency |
| Latency and throughput at concurrency 1, measured over 50 utterances. **TTFP** is the wall-clock time from request to the first emitted audio packet; **RTF** (real-time factor) is generation time divided by synthesized audio duration — lower is faster, and `RTF < 1` means faster than real time. |
|
|
| | Config (25 Hz, 0.5B) | TTFP (ms) ↓ | RTF ↓ | |
| | --- | :---: | :---: | |
| | `D = 16`, `α = 0.5` (default) | 118 | 0.107 | |
| | `D = 16`, `α = 0.75` | 106 | 0.091 | |
| | `D = 24`, `α = 0.5` | 119 | 0.100 | |
| | `D = 24`, `α = 0.75` | 105 | 0.084 | |
| | `D = 32`, `α = 0.5` | 115 | 0.090 | |
| | `D = 32`, `α = 0.75` | **103** | **0.076** | |
|
|
| Even on a single concurrent request, Chatterbox-Flash sustains roughly **9× real-time** synthesis at the default setting and **~13× real-time** at `D = 32`, `α = 0.75`, while keeping time-to-first-packet low enough for interactive streaming. |
|
|
| # Apple Silicon (MLX) |
| Chatterbox-Flash also runs locally on Apple Silicon via [MLX](https://github.com/ml-explore/mlx). The numbers below were measured on a **Mac M4** at the default configuration; both stay comfortably under real time (`RTF < 1`), and 4-bit quantization gives a further speedup. |
|
|
| | Backend | RTF ↓ | |
| | --- | :---: | |
| | MLX | 0.778 | |
| | MLX (4-bit quantized) | **0.665** | |
|
|
| # Quality (Seed-TTS test-en) |
| Zero-shot TTS quality on the Seed-TTS English benchmark, under the canonical configuration (`D = 16`). **SIM-o** is speaker similarity to the reference (higher is better), **WER** is word error rate from ASR transcription (lower is better), and **UTMOS** is a predicted naturalness score (higher is better). Results are shown for our two main decoding settings against the Chatterbox backbone and ground-truth audio for reference. |
|
|
| | System | SIM-o ↑ | WER ↓ | UTMOS ↑ | |
| | --- | :---: | :---: | :---: | |
| | Ground-truth | 0.734 | 2.14 | 3.52 | |
| | Chatterbox (AR backbone) | 0.685 | 2.20 | 4.10 | |
| | Chatterbox-Flash (`α = 0`) | **0.704** | **1.96** | **4.09** | |
| | Chatterbox-Flash (`α = 0.5`, early decoding) | **0.704** | 2.04 | 4.08 | |
|
|
| Converting the autoregressive backbone into a block-diffusion decoder improves both speaker similarity (`0.685 → 0.704`) and intelligibility (`2.20 → 1.96` WER) while keeping naturalness essentially unchanged — all while unlocking parallel, streaming-friendly decoding. |
|
|
| # License |
| MIT — see `LICENSE` in the source repository. |
