|
Download README.md from FireRedTeam/FireRedTTS3-Base-HF: direct link, hf CLI and curl.
- Browser
- Download file 9.32 kB
-
https://huggingface.co/FireRedTeam/FireRedTTS3-Base-HF/resolve/main/README.md
- Command line
-
hf download hf://FireRedTeam/FireRedTTS3-Base-HF/README.md
-
curl -L -o README.md https://huggingface.co/FireRedTeam/FireRedTTS3-Base-HF/resolve/main/README.md
9.32 kB
| license: apache-2.0 | |
| base_model: FireRedTeam/FireRedTTS3 | |
| pipeline_tag: text-to-speech | |
| tags: | |
| - sglang-omni | |
| - fireredtts3 | |
| - text-to-speech | |
| - voice-cloning | |
| # FireRedTTS3-Base for SGLang Omni | |
| This repository contains the officially released HF-format FireRedTTS3-Base | |
| checkpoint for inference with | |
| [SGLang Omni](https://github.com/sgl-project/sglang-omni). It supports | |
| zero-shot voice cloning from one reference recording and its transcript. | |
| Audio output is 24 kHz mono, with WAV responses and streaming PCM available | |
| through the `/v1/audio/speech` API. | |
| Our model code and original checkpoint release: | |
| - [FireRedTTS3 source code](https://github.com/FireRedTeam/FireRedTTS3) | |
| - [Original FireRedTTS3 weights and model card](https://huggingface.co/FireRedTeam/FireRedTTS3) | |
| This card documents the SGLang Omni loading and serving workflow. | |
| ## Install SGLang Omni | |
| Use Linux with an NVIDIA CUDA GPU and Python 3.12. Follow the | |
| [SGLang Omni installation guide](https://github.com/sgl-project/sglang-omni/blob/main/docs/get_started/installation.md) | |
| for the CUDA, UCX, and FlashAttention prerequisites, then install from source: | |
| ```bash | |
| git clone https://github.com/sgl-project/sglang-omni.git | |
| cd sglang-omni | |
| pip install uv | |
| uv venv .venv -p 3.12 | |
| source .venv/bin/activate | |
| uv pip install --prerelease=allow -e '.[firered-tts3]' | |
| ``` | |
| Run the following commands from the `sglang-omni` repository root. | |
| ## Download this checkpoint | |
| Download the official HF-format checkpoint from | |
| [FireRedTeam/FireRedTTS3-Base-HF](https://huggingface.co/FireRedTeam/FireRedTTS3-Base-HF). | |
| ```bash | |
| hf download FireRedTeam/FireRedTTS3-Base-HF \ | |
| --local-dir ./models/FireRedTTS3-Base | |
| ``` | |
| Download the complete repository, including configuration, tokenizer, and | |
| `redae/` assets, plus `campp/`. Loading only the root | |
| `model.safetensors` file is insufficient. If you already have the complete | |
| checkpoint locally, use that directory directly. | |
| ## Start the server | |
| ```bash | |
| CUDA_VISIBLE_DEVICES=0 sgl-omni serve \ | |
| --model-path ./models/FireRedTTS3-Base \ | |
| --config examples/configs/firered_tts3_base.yaml \ | |
| --allowed-local-media-path "$PWD/docs/_static/audio" \ | |
| --port 8000 | |
| ``` | |
| Wait for `/health` to return HTTP 200 before sending requests. Initial model | |
| loading, compilation, and warmup can take several minutes. | |
| The reference clips below are included in the SGLang Omni repository. Local | |
| paths are resolved on the **server**, relative to its working directory. | |
| For your own clips, allow their directory with `--allowed-local-media-path` | |
| and supply the matching transcript for cloning. | |
| ## Voice cloning | |
| Provide exactly one reference recording and its actual transcript. The example | |
| uses the repository's `ref_voice.wav`: | |
| ```bash | |
| curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \ | |
| -H 'Content-Type: application/json' \ | |
| -d '{ | |
| "model": "FireRedTTS3-Base", | |
| "input": "Hello, welcome to SGLang Omni.", | |
| "language": "English", | |
| "references": [{ | |
| "audio_path": "docs/_static/audio/ref_voice.wav", | |
| "text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year." | |
| }], | |
| "seed": 42 | |
| }' --output clone.wav | |
| ``` | |
| ## Streaming | |
| To stream audio, add `"stream": true` and `"response_format": "pcm"` to | |
| the request and use `curl -N`. The response contains raw **24 kHz, signed | |
| 16-bit little-endian, mono PCM**. Save it as `output.pcm` and convert it with: | |
| ```bash | |
| ffmpeg -f s16le -ar 24000 -ac 1 -i output.pcm output.wav | |
| ``` | |
| The server also exposes `/v1/audio/speech/stream` over WebSocket. See the | |
| [WebSocket protocol](https://github.com/sgl-project/sglang-omni/blob/main/docs/basic_usage/tts.md#websocket-speech-streaming) | |
| for `session.config`, `input.text`, `input.commit`, and `input.done` events. | |
| A complete streaming request: | |
| ```bash | |
| curl --fail-with-body -N -X POST http://localhost:8000/v1/audio/speech \ | |
| -H 'Content-Type: application/json' \ | |
| -d '{ | |
| "model": "FireRedTTS3-Base", | |
| "input": "Hello, welcome to SGLang Omni.", | |
| "language": "English", | |
| "ref_audio": "docs/_static/audio/ref_voice.wav", | |
| "ref_text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year.", | |
| "stream": true, | |
| "response_format": "pcm", | |
| "seed": 42 | |
| }' --output output.pcm | |
| ``` | |
| ## Offline Python SDK | |
| Save this as `offline_tts.py` in the SGLang Omni repository root and run | |
| `CUDA_VISIBLE_DEVICES=0 python offline_tts.py`. This starts the pipeline | |
| locally without an HTTP server. | |
| ```python | |
| import asyncio | |
| import multiprocessing as mp | |
| from pathlib import Path | |
| from sglang_omni.client import Client, GenerateRequest | |
| from sglang_omni.models.firered_tts3.config import FireRedTTS3PipelineConfig | |
| from sglang_omni.pipeline.mp_runner import MultiProcessPipelineRunner | |
| async def main() -> None: | |
| config = FireRedTTS3PipelineConfig( | |
| model_path="./models/FireRedTTS3-Base", | |
| ) | |
| runner = MultiProcessPipelineRunner(config) | |
| await runner.start(timeout=600) | |
| try: | |
| client = Client(runner.coordinator) | |
| result = await client.speech( | |
| GenerateRequest( | |
| prompt={ | |
| "input": "Hello, welcome to SGLang Omni.", | |
| "language": "English", | |
| "references": [{ | |
| "audio_path": "docs/_static/audio/ref_voice.wav", | |
| "text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year.", | |
| }], | |
| "seed": 42, | |
| }, | |
| stream=False, | |
| ), | |
| request_id="demo-1", | |
| ) | |
| Path("offline.wav").write_bytes(result.audio_bytes) | |
| finally: | |
| await runner.stop() | |
| if __name__ == "__main__": | |
| mp.set_start_method("spawn", force=True) | |
| asyncio.run(main()) | |
| ``` | |
| ## Serving notes | |
| - The current deployment supports a single GPU (`tp_size=1`). | |
| - The `input` text is limited to 400 characters. | |
| - Use an explicit language: `Chinese` (default), `English`, `Japanese`, `Korean`, | |
| `German`, `French`, `Russian`, `Portuguese`, `Spanish`, or `Italian`. | |
| Automatic language detection (`Auto`) is unsupported. These are the stock | |
| serving API's accepted language values; FireRedTTS3 supports additional | |
| languages and dialects through our standalone inference pipeline. | |
| - Submit spoken-form text when normalization is needed. The FastText and LLM | |
| text-normalization setup in our standalone pipeline is not used in this quick start. | |
| - Put flow-solver settings under `stage_params.talker`, for example | |
| `{"n_timesteps": 10, "inference_cfg": 2.0, "max_patches": 200}`. Unknown | |
| top-level fields are not forwarded as model settings. | |
| - HTTP, batch, and WebSocket requests do not forward `extra_params`. The SDK | |
| `GenerateRequest.extra_params` field remains available for SDK requests. | |
| - `seed` controls the per-request flow sampler. Different precisions, hardware, | |
| or batching schedules can produce different results with the same seed. | |
| - For our published model-quality results, see the | |
| [FireRedTTS3 model card](https://huggingface.co/FireRedTeam/FireRedTTS3). | |
| SGLang Omni runtime accuracy and performance are evaluated separately. | |
| - The audio phase generates continuous latents. Token-sampling settings | |
| such as `temperature`, `top_p`, and `top_k` do not control its audio output. | |
| See the [FireRedTTS3 cookbook](https://github.com/sgl-project/sglang-omni/blob/main/docs/cookbook/firered_tts3.md) | |
| for the full request parameters and deployment configuration. | |
| ## Usage Disclaimer | |
| - The project incorporates zero-shot voice cloning functionality; Please note that this capability is intended **solely for academic research purposes**. | |
| - **DO NOT** use this model for **ANY illegal activities**❗️❗️ | |
| - The developers assume no liability for any misuse of this model. | |
| - If you identify any instances of **abuse**, **misuse**, or **fraudulent** activities related to this project, **please report them to our team immediately.** | |
| ## Citation | |
| ```bib | |
| @article{fireredtts3, | |
| title = {FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations}, | |
| author = {FireRed Team}, | |
| journal = {arXiv preprint}, | |
| year = {2026}, | |
| } | |
| ``` | |
| ## Acknowledgements | |
| - [Qwen3](https://github.com/QwenLM/Qwen3) and [Qwen2-Audio](https://github.com/QwenLM/Qwen2-Audio) for the language model and audio understanding foundations | |
| - <u>DiTAR</u> for the patch-level diffusion autoregressive formulation | |
| - [X-Codec](https://github.com/zhenye234/xcodec) for the discriminator design used in RedAE training | |
| - [CAM++](https://modelscope.cn/models/iic/speech_campplus_sv_en_voxceleb_16k) for speaker embedding extraction | |
| - [fastText](https://fasttext.cc/docs/en/language-identification.html) for automatic language identification | |
| - [WeTextProcessing](https://github.com/wenet-e2e/WeTextProcessing) (wetext) for the Chinese / English text normalization front-end | |
| ## License | |
| FireRedTTS3 is released under [Apache-2.0](https://github.com/FireRedTeam/FireRedTTS3/blob/main/LICENSE). | |
| See our [original model card](https://huggingface.co/FireRedTeam/FireRedTTS3) | |
| for usage guidance and model evaluation details. | |