|
Download README.md from FireRedTeam/FireRedTTS3-Instruct-HF: direct link, hf CLI and curl.
- Browser
- Download file 11.6 kB
-
https://huggingface.co/FireRedTeam/FireRedTTS3-Instruct-HF/resolve/main/README.md
- Command line
-
hf download hf://FireRedTeam/FireRedTTS3-Instruct-HF/README.md
-
curl -L -o README.md https://huggingface.co/FireRedTeam/FireRedTTS3-Instruct-HF/resolve/main/README.md
11.6 kB
| license: apache-2.0 | |
| base_model: FireRedTeam/FireRedTTS3 | |
| pipeline_tag: text-to-speech | |
| tags: | |
| - sglang-omni | |
| - fireredtts3 | |
| - text-to-speech | |
| - voice-cloning | |
| # FireRedTTS3-Instruct for SGLang Omni | |
| This repository contains the officially released HF-format FireRedTTS3-Instruct | |
| checkpoint for inference with | |
| [SGLang Omni](https://github.com/sgl-project/sglang-omni). It supports | |
| reference-audio voice cloning, instruction-guided voice design, semantic editing, and acoustic editing. | |
| Audio output is 24 kHz mono, with WAV responses and streaming PCM available | |
| through the `/v1/audio/speech` API. | |
| Our model code and original checkpoint release: | |
| - [FireRedTTS3 source code](https://github.com/FireRedTeam/FireRedTTS3) | |
| - [Original FireRedTTS3 weights and model card](https://huggingface.co/FireRedTeam/FireRedTTS3) | |
| This card documents the SGLang Omni loading and serving workflow. | |
| ## Install SGLang Omni | |
| Use Linux with an NVIDIA CUDA GPU and Python 3.12. Follow the | |
| [SGLang Omni installation guide](https://github.com/sgl-project/sglang-omni/blob/main/docs/get_started/installation.md) | |
| for the CUDA, UCX, and FlashAttention prerequisites, then install from source: | |
| ```bash | |
| git clone https://github.com/sgl-project/sglang-omni.git | |
| cd sglang-omni | |
| pip install uv | |
| uv venv .venv -p 3.12 | |
| source .venv/bin/activate | |
| uv pip install --prerelease=allow -e '.[firered-tts3]' | |
| ``` | |
| Run the following commands from the `sglang-omni` repository root. | |
| ## Download this checkpoint | |
| Download the official HF-format checkpoint from | |
| [FireRedTeam/FireRedTTS3-Instruct-HF](https://huggingface.co/FireRedTeam/FireRedTTS3-Instruct-HF). | |
| ```bash | |
| hf download FireRedTeam/FireRedTTS3-Instruct-HF \ | |
| --local-dir ./models/FireRedTTS3-Instruct | |
| ``` | |
| Download the complete repository, including configuration, tokenizer, and | |
| `redae/` assets. Loading only the root | |
| `model.safetensors` file is insufficient. If you already have the complete | |
| checkpoint locally, use that directory directly. | |
| ## Start the server | |
| ```bash | |
| CUDA_VISIBLE_DEVICES=0 sgl-omni serve \ | |
| --model-path ./models/FireRedTTS3-Instruct \ | |
| --config examples/configs/firered_tts3_instruct.yaml \ | |
| --allowed-local-media-path "$PWD/docs/_static/audio" \ | |
| --port 8000 | |
| ``` | |
| Wait for `/health` to return HTTP 200 before sending requests. Initial model | |
| loading, compilation, and warmup can take several minutes. | |
| The reference clips below are included in the SGLang Omni repository. Local | |
| paths are resolved on the **server**, relative to its working directory. | |
| For your own clips, allow their directory with `--allowed-local-media-path` | |
| and supply the matching transcript for cloning. | |
| ## Voice cloning | |
| Provide exactly one reference recording and its actual transcript. The example | |
| uses the repository's `ref_voice.wav`: | |
| ```bash | |
| curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \ | |
| -H 'Content-Type: application/json' \ | |
| -d '{ | |
| "model": "FireRedTTS3-Instruct", | |
| "input": "Hello, welcome to SGLang Omni.", | |
| "references": [{ | |
| "audio_path": "docs/_static/audio/ref_voice.wav", | |
| "text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year." | |
| }], | |
| "seed": 42 | |
| }' --output clone.wav | |
| ``` | |
| Omit `task_type` for cloning, or set it to `Base`. The value `tts` is not an | |
| accepted serving task name. | |
| ## Voice design | |
| Generate a voice from a description, without a reference recording: | |
| ```bash | |
| curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \ | |
| -H 'Content-Type: application/json' \ | |
| -d '{ | |
| "model": "FireRedTTS3-Instruct", | |
| "input": "今天天气很好,我们一起去公园散步吧。", | |
| "task_type": "VoiceDesign", | |
| "instructions": "一个年轻女性的温柔嗓音,语速稍慢,带一点俏皮。", | |
| "seed": 42 | |
| }' --output voice_design.wav | |
| ``` | |
| ## Semantic editing | |
| Use the reference recording as the source to edit. Its content supplies the | |
| input, so `input` may be empty; no reference transcript is required: | |
| ```bash | |
| curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \ | |
| -H 'Content-Type: application/json' \ | |
| --data-binary @- --output semantic_edit.wav <<'JSON' | |
| { | |
| "model": "FireRedTTS3-Instruct", | |
| "input": "", | |
| "task_type": "SemanticEdit", | |
| "instructions": "Replace 'birthday' with 'anniversary'.", | |
| "ref_audio": "docs/_static/audio/ref_voice.wav", | |
| "seed": 42 | |
| } | |
| JSON | |
| ``` | |
| For substitutions, use the quoted form `Replace 'source' with 'target'.` | |
| The quoted example above was verified by transcribing the generated audio; | |
| omitting the quotes did not perform the intended replacement in this test. | |
| The `x-gen-text` response header contains text with edit masks, so the target | |
| word may appear only in the audio. Validate the rendered audio rather than | |
| requiring the target word to appear in that header. | |
| ## Acoustic editing | |
| Adjust the source recording's speed, pitch, or volume. This example requests | |
| 0.8x speed through the model's native acoustic-edit task: | |
| ```bash | |
| curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \ | |
| -H 'Content-Type: application/json' \ | |
| -d '{ | |
| "model": "FireRedTTS3-Instruct", | |
| "input": "", | |
| "task_type": "AcousticEdit", | |
| "ref_audio": "docs/_static/audio/ref_voice.wav", | |
| "speed": 0.8, | |
| "seed": 42 | |
| }' --output acoustic_edit.wav | |
| ``` | |
| Raw acoustic instructions must follow the trained templates rather than | |
| free-form descriptions. For pitch or volume, use, for example, | |
| `"instructions": "shift the pitch by 2 steps"` or | |
| `"instructions": "adjust the volume to 0.8"`. An explicit `instructions` | |
| string takes precedence over composed acoustic instructions. Without explicit | |
| `instructions`, the composed `acoustic_attribute` must be supported (`Speed`, | |
| `Pitch`, or `Volume`) and its value must be a finite number. Set | |
| `acoustic_attribute` and `acoustic_value` under `stage_params.preprocessing`. | |
| For example: | |
| ```json | |
| { | |
| "stage_params": { | |
| "preprocessing": {"acoustic_attribute": "Pitch", "acoustic_value": 2} | |
| } | |
| } | |
| ``` | |
| The top-level `speed` field supplies the Speed value for `AcousticEdit`; it | |
| does not change cloning or voice design. | |
| Edit sources are capped at 20 seconds. VoiceDesign and SemanticEdit generate | |
| text before audio, so their first-audio latency differs from cloning. | |
| ## Streaming | |
| To stream audio, add `"stream": true` and `"response_format": "pcm"` to | |
| the request and use `curl -N`. The response contains raw **24 kHz, signed | |
| 16-bit little-endian, mono PCM**. Save it as `output.pcm` and convert it with: | |
| ```bash | |
| ffmpeg -f s16le -ar 24000 -ac 1 -i output.pcm output.wav | |
| ``` | |
| The server also exposes `/v1/audio/speech/stream` over WebSocket. See the | |
| [WebSocket protocol](https://github.com/sgl-project/sglang-omni/blob/main/docs/basic_usage/tts.md#websocket-speech-streaming) | |
| for `session.config`, `input.text`, `input.commit`, and `input.done` events. | |
| ## Offline Python SDK | |
| Save this as `offline_tts.py` in the SGLang Omni repository root and run | |
| `CUDA_VISIBLE_DEVICES=0 python offline_tts.py`. This starts the pipeline | |
| locally without an HTTP server. | |
| ```python | |
| import asyncio | |
| import multiprocessing as mp | |
| from pathlib import Path | |
| from sglang_omni.client import Client, GenerateRequest | |
| from sglang_omni.models.firered_tts3.config import FireRedTTS3PipelineConfig | |
| from sglang_omni.pipeline.mp_runner import MultiProcessPipelineRunner | |
| async def main() -> None: | |
| config = FireRedTTS3PipelineConfig( | |
| model_path="./models/FireRedTTS3-Instruct", | |
| ) | |
| config.stage_named("talker").factory.flow_mode = "fp32" | |
| runner = MultiProcessPipelineRunner(config) | |
| await runner.start(timeout=600) | |
| try: | |
| client = Client(runner.coordinator) | |
| result = await client.speech( | |
| GenerateRequest( | |
| prompt={ | |
| "input": "今天天气很好,我们一起去公园散步吧。", | |
| "task_type": "VoiceDesign", | |
| "instructions": "一个年轻女性的温柔嗓音,语速稍慢,带一点俏皮。", | |
| "seed": 42, | |
| }, | |
| stream=False, | |
| ), | |
| request_id="demo-1", | |
| ) | |
| Path("offline.wav").write_bytes(result.audio_bytes) | |
| finally: | |
| await runner.stop() | |
| if __name__ == "__main__": | |
| mp.set_start_method("spawn", force=True) | |
| asyncio.run(main()) | |
| ``` | |
| ## Serving notes | |
| - The current deployment supports a single GPU (`tp_size=1`). | |
| - The `input` text is limited to 400 characters. Editing requests may use an empty | |
| `input` as shown above. | |
| - Instruct does not use the Base model's language tag. | |
| - Submit spoken-form text when normalization is needed. The FastText and LLM | |
| text-normalization setup in our standalone pipeline is not used in this quick start. | |
| - Put flow-solver settings under `stage_params.talker`, for example | |
| `{"n_timesteps": 10, "inference_cfg": 2.0, "max_patches": 200}`. Unknown | |
| top-level fields are not forwarded as model settings. | |
| - HTTP, batch, and WebSocket requests do not forward `extra_params`. The SDK | |
| `GenerateRequest.extra_params` field remains available for SDK requests. | |
| - `seed` controls the per-request flow sampler. Different precisions, hardware, | |
| or batching schedules can produce different results with the same seed. | |
| - For our published model-quality results, see the | |
| [FireRedTTS3 model card](https://huggingface.co/FireRedTeam/FireRedTTS3). | |
| SGLang Omni runtime accuracy and performance are evaluated separately. | |
| - Keep the example Instruct configuration's `flow_mode: fp32` for the | |
| supported default. The offline example sets it explicitly because it does | |
| not load the example YAML. | |
| - `temperature`, `top_p`, and `top_k` affect the VoiceDesign/SemanticEdit text | |
| phase. They do not control continuous-latent audio generation. | |
| See the [FireRedTTS3 cookbook](https://github.com/sgl-project/sglang-omni/blob/main/docs/cookbook/firered_tts3.md) | |
| for the full request parameters and deployment configuration. | |
| ## Usage Disclaimer | |
| - The project incorporates zero-shot voice cloning functionality; Please note that this capability is intended **solely for academic research purposes**. | |
| - **DO NOT** use this model for **ANY illegal activities**❗️❗️ | |
| - The developers assume no liability for any misuse of this model. | |
| - If you identify any instances of **abuse**, **misuse**, or **fraudulent** activities related to this project, **please report them to our team immediately.** | |
| ## Citation | |
| ```bib | |
| @article{fireredtts3, | |
| title = {FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations}, | |
| author = {FireRed Team}, | |
| journal = {arXiv preprint}, | |
| year = {2026}, | |
| } | |
| ``` | |
| ## Acknowledgements | |
| - [Qwen3](https://github.com/QwenLM/Qwen3) and [Qwen2-Audio](https://github.com/QwenLM/Qwen2-Audio) for the language model and audio understanding foundations | |
| - <u>DiTAR</u> for the patch-level diffusion autoregressive formulation | |
| - [X-Codec](https://github.com/zhenye234/xcodec) for the discriminator design used in RedAE training | |
| - [CAM++](https://modelscope.cn/models/iic/speech_campplus_sv_en_voxceleb_16k) for speaker embedding extraction | |
| - [fastText](https://fasttext.cc/docs/en/language-identification.html) for automatic language identification | |
| - [WeTextProcessing](https://github.com/wenet-e2e/WeTextProcessing) (wetext) for the Chinese / English text normalization front-end | |
| ## License | |
| FireRedTTS3 is released under [Apache-2.0](https://github.com/FireRedTeam/FireRedTTS3/blob/main/LICENSE). | |
| See our [original model card](https://huggingface.co/FireRedTeam/FireRedTTS3) | |
| for usage guidance and model evaluation details. | |