FireRedTTS3-Base-HF / README.md
hukyhukygo's picture
Update official model card
e4133ef verified
|
Raw History Blame Contribute Delete
9.32 kB
---
license: apache-2.0
base_model: FireRedTeam/FireRedTTS3
pipeline_tag: text-to-speech
tags:
- sglang-omni
- fireredtts3
- text-to-speech
- voice-cloning
---
# FireRedTTS3-Base for SGLang Omni
This repository contains the officially released HF-format FireRedTTS3-Base
checkpoint for inference with
[SGLang Omni](https://github.com/sgl-project/sglang-omni). It supports
zero-shot voice cloning from one reference recording and its transcript.
Audio output is 24 kHz mono, with WAV responses and streaming PCM available
through the `/v1/audio/speech` API.
Our model code and original checkpoint release:
- [FireRedTTS3 source code](https://github.com/FireRedTeam/FireRedTTS3)
- [Original FireRedTTS3 weights and model card](https://huggingface.co/FireRedTeam/FireRedTTS3)
This card documents the SGLang Omni loading and serving workflow.
## Install SGLang Omni
Use Linux with an NVIDIA CUDA GPU and Python 3.12. Follow the
[SGLang Omni installation guide](https://github.com/sgl-project/sglang-omni/blob/main/docs/get_started/installation.md)
for the CUDA, UCX, and FlashAttention prerequisites, then install from source:
```bash
git clone https://github.com/sgl-project/sglang-omni.git
cd sglang-omni
pip install uv
uv venv .venv -p 3.12
source .venv/bin/activate
uv pip install --prerelease=allow -e '.[firered-tts3]'
```
Run the following commands from the `sglang-omni` repository root.
## Download this checkpoint
Download the official HF-format checkpoint from
[FireRedTeam/FireRedTTS3-Base-HF](https://huggingface.co/FireRedTeam/FireRedTTS3-Base-HF).
```bash
hf download FireRedTeam/FireRedTTS3-Base-HF \
--local-dir ./models/FireRedTTS3-Base
```
Download the complete repository, including configuration, tokenizer, and
`redae/` assets, plus `campp/`. Loading only the root
`model.safetensors` file is insufficient. If you already have the complete
checkpoint locally, use that directory directly.
## Start the server
```bash
CUDA_VISIBLE_DEVICES=0 sgl-omni serve \
--model-path ./models/FireRedTTS3-Base \
--config examples/configs/firered_tts3_base.yaml \
--allowed-local-media-path "$PWD/docs/_static/audio" \
--port 8000
```
Wait for `/health` to return HTTP 200 before sending requests. Initial model
loading, compilation, and warmup can take several minutes.
The reference clips below are included in the SGLang Omni repository. Local
paths are resolved on the **server**, relative to its working directory.
For your own clips, allow their directory with `--allowed-local-media-path`
and supply the matching transcript for cloning.
## Voice cloning
Provide exactly one reference recording and its actual transcript. The example
uses the repository's `ref_voice.wav`:
```bash
curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
-H 'Content-Type: application/json' \
-d '{
"model": "FireRedTTS3-Base",
"input": "Hello, welcome to SGLang Omni.",
"language": "English",
"references": [{
"audio_path": "docs/_static/audio/ref_voice.wav",
"text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year."
}],
"seed": 42
}' --output clone.wav
```
## Streaming
To stream audio, add `"stream": true` and `"response_format": "pcm"` to
the request and use `curl -N`. The response contains raw **24 kHz, signed
16-bit little-endian, mono PCM**. Save it as `output.pcm` and convert it with:
```bash
ffmpeg -f s16le -ar 24000 -ac 1 -i output.pcm output.wav
```
The server also exposes `/v1/audio/speech/stream` over WebSocket. See the
[WebSocket protocol](https://github.com/sgl-project/sglang-omni/blob/main/docs/basic_usage/tts.md#websocket-speech-streaming)
for `session.config`, `input.text`, `input.commit`, and `input.done` events.
A complete streaming request:
```bash
curl --fail-with-body -N -X POST http://localhost:8000/v1/audio/speech \
-H 'Content-Type: application/json' \
-d '{
"model": "FireRedTTS3-Base",
"input": "Hello, welcome to SGLang Omni.",
"language": "English",
"ref_audio": "docs/_static/audio/ref_voice.wav",
"ref_text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year.",
"stream": true,
"response_format": "pcm",
"seed": 42
}' --output output.pcm
```
## Offline Python SDK
Save this as `offline_tts.py` in the SGLang Omni repository root and run
`CUDA_VISIBLE_DEVICES=0 python offline_tts.py`. This starts the pipeline
locally without an HTTP server.
```python
import asyncio
import multiprocessing as mp
from pathlib import Path
from sglang_omni.client import Client, GenerateRequest
from sglang_omni.models.firered_tts3.config import FireRedTTS3PipelineConfig
from sglang_omni.pipeline.mp_runner import MultiProcessPipelineRunner
async def main() -> None:
config = FireRedTTS3PipelineConfig(
model_path="./models/FireRedTTS3-Base",
)
runner = MultiProcessPipelineRunner(config)
await runner.start(timeout=600)
try:
client = Client(runner.coordinator)
result = await client.speech(
GenerateRequest(
prompt={
"input": "Hello, welcome to SGLang Omni.",
"language": "English",
"references": [{
"audio_path": "docs/_static/audio/ref_voice.wav",
"text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year.",
}],
"seed": 42,
},
stream=False,
),
request_id="demo-1",
)
Path("offline.wav").write_bytes(result.audio_bytes)
finally:
await runner.stop()
if __name__ == "__main__":
mp.set_start_method("spawn", force=True)
asyncio.run(main())
```
## Serving notes
- The current deployment supports a single GPU (`tp_size=1`).
- The `input` text is limited to 400 characters.
- Use an explicit language: `Chinese` (default), `English`, `Japanese`, `Korean`,
`German`, `French`, `Russian`, `Portuguese`, `Spanish`, or `Italian`.
Automatic language detection (`Auto`) is unsupported. These are the stock
serving API's accepted language values; FireRedTTS3 supports additional
languages and dialects through our standalone inference pipeline.
- Submit spoken-form text when normalization is needed. The FastText and LLM
text-normalization setup in our standalone pipeline is not used in this quick start.
- Put flow-solver settings under `stage_params.talker`, for example
`{"n_timesteps": 10, "inference_cfg": 2.0, "max_patches": 200}`. Unknown
top-level fields are not forwarded as model settings.
- HTTP, batch, and WebSocket requests do not forward `extra_params`. The SDK
`GenerateRequest.extra_params` field remains available for SDK requests.
- `seed` controls the per-request flow sampler. Different precisions, hardware,
or batching schedules can produce different results with the same seed.
- For our published model-quality results, see the
[FireRedTTS3 model card](https://huggingface.co/FireRedTeam/FireRedTTS3).
SGLang Omni runtime accuracy and performance are evaluated separately.
- The audio phase generates continuous latents. Token-sampling settings
such as `temperature`, `top_p`, and `top_k` do not control its audio output.
See the [FireRedTTS3 cookbook](https://github.com/sgl-project/sglang-omni/blob/main/docs/cookbook/firered_tts3.md)
for the full request parameters and deployment configuration.
## Usage Disclaimer
- The project incorporates zero-shot voice cloning functionality; Please note that this capability is intended **solely for academic research purposes**.
- **DO NOT** use this model for **ANY illegal activities**❗️❗️
- The developers assume no liability for any misuse of this model.
- If you identify any instances of **abuse**, **misuse**, or **fraudulent** activities related to this project, **please report them to our team immediately.**
## Citation
```bib
@article{fireredtts3,
title = {FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations},
author = {FireRed Team},
journal = {arXiv preprint},
year = {2026},
}
```
## Acknowledgements
- [Qwen3](https://github.com/QwenLM/Qwen3) and [Qwen2-Audio](https://github.com/QwenLM/Qwen2-Audio) for the language model and audio understanding foundations
- <u>DiTAR</u> for the patch-level diffusion autoregressive formulation
- [X-Codec](https://github.com/zhenye234/xcodec) for the discriminator design used in RedAE training
- [CAM++](https://modelscope.cn/models/iic/speech_campplus_sv_en_voxceleb_16k) for speaker embedding extraction
- [fastText](https://fasttext.cc/docs/en/language-identification.html) for automatic language identification
- [WeTextProcessing](https://github.com/wenet-e2e/WeTextProcessing) (wetext) for the Chinese / English text normalization front-end
## License
FireRedTTS3 is released under [Apache-2.0](https://github.com/FireRedTeam/FireRedTTS3/blob/main/LICENSE).
See our [original model card](https://huggingface.co/FireRedTeam/FireRedTTS3)
for usage guidance and model evaluation details.