File size: 11,638 Bytes
95b508f
 
 
 
 
 
 
 
 
 
 
 
 
1fa25f9
 
95b508f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1fa25f9
95b508f
 
 
1fa25f9
 
95b508f
 
1fa25f9
95b508f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1fa25f9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
95b508f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1fa25f9
95b508f
1fa25f9
95b508f
 
 
 
 
1fa25f9
 
95b508f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
---
license: apache-2.0
base_model: FireRedTeam/FireRedTTS3
pipeline_tag: text-to-speech
tags:
  - sglang-omni
  - fireredtts3
  - text-to-speech
  - voice-cloning
---

# FireRedTTS3-Instruct for SGLang Omni

This repository contains the officially released HF-format FireRedTTS3-Instruct
checkpoint for inference with
[SGLang Omni](https://github.com/sgl-project/sglang-omni). It supports
reference-audio voice cloning, instruction-guided voice design, semantic editing, and acoustic editing.
Audio output is 24 kHz mono, with WAV responses and streaming PCM available
through the `/v1/audio/speech` API.

Our model code and original checkpoint release:

- [FireRedTTS3 source code](https://github.com/FireRedTeam/FireRedTTS3)
- [Original FireRedTTS3 weights and model card](https://huggingface.co/FireRedTeam/FireRedTTS3)

This card documents the SGLang Omni loading and serving workflow.

## Install SGLang Omni

Use Linux with an NVIDIA CUDA GPU and Python 3.12. Follow the
[SGLang Omni installation guide](https://github.com/sgl-project/sglang-omni/blob/main/docs/get_started/installation.md)
for the CUDA, UCX, and FlashAttention prerequisites, then install from source:

```bash
git clone https://github.com/sgl-project/sglang-omni.git
cd sglang-omni
pip install uv
uv venv .venv -p 3.12
source .venv/bin/activate
uv pip install --prerelease=allow -e '.[firered-tts3]'
```

Run the following commands from the `sglang-omni` repository root.

## Download this checkpoint

Download the official HF-format checkpoint from
[FireRedTeam/FireRedTTS3-Instruct-HF](https://huggingface.co/FireRedTeam/FireRedTTS3-Instruct-HF).

```bash
hf download FireRedTeam/FireRedTTS3-Instruct-HF \
  --local-dir ./models/FireRedTTS3-Instruct
```

Download the complete repository, including configuration, tokenizer, and
`redae/` assets. Loading only the root
`model.safetensors` file is insufficient. If you already have the complete
checkpoint locally, use that directory directly.

## Start the server

```bash
CUDA_VISIBLE_DEVICES=0 sgl-omni serve \
  --model-path ./models/FireRedTTS3-Instruct \
  --config examples/configs/firered_tts3_instruct.yaml \
  --allowed-local-media-path "$PWD/docs/_static/audio" \
  --port 8000
```

Wait for `/health` to return HTTP 200 before sending requests. Initial model
loading, compilation, and warmup can take several minutes.

The reference clips below are included in the SGLang Omni repository. Local
paths are resolved on the **server**, relative to its working directory.
For your own clips, allow their directory with `--allowed-local-media-path`
and supply the matching transcript for cloning.

## Voice cloning

Provide exactly one reference recording and its actual transcript. The example
uses the repository's `ref_voice.wav`:

```bash
curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "FireRedTTS3-Instruct",
    "input": "Hello, welcome to SGLang Omni.",
    "references": [{
      "audio_path": "docs/_static/audio/ref_voice.wav",
      "text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year."
    }],
    "seed": 42
  }' --output clone.wav
```

Omit `task_type` for cloning, or set it to `Base`. The value `tts` is not an
accepted serving task name.

## Voice design

Generate a voice from a description, without a reference recording:

```bash
curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "FireRedTTS3-Instruct",
    "input": "今天天气很好,我们一起去公园散步吧。",
    "task_type": "VoiceDesign",
    "instructions": "一个年轻女性的温柔嗓音,语速稍慢,带一点俏皮。",
    "seed": 42
  }' --output voice_design.wav
```

## Semantic editing

Use the reference recording as the source to edit. Its content supplies the
input, so `input` may be empty; no reference transcript is required:

```bash
curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  --data-binary @- --output semantic_edit.wav <<'JSON'
{
  "model": "FireRedTTS3-Instruct",
  "input": "",
  "task_type": "SemanticEdit",
  "instructions": "Replace 'birthday' with 'anniversary'.",
  "ref_audio": "docs/_static/audio/ref_voice.wav",
  "seed": 42
}
JSON
```

For substitutions, use the quoted form `Replace 'source' with 'target'.`
The quoted example above was verified by transcribing the generated audio;
omitting the quotes did not perform the intended replacement in this test.
The `x-gen-text` response header contains text with edit masks, so the target
word may appear only in the audio. Validate the rendered audio rather than
requiring the target word to appear in that header.

## Acoustic editing

Adjust the source recording's speed, pitch, or volume. This example requests
0.8x speed through the model's native acoustic-edit task:

```bash
curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "FireRedTTS3-Instruct",
    "input": "",
    "task_type": "AcousticEdit",
    "ref_audio": "docs/_static/audio/ref_voice.wav",
    "speed": 0.8,
    "seed": 42
  }' --output acoustic_edit.wav
```

Raw acoustic instructions must follow the trained templates rather than
free-form descriptions. For pitch or volume, use, for example,
`"instructions": "shift the pitch by 2 steps"` or
`"instructions": "adjust the volume to 0.8"`. An explicit `instructions`
string takes precedence over composed acoustic instructions. Without explicit
`instructions`, the composed `acoustic_attribute` must be supported (`Speed`,
`Pitch`, or `Volume`) and its value must be a finite number. Set
`acoustic_attribute` and `acoustic_value` under `stage_params.preprocessing`.
For example:

```json
{
  "stage_params": {
    "preprocessing": {"acoustic_attribute": "Pitch", "acoustic_value": 2}
  }
}
```

The top-level `speed` field supplies the Speed value for `AcousticEdit`; it
does not change cloning or voice design.
Edit sources are capped at 20 seconds. VoiceDesign and SemanticEdit generate
text before audio, so their first-audio latency differs from cloning.

## Streaming

To stream audio, add `"stream": true` and `"response_format": "pcm"` to
the request and use `curl -N`. The response contains raw **24 kHz, signed
16-bit little-endian, mono PCM**. Save it as `output.pcm` and convert it with:

```bash
ffmpeg -f s16le -ar 24000 -ac 1 -i output.pcm output.wav
```

The server also exposes `/v1/audio/speech/stream` over WebSocket. See the
[WebSocket protocol](https://github.com/sgl-project/sglang-omni/blob/main/docs/basic_usage/tts.md#websocket-speech-streaming)
for `session.config`, `input.text`, `input.commit`, and `input.done` events.

## Offline Python SDK

Save this as `offline_tts.py` in the SGLang Omni repository root and run
`CUDA_VISIBLE_DEVICES=0 python offline_tts.py`. This starts the pipeline
locally without an HTTP server.

```python
import asyncio
import multiprocessing as mp
from pathlib import Path

from sglang_omni.client import Client, GenerateRequest
from sglang_omni.models.firered_tts3.config import FireRedTTS3PipelineConfig
from sglang_omni.pipeline.mp_runner import MultiProcessPipelineRunner


async def main() -> None:
    config = FireRedTTS3PipelineConfig(
        model_path="./models/FireRedTTS3-Instruct",
    )
    config.stage_named("talker").factory.flow_mode = "fp32"
    runner = MultiProcessPipelineRunner(config)
    await runner.start(timeout=600)
    try:
        client = Client(runner.coordinator)
        result = await client.speech(
            GenerateRequest(
                prompt={
                    "input": "今天天气很好,我们一起去公园散步吧。",
                    "task_type": "VoiceDesign",
                    "instructions": "一个年轻女性的温柔嗓音,语速稍慢,带一点俏皮。",
                    "seed": 42,
                },
                stream=False,
            ),
            request_id="demo-1",
        )
        Path("offline.wav").write_bytes(result.audio_bytes)
    finally:
        await runner.stop()


if __name__ == "__main__":
    mp.set_start_method("spawn", force=True)
    asyncio.run(main())
```

## Serving notes

- The current deployment supports a single GPU (`tp_size=1`).
- The `input` text is limited to 400 characters. Editing requests may use an empty
  `input` as shown above.
- Instruct does not use the Base model's language tag.
- Submit spoken-form text when normalization is needed. The FastText and LLM
  text-normalization setup in our standalone pipeline is not used in this quick start.
- Put flow-solver settings under `stage_params.talker`, for example
  `{"n_timesteps": 10, "inference_cfg": 2.0, "max_patches": 200}`. Unknown
  top-level fields are not forwarded as model settings.
- HTTP, batch, and WebSocket requests do not forward `extra_params`. The SDK
  `GenerateRequest.extra_params` field remains available for SDK requests.
- `seed` controls the per-request flow sampler. Different precisions, hardware,
  or batching schedules can produce different results with the same seed.
- For our published model-quality results, see the
  [FireRedTTS3 model card](https://huggingface.co/FireRedTeam/FireRedTTS3).
  SGLang Omni runtime accuracy and performance are evaluated separately.
- Keep the example Instruct configuration's `flow_mode: fp32` for the
  supported default. The offline example sets it explicitly because it does
  not load the example YAML.
- `temperature`, `top_p`, and `top_k` affect the VoiceDesign/SemanticEdit text
  phase. They do not control continuous-latent audio generation.

See the [FireRedTTS3 cookbook](https://github.com/sgl-project/sglang-omni/blob/main/docs/cookbook/firered_tts3.md)
for the full request parameters and deployment configuration.

## Usage Disclaimer

- The project incorporates zero-shot voice cloning functionality; Please note that this capability is intended **solely for academic research purposes**.
- **DO NOT** use this model for **ANY illegal activities**❗️❗️
- The developers assume no liability for any misuse of this model.
- If you identify any instances of **abuse**, **misuse**, or **fraudulent** activities related to this project, **please report them to our team immediately.**

## Citation

```bib
@article{fireredtts3,
  title   = {FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations},
  author  = {FireRed Team},
  journal = {arXiv preprint},
  year    = {2026},
}
```

## Acknowledgements

- [Qwen3](https://github.com/QwenLM/Qwen3) and [Qwen2-Audio](https://github.com/QwenLM/Qwen2-Audio) for the language model and audio understanding foundations
- <u>DiTAR</u> for the patch-level diffusion autoregressive formulation
- [X-Codec](https://github.com/zhenye234/xcodec) for the discriminator design used in RedAE training
- [CAM++](https://modelscope.cn/models/iic/speech_campplus_sv_en_voxceleb_16k) for speaker embedding extraction
- [fastText](https://fasttext.cc/docs/en/language-identification.html) for automatic language identification
- [WeTextProcessing](https://github.com/wenet-e2e/WeTextProcessing) (wetext) for the Chinese / English text normalization front-end

## License

FireRedTTS3 is released under [Apache-2.0](https://github.com/FireRedTeam/FireRedTTS3/blob/main/LICENSE).
See our [original model card](https://huggingface.co/FireRedTeam/FireRedTTS3)
for usage guidance and model evaluation details.