hukyhukygo commited on
Commit
de8829c
·
verified ·
1 Parent(s): 1b20fc8

Upload FireRedTTS3 checkpoint

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,238 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: FireRedTeam/FireRedTTS3
4
+ pipeline_tag: text-to-speech
5
+ tags:
6
+ - sglang-omni
7
+ - fireredtts3
8
+ - text-to-speech
9
+ - voice-cloning
10
+ ---
11
+
12
+ # FireRedTTS3-Base for SGLang Omni
13
+
14
+ We provide our FireRedTTS3-Base checkpoint for inference with
15
+ [SGLang Omni](https://github.com/sgl-project/sglang-omni). It supports
16
+ zero-shot voice cloning from one reference recording and its transcript.
17
+ Audio output is 24 kHz mono, with WAV responses and streaming PCM available
18
+ through the `/v1/audio/speech` API.
19
+
20
+ Our model code and original checkpoint release:
21
+
22
+ - [FireRedTTS3 source code](https://github.com/FireRedTeam/FireRedTTS3)
23
+ - [Original FireRedTTS3 weights and model card](https://huggingface.co/FireRedTeam/FireRedTTS3)
24
+
25
+ This card documents the SGLang Omni loading and serving workflow.
26
+
27
+ ## Install SGLang Omni
28
+
29
+ Use Linux with an NVIDIA CUDA GPU and Python 3.12. Follow the
30
+ [SGLang Omni installation guide](https://github.com/sgl-project/sglang-omni/blob/main/docs/get_started/installation.md)
31
+ for the CUDA, UCX, and FlashAttention prerequisites, then install from source:
32
+
33
+ ```bash
34
+ git clone https://github.com/sgl-project/sglang-omni.git
35
+ cd sglang-omni
36
+ pip install uv
37
+ uv venv .venv -p 3.12
38
+ source .venv/bin/activate
39
+ uv pip install --prerelease=allow -e '.[firered-tts3]'
40
+ ```
41
+
42
+ Use a revision containing the FireRedTTS3 integration. Before that integration
43
+ is merged, check out the corresponding PR revision before installing; an older
44
+ PyPI release or checkout does not provide this model. Run the following commands
45
+ from the `sglang-omni` repository root.
46
+
47
+ ## Download this checkpoint
48
+
49
+ For PR testing, use our temporary checkpoint at
50
+ [CKermit/FireRedTTS3-Base](https://huggingface.co/CKermit/FireRedTTS3-Base).
51
+ After this PR, we plan to publish the official checkpoint at
52
+ [FireRedTeam/FireRedTTS3-Base](https://huggingface.co/FireRedTeam/FireRedTTS3-Base).
53
+
54
+ ```bash
55
+ hf download CKermit/FireRedTTS3-Base \
56
+ --local-dir ./models/FireRedTTS3-Base
57
+ ```
58
+
59
+ Download the complete repository, including configuration, tokenizer, and
60
+ `redae/` assets, plus `campp/`. Loading only the root
61
+ `model.safetensors` file is insufficient. If you already have the complete
62
+ checkpoint locally, use that directory directly.
63
+
64
+ ## Start the server
65
+
66
+ ```bash
67
+ CUDA_VISIBLE_DEVICES=0 sgl-omni serve \
68
+ --model-path ./models/FireRedTTS3-Base \
69
+ --config examples/configs/firered_tts3_base.yaml \
70
+ --allowed-local-media-path "$PWD/docs/_static/audio" \
71
+ --port 8000
72
+ ```
73
+
74
+ Wait for `/health` to return HTTP 200 before sending requests. Initial model
75
+ loading, compilation, and warmup can take several minutes.
76
+
77
+ The reference clips below are included in the SGLang Omni repository. Local
78
+ paths are resolved on the **server**, relative to its working directory.
79
+ For your own clips, allow their directory with `--allowed-local-media-path`
80
+ and supply the matching transcript for cloning.
81
+
82
+ ## Voice cloning
83
+
84
+ Provide exactly one reference recording and its actual transcript. The example
85
+ uses the repository's `ref_voice.wav`:
86
+
87
+ ```bash
88
+ curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
89
+ -H 'Content-Type: application/json' \
90
+ -d '{
91
+ "model": "FireRedTTS3-Base",
92
+ "input": "Hello, welcome to SGLang Omni.",
93
+ "language": "English",
94
+ "references": [{
95
+ "audio_path": "docs/_static/audio/ref_voice.wav",
96
+ "text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year."
97
+ }],
98
+ "seed": 42
99
+ }' --output clone.wav
100
+ ```
101
+
102
+ ## Streaming
103
+
104
+ To stream audio, add `"stream": true` and `"response_format": "pcm"` to
105
+ the request and use `curl -N`. The response contains raw **24 kHz, signed
106
+ 16-bit little-endian, mono PCM**. Save it as `output.pcm` and convert it with:
107
+
108
+ ```bash
109
+ ffmpeg -f s16le -ar 24000 -ac 1 -i output.pcm output.wav
110
+ ```
111
+
112
+ The server also exposes `/v1/audio/speech/stream` over WebSocket. See the
113
+ [WebSocket protocol](https://github.com/sgl-project/sglang-omni/blob/main/docs/basic_usage/tts.md#websocket-speech-streaming)
114
+ for `session.config`, `input.text`, `input.commit`, and `input.done` events.
115
+
116
+ A complete streaming request:
117
+
118
+ ```bash
119
+ curl --fail-with-body -N -X POST http://localhost:8000/v1/audio/speech \
120
+ -H 'Content-Type: application/json' \
121
+ -d '{
122
+ "model": "FireRedTTS3-Base",
123
+ "input": "Hello, welcome to SGLang Omni.",
124
+ "language": "English",
125
+ "ref_audio": "docs/_static/audio/ref_voice.wav",
126
+ "ref_text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year.",
127
+ "stream": true,
128
+ "response_format": "pcm",
129
+ "seed": 42
130
+ }' --output output.pcm
131
+ ```
132
+
133
+ ## Offline Python SDK
134
+
135
+ Save this as `offline_tts.py` in the SGLang Omni repository root and run
136
+ `CUDA_VISIBLE_DEVICES=0 python offline_tts.py`. This starts the pipeline
137
+ locally without an HTTP server.
138
+
139
+ ```python
140
+ import asyncio
141
+ import multiprocessing as mp
142
+ from pathlib import Path
143
+
144
+ from sglang_omni.client import Client, GenerateRequest
145
+ from sglang_omni.models.firered_tts3.config import FireRedTTS3PipelineConfig
146
+ from sglang_omni.pipeline.mp_runner import MultiProcessPipelineRunner
147
+
148
+
149
+ async def main() -> None:
150
+ config = FireRedTTS3PipelineConfig(
151
+ model_path="./models/FireRedTTS3-Base",
152
+ )
153
+ runner = MultiProcessPipelineRunner(config)
154
+ await runner.start(timeout=600)
155
+ try:
156
+ client = Client(runner.coordinator)
157
+ result = await client.speech(
158
+ GenerateRequest(
159
+ prompt={
160
+ "input": "Hello, welcome to SGLang Omni.",
161
+ "language": "English",
162
+ "references": [{
163
+ "audio_path": "docs/_static/audio/ref_voice.wav",
164
+ "text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year.",
165
+ }],
166
+ "seed": 42,
167
+ },
168
+ stream=False,
169
+ ),
170
+ request_id="demo-1",
171
+ )
172
+ Path("offline.wav").write_bytes(result.audio_bytes)
173
+ finally:
174
+ await runner.stop()
175
+
176
+
177
+ if __name__ == "__main__":
178
+ mp.set_start_method("spawn", force=True)
179
+ asyncio.run(main())
180
+ ```
181
+
182
+ ## Serving notes
183
+
184
+ - The current deployment supports a single GPU (`tp_size=1`).
185
+ - HTTP `input` is limited to 400 characters.
186
+ - Use an explicit language: `Chinese` (default), `English`, `Japanese`, `Korean`,
187
+ `German`, `French`, `Russian`, `Portuguese`, `Spanish`, or `Italian`.
188
+ Automatic language detection (`Auto`) is unsupported. These are the stock
189
+ serving API's accepted language values; FireRedTTS3 supports additional
190
+ languages and dialects through our standalone inference pipeline.
191
+ - Submit spoken-form text when normalization is needed. The FastText and LLM
192
+ text-normalization setup in our standalone pipeline is not used in this quick start.
193
+ - Put flow-solver settings under `stage_params.talker`, for example
194
+ `{"n_timesteps": 10, "inference_cfg": 2.0, "max_patches": 200}`. Unknown
195
+ top-level fields are not forwarded as model settings.
196
+ - `seed` controls the per-request flow sampler. Different precisions, hardware,
197
+ or batching schedules can produce different results with the same seed.
198
+ - For our published model-quality results, see the
199
+ [FireRedTTS3 model card](https://huggingface.co/FireRedTeam/FireRedTTS3).
200
+ SGLang Omni runtime accuracy and performance are evaluated separately.
201
+ - The audio phase generates continuous latents. Token-sampling settings
202
+ such as `temperature`, `top_p`, and `top_k` do not control its audio output.
203
+
204
+ See the [FireRedTTS3 cookbook](https://github.com/sgl-project/sglang-omni/blob/main/docs/cookbook/firered_tts3.md)
205
+ for the full request parameters and deployment configuration.
206
+
207
+ ## Usage Disclaimer
208
+
209
+ - The project incorporates zero-shot voice cloning functionality; Please note that this capability is intended **solely for academic research purposes**.
210
+ - **DO NOT** use this model for **ANY illegal activities**❗️❗️
211
+ - The developers assume no liability for any misuse of this model.
212
+ - If you identify any instances of **abuse**, **misuse**, or **fraudulent** activities related to this project, **please report them to our team immediately.**
213
+
214
+ ## Citation
215
+
216
+ ```bib
217
+ @article{fireredtts3,
218
+ title = {FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations},
219
+ author = {FireRed Team},
220
+ journal = {arXiv preprint},
221
+ year = {2026},
222
+ }
223
+ ```
224
+
225
+ ## Acknowledgements
226
+
227
+ - [Qwen3](https://github.com/QwenLM/Qwen3) and [Qwen2-Audio](https://github.com/QwenLM/Qwen2-Audio) for the language model and audio understanding foundations
228
+ - <u>DiTAR</u> for the patch-level diffusion autoregressive formulation
229
+ - [X-Codec](https://github.com/zhenye234/xcodec) for the discriminator design used in RedAE training
230
+ - [CAM++](https://modelscope.cn/models/iic/speech_campplus_sv_en_voxceleb_16k) for speaker embedding extraction
231
+ - [fastText](https://fasttext.cc/docs/en/language-identification.html) for automatic language identification
232
+ - [WeTextProcessing](https://github.com/wenet-e2e/WeTextProcessing) (wetext) for the Chinese / English text normalization front-end
233
+
234
+ ## License
235
+
236
+ FireRedTTS3 is released under [Apache-2.0](https://github.com/FireRedTeam/FireRedTTS3/blob/main/LICENSE).
237
+ See our [original model card](https://huggingface.co/FireRedTeam/FireRedTTS3)
238
+ for usage guidance and model evaluation details.
added_tokens.json ADDED
@@ -0,0 +1,283 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "<|endoftext|>": 151643,
3
+ "<|im_start|>": 151644,
4
+ "<|im_end|>": 151645,
5
+ "<|object_ref_start|>": 151646,
6
+ "<|object_ref_end|>": 151647,
7
+ "<|box_start|>": 151648,
8
+ "<|box_end|>": 151649,
9
+ "<|quad_start|>": 151650,
10
+ "<|quad_end|>": 151651,
11
+ "<|vision_start|>": 151652,
12
+ "<|vision_end|>": 151653,
13
+ "<|vision_pad|>": 151654,
14
+ "<|image_pad|>": 151655,
15
+ "<|video_pad|>": 151656,
16
+ "<tool_call>": 151657,
17
+ "</tool_call>": 151658,
18
+ "<|fim_prefix|>": 151659,
19
+ "<|fim_middle|>": 151660,
20
+ "<|fim_suffix|>": 151661,
21
+ "<|fim_pad|>": 151662,
22
+ "<|repo_name|>": 151663,
23
+ "<|file_sep|>": 151664,
24
+ "<tool_response>": 151665,
25
+ "</tool_response>": 151666,
26
+ "<think>": 151667,
27
+ "</think>": 151668,
28
+ "<|sosp|>": 151669,
29
+ "<|eosp|>": 151670,
30
+ "<|empty|>": 151671,
31
+ "<|Human|>": 151672,
32
+ "<|SpeechLM|>": 151673,
33
+ "<|sostm|>": 151674,
34
+ "<|eostm|>": 151675,
35
+ "<|sot|>": 151676,
36
+ "<|eot|>": 151677,
37
+ "<|TEXT_ONLY|>": 151678,
38
+ "<|AUDIO_ONLY|>": 151679,
39
+ "<|ASR|>": 151680,
40
+ "<|TTS|>": 151681,
41
+ "<|INTERLEAVE|>": 151682,
42
+ "<|UNDERSTANDING|>": 151683,
43
+ "<|placeholder_001|>": 151684,
44
+ "<|placeholder_002|>": 151685,
45
+ "<|placeholder_003|>": 151686,
46
+ "<|placeholder_004|>": 151687,
47
+ "<|placeholder_005|>": 151688,
48
+ "<|placeholder_006|>": 151689,
49
+ "<|placeholder_007|>": 151690,
50
+ "<|placeholder_008|>": 151691,
51
+ "<|placeholder_009|>": 151692,
52
+ "<|placeholder_010|>": 151693,
53
+ "<|placeholder_011|>": 151694,
54
+ "<|placeholder_012|>": 151695,
55
+ "<|placeholder_013|>": 151696,
56
+ "<|placeholder_014|>": 151697,
57
+ "<|placeholder_015|>": 151698,
58
+ "<|placeholder_016|>": 151699,
59
+ "<|placeholder_017|>": 151700,
60
+ "<|placeholder_018|>": 151701,
61
+ "<|placeholder_019|>": 151702,
62
+ "<|placeholder_020|>": 151703,
63
+ "<|placeholder_021|>": 151704,
64
+ "<|placeholder_022|>": 151705,
65
+ "<|placeholder_023|>": 151706,
66
+ "<|placeholder_024|>": 151707,
67
+ "<|placeholder_025|>": 151708,
68
+ "<|placeholder_026|>": 151709,
69
+ "<|placeholder_027|>": 151710,
70
+ "<|placeholder_028|>": 151711,
71
+ "<|placeholder_029|>": 151712,
72
+ "<|placeholder_030|>": 151713,
73
+ "<|placeholder_031|>": 151714,
74
+ "<|placeholder_032|>": 151715,
75
+ "<|placeholder_033|>": 151716,
76
+ "<|placeholder_034|>": 151717,
77
+ "<|placeholder_035|>": 151718,
78
+ "<|placeholder_036|>": 151719,
79
+ "<|placeholder_037|>": 151720,
80
+ "<|placeholder_038|>": 151721,
81
+ "<|placeholder_039|>": 151722,
82
+ "<|placeholder_040|>": 151723,
83
+ "<|placeholder_041|>": 151724,
84
+ "<|placeholder_042|>": 151725,
85
+ "<|placeholder_043|>": 151726,
86
+ "<|placeholder_044|>": 151727,
87
+ "<|placeholder_045|>": 151728,
88
+ "<|placeholder_046|>": 151729,
89
+ "<|placeholder_047|>": 151730,
90
+ "<|placeholder_048|>": 151731,
91
+ "<|placeholder_049|>": 151732,
92
+ "<|placeholder_050|>": 151733,
93
+ "<|placeholder_051|>": 151734,
94
+ "<|placeholder_052|>": 151735,
95
+ "<|placeholder_053|>": 151736,
96
+ "<|placeholder_054|>": 151737,
97
+ "<|placeholder_055|>": 151738,
98
+ "<|placeholder_056|>": 151739,
99
+ "<|placeholder_057|>": 151740,
100
+ "<|placeholder_058|>": 151741,
101
+ "<|placeholder_059|>": 151742,
102
+ "<|placeholder_060|>": 151743,
103
+ "<|placeholder_061|>": 151744,
104
+ "<|placeholder_062|>": 151745,
105
+ "<|placeholder_063|>": 151746,
106
+ "<|placeholder_064|>": 151747,
107
+ "<|placeholder_065|>": 151748,
108
+ "<|placeholder_066|>": 151749,
109
+ "<|placeholder_067|>": 151750,
110
+ "<|placeholder_068|>": 151751,
111
+ "<|placeholder_069|>": 151752,
112
+ "<|placeholder_070|>": 151753,
113
+ "<|placeholder_071|>": 151754,
114
+ "<|placeholder_072|>": 151755,
115
+ "<|placeholder_073|>": 151756,
116
+ "<|placeholder_074|>": 151757,
117
+ "<|placeholder_075|>": 151758,
118
+ "<|placeholder_076|>": 151759,
119
+ "<|placeholder_077|>": 151760,
120
+ "<|placeholder_078|>": 151761,
121
+ "<|placeholder_079|>": 151762,
122
+ "<|placeholder_080|>": 151763,
123
+ "<|placeholder_081|>": 151764,
124
+ "<|placeholder_082|>": 151765,
125
+ "<|placeholder_083|>": 151766,
126
+ "<|placeholder_084|>": 151767,
127
+ "<|placeholder_085|>": 151768,
128
+ "<|placeholder_086|>": 151769,
129
+ "<|placeholder_087|>": 151770,
130
+ "<|placeholder_088|>": 151771,
131
+ "<|placeholder_089|>": 151772,
132
+ "<|placeholder_090|>": 151773,
133
+ "<|placeholder_091|>": 151774,
134
+ "<|placeholder_092|>": 151775,
135
+ "<|placeholder_093|>": 151776,
136
+ "<|placeholder_094|>": 151777,
137
+ "<|placeholder_095|>": 151778,
138
+ "<|placeholder_096|>": 151779,
139
+ "<|placeholder_097|>": 151780,
140
+ "<|placeholder_098|>": 151781,
141
+ "<|placeholder_099|>": 151782,
142
+ "<|placeholder_100|>": 151783,
143
+ "<|placeholder_101|>": 151784,
144
+ "<|placeholder_102|>": 151785,
145
+ "<|placeholder_103|>": 151786,
146
+ "<|placeholder_104|>": 151787,
147
+ "<|placeholder_105|>": 151788,
148
+ "<|placeholder_106|>": 151789,
149
+ "<|placeholder_107|>": 151790,
150
+ "<|placeholder_108|>": 151791,
151
+ "<|placeholder_109|>": 151792,
152
+ "<|placeholder_110|>": 151793,
153
+ "<|placeholder_111|>": 151794,
154
+ "<|placeholder_112|>": 151795,
155
+ "<|placeholder_113|>": 151796,
156
+ "<|placeholder_114|>": 151797,
157
+ "<|placeholder_115|>": 151798,
158
+ "<|placeholder_116|>": 151799,
159
+ "<|placeholder_117|>": 151800,
160
+ "<|placeholder_118|>": 151801,
161
+ "<|placeholder_119|>": 151802,
162
+ "<|placeholder_120|>": 151803,
163
+ "<|placeholder_121|>": 151804,
164
+ "<|placeholder_122|>": 151805,
165
+ "<|placeholder_123|>": 151806,
166
+ "<|placeholder_124|>": 151807,
167
+ "<|placeholder_125|>": 151808,
168
+ "<|placeholder_126|>": 151809,
169
+ "<|placeholder_127|>": 151810,
170
+ "<|placeholder_128|>": 151811,
171
+ "<|placeholder_129|>": 151812,
172
+ "<|placeholder_130|>": 151813,
173
+ "<|placeholder_131|>": 151814,
174
+ "<|placeholder_132|>": 151815,
175
+ "<|placeholder_133|>": 151816,
176
+ "<|placeholder_134|>": 151817,
177
+ "<|placeholder_135|>": 151818,
178
+ "<|placeholder_136|>": 151819,
179
+ "<|placeholder_137|>": 151820,
180
+ "<|placeholder_138|>": 151821,
181
+ "<|placeholder_139|>": 151822,
182
+ "<|placeholder_140|>": 151823,
183
+ "<|placeholder_141|>": 151824,
184
+ "<|placeholder_142|>": 151825,
185
+ "<|placeholder_143|>": 151826,
186
+ "<|placeholder_144|>": 151827,
187
+ "<|placeholder_145|>": 151828,
188
+ "<|placeholder_146|>": 151829,
189
+ "<|placeholder_147|>": 151830,
190
+ "<|placeholder_148|>": 151831,
191
+ "<|placeholder_149|>": 151832,
192
+ "<|placeholder_150|>": 151833,
193
+ "<|placeholder_151|>": 151834,
194
+ "<|placeholder_152|>": 151835,
195
+ "<|placeholder_153|>": 151836,
196
+ "<|placeholder_154|>": 151837,
197
+ "<|placeholder_155|>": 151838,
198
+ "<|placeholder_156|>": 151839,
199
+ "<|placeholder_157|>": 151840,
200
+ "<|placeholder_158|>": 151841,
201
+ "<|placeholder_159|>": 151842,
202
+ "<|placeholder_160|>": 151843,
203
+ "<|placeholder_161|>": 151844,
204
+ "<|placeholder_162|>": 151845,
205
+ "<|placeholder_163|>": 151846,
206
+ "<|placeholder_164|>": 151847,
207
+ "<|placeholder_165|>": 151848,
208
+ "<|placeholder_166|>": 151849,
209
+ "<|placeholder_167|>": 151850,
210
+ "<|placeholder_168|>": 151851,
211
+ "<|placeholder_169|>": 151852,
212
+ "<|placeholder_170|>": 151853,
213
+ "<|placeholder_171|>": 151854,
214
+ "<|placeholder_172|>": 151855,
215
+ "<|placeholder_173|>": 151856,
216
+ "<|placeholder_174|>": 151857,
217
+ "<|placeholder_175|>": 151858,
218
+ "<|placeholder_176|>": 151859,
219
+ "<|placeholder_177|>": 151860,
220
+ "<|placeholder_178|>": 151861,
221
+ "<|placeholder_179|>": 151862,
222
+ "<|placeholder_180|>": 151863,
223
+ "<|placeholder_181|>": 151864,
224
+ "<|placeholder_182|>": 151865,
225
+ "<|placeholder_183|>": 151866,
226
+ "<|placeholder_184|>": 151867,
227
+ "<|placeholder_185|>": 151868,
228
+ "<|placeholder_186|>": 151869,
229
+ "<|placeholder_187|>": 151870,
230
+ "<|placeholder_188|>": 151871,
231
+ "<|placeholder_189|>": 151872,
232
+ "<|placeholder_190|>": 151873,
233
+ "<|placeholder_191|>": 151874,
234
+ "<|placeholder_192|>": 151875,
235
+ "<|Chinese|>": 151876,
236
+ "<|English|>": 151877,
237
+ "<|Cantonese|>": 151878,
238
+ "<|Japanese|>": 151879,
239
+ "<|Korean|>": 151880,
240
+ "<|Spanish|>": 151881,
241
+ "<|French|>": 151882,
242
+ "<|Russian|>": 151883,
243
+ "<|Arabic|>": 151884,
244
+ "<|Turkish|>": 151885,
245
+ "<|Indonesian|>": 151886,
246
+ "<|Portuguese|>": 151887,
247
+ "<|Italian|>": 151888,
248
+ "<|Dutch|>": 151889,
249
+ "<|Vietnamese|>": 151890,
250
+ "<|German|>": 151891,
251
+ "<|Ukrainian|>": 151892,
252
+ "<|Thai|>": 151893,
253
+ "<|Polish|>": 151894,
254
+ "<|Romanian|>": 151895,
255
+ "<|Greek|>": 151896,
256
+ "<|Czech|>": 151897,
257
+ "<|Finnish|>": 151898,
258
+ "<|Hindi|>": 151899,
259
+ "<|ZH_Anhui|>": 151900,
260
+ "<|ZH_Fujian|>": 151901,
261
+ "<|ZH_Gansu|>": 151902,
262
+ "<|ZH_Guizhou|>": 151903,
263
+ "<|ZH_Hebei|>": 151904,
264
+ "<|ZH_Henan|>": 151905,
265
+ "<|ZH_Hubei|>": 151906,
266
+ "<|ZH_Hunan|>": 151907,
267
+ "<|ZH_Jiangxi|>": 151908,
268
+ "<|ZH_Liaoning|>": 151909,
269
+ "<|ZH_Minnan|>": 151910,
270
+ "<|ZH_Ningxia|>": 151911,
271
+ "<|ZH_Shaanxi|>": 151912,
272
+ "<|ZH_Shandong|>": 151913,
273
+ "<|ZH_Shanghai|>": 151914,
274
+ "<|ZH_Shanxi|>": 151915,
275
+ "<|ZH_Sichuan|>": 151916,
276
+ "<|ZH_Tianjin|>": 151917,
277
+ "<|ZH_Wenzhou|>": 151918,
278
+ "<|ZH_Wu|>": 151919,
279
+ "<|ZH_Yunnan|>": 151920,
280
+ "<|edit|>": 151921,
281
+ "<|frame_patch|>": 151922,
282
+ "<|end_edit|>": 151923
283
+ }
campp/campplus_voxceleb.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5b1a88b6f8d85826fabef804779c3372b42f3af21457fa48bd5c097c0686b2de
3
+ size 29357703
chat_template.jinja ADDED
@@ -0,0 +1,89 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- if tools %}
2
+ {{- '<|im_start|>system\n' }}
3
+ {%- if messages[0].role == 'system' %}
4
+ {{- messages[0].content + '\n\n' }}
5
+ {%- endif %}
6
+ {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
7
+ {%- for tool in tools %}
8
+ {{- "\n" }}
9
+ {{- tool | tojson }}
10
+ {%- endfor %}
11
+ {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
12
+ {%- else %}
13
+ {%- if messages[0].role == 'system' %}
14
+ {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
15
+ {%- endif %}
16
+ {%- endif %}
17
+ {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
18
+ {%- for message in messages[::-1] %}
19
+ {%- set index = (messages|length - 1) - loop.index0 %}
20
+ {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
21
+ {%- set ns.multi_step_tool = false %}
22
+ {%- set ns.last_query_index = index %}
23
+ {%- endif %}
24
+ {%- endfor %}
25
+ {%- for message in messages %}
26
+ {%- if message.content is string %}
27
+ {%- set content = message.content %}
28
+ {%- else %}
29
+ {%- set content = '' %}
30
+ {%- endif %}
31
+ {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
32
+ {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
33
+ {%- elif message.role == "assistant" %}
34
+ {%- set reasoning_content = '' %}
35
+ {%- if message.reasoning_content is string %}
36
+ {%- set reasoning_content = message.reasoning_content %}
37
+ {%- else %}
38
+ {%- if '</think>' in content %}
39
+ {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
40
+ {%- set content = content.split('</think>')[-1].lstrip('\n') %}
41
+ {%- endif %}
42
+ {%- endif %}
43
+ {%- if loop.index0 > ns.last_query_index %}
44
+ {%- if loop.last or (not loop.last and reasoning_content) %}
45
+ {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
46
+ {%- else %}
47
+ {{- '<|im_start|>' + message.role + '\n' + content }}
48
+ {%- endif %}
49
+ {%- else %}
50
+ {{- '<|im_start|>' + message.role + '\n' + content }}
51
+ {%- endif %}
52
+ {%- if message.tool_calls %}
53
+ {%- for tool_call in message.tool_calls %}
54
+ {%- if (loop.first and content) or (not loop.first) %}
55
+ {{- '\n' }}
56
+ {%- endif %}
57
+ {%- if tool_call.function %}
58
+ {%- set tool_call = tool_call.function %}
59
+ {%- endif %}
60
+ {{- '<tool_call>\n{"name": "' }}
61
+ {{- tool_call.name }}
62
+ {{- '", "arguments": ' }}
63
+ {%- if tool_call.arguments is string %}
64
+ {{- tool_call.arguments }}
65
+ {%- else %}
66
+ {{- tool_call.arguments | tojson }}
67
+ {%- endif %}
68
+ {{- '}\n</tool_call>' }}
69
+ {%- endfor %}
70
+ {%- endif %}
71
+ {{- '<|im_end|>\n' }}
72
+ {%- elif message.role == "tool" %}
73
+ {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
74
+ {{- '<|im_start|>user' }}
75
+ {%- endif %}
76
+ {{- '\n<tool_response>\n' }}
77
+ {{- content }}
78
+ {{- '\n</tool_response>' }}
79
+ {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
80
+ {{- '<|im_end|>\n' }}
81
+ {%- endif %}
82
+ {%- endif %}
83
+ {%- endfor %}
84
+ {%- if add_generation_prompt %}
85
+ {{- '<|im_start|>assistant\n' }}
86
+ {%- if enable_thinking is defined and enable_thinking is false %}
87
+ {{- '<think>\n\n</think>\n\n' }}
88
+ {%- endif %}
89
+ {%- endif %}
config.json ADDED
@@ -0,0 +1,87 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "FireRedTTS3ForConditionalGeneration"
4
+ ],
5
+ "model_type": "fireredtts3",
6
+ "redae_dim": 64,
7
+ "num_history_patches": 2,
8
+ "patch_size": 4,
9
+ "patch_encoder_hidden_size": 1024,
10
+ "patch_encoder_mlp_ratio": 4,
11
+ "patch_encoder_depth": 8,
12
+ "patch_encoder_num_heads": 16,
13
+ "dit_mlp_ratio": 3,
14
+ "dit_depth": 11,
15
+ "dit_num_heads": 16,
16
+ "dit_hidden_size": 1024,
17
+ "dtype": "float32",
18
+ "spk_in_dim": 512,
19
+ "dit_in_channels": 1600,
20
+ "audio_patch_size": 480,
21
+ "audio_sample_rate": 24000,
22
+ "bottleneck_dim": 64,
23
+ "dec_hidden_size": 896,
24
+ "dec_intermediate_size": 3584,
25
+ "dec_max_position_embeddings": 32768,
26
+ "dec_max_window_layers": 0,
27
+ "dec_num_attention_heads": 14,
28
+ "dec_num_hidden_layers": 18,
29
+ "dec_num_key_value_heads": 2,
30
+ "dec_sliding_window": 64,
31
+ "dec_use_sliding_window": true,
32
+ "enc_downsample_num_hidden_layers": 4,
33
+ "enc_extra_downsample_rate": 2,
34
+ "enc_hidden_size": 896,
35
+ "enc_intermediate_size": 3584,
36
+ "enc_max_position_embeddings": 32768,
37
+ "enc_max_window_layers": 0,
38
+ "enc_num_attention_heads": 14,
39
+ "enc_num_hidden_layers": 18,
40
+ "enc_num_key_value_heads": 2,
41
+ "enc_sliding_window": 64,
42
+ "enc_use_sliding_window": true,
43
+ "vocab_size": 151936,
44
+ "hidden_size": 2048,
45
+ "intermediate_size": 6144,
46
+ "num_hidden_layers": 28,
47
+ "num_attention_heads": 16,
48
+ "num_key_value_heads": 8,
49
+ "head_dim": 128,
50
+ "hidden_act": "silu",
51
+ "max_position_embeddings": 40960,
52
+ "rms_norm_eps": 1e-06,
53
+ "rope_theta": 1000000.0,
54
+ "tie_word_embeddings": true,
55
+ "attention_bias": false,
56
+ "max_window_layers": 28,
57
+ "use_sliding_window": false,
58
+ "sliding_window": null,
59
+ "bos_token_id": 151643,
60
+ "eos_token_id": 151645,
61
+ "use_lm_head": false,
62
+ "redae_scale": 1.0,
63
+ "speaker_token": null,
64
+ "text_latent_interleave_ratio": null,
65
+ "special_tokens": {
66
+ "text_start": "<|sot|>",
67
+ "text_end": "<|eot|>",
68
+ "audio_sos": null,
69
+ "audio_eos": null,
70
+ "audio_in_placeholder": null,
71
+ "icl_audio_placeholder": null,
72
+ "audio_holder": null,
73
+ "text_holder": null,
74
+ "text_pad": null,
75
+ "audio_start": null,
76
+ "audio_end": null
77
+ },
78
+ "generation_defaults": {
79
+ "n_timesteps": 10,
80
+ "inference_cfg": 2.0,
81
+ "stop_threshold": 0.5,
82
+ "min_patches": 6,
83
+ "max_patches": 400,
84
+ "flow_mode": "compile-bf16"
85
+ },
86
+ "transformers_version": "5.14.1"
87
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d6ceed109a04207ef48bc669fb68248d3a44d990fcf2b9641990a6134d2ebb8a
3
+ size 8482608484
redae/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0723e87fdaf46d377f01aa32ce56b43f43f9b249415bc8b8f81040eaa8b14abe
3
+ size 3775160672
special_tokens_map.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": null,
3
+ "eos_token": "<|im_end|>",
4
+ "unk_token": null,
5
+ "pad_token": "<|endoftext|>",
6
+ "additional_special_tokens": [
7
+ "<|im_start|>",
8
+ "<|im_end|>",
9
+ "<|object_ref_start|>",
10
+ "<|object_ref_end|>",
11
+ "<|box_start|>",
12
+ "<|box_end|>",
13
+ "<|quad_start|>",
14
+ "<|quad_end|>",
15
+ "<|vision_start|>",
16
+ "<|vision_end|>",
17
+ "<|vision_pad|>",
18
+ "<|image_pad|>",
19
+ "<|video_pad|>"
20
+ ]
21
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d84d0d2d4926a03d2d5bb3fed5b3fedb2299d0ca7e933bed7e8a3ad2ce73f3d0
3
+ size 11472157
tokenizer_config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": null,
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|im_end|>",
7
+ "errors": "replace",
8
+ "extra_special_tokens": [
9
+ "<|im_start|>",
10
+ "<|im_end|>",
11
+ "<|object_ref_start|>",
12
+ "<|object_ref_end|>",
13
+ "<|box_start|>",
14
+ "<|box_end|>",
15
+ "<|quad_start|>",
16
+ "<|quad_end|>",
17
+ "<|vision_start|>",
18
+ "<|vision_end|>",
19
+ "<|vision_pad|>",
20
+ "<|image_pad|>",
21
+ "<|video_pad|>"
22
+ ],
23
+ "is_local": true,
24
+ "local_files_only": false,
25
+ "model_max_length": 131072,
26
+ "pad_token": "<|endoftext|>",
27
+ "split_special_tokens": false,
28
+ "tokenizer_class": "Qwen2Tokenizer",
29
+ "unk_token": null
30
+ }
vocab.json ADDED
The diff for this file is too large to render. See raw diff