Upload FireRedTTS3 checkpoint
Browse files- .gitattributes +1 -0
- README.md +296 -0
- added_tokens.json +283 -0
- chat_template.jinja +89 -0
- config.json +85 -0
- merges.txt +0 -0
- model.safetensors +3 -0
- redae/model.safetensors +3 -0
- special_tokens_map.json +21 -0
- tokenizer.json +3 -0
- tokenizer_config.json +30 -0
- vocab.json +0 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,296 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: FireRedTeam/FireRedTTS3
|
| 4 |
+
pipeline_tag: text-to-speech
|
| 5 |
+
tags:
|
| 6 |
+
- sglang-omni
|
| 7 |
+
- fireredtts3
|
| 8 |
+
- text-to-speech
|
| 9 |
+
- voice-cloning
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# FireRedTTS3-Instruct for SGLang Omni
|
| 13 |
+
|
| 14 |
+
We provide our FireRedTTS3-Instruct checkpoint for inference with
|
| 15 |
+
[SGLang Omni](https://github.com/sgl-project/sglang-omni). It supports
|
| 16 |
+
reference-audio voice cloning, instruction-guided voice design, semantic editing, and acoustic editing.
|
| 17 |
+
Audio output is 24 kHz mono, with WAV responses and streaming PCM available
|
| 18 |
+
through the `/v1/audio/speech` API.
|
| 19 |
+
|
| 20 |
+
Our model code and original checkpoint release:
|
| 21 |
+
|
| 22 |
+
- [FireRedTTS3 source code](https://github.com/FireRedTeam/FireRedTTS3)
|
| 23 |
+
- [Original FireRedTTS3 weights and model card](https://huggingface.co/FireRedTeam/FireRedTTS3)
|
| 24 |
+
|
| 25 |
+
This card documents the SGLang Omni loading and serving workflow.
|
| 26 |
+
|
| 27 |
+
## Install SGLang Omni
|
| 28 |
+
|
| 29 |
+
Use Linux with an NVIDIA CUDA GPU and Python 3.12. Follow the
|
| 30 |
+
[SGLang Omni installation guide](https://github.com/sgl-project/sglang-omni/blob/main/docs/get_started/installation.md)
|
| 31 |
+
for the CUDA, UCX, and FlashAttention prerequisites, then install from source:
|
| 32 |
+
|
| 33 |
+
```bash
|
| 34 |
+
git clone https://github.com/sgl-project/sglang-omni.git
|
| 35 |
+
cd sglang-omni
|
| 36 |
+
pip install uv
|
| 37 |
+
uv venv .venv -p 3.12
|
| 38 |
+
source .venv/bin/activate
|
| 39 |
+
uv pip install --prerelease=allow -e '.[firered-tts3]'
|
| 40 |
+
```
|
| 41 |
+
|
| 42 |
+
Use a revision containing the FireRedTTS3 integration. Before that integration
|
| 43 |
+
is merged, check out the corresponding PR revision before installing; an older
|
| 44 |
+
PyPI release or checkout does not provide this model. Run the following commands
|
| 45 |
+
from the `sglang-omni` repository root.
|
| 46 |
+
|
| 47 |
+
## Download this checkpoint
|
| 48 |
+
|
| 49 |
+
For PR testing, use our temporary checkpoint at
|
| 50 |
+
[CKermit/FireRedTTS3-Instruct](https://huggingface.co/CKermit/FireRedTTS3-Instruct).
|
| 51 |
+
After this PR, we plan to publish the official checkpoint at
|
| 52 |
+
[FireRedTeam/FireRedTTS3-Instruct](https://huggingface.co/FireRedTeam/FireRedTTS3-Instruct).
|
| 53 |
+
|
| 54 |
+
```bash
|
| 55 |
+
hf download CKermit/FireRedTTS3-Instruct \
|
| 56 |
+
--local-dir ./models/FireRedTTS3-Instruct
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
Download the complete repository, including configuration, tokenizer, and
|
| 60 |
+
`redae/` assets. Loading only the root
|
| 61 |
+
`model.safetensors` file is insufficient. If you already have the complete
|
| 62 |
+
checkpoint locally, use that directory directly.
|
| 63 |
+
|
| 64 |
+
## Start the server
|
| 65 |
+
|
| 66 |
+
```bash
|
| 67 |
+
CUDA_VISIBLE_DEVICES=0 sgl-omni serve \
|
| 68 |
+
--model-path ./models/FireRedTTS3-Instruct \
|
| 69 |
+
--config examples/configs/firered_tts3_instruct.yaml \
|
| 70 |
+
--allowed-local-media-path "$PWD/docs/_static/audio" \
|
| 71 |
+
--port 8000
|
| 72 |
+
```
|
| 73 |
+
|
| 74 |
+
Wait for `/health` to return HTTP 200 before sending requests. Initial model
|
| 75 |
+
loading, compilation, and warmup can take several minutes.
|
| 76 |
+
|
| 77 |
+
The reference clips below are included in the SGLang Omni repository. Local
|
| 78 |
+
paths are resolved on the **server**, relative to its working directory.
|
| 79 |
+
For your own clips, allow their directory with `--allowed-local-media-path`
|
| 80 |
+
and supply the matching transcript for cloning.
|
| 81 |
+
|
| 82 |
+
## Voice cloning
|
| 83 |
+
|
| 84 |
+
Provide exactly one reference recording and its actual transcript. The example
|
| 85 |
+
uses the repository's `ref_voice.wav`:
|
| 86 |
+
|
| 87 |
+
```bash
|
| 88 |
+
curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
|
| 89 |
+
-H 'Content-Type: application/json' \
|
| 90 |
+
-d '{
|
| 91 |
+
"model": "FireRedTTS3-Instruct",
|
| 92 |
+
"input": "Hello, welcome to SGLang Omni.",
|
| 93 |
+
"language": "English",
|
| 94 |
+
"references": [{
|
| 95 |
+
"audio_path": "docs/_static/audio/ref_voice.wav",
|
| 96 |
+
"text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year."
|
| 97 |
+
}],
|
| 98 |
+
"seed": 42
|
| 99 |
+
}' --output clone.wav
|
| 100 |
+
```
|
| 101 |
+
|
| 102 |
+
Omit `task_type` for cloning, or set it to `Base`. The value `tts` is not an
|
| 103 |
+
accepted serving task name.
|
| 104 |
+
|
| 105 |
+
## Voice design
|
| 106 |
+
|
| 107 |
+
Generate a voice from a description, without a reference recording:
|
| 108 |
+
|
| 109 |
+
```bash
|
| 110 |
+
curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
|
| 111 |
+
-H 'Content-Type: application/json' \
|
| 112 |
+
-d '{
|
| 113 |
+
"model": "FireRedTTS3-Instruct",
|
| 114 |
+
"input": "今天天气很好,我们一起去公园散步吧。",
|
| 115 |
+
"task_type": "VoiceDesign",
|
| 116 |
+
"instructions": "一个年轻女性的温柔嗓音,语速稍慢,带一点俏皮。",
|
| 117 |
+
"seed": 42
|
| 118 |
+
}' --output voice_design.wav
|
| 119 |
+
```
|
| 120 |
+
|
| 121 |
+
## Semantic editing
|
| 122 |
+
|
| 123 |
+
Use the reference recording as the source to edit. Its content supplies the
|
| 124 |
+
input, so `input` may be empty; no reference transcript is required:
|
| 125 |
+
|
| 126 |
+
```bash
|
| 127 |
+
curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
|
| 128 |
+
-H 'Content-Type: application/json' \
|
| 129 |
+
--data-binary @- --output semantic_edit.wav <<'JSON'
|
| 130 |
+
{
|
| 131 |
+
"model": "FireRedTTS3-Instruct",
|
| 132 |
+
"input": "",
|
| 133 |
+
"task_type": "SemanticEdit",
|
| 134 |
+
"instructions": "Replace 'birthday' with 'anniversary'.",
|
| 135 |
+
"ref_audio": "docs/_static/audio/ref_voice.wav",
|
| 136 |
+
"seed": 42
|
| 137 |
+
}
|
| 138 |
+
JSON
|
| 139 |
+
```
|
| 140 |
+
|
| 141 |
+
For substitutions, use the quoted form `Replace 'source' with 'target'.`
|
| 142 |
+
The quoted example above was verified by transcribing the generated audio;
|
| 143 |
+
omitting the quotes did not perform the intended replacement in this test.
|
| 144 |
+
The `x-gen-text` response header contains text with edit masks, so the target
|
| 145 |
+
word may appear only in the audio. Validate the rendered audio rather than
|
| 146 |
+
requiring the target word to appear in that header.
|
| 147 |
+
|
| 148 |
+
## Acoustic editing
|
| 149 |
+
|
| 150 |
+
Adjust the source recording's speed, pitch, or volume. This example requests
|
| 151 |
+
0.8x speed through the model's native acoustic-edit task:
|
| 152 |
+
|
| 153 |
+
```bash
|
| 154 |
+
curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
|
| 155 |
+
-H 'Content-Type: application/json' \
|
| 156 |
+
-d '{
|
| 157 |
+
"model": "FireRedTTS3-Instruct",
|
| 158 |
+
"input": "",
|
| 159 |
+
"task_type": "AcousticEdit",
|
| 160 |
+
"ref_audio": "docs/_static/audio/ref_voice.wav",
|
| 161 |
+
"speed": 0.8,
|
| 162 |
+
"seed": 42
|
| 163 |
+
}' --output acoustic_edit.wav
|
| 164 |
+
```
|
| 165 |
+
|
| 166 |
+
Raw acoustic instructions must follow the trained templates rather than
|
| 167 |
+
free-form descriptions. For pitch or volume, use, for example,
|
| 168 |
+
`"instructions": "shift the pitch by 2 steps"` or
|
| 169 |
+
`"instructions": "adjust the volume to 0.8"`. An explicit `instructions`
|
| 170 |
+
string takes precedence over the composed speed instruction. The `speed`
|
| 171 |
+
field applies to `AcousticEdit`; it does not change cloning or voice design.
|
| 172 |
+
Edit sources are capped at 20 seconds. VoiceDesign and SemanticEdit generate
|
| 173 |
+
text before audio, so their first-audio latency differs from cloning.
|
| 174 |
+
|
| 175 |
+
## Streaming
|
| 176 |
+
|
| 177 |
+
To stream audio, add `"stream": true` and `"response_format": "pcm"` to
|
| 178 |
+
the request and use `curl -N`. The response contains raw **24 kHz, signed
|
| 179 |
+
16-bit little-endian, mono PCM**. Save it as `output.pcm` and convert it with:
|
| 180 |
+
|
| 181 |
+
```bash
|
| 182 |
+
ffmpeg -f s16le -ar 24000 -ac 1 -i output.pcm output.wav
|
| 183 |
+
```
|
| 184 |
+
|
| 185 |
+
The server also exposes `/v1/audio/speech/stream` over WebSocket. See the
|
| 186 |
+
[WebSocket protocol](https://github.com/sgl-project/sglang-omni/blob/main/docs/basic_usage/tts.md#websocket-speech-streaming)
|
| 187 |
+
for `session.config`, `input.text`, `input.commit`, and `input.done` events.
|
| 188 |
+
|
| 189 |
+
## Offline Python SDK
|
| 190 |
+
|
| 191 |
+
Save this as `offline_tts.py` in the SGLang Omni repository root and run
|
| 192 |
+
`CUDA_VISIBLE_DEVICES=0 python offline_tts.py`. This starts the pipeline
|
| 193 |
+
locally without an HTTP server.
|
| 194 |
+
|
| 195 |
+
```python
|
| 196 |
+
import asyncio
|
| 197 |
+
import multiprocessing as mp
|
| 198 |
+
from pathlib import Path
|
| 199 |
+
|
| 200 |
+
from sglang_omni.client import Client, GenerateRequest
|
| 201 |
+
from sglang_omni.models.firered_tts3.config import FireRedTTS3PipelineConfig
|
| 202 |
+
from sglang_omni.pipeline.mp_runner import MultiProcessPipelineRunner
|
| 203 |
+
|
| 204 |
+
|
| 205 |
+
async def main() -> None:
|
| 206 |
+
config = FireRedTTS3PipelineConfig(
|
| 207 |
+
model_path="./models/FireRedTTS3-Instruct",
|
| 208 |
+
)
|
| 209 |
+
config.stage_named("talker").factory.flow_mode = "fp32"
|
| 210 |
+
runner = MultiProcessPipelineRunner(config)
|
| 211 |
+
await runner.start(timeout=600)
|
| 212 |
+
try:
|
| 213 |
+
client = Client(runner.coordinator)
|
| 214 |
+
result = await client.speech(
|
| 215 |
+
GenerateRequest(
|
| 216 |
+
prompt={
|
| 217 |
+
"input": "今天天气很好,我们一起去公园散步吧。",
|
| 218 |
+
"task_type": "VoiceDesign",
|
| 219 |
+
"instructions": "一个年轻女性的温柔嗓音,语速稍慢,带一点俏皮。",
|
| 220 |
+
"seed": 42,
|
| 221 |
+
},
|
| 222 |
+
stream=False,
|
| 223 |
+
),
|
| 224 |
+
request_id="demo-1",
|
| 225 |
+
)
|
| 226 |
+
Path("offline.wav").write_bytes(result.audio_bytes)
|
| 227 |
+
finally:
|
| 228 |
+
await runner.stop()
|
| 229 |
+
|
| 230 |
+
|
| 231 |
+
if __name__ == "__main__":
|
| 232 |
+
mp.set_start_method("spawn", force=True)
|
| 233 |
+
asyncio.run(main())
|
| 234 |
+
```
|
| 235 |
+
|
| 236 |
+
## Serving notes
|
| 237 |
+
|
| 238 |
+
- The current deployment supports a single GPU (`tp_size=1`).
|
| 239 |
+
- HTTP `input` is limited to 400 characters. Editing requests may use an empty
|
| 240 |
+
`input` as shown above.
|
| 241 |
+
- Use an explicit language: `Chinese` (default), `English`, `Japanese`, `Korean`,
|
| 242 |
+
`German`, `French`, `Russian`, `Portuguese`, `Spanish`, or `Italian`.
|
| 243 |
+
Automatic language detection (`Auto`) is unsupported. These are the stock
|
| 244 |
+
serving API's accepted language values; FireRedTTS3 supports additional
|
| 245 |
+
languages and dialects through our standalone inference pipeline.
|
| 246 |
+
- Submit spoken-form text when normalization is needed. The FastText and LLM
|
| 247 |
+
text-normalization setup in our standalone pipeline is not used in this quick start.
|
| 248 |
+
- Put flow-solver settings under `stage_params.talker`, for example
|
| 249 |
+
`{"n_timesteps": 10, "inference_cfg": 2.0, "max_patches": 200}`. Unknown
|
| 250 |
+
top-level fields are not forwarded as model settings.
|
| 251 |
+
- `seed` controls the per-request flow sampler. Different precisions, hardware,
|
| 252 |
+
or batching schedules can produce different results with the same seed.
|
| 253 |
+
- For our published model-quality results, see the
|
| 254 |
+
[FireRedTTS3 model card](https://huggingface.co/FireRedTeam/FireRedTTS3).
|
| 255 |
+
SGLang Omni runtime accuracy and performance are evaluated separately.
|
| 256 |
+
- Keep the example Instruct configuration's `flow_mode: fp32` for the
|
| 257 |
+
supported default. The offline example sets it explicitly because it does
|
| 258 |
+
not load the example YAML.
|
| 259 |
+
- `temperature`, `top_p`, and `top_k` affect the VoiceDesign/SemanticEdit text
|
| 260 |
+
phase. They do not control continuous-latent audio generation.
|
| 261 |
+
|
| 262 |
+
See the [FireRedTTS3 cookbook](https://github.com/sgl-project/sglang-omni/blob/main/docs/cookbook/firered_tts3.md)
|
| 263 |
+
for the full request parameters and deployment configuration.
|
| 264 |
+
|
| 265 |
+
## Usage Disclaimer
|
| 266 |
+
|
| 267 |
+
- The project incorporates zero-shot voice cloning functionality; Please note that this capability is intended **solely for academic research purposes**.
|
| 268 |
+
- **DO NOT** use this model for **ANY illegal activities**❗️❗️
|
| 269 |
+
- The developers assume no liability for any misuse of this model.
|
| 270 |
+
- If you identify any instances of **abuse**, **misuse**, or **fraudulent** activities related to this project, **please report them to our team immediately.**
|
| 271 |
+
|
| 272 |
+
## Citation
|
| 273 |
+
|
| 274 |
+
```bib
|
| 275 |
+
@article{fireredtts3,
|
| 276 |
+
title = {FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations},
|
| 277 |
+
author = {FireRed Team},
|
| 278 |
+
journal = {arXiv preprint},
|
| 279 |
+
year = {2026},
|
| 280 |
+
}
|
| 281 |
+
```
|
| 282 |
+
|
| 283 |
+
## Acknowledgements
|
| 284 |
+
|
| 285 |
+
- [Qwen3](https://github.com/QwenLM/Qwen3) and [Qwen2-Audio](https://github.com/QwenLM/Qwen2-Audio) for the language model and audio understanding foundations
|
| 286 |
+
- <u>DiTAR</u> for the patch-level diffusion autoregressive formulation
|
| 287 |
+
- [X-Codec](https://github.com/zhenye234/xcodec) for the discriminator design used in RedAE training
|
| 288 |
+
- [CAM++](https://modelscope.cn/models/iic/speech_campplus_sv_en_voxceleb_16k) for speaker embedding extraction
|
| 289 |
+
- [fastText](https://fasttext.cc/docs/en/language-identification.html) for automatic language identification
|
| 290 |
+
- [WeTextProcessing](https://github.com/wenet-e2e/WeTextProcessing) (wetext) for the Chinese / English text normalization front-end
|
| 291 |
+
|
| 292 |
+
## License
|
| 293 |
+
|
| 294 |
+
FireRedTTS3 is released under [Apache-2.0](https://github.com/FireRedTeam/FireRedTTS3/blob/main/LICENSE).
|
| 295 |
+
See our [original model card](https://huggingface.co/FireRedTeam/FireRedTTS3)
|
| 296 |
+
for usage guidance and model evaluation details.
|
added_tokens.json
ADDED
|
@@ -0,0 +1,283 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"<|endoftext|>": 151643,
|
| 3 |
+
"<|im_start|>": 151644,
|
| 4 |
+
"<|im_end|>": 151645,
|
| 5 |
+
"<|object_ref_start|>": 151646,
|
| 6 |
+
"<|object_ref_end|>": 151647,
|
| 7 |
+
"<|box_start|>": 151648,
|
| 8 |
+
"<|box_end|>": 151649,
|
| 9 |
+
"<|quad_start|>": 151650,
|
| 10 |
+
"<|quad_end|>": 151651,
|
| 11 |
+
"<|vision_start|>": 151652,
|
| 12 |
+
"<|vision_end|>": 151653,
|
| 13 |
+
"<|vision_pad|>": 151654,
|
| 14 |
+
"<|image_pad|>": 151655,
|
| 15 |
+
"<|video_pad|>": 151656,
|
| 16 |
+
"<tool_call>": 151657,
|
| 17 |
+
"</tool_call>": 151658,
|
| 18 |
+
"<|fim_prefix|>": 151659,
|
| 19 |
+
"<|fim_middle|>": 151660,
|
| 20 |
+
"<|fim_suffix|>": 151661,
|
| 21 |
+
"<|fim_pad|>": 151662,
|
| 22 |
+
"<|repo_name|>": 151663,
|
| 23 |
+
"<|file_sep|>": 151664,
|
| 24 |
+
"<tool_response>": 151665,
|
| 25 |
+
"</tool_response>": 151666,
|
| 26 |
+
"<think>": 151667,
|
| 27 |
+
"</think>": 151668,
|
| 28 |
+
"<|sosp|>": 151669,
|
| 29 |
+
"<|eosp|>": 151670,
|
| 30 |
+
"<|empty|>": 151671,
|
| 31 |
+
"<|Human|>": 151672,
|
| 32 |
+
"<|SpeechLM|>": 151673,
|
| 33 |
+
"<|sostm|>": 151674,
|
| 34 |
+
"<|eostm|>": 151675,
|
| 35 |
+
"<|sot|>": 151676,
|
| 36 |
+
"<|eot|>": 151677,
|
| 37 |
+
"<|TEXT_ONLY|>": 151678,
|
| 38 |
+
"<|AUDIO_ONLY|>": 151679,
|
| 39 |
+
"<|ASR|>": 151680,
|
| 40 |
+
"<|TTS|>": 151681,
|
| 41 |
+
"<|INTERLEAVE|>": 151682,
|
| 42 |
+
"<|UNDERSTANDING|>": 151683,
|
| 43 |
+
"<|placeholder_001|>": 151684,
|
| 44 |
+
"<|placeholder_002|>": 151685,
|
| 45 |
+
"<|placeholder_003|>": 151686,
|
| 46 |
+
"<|placeholder_004|>": 151687,
|
| 47 |
+
"<|placeholder_005|>": 151688,
|
| 48 |
+
"<|placeholder_006|>": 151689,
|
| 49 |
+
"<|placeholder_007|>": 151690,
|
| 50 |
+
"<|placeholder_008|>": 151691,
|
| 51 |
+
"<|placeholder_009|>": 151692,
|
| 52 |
+
"<|placeholder_010|>": 151693,
|
| 53 |
+
"<|placeholder_011|>": 151694,
|
| 54 |
+
"<|placeholder_012|>": 151695,
|
| 55 |
+
"<|placeholder_013|>": 151696,
|
| 56 |
+
"<|placeholder_014|>": 151697,
|
| 57 |
+
"<|placeholder_015|>": 151698,
|
| 58 |
+
"<|placeholder_016|>": 151699,
|
| 59 |
+
"<|placeholder_017|>": 151700,
|
| 60 |
+
"<|placeholder_018|>": 151701,
|
| 61 |
+
"<|placeholder_019|>": 151702,
|
| 62 |
+
"<|placeholder_020|>": 151703,
|
| 63 |
+
"<|placeholder_021|>": 151704,
|
| 64 |
+
"<|placeholder_022|>": 151705,
|
| 65 |
+
"<|placeholder_023|>": 151706,
|
| 66 |
+
"<|placeholder_024|>": 151707,
|
| 67 |
+
"<|placeholder_025|>": 151708,
|
| 68 |
+
"<|placeholder_026|>": 151709,
|
| 69 |
+
"<|placeholder_027|>": 151710,
|
| 70 |
+
"<|placeholder_028|>": 151711,
|
| 71 |
+
"<|placeholder_029|>": 151712,
|
| 72 |
+
"<|placeholder_030|>": 151713,
|
| 73 |
+
"<|placeholder_031|>": 151714,
|
| 74 |
+
"<|placeholder_032|>": 151715,
|
| 75 |
+
"<|placeholder_033|>": 151716,
|
| 76 |
+
"<|placeholder_034|>": 151717,
|
| 77 |
+
"<|placeholder_035|>": 151718,
|
| 78 |
+
"<|placeholder_036|>": 151719,
|
| 79 |
+
"<|placeholder_037|>": 151720,
|
| 80 |
+
"<|placeholder_038|>": 151721,
|
| 81 |
+
"<|placeholder_039|>": 151722,
|
| 82 |
+
"<|placeholder_040|>": 151723,
|
| 83 |
+
"<|placeholder_041|>": 151724,
|
| 84 |
+
"<|placeholder_042|>": 151725,
|
| 85 |
+
"<|placeholder_043|>": 151726,
|
| 86 |
+
"<|placeholder_044|>": 151727,
|
| 87 |
+
"<|placeholder_045|>": 151728,
|
| 88 |
+
"<|placeholder_046|>": 151729,
|
| 89 |
+
"<|placeholder_047|>": 151730,
|
| 90 |
+
"<|placeholder_048|>": 151731,
|
| 91 |
+
"<|placeholder_049|>": 151732,
|
| 92 |
+
"<|placeholder_050|>": 151733,
|
| 93 |
+
"<|placeholder_051|>": 151734,
|
| 94 |
+
"<|placeholder_052|>": 151735,
|
| 95 |
+
"<|placeholder_053|>": 151736,
|
| 96 |
+
"<|placeholder_054|>": 151737,
|
| 97 |
+
"<|placeholder_055|>": 151738,
|
| 98 |
+
"<|placeholder_056|>": 151739,
|
| 99 |
+
"<|placeholder_057|>": 151740,
|
| 100 |
+
"<|placeholder_058|>": 151741,
|
| 101 |
+
"<|placeholder_059|>": 151742,
|
| 102 |
+
"<|placeholder_060|>": 151743,
|
| 103 |
+
"<|placeholder_061|>": 151744,
|
| 104 |
+
"<|placeholder_062|>": 151745,
|
| 105 |
+
"<|placeholder_063|>": 151746,
|
| 106 |
+
"<|placeholder_064|>": 151747,
|
| 107 |
+
"<|placeholder_065|>": 151748,
|
| 108 |
+
"<|placeholder_066|>": 151749,
|
| 109 |
+
"<|placeholder_067|>": 151750,
|
| 110 |
+
"<|placeholder_068|>": 151751,
|
| 111 |
+
"<|placeholder_069|>": 151752,
|
| 112 |
+
"<|placeholder_070|>": 151753,
|
| 113 |
+
"<|placeholder_071|>": 151754,
|
| 114 |
+
"<|placeholder_072|>": 151755,
|
| 115 |
+
"<|placeholder_073|>": 151756,
|
| 116 |
+
"<|placeholder_074|>": 151757,
|
| 117 |
+
"<|placeholder_075|>": 151758,
|
| 118 |
+
"<|placeholder_076|>": 151759,
|
| 119 |
+
"<|placeholder_077|>": 151760,
|
| 120 |
+
"<|placeholder_078|>": 151761,
|
| 121 |
+
"<|placeholder_079|>": 151762,
|
| 122 |
+
"<|placeholder_080|>": 151763,
|
| 123 |
+
"<|placeholder_081|>": 151764,
|
| 124 |
+
"<|placeholder_082|>": 151765,
|
| 125 |
+
"<|placeholder_083|>": 151766,
|
| 126 |
+
"<|placeholder_084|>": 151767,
|
| 127 |
+
"<|placeholder_085|>": 151768,
|
| 128 |
+
"<|placeholder_086|>": 151769,
|
| 129 |
+
"<|placeholder_087|>": 151770,
|
| 130 |
+
"<|placeholder_088|>": 151771,
|
| 131 |
+
"<|placeholder_089|>": 151772,
|
| 132 |
+
"<|placeholder_090|>": 151773,
|
| 133 |
+
"<|placeholder_091|>": 151774,
|
| 134 |
+
"<|placeholder_092|>": 151775,
|
| 135 |
+
"<|placeholder_093|>": 151776,
|
| 136 |
+
"<|placeholder_094|>": 151777,
|
| 137 |
+
"<|placeholder_095|>": 151778,
|
| 138 |
+
"<|placeholder_096|>": 151779,
|
| 139 |
+
"<|placeholder_097|>": 151780,
|
| 140 |
+
"<|placeholder_098|>": 151781,
|
| 141 |
+
"<|placeholder_099|>": 151782,
|
| 142 |
+
"<|placeholder_100|>": 151783,
|
| 143 |
+
"<|placeholder_101|>": 151784,
|
| 144 |
+
"<|placeholder_102|>": 151785,
|
| 145 |
+
"<|placeholder_103|>": 151786,
|
| 146 |
+
"<|placeholder_104|>": 151787,
|
| 147 |
+
"<|placeholder_105|>": 151788,
|
| 148 |
+
"<|placeholder_106|>": 151789,
|
| 149 |
+
"<|placeholder_107|>": 151790,
|
| 150 |
+
"<|placeholder_108|>": 151791,
|
| 151 |
+
"<|placeholder_109|>": 151792,
|
| 152 |
+
"<|placeholder_110|>": 151793,
|
| 153 |
+
"<|placeholder_111|>": 151794,
|
| 154 |
+
"<|placeholder_112|>": 151795,
|
| 155 |
+
"<|placeholder_113|>": 151796,
|
| 156 |
+
"<|placeholder_114|>": 151797,
|
| 157 |
+
"<|placeholder_115|>": 151798,
|
| 158 |
+
"<|placeholder_116|>": 151799,
|
| 159 |
+
"<|placeholder_117|>": 151800,
|
| 160 |
+
"<|placeholder_118|>": 151801,
|
| 161 |
+
"<|placeholder_119|>": 151802,
|
| 162 |
+
"<|placeholder_120|>": 151803,
|
| 163 |
+
"<|placeholder_121|>": 151804,
|
| 164 |
+
"<|placeholder_122|>": 151805,
|
| 165 |
+
"<|placeholder_123|>": 151806,
|
| 166 |
+
"<|placeholder_124|>": 151807,
|
| 167 |
+
"<|placeholder_125|>": 151808,
|
| 168 |
+
"<|placeholder_126|>": 151809,
|
| 169 |
+
"<|placeholder_127|>": 151810,
|
| 170 |
+
"<|placeholder_128|>": 151811,
|
| 171 |
+
"<|placeholder_129|>": 151812,
|
| 172 |
+
"<|placeholder_130|>": 151813,
|
| 173 |
+
"<|placeholder_131|>": 151814,
|
| 174 |
+
"<|placeholder_132|>": 151815,
|
| 175 |
+
"<|placeholder_133|>": 151816,
|
| 176 |
+
"<|placeholder_134|>": 151817,
|
| 177 |
+
"<|placeholder_135|>": 151818,
|
| 178 |
+
"<|placeholder_136|>": 151819,
|
| 179 |
+
"<|placeholder_137|>": 151820,
|
| 180 |
+
"<|placeholder_138|>": 151821,
|
| 181 |
+
"<|placeholder_139|>": 151822,
|
| 182 |
+
"<|placeholder_140|>": 151823,
|
| 183 |
+
"<|placeholder_141|>": 151824,
|
| 184 |
+
"<|placeholder_142|>": 151825,
|
| 185 |
+
"<|placeholder_143|>": 151826,
|
| 186 |
+
"<|placeholder_144|>": 151827,
|
| 187 |
+
"<|placeholder_145|>": 151828,
|
| 188 |
+
"<|placeholder_146|>": 151829,
|
| 189 |
+
"<|placeholder_147|>": 151830,
|
| 190 |
+
"<|placeholder_148|>": 151831,
|
| 191 |
+
"<|placeholder_149|>": 151832,
|
| 192 |
+
"<|placeholder_150|>": 151833,
|
| 193 |
+
"<|placeholder_151|>": 151834,
|
| 194 |
+
"<|placeholder_152|>": 151835,
|
| 195 |
+
"<|placeholder_153|>": 151836,
|
| 196 |
+
"<|placeholder_154|>": 151837,
|
| 197 |
+
"<|placeholder_155|>": 151838,
|
| 198 |
+
"<|placeholder_156|>": 151839,
|
| 199 |
+
"<|placeholder_157|>": 151840,
|
| 200 |
+
"<|placeholder_158|>": 151841,
|
| 201 |
+
"<|placeholder_159|>": 151842,
|
| 202 |
+
"<|placeholder_160|>": 151843,
|
| 203 |
+
"<|placeholder_161|>": 151844,
|
| 204 |
+
"<|placeholder_162|>": 151845,
|
| 205 |
+
"<|placeholder_163|>": 151846,
|
| 206 |
+
"<|placeholder_164|>": 151847,
|
| 207 |
+
"<|placeholder_165|>": 151848,
|
| 208 |
+
"<|placeholder_166|>": 151849,
|
| 209 |
+
"<|placeholder_167|>": 151850,
|
| 210 |
+
"<|placeholder_168|>": 151851,
|
| 211 |
+
"<|placeholder_169|>": 151852,
|
| 212 |
+
"<|placeholder_170|>": 151853,
|
| 213 |
+
"<|placeholder_171|>": 151854,
|
| 214 |
+
"<|placeholder_172|>": 151855,
|
| 215 |
+
"<|placeholder_173|>": 151856,
|
| 216 |
+
"<|placeholder_174|>": 151857,
|
| 217 |
+
"<|placeholder_175|>": 151858,
|
| 218 |
+
"<|placeholder_176|>": 151859,
|
| 219 |
+
"<|placeholder_177|>": 151860,
|
| 220 |
+
"<|placeholder_178|>": 151861,
|
| 221 |
+
"<|placeholder_179|>": 151862,
|
| 222 |
+
"<|placeholder_180|>": 151863,
|
| 223 |
+
"<|placeholder_181|>": 151864,
|
| 224 |
+
"<|placeholder_182|>": 151865,
|
| 225 |
+
"<|placeholder_183|>": 151866,
|
| 226 |
+
"<|placeholder_184|>": 151867,
|
| 227 |
+
"<|placeholder_185|>": 151868,
|
| 228 |
+
"<|placeholder_186|>": 151869,
|
| 229 |
+
"<|placeholder_187|>": 151870,
|
| 230 |
+
"<|placeholder_188|>": 151871,
|
| 231 |
+
"<|placeholder_189|>": 151872,
|
| 232 |
+
"<|placeholder_190|>": 151873,
|
| 233 |
+
"<|placeholder_191|>": 151874,
|
| 234 |
+
"<|placeholder_192|>": 151875,
|
| 235 |
+
"<|Chinese|>": 151876,
|
| 236 |
+
"<|English|>": 151877,
|
| 237 |
+
"<|Cantonese|>": 151878,
|
| 238 |
+
"<|Japanese|>": 151879,
|
| 239 |
+
"<|Korean|>": 151880,
|
| 240 |
+
"<|Spanish|>": 151881,
|
| 241 |
+
"<|French|>": 151882,
|
| 242 |
+
"<|Russian|>": 151883,
|
| 243 |
+
"<|Arabic|>": 151884,
|
| 244 |
+
"<|Turkish|>": 151885,
|
| 245 |
+
"<|Indonesian|>": 151886,
|
| 246 |
+
"<|Portuguese|>": 151887,
|
| 247 |
+
"<|Italian|>": 151888,
|
| 248 |
+
"<|Dutch|>": 151889,
|
| 249 |
+
"<|Vietnamese|>": 151890,
|
| 250 |
+
"<|German|>": 151891,
|
| 251 |
+
"<|Ukrainian|>": 151892,
|
| 252 |
+
"<|Thai|>": 151893,
|
| 253 |
+
"<|Polish|>": 151894,
|
| 254 |
+
"<|Romanian|>": 151895,
|
| 255 |
+
"<|Greek|>": 151896,
|
| 256 |
+
"<|Czech|>": 151897,
|
| 257 |
+
"<|Finnish|>": 151898,
|
| 258 |
+
"<|Hindi|>": 151899,
|
| 259 |
+
"<|ZH_Anhui|>": 151900,
|
| 260 |
+
"<|ZH_Fujian|>": 151901,
|
| 261 |
+
"<|ZH_Gansu|>": 151902,
|
| 262 |
+
"<|ZH_Guizhou|>": 151903,
|
| 263 |
+
"<|ZH_Hebei|>": 151904,
|
| 264 |
+
"<|ZH_Henan|>": 151905,
|
| 265 |
+
"<|ZH_Hubei|>": 151906,
|
| 266 |
+
"<|ZH_Hunan|>": 151907,
|
| 267 |
+
"<|ZH_Jiangxi|>": 151908,
|
| 268 |
+
"<|ZH_Liaoning|>": 151909,
|
| 269 |
+
"<|ZH_Minnan|>": 151910,
|
| 270 |
+
"<|ZH_Ningxia|>": 151911,
|
| 271 |
+
"<|ZH_Shaanxi|>": 151912,
|
| 272 |
+
"<|ZH_Shandong|>": 151913,
|
| 273 |
+
"<|ZH_Shanghai|>": 151914,
|
| 274 |
+
"<|ZH_Shanxi|>": 151915,
|
| 275 |
+
"<|ZH_Sichuan|>": 151916,
|
| 276 |
+
"<|ZH_Tianjin|>": 151917,
|
| 277 |
+
"<|ZH_Wenzhou|>": 151918,
|
| 278 |
+
"<|ZH_Wu|>": 151919,
|
| 279 |
+
"<|ZH_Yunnan|>": 151920,
|
| 280 |
+
"<|edit|>": 151921,
|
| 281 |
+
"<|frame_patch|>": 151922,
|
| 282 |
+
"<|end_edit|>": 151923
|
| 283 |
+
}
|
chat_template.jinja
ADDED
|
@@ -0,0 +1,89 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{%- if tools %}
|
| 2 |
+
{{- '<|im_start|>system\n' }}
|
| 3 |
+
{%- if messages[0].role == 'system' %}
|
| 4 |
+
{{- messages[0].content + '\n\n' }}
|
| 5 |
+
{%- endif %}
|
| 6 |
+
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
| 7 |
+
{%- for tool in tools %}
|
| 8 |
+
{{- "\n" }}
|
| 9 |
+
{{- tool | tojson }}
|
| 10 |
+
{%- endfor %}
|
| 11 |
+
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
| 12 |
+
{%- else %}
|
| 13 |
+
{%- if messages[0].role == 'system' %}
|
| 14 |
+
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
|
| 15 |
+
{%- endif %}
|
| 16 |
+
{%- endif %}
|
| 17 |
+
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
| 18 |
+
{%- for message in messages[::-1] %}
|
| 19 |
+
{%- set index = (messages|length - 1) - loop.index0 %}
|
| 20 |
+
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
|
| 21 |
+
{%- set ns.multi_step_tool = false %}
|
| 22 |
+
{%- set ns.last_query_index = index %}
|
| 23 |
+
{%- endif %}
|
| 24 |
+
{%- endfor %}
|
| 25 |
+
{%- for message in messages %}
|
| 26 |
+
{%- if message.content is string %}
|
| 27 |
+
{%- set content = message.content %}
|
| 28 |
+
{%- else %}
|
| 29 |
+
{%- set content = '' %}
|
| 30 |
+
{%- endif %}
|
| 31 |
+
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
|
| 32 |
+
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
| 33 |
+
{%- elif message.role == "assistant" %}
|
| 34 |
+
{%- set reasoning_content = '' %}
|
| 35 |
+
{%- if message.reasoning_content is string %}
|
| 36 |
+
{%- set reasoning_content = message.reasoning_content %}
|
| 37 |
+
{%- else %}
|
| 38 |
+
{%- if '</think>' in content %}
|
| 39 |
+
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
|
| 40 |
+
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
|
| 41 |
+
{%- endif %}
|
| 42 |
+
{%- endif %}
|
| 43 |
+
{%- if loop.index0 > ns.last_query_index %}
|
| 44 |
+
{%- if loop.last or (not loop.last and reasoning_content) %}
|
| 45 |
+
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
|
| 46 |
+
{%- else %}
|
| 47 |
+
{{- '<|im_start|>' + message.role + '\n' + content }}
|
| 48 |
+
{%- endif %}
|
| 49 |
+
{%- else %}
|
| 50 |
+
{{- '<|im_start|>' + message.role + '\n' + content }}
|
| 51 |
+
{%- endif %}
|
| 52 |
+
{%- if message.tool_calls %}
|
| 53 |
+
{%- for tool_call in message.tool_calls %}
|
| 54 |
+
{%- if (loop.first and content) or (not loop.first) %}
|
| 55 |
+
{{- '\n' }}
|
| 56 |
+
{%- endif %}
|
| 57 |
+
{%- if tool_call.function %}
|
| 58 |
+
{%- set tool_call = tool_call.function %}
|
| 59 |
+
{%- endif %}
|
| 60 |
+
{{- '<tool_call>\n{"name": "' }}
|
| 61 |
+
{{- tool_call.name }}
|
| 62 |
+
{{- '", "arguments": ' }}
|
| 63 |
+
{%- if tool_call.arguments is string %}
|
| 64 |
+
{{- tool_call.arguments }}
|
| 65 |
+
{%- else %}
|
| 66 |
+
{{- tool_call.arguments | tojson }}
|
| 67 |
+
{%- endif %}
|
| 68 |
+
{{- '}\n</tool_call>' }}
|
| 69 |
+
{%- endfor %}
|
| 70 |
+
{%- endif %}
|
| 71 |
+
{{- '<|im_end|>\n' }}
|
| 72 |
+
{%- elif message.role == "tool" %}
|
| 73 |
+
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
|
| 74 |
+
{{- '<|im_start|>user' }}
|
| 75 |
+
{%- endif %}
|
| 76 |
+
{{- '\n<tool_response>\n' }}
|
| 77 |
+
{{- content }}
|
| 78 |
+
{{- '\n</tool_response>' }}
|
| 79 |
+
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
| 80 |
+
{{- '<|im_end|>\n' }}
|
| 81 |
+
{%- endif %}
|
| 82 |
+
{%- endif %}
|
| 83 |
+
{%- endfor %}
|
| 84 |
+
{%- if add_generation_prompt %}
|
| 85 |
+
{{- '<|im_start|>assistant\n' }}
|
| 86 |
+
{%- if enable_thinking is defined and enable_thinking is false %}
|
| 87 |
+
{{- '<think>\n\n</think>\n\n' }}
|
| 88 |
+
{%- endif %}
|
| 89 |
+
{%- endif %}
|
config.json
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"FireRedTTS3ForConditionalGeneration"
|
| 4 |
+
],
|
| 5 |
+
"model_type": "fireredtts3",
|
| 6 |
+
"redae_dim": 64,
|
| 7 |
+
"num_history_patches": 2,
|
| 8 |
+
"patch_size": 4,
|
| 9 |
+
"patch_encoder_hidden_size": 1024,
|
| 10 |
+
"patch_encoder_mlp_ratio": 4,
|
| 11 |
+
"patch_encoder_depth": 8,
|
| 12 |
+
"patch_encoder_num_heads": 16,
|
| 13 |
+
"dit_mlp_ratio": 3,
|
| 14 |
+
"dit_depth": 11,
|
| 15 |
+
"dit_num_heads": 16,
|
| 16 |
+
"dit_hidden_size": 1024,
|
| 17 |
+
"dtype": "float32",
|
| 18 |
+
"dit_in_channels": 1088,
|
| 19 |
+
"audio_patch_size": 480,
|
| 20 |
+
"audio_sample_rate": 24000,
|
| 21 |
+
"bottleneck_dim": 64,
|
| 22 |
+
"dec_hidden_size": 896,
|
| 23 |
+
"dec_intermediate_size": 3584,
|
| 24 |
+
"dec_max_position_embeddings": 32768,
|
| 25 |
+
"dec_max_window_layers": 0,
|
| 26 |
+
"dec_num_attention_heads": 14,
|
| 27 |
+
"dec_num_hidden_layers": 18,
|
| 28 |
+
"dec_num_key_value_heads": 2,
|
| 29 |
+
"dec_sliding_window": 64,
|
| 30 |
+
"dec_use_sliding_window": true,
|
| 31 |
+
"enc_downsample_num_hidden_layers": 4,
|
| 32 |
+
"enc_extra_downsample_rate": 2,
|
| 33 |
+
"enc_hidden_size": 896,
|
| 34 |
+
"enc_intermediate_size": 3584,
|
| 35 |
+
"enc_max_position_embeddings": 32768,
|
| 36 |
+
"enc_max_window_layers": 0,
|
| 37 |
+
"enc_num_attention_heads": 14,
|
| 38 |
+
"enc_num_hidden_layers": 18,
|
| 39 |
+
"enc_num_key_value_heads": 2,
|
| 40 |
+
"enc_sliding_window": 64,
|
| 41 |
+
"enc_use_sliding_window": true,
|
| 42 |
+
"vocab_size": 151936,
|
| 43 |
+
"hidden_size": 2048,
|
| 44 |
+
"intermediate_size": 6144,
|
| 45 |
+
"num_hidden_layers": 28,
|
| 46 |
+
"num_attention_heads": 16,
|
| 47 |
+
"num_key_value_heads": 8,
|
| 48 |
+
"head_dim": 128,
|
| 49 |
+
"hidden_act": "silu",
|
| 50 |
+
"max_position_embeddings": 40960,
|
| 51 |
+
"rms_norm_eps": 1e-06,
|
| 52 |
+
"rope_theta": 1000000.0,
|
| 53 |
+
"tie_word_embeddings": true,
|
| 54 |
+
"attention_bias": false,
|
| 55 |
+
"max_window_layers": 28,
|
| 56 |
+
"use_sliding_window": false,
|
| 57 |
+
"sliding_window": null,
|
| 58 |
+
"bos_token_id": 151643,
|
| 59 |
+
"eos_token_id": 151645,
|
| 60 |
+
"use_lm_head": true,
|
| 61 |
+
"redae_scale": 0.4,
|
| 62 |
+
"speaker_token": null,
|
| 63 |
+
"text_latent_interleave_ratio": null,
|
| 64 |
+
"special_tokens": {
|
| 65 |
+
"text_start": "<|sot|>",
|
| 66 |
+
"text_end": "<|eot|>",
|
| 67 |
+
"audio_sos": "<|sosp|>",
|
| 68 |
+
"audio_eos": "<|eosp|>",
|
| 69 |
+
"audio_in_placeholder": "<|image_pad|>",
|
| 70 |
+
"icl_audio_placeholder": "<|video_pad|>",
|
| 71 |
+
"audio_holder": null,
|
| 72 |
+
"text_holder": null,
|
| 73 |
+
"text_pad": null,
|
| 74 |
+
"audio_start": null,
|
| 75 |
+
"audio_end": null
|
| 76 |
+
},
|
| 77 |
+
"generation_defaults": {
|
| 78 |
+
"n_timesteps": 10,
|
| 79 |
+
"inference_cfg": 2.0,
|
| 80 |
+
"stop_threshold": 0.5,
|
| 81 |
+
"min_patches": 6,
|
| 82 |
+
"max_patches": 400
|
| 83 |
+
},
|
| 84 |
+
"transformers_version": "5.14.1"
|
| 85 |
+
}
|
merges.txt
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a145074153202871f759f52480bbf022bdb70c062a1da90a94b797db8511d889
|
| 3 |
+
size 8475259708
|
redae/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0723e87fdaf46d377f01aa32ce56b43f43f9b249415bc8b8f81040eaa8b14abe
|
| 3 |
+
size 3775160672
|
special_tokens_map.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token": null,
|
| 3 |
+
"eos_token": "<|im_end|>",
|
| 4 |
+
"unk_token": null,
|
| 5 |
+
"pad_token": "<|endoftext|>",
|
| 6 |
+
"additional_special_tokens": [
|
| 7 |
+
"<|im_start|>",
|
| 8 |
+
"<|im_end|>",
|
| 9 |
+
"<|object_ref_start|>",
|
| 10 |
+
"<|object_ref_end|>",
|
| 11 |
+
"<|box_start|>",
|
| 12 |
+
"<|box_end|>",
|
| 13 |
+
"<|quad_start|>",
|
| 14 |
+
"<|quad_end|>",
|
| 15 |
+
"<|vision_start|>",
|
| 16 |
+
"<|vision_end|>",
|
| 17 |
+
"<|vision_pad|>",
|
| 18 |
+
"<|image_pad|>",
|
| 19 |
+
"<|video_pad|>"
|
| 20 |
+
]
|
| 21 |
+
}
|
tokenizer.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d84d0d2d4926a03d2d5bb3fed5b3fedb2299d0ca7e933bed7e8a3ad2ce73f3d0
|
| 3 |
+
size 11472157
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"add_prefix_space": false,
|
| 3 |
+
"backend": "tokenizers",
|
| 4 |
+
"bos_token": null,
|
| 5 |
+
"clean_up_tokenization_spaces": false,
|
| 6 |
+
"eos_token": "<|im_end|>",
|
| 7 |
+
"errors": "replace",
|
| 8 |
+
"extra_special_tokens": [
|
| 9 |
+
"<|im_start|>",
|
| 10 |
+
"<|im_end|>",
|
| 11 |
+
"<|object_ref_start|>",
|
| 12 |
+
"<|object_ref_end|>",
|
| 13 |
+
"<|box_start|>",
|
| 14 |
+
"<|box_end|>",
|
| 15 |
+
"<|quad_start|>",
|
| 16 |
+
"<|quad_end|>",
|
| 17 |
+
"<|vision_start|>",
|
| 18 |
+
"<|vision_end|>",
|
| 19 |
+
"<|vision_pad|>",
|
| 20 |
+
"<|image_pad|>",
|
| 21 |
+
"<|video_pad|>"
|
| 22 |
+
],
|
| 23 |
+
"is_local": true,
|
| 24 |
+
"local_files_only": false,
|
| 25 |
+
"model_max_length": 131072,
|
| 26 |
+
"pad_token": "<|endoftext|>",
|
| 27 |
+
"split_special_tokens": false,
|
| 28 |
+
"tokenizer_class": "Qwen2Tokenizer",
|
| 29 |
+
"unk_token": null
|
| 30 |
+
}
|
vocab.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|