tag-tts

tag-tts makes CosyVoice3 actually do what its tags say.

Fun-CosyVoice3-0.5B-2512 supports inline paralinguistic tags such as [laughter], [cough], [sigh], <laughter>…</laughter> and <strong>…</strong>, but in real-time (bi-streaming) synthesis it ignores most of them. tag-tts is a drop-in replacement for its language model (llm.pt), trained with reinforcement learning so that the tag you write is the sound you hear.

The judge in the RL loop is our paralinguistic speech-understanding model DuplexJev: it listens to every candidate and decides whether the laugh, cough or sigh really happened and which word carries the stress.

中文说明 · Audio examples · Usage · Changelog

v1.1 (2026-10-30): two more RL rounds (GRPO + DuplexJev-guided positives). Stress on the tagged word 35.0% → 40.7% on held-out sentences, sigh 15.0% → 17.9%, tongue-twister error back down (6.42 → 6.24).

Results

Same text, same cloned voice, token-by-token bi-streaming, no style instruction. "Hit" = the event is clearly present. DuplexJev hit = DuplexJev-32B-Para picks the right answer in a multiple-choice question ("Which sound does the speaker make besides speaking?", options rotated); AST hit = AudioSet-AST event probability above a fixed threshold. 24 utterances per tag (tag at sentence start / middle / end, Chinese and English, 2 samples each).

Tag	CosyVoice3	tag-tts v1.0	tag-tts v1.1
[laughter]	62%	100%	100%
<laughter>…</laughter>	12%	88%	92%
[cough]	42%	79%	71%
[sigh]	4%	17%	17%
[breath]	25%	25%	17%
no tag (nothing extra heard)	100%	88%	88%

24 utterances per tag, so differences of one or two utterances (±8%) are noise.

Stress, held-out sentences (70 sentences × 4 samples; one word wrapped in <strong>…</strong>; DuplexJev answers "Which word does the speaker emphasise?" among the target word, 2–3 other words and "no word is emphasised"):

	CosyVoice3	tag-tts v1.0	tag-tts v1.1
<strong> – stress lands on the tagged word	28.6%	35.0%	40.7%
[sigh] – sigh heard (DuplexJev)	4.3%	15.0%	17.9%

Basic quality holds (zh, en and the tongue-twister set are all at or below the official model's error, except zh-hard which is 0.2 points worse). Seed-TTS-eval in the same deployment setting (bi-streaming, voice cloned from the prompt audio only, no prompt text; ASR: Paraformer-zh / Whisper-large-v3; speaker similarity with CAM++, so SIM is not comparable to numbers in papers):

	zh CER ↓	en WER ↓	zh-hard CER ↓	SIM zh / en ↑
CosyVoice3 (official llm.pt)	1.53	2.58	6.00	0.776 / 0.814
tag-tts v1.0	1.29	2.02	6.42	0.776 / 0.815
tag-tts v1.1	1.27	1.96	6.24	0.777 / 0.814

tag-tts also inherits our earlier RL round for sentence-level style instructions ("say this happily / sadly / angrily / louder / slower …"). On 160 ESD sentences with an emotion instruction, the emotion recognised by emotion2vec matches the instruction in 66.2% / 71.9% of cases with v1.1 (default voice / cloned ESD voice; v1.0 70.0% / 72.5%), vs 49.4% / 47.5% for the official model.

How it was trained

1. Sampling. A tag is inserted into each training sentence (start / middle / end, or one word wrapped in <strong>). The current model synthesises 8 candidates in the production setting (bi-streaming, 10 cloned voices, mostly without style instruction).
2. Judging. DuplexJev-32B answers multiple-choice questions about every candidate: which word is stressed, which non-verbal sound do you hear. AudioSet-AST is a second detector for laughter / cough / breath. Candidates that mispronounce (ASR error > 10%), drift in timbre or speaking rate are ranked last.
3. Preference optimisation. Best-vs-worst pairs within each group train the 0.5B language model with DPO (β = 0.1, lr 1e-6, chosen-NLL regulariser 0.1). Flow matching and vocoder are unchanged. Three on-policy rounds (2,100–2,800 preference pairs each) gave v1.0.
4. v1.1: GRPO with guided positives. Stress is the hardest target: for a third of the sentences the model never stresses the tagged word in 12 tries, so preference learning has nothing to work with. For those sentences we sample with a text hint (a short pause before the tagged word) and keep the candidates DuplexJev judges as stressed; training then sees the original text, so the model learns to stress from <strong> alone. These positives are added as an NLL term to on-policy GRPO (8 samples per prompt, group-normalised advantage, token-level clipping, KL to the previous round). Two rounds; the second did not improve further, so v1.1 is the first.

The RL rounds use model-generated audio only; no external recordings are used for training.

Lessons we learned on the way (they matter for anyone using CosyVoice3 tags):

• With no instruction the prefix must be exactly You are a helpful assistant.<|endofprompt|>. A single extra space before <|endofprompt|> silently disables most tags (laughter detection 0.24 → 0.007).
• In bi-streaming, send every tag as one complete text chunk. A tag split across chunks ([laugh + ter]) is read out loud.
• Malformed or unknown tags ([laugh], (laughs), [happy], 【笑声】…) are read out loud. Whitelist your tags.
• An LLM-based judge must be queried in exactly the format it was trained with; with a slightly different prompt DuplexJev's stress accuracy dropped from 90% to 5%.

Usage

Use the official CosyVoice code. This repository is a complete model directory (only llm.pt differs from the official release), so it loads exactly like the official model:

import torchaudio
from cosyvoice.cli.cosyvoice import AutoModel

tts = AutoModel(model_dir="pretrained_models/tag-tts")   # this repo
prompt_wav = "prompt.wav"                                 # 3–10 s of the target voice

text = "You are a helpful assistant.<|endofprompt|>[laughter]哎呀，今天真是太累了，从早上忙到现在。"
for i, out in enumerate(tts.inference_cross_lingual(text, prompt_wav, stream=False)):
    torchaudio.save(f"out_{i}.wav", out["tts_speech"], tts.sample_rate)

# with a style instruction (note the space after the full stop)
for out in tts.inference_instruct2("Others were hiding in fear for their <strong>lives</strong>.",
                                   "You are a helpful assistant. Please say this in a sad tone.<|endofprompt|>",
                                   prompt_wav, stream=False):
    ...

Already using the official model? Just replace llm.pt with the one from this repo.

Recommended tags: [laughter], <laughter>…</laughter>, [cough], <strong>…</strong>, [breath]. [sigh] improved but is still unreliable.

Audio examples

Official CosyVoice3 vs tag-tts (v1.0 samples), same text and voice (in examples/). Numbers are DuplexJev's probability for the intended answer.

Text	CosyVoice3	tag-tts
[laughter]哎呀，今天真是太累了，从早上忙到现在。	▶ 0.00	▶ 1.00
Okay, <laughter>I see what you mean</laughter>, let's take it step by step.	▶ 0.00	▶ 1.00
[cough]Oh well, what a long day, I've been busy since this morning.	▶ 0.00	▶ 1.00
Others were hiding in fear for their <strong>lives</strong>.	▶ 0.00	▶ 1.00
[sigh]我支持你，他是需要重做	▶ 0.00	▶ 0.99
Overcast skies are expected in the area, [sigh]though.	▶ 0.00	▶ 0.89

Limitations

• [sigh], [breath] and [quick_breath] are still weak; [lipsmack], [mn], [noise], [accent], [clucking], [hissing], [vocalized-noise] were not trained ([vocalized-noise] can hurt pronunciation).
• <strong> works in about 40% of cases. Stress is the hardest target and the judge is strict.
• Tags work best where the sentence supports them (a laugh after a joke, a sigh after bad news).
• Tongue twisters (Seed-TTS zh-hard) are slightly worse than the official model (CER 6.00 → 6.24).
• [cough] and [breath] hit rates moved down by one or two of 24 bench utterances in v1.1 (within noise).
• Evaluated on Chinese and English only.

Related

• DuplexJev – full-duplex speech-understanding models that hear emotion, stress, non-verbal sounds and turn-taking; the judge used to train tag-tts. GitHub · Hugging Face · ModelScope
• Base model: Fun-CosyVoice3-0.5B-2512

Changelog

• v1.1 (2026-10-30) – two GRPO rounds with DuplexJev rewards and text-guided positives for stress. Stress 35.0% → 40.7%, sigh 15.0% → 17.9% (held-out), zh-hard 6.42 → 6.24. v1.0 weights stay available as the v1.0 revision.
• v1.0 (2026-10-29) – three DPO rounds judged by DuplexJev.

License

Apache-2.0, same as the base model. tag-tts only changes llm.pt; all other files are from Fun-CosyVoice3-0.5B-2512 by Tongyi Lab / FunAudioLLM.

中文说明

tag-tts 让 CosyVoice3 的标签真正生效。

CosyVoice3 支持在文本里插入 [laughter]、[cough]、[sigh]、<laughter>…</laughter>、<strong>…</strong> 等副语言标签，但在实时逐字双流合成里，大部分标签会被忽略。tag-tts 只替换它的语言模型（llm.pt）， 用强化学习训练，让“写了笑就真的笑出来”。

强化学习的裁判是我们的副语言语音理解模型 DuplexJev： 它逐条听合成结果，判断笑、咳嗽、叹气有没有真的出现，以及重音落在哪个词上。

• 笑声：DuplexJev 判定命中 62% → **100%**；成对写法 <laughter>…</laughter> 12% → 88%
• 咳嗽：42% → **71%**（v1.0 79%，在误差内）；叹气（留出句）：4.3% → 17.9%
• 重音（留出句，一个词加 <strong>）：重音落在目标词上 28.6% → **40.7%**（v1.0 为 35.0%）
• 基础能力不退化：Seed-TTS-eval 中文字错率 1.53 → 1.27，英文 2.58 → 1.96（绕口令难句 6.00 → 6.24，略差）
• 同时包含我们此前的整句情绪指令强化学习：情绪指令的命中率（emotion2vec）49.4% → **66.2%**（ESD 克隆音色 47.5% → **71.9%**）

训练方法：每句插入标签 → 当前模型在线上同款流式设置下采样 8 个候选 → DuplexJev-32B 做选择题打分 （AudioSet-AST 辅助）→ 念错字、音色或语速跑偏的直接判最差 → 组内好坏对做 DPO，三轮（v1.0）。v1.1 再做两轮 GRPO：对模型 12 次都读不出重音的句子，采样时先在目标词前加停顿作引导，挑出 DuplexJev 判为重读的样本，训练时换回原文本，让模型学会只看 <strong> 就重读。只更新 0.5B 语言模型， flow 与声码器不变；训练只用模型自己生成的音频。

使用注意：

1. 不带指令时前缀必须是 You are a helpful assistant.<|endofprompt|>，<|endofprompt|> 前多一个空格会让大部分标签失效；
2. 流式送字时，标签要作为一整块送入，拆开的标签会被念出来；
3. 只使用白名单标签，格式错误的标签会被念出来。

许可：Apache-2.0，与基座模型一致。除 llm.pt 外，其余文件均来自通义实验室 FunAudioLLM 的 Fun-CosyVoice3-0.5B-2512。