Text-Cue-Guided TS-ASR-Qwen

Text-Cue-Guided TS-ASR-Qwen is a target-speaker speech recognizer. Given a recording where several people talk at once and a short text cue (a few words the target speaker says), it transcribes only the speaker who says the cue and ignores everyone else. If nobody says the cue, it rejects the request.

Demo: Unified-TS-ASR Space (select "Text-Cue-Guided TS-ASR").
Inference code: the tcg_tsasr package in the Space repository.
Paper: Unified Target-Speaker ASR with Text and Enrollment Speech Cues (arXiv:2609.33853)
Code: github.com/YuCeong-May/Unified-TS-ASR
Unified-TS-ASR model: YuCeong-May/Unified-TS-ASR
Language: Mandarin Chinese, 16 kHz audio. Audio within 20 seconds is recommended.

Model

text cue --> CueTokenizer --> CueEncoder --------------------.
                                                             v  cross-attention in every layer
16 kHz audio --> 80-dim fbank --> AcousticEncoder (9 x Conformer) --> AudioProjector --> Qwen2.5-0.5B + LoRA

Component	Details	Parameters
Front end	80-dim kaldi fbank, 25 ms window, 10 ms shift	-
CueEncoder	8406-unit cue vocabulary (Chinese characters and subwords) + 4 Transformer layers, dim 256	7.5 M
AcousticEncoder	2 x Conv2d (stride 2, 4x subsampling to 25 Hz) + 9 Conformer layers (dim 256, 4 heads, FFN 2048, conv kernel 15); every layer cross-attends to the cue	27.5 M
AudioProjector	Gated 4-frame Conv1d, stride 2 (12.5 audio tokens/s), bottleneck, projected to 896	3.6 M
LLM	Qwen2.5-0.5B (24 layers, hidden 896) with LoRA r = 64, alpha = 128 on q/k/v/o, gate/up/down and lm_head	630 M base + 45 M LoRA

The LLM is prompted with the cue and generates greedily:

<psok>cue<peok>transcript when the target speaker is found. The cue is bracketed where it occurs; the demo strips the brackets.
<reject/> when the cue is not in the audio, or <psok>cue<peok><reject/> when the cue is heard but not from the target speaker.

model.pt (1.6 GB) holds every weight the model needs, including the Qwen2.5-0.5B base weights, so nothing else is downloaded. The checkpoint also contains a training-only speaker-classification head (1.8 M), which inference ignores.

SHA-256: 870aef70a8a8c2b292b3eb959ea4a0907edc751848729070aea04465b449fcf5

from tcg_tsasr import TextCueGuidedTSASR  # from the Space repository

asr = TextCueGuidedTSASR.from_pretrained("YuCeong-May/Text-Cue-Guided-TS-ASR-Qwen")
out = asr.transcribe("examples/0.wav", "附近居民")
print(out["text"], out["rejected"])

Results on TS-Aishell

TS-Aishell mixes two speakers from AISHELL-1 and the AISHELL-2 test sets (iOS, Android, Mic, and a cross-device mix). Each of the five test sets has 6000 mixtures. The text cue is 2-5 consecutive characters taken from the target transcript. Decoding is greedy.

Text-Cue-Guided TS-ASR-Qwen, by cue length

CER / SER / reject rate in %. A rejected utterance is scored on whatever text the model produced.

Test set	2 chars	3 chars	4 chars	5 chars
aishell1-ts	4.21 / 20.32 / 0.92	2.69 / 17.08 / 0.35	2.09 / 15.23 / 0.32	1.83 / 13.90 / 0.57
aishell-ios-ts	11.56 / 44.55 / 3.67	8.77 / 38.73 / 3.12	7.22 / 34.32 / 4.47	6.34 / 29.25 / 7.58
aishell-android-ts	15.00 / 49.98 / 4.70	11.26 / 43.78 / 3.57	9.60 / 38.75 / 4.18	8.76 / 34.58 / 6.50
aishell-mic-ts	12.21 / 46.02 / 3.85	9.33 / 39.63 / 3.23	7.62 / 34.93 / 4.10	6.42 / 30.23 / 6.68
aishell-mix-ts	8.35 / 37.55 / 2.78	5.91 / 32.37 / 2.63	4.76 / 28.55 / 3.32	3.94 / 23.70 / 5.40

Longer cues help: going from 2 to 5 characters cuts CER by 42-57 % relative.

Comparison with Unified-TS-ASR

Unified-TS-ASR (Conformer + CTC, 37 M parameters, also in the demo) attends to a text cue and an enrollment recording of the target speaker (first 5 s). The table compares both models at the 5-character cue; Unified-TS-ASR additionally gets the enrollment. Both use the same 6000 mixtures per set, and both CERs are computed on the output text, including rejected utterances.

Test set	Text-Cue-Guided TS-ASR-Qwen text cue	Unified-TS-ASR text cue + enrollment
	CER / SER / reject %	CER / SER / reject %
aishell1-ts	1.83 / 13.90 / 0.57	5.12 / 37.98 / 0.13
aishell-ios-ts	6.34 / 29.25 / 7.58	9.94 / 49.23 / 0.32
aishell-android-ts	8.76 / 34.58 / 6.50	12.06 / 55.57 / 0.80
aishell-mic-ts	6.42 / 30.23 / 6.68	9.31 / 47.75 / 0.20
aishell-mix-ts	3.94 / 23.70 / 5.40	9.27 / 47.77 / 25.53

Text-Cue-Guided TS-ASR-Qwen has lower CER on every set (2.9-5.3 points absolute) and much lower SER, at about 19x the parameters (715 M vs 37 M) and autoregressive decoding.
It also rejects more often (5-8 % on AISHELL-2 sets, vs under 1 % for Unified-TS-ASR), so it is more conservative when the cue is unclear.
Unified-TS-ASR's 25.5 % reject rate on aishell-mix-ts comes from enrollments recorded on a different device than the target speech: 33.7 % of those mismatched pairs (1502 / 4455) are rejected as "speaker not found", vs 0.9 % (14 / 1545) for same-device pairs.
The released Unified-TS-ASR checkpoint was trained and evaluated with both cues present; text-only or enrollment-only use in the demo is outside that setting.

Limitations

Mandarin only; the cue must be words the target speaker actually says.
Tested on two-speaker mixtures. More speakers, heavy noise or long audio (over 20 s) are not evaluated.
Cues of 2-5 characters are recommended; longer cues are accepted but untested.