--- license: apache-2.0 language: - en - zh - ja - ko - fr - es - pt - ru - vi - id pipeline_tag: automatic-speech-recognition tags: - tta - speech - translation - alignment - multilingual - retrieval --- # TTA: Transcribe, Translate and Alignment for Cross-lingual Speech Representation **TTA** is a multilingual model that jointly supports **transcribe**, **translate**, and **align** tasks. It provides strong multilingual ASR/ST performance and cross-lingual speech retrieval capability. ๐Ÿ”— **Paper**: https://arxiv.org/abs/2511.14410 ๐Ÿ”— **Model**: https://huggingface.co/AudenAI/auden-tta-m10 ๐Ÿ”— **Encoder**: https://huggingface.co/AudenAI/auden-encoder-tta-m10 ๐Ÿ”— **Code**: https://github.com/AudenAI/Auden/tree/main/examples/tta ## ๐Ÿ” What Can This Model Do? - ๐ŸŽ™๏ธ **Multilingual ASR** (transcribe) - ๐ŸŒ **Speech translation** (translate) - ๐Ÿงฉ **Audioโ€“text alignment** (align) - ๐Ÿ”Ž **Cross-lingual speech retrieval** ## Quick Start ### TTA model ```python from auden.auto.auto_model import AutoModel # 1) Load a model checkpoint directory (contains config.json + weights) model_dir = "AudenAI/auden-tta-m10" # or any exported directory / HF repo id model = AutoModel.from_pretrained(model_dir) model = model.to("cuda") model.eval() # 2) Prepare input features (x, x_lens). If you have raw audio, you can use # model.speech_encoder.extract_feature(wav) to get (x, x_lens). x, x_lens = ... # Tensor shapes: (B, T, F), (B,) inputs = (x, x_lens) # Alternatively, you can pass WAV inputs directly: # - List of WAV paths (str): # inputs = ["/abs/a.wav", "/abs/b.wav"] # - List of mono waveforms (Tensor/ndarray), 16 kHz: # inputs = [torch.randn(16000*5), torch.randn(16000*3)] # 3a) Transcribe (RNNT greedy) out = model.generate(inputs, task="transcribe", blank_penalty=0.0, return_timestamps=False) print(out["hypotheses"]) # list[str] # 3b) Translate (attention beam search). Language can be a single str or a list[str] per utterance out = model.generate( inputs, task="translate", beam_size=5, source_language=["zh"] * x.size(0), target_language=["en"] * x.size(0), ) print(out["hypotheses"]) # list[str] print(out["source_language"]) # list[str], model-predicted or provided print(out["target_language"]) # list[str], model-predicted or provided # 3c) Align (audio-text similarity) texts = ["hello world", "good morning"] out = model.generate(inputs, task="align", texts=texts) print(out["similarities"]) # (B, len(texts)) print(out["audio_emb"]) # (B, emb_dim) print(out["text_emb"]) # (B, emb_dim) ``` ### TTA encoder ```python from auden.auto.auto_model import AutoModel encoder = AutoModel.from_pretrained("AudenAI/auden-encoder-tta-m10") encoder = encoder.to("cuda") # 2) Prepare input features (x, x_lens). If you have raw audio, you can use # encoder.extract_feature(wav) to get (x, x_lens). x, x_lens = ... # Tensor shapes: (B, T, F), (B,) encoder_output = encoder(x, x_lens) print(encoder_output["encoder_out"]) # (B, T//4, D) print(encoder_output["encoder_out_lens"]) # (B) ``` ## ๐Ÿ“Œ Model Characteristics - Input: Raw audio waveform (16 kHz recommended) - Output: Transcription, translation, or alignment scores - Encoder: TTA encoder (`AudenAI/auden-encoder-tta-m10`) - Tasks: transcribe / translate / align ## ๐Ÿ“Š Evaluation ### Multilingual ASR & ST | Model | #Params | AISHELL1/2 (CERโ†“) | Wenet (CERโ†“) | LibriSpeech (WERโ†“) | CommonVoice (WERโ†“) | MLS (WERโ†“) | VoxPopuli (WERโ†“) | FLEURS (WERโ†“) | CoVoSTv2 (BLEUโ†‘) | |--------|----------|------------------|---------------|---------------------|--------------------|-------------|-------------------|----------------|-------------------| | **Whisper Medium** | 762M | 6.74 / 6.23 | 11.00 / 22.68 | 2.88 / 6.08 | 11.86 | 7.27 | 12.08 | 6.62 | 35.12 | | **Whisper Large-v2** | 1.54B | 5.90 / 5.24 | 9.47 / 22.77 | 2.64 / 5.14 | 9.70 | 5.65 | 11.90 | 5.20 | **38.80** | | **Whisper Large-v3** | 1.54B | 5.33 / 4.76 | 9.00 / 15.68 | 2.01 / 3.89 | 8.30 | 4.48 | 13.78 | 4.51 | 37.60 | | **ZT (ASR)** | 199M | 1.89 / 3.14 | 6.91 / 6.08 | 1.58 / 3.62 | 6.92 | 5.82 | 11.12 | 6.35 | โ€“ | | **ZT-AED (ASR)** | 246M | 1.82 / 3.07 | 6.89 / 6.18 | 1.54 / 3.59 | 6.70 | 5.71 | 10.78 | 6.18 | โ€“ | | **ZT-AED (Full)** | 246M | 1.80 / 3.03 | 6.96 / 5.94 | 1.56 / 3.76 | 6.69 | 5.72 | 10.88 | 6.17 | 34.72 | | **๐Ÿ”ฅ TTA (Ours)** | **247M** | **1.85 / 3.09** | **7.06 / 6.44** | **1.58 / 3.85** | **6.76** | **5.74** | **10.87** | **6.19** | **35.28** | ### TTA Encoder (LLM-ASR Encoder Evaluation) | Encoder | Aishell CERโ†“ | LibriSpeech WERโ†“ | |----------|---------------|------------------| | Whisper-Medium | 5.47 | 4.66 | | Whisper-Large | 4.87 | 3.64 | | ZT-AED | 2.92 | 2.30 | | **TTA (Ours)** | **1.92** | **1.95** | ## Training Data Full data composition (open-source links + in-house aggregation): | Language | Data Source | Type | Hours | Total Hours | Share | | :--- | :--- | :--- | :--- | :--- | :--- | | **Chinese (Zh)** | [WenetSpeech](https://github.com/wenet-e2e/WenetSpeech) | Open Source | 10,005 | 129,265 | 37.1% | | | [AISHELL-2](https://www.aishelltech.com/aishell_2) | Open Source | 1,000 | | | [AISHELL-1](https://huggingface.co/datasets/AISHELL/AISHELL-1) | Open Source | 150 | | | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 237 | | | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 222 | | | *In-house Data* | In-house | 117,651 | | **Code-Switch** | [TALCS](https://github.com/SpeechClub/TALCS) | Open Source | 555 | 8,924 | 2.6% | | | *In-house Data* | In-house | 8,369 | | **English (En)** | [Libriheavy](https://huggingface.co/datasets/pkufool/libriheavy) | Open Source | 45,751 | 107,626 | 30.9% | | | [Multilingual LibriSpeech (MLS)](https://huggingface.co/datasets/facebook/multilingual_librispeech) | Open Source | 44,659 | | | [GigaSpeech](https://huggingface.co/datasets/speechcolab/gigaspeech) | Open Source | 10,000 | | | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 3,426 | | | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 1,778 | | | [LibriSpeech](https://huggingface.co/datasets/librispeech_asr) | Open Source | 960 | | | [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) | Open Source | 522 | | | [TED-LIUM](https://huggingface.co/datasets/LIUM/tedlium) | Open Source | 453 | | | [AMI Corpus](https://huggingface.co/datasets/edinburgh-cstr/ami) | Open Source | 77 | | **Japanese (Ja)** | [ReazonSpeech](https://huggingface.co/datasets/reazon-research/reazonspeech) | Open Source | 35,389 | 40,426 | 11.6% | | | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 499 | | | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 19 | | | *In-house Data* | In-house | 4,519 | | **Korean (Ko)** | [KsponSpeech (AIHub)](https://huggingface.co/datasets/cheulyop/ksponspeech) | Open Source | 965 | 20,095 | 5.8% | | | [KrespSpeech (AIHub)](https://aihub.or.kr/) | Open Source | 2,906 | | | [KconfSpeech (AIHub)](https://aihub.or.kr/) | Open Source | 2,928 | | | [MeetingSpeech (AIHub)](https://aihub.or.kr/) | Open Source | 4,962 | | | [GyeongsangSpeech (AIHub)](https://aihub.or.kr/) | Open Source | 2,481 | | | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 1,528 | | | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 1 | | | *In-house Data (Aggregated)* | In-house | 4,324 | | **Russian (Ru)** | [Golos](https://huggingface.co/datasets/SberDevices/Golos) | Open Source | 1,221 | 15,246 | 4.4% | | | [Public Speech & Radio](https://huggingface.co/datasets/bond005/sberdevices_golos_10h) | Open Source | 1,651 | | | [Buriy Audiobook](https://huggingface.co/datasets/bond005/audio_books_russian) | Open Source | 874 | | | Public Youtube Dataset | Open Source | 809 | | | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 2,606 | | | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 37 | | | *In-house Data* | In-house | 8,048 | | **Vietnamese (Vi)** | [GigaSpeech 2](https://huggingface.co/datasets/speechcolab/gigaspeech2) | Open Source | 6,048 | 8,390 | 2.4% | | | [Bud500](https://huggingface.co/datasets/linhtran92/viet_bud500) | Open Source | 324 | | | [VLSP 2020](https://vlsp.org.vn/vlsp2020) | Open Source | 101 | | | [ViMD](https://github.com/NhutP/ViMD) | Open Source | 81 | | | [LSVSC](https://huggingface.co/datasets/doof-ferb/LSVSC) | Open Source | 80 | | | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 140 | | | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 2 | | | *In-house Data* | In-house | 1,614 | | **Indonesian (Id)** | [GigaSpeech 2](https://huggingface.co/datasets/speechcolab/gigaspeech2) | Open Source | 6,352 | 8,238 | 2.4% | | | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 442 | | | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 7 | | | *In-house Data* | In-house | 1,437 | | **French (Fr)** | [Multilingual LibriSpeech (MLS)](https://huggingface.co/datasets/facebook/multilingual_librispeech) | Open Source | 1,076 | 4,124 | 1.2% | | | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 1,423 | | | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 831 | | | [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) | Open Source | 205 | | | *In-house Data* | In-house | 589 | | **Spanish (Es)** | [Multilingual LibriSpeech (MLS)](https://huggingface.co/datasets/facebook/multilingual_librispeech) | Open Source | 917 | 4,596 | 1.3% | | | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 2,399 | | | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 502 | | | [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) | Open Source | 151 | | | *In-house Data* | In-house | 627 | | **Portuguese (Pt)** | [Multilingual LibriSpeech (MLS)](https://huggingface.co/datasets/facebook/multilingual_librispeech) | Open Source | 160 | 1,602 | 0.5% | | | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 852 | | | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 25 | | | *In-house Data* | In-house | 565 | Language totals from the same table: | Language | Total Hours | Share | | :--- | ---: | ---: | | Chinese (Zh) | 129,265 | 37.1% | | English (En) | 107,626 | 30.9% | | Japanese (Ja) | 40,426 | 11.6% | | Korean (Ko) | 20,095 | 5.8% | | Russian (Ru) | 15,246 | 4.4% | | Code-Switch | 8,924 | 2.6% | | Vietnamese (Vi) | 8,390 | 2.4% | | Indonesian (Id) | 8,238 | 2.4% | | Spanish (Es) | 4,596 | 1.3% | | French (Fr) | 4,124 | 1.2% | | Portuguese (Pt) | 1,602 | 0.5% | ## โš ๏ธ Limitations - Performance depends on audio quality and recording conditions. - For long-form audio, chunking and post-processing might be required for optimal performance. - Not designed for safety-critical applications. ## Citation If you use this model in your research, please cite: ```bibtex @article{liu2025tta, title={TTA: Transcribe, Translate and Alignment for Cross-lingual Speech Representation}, author={Liu, Wei and Li, Jiahong and Shao, Yiwen and Yu, Dong}, journal={arXiv preprint arXiv:2511.14410}, year={2025} } ```