README.md

---
license: apache-2.0
language:
  - en
  - zh
  - ja
  - ko
  - fr
  - es
  - pt
  - ru
  - vi
  - id
pipeline_tag: automatic-speech-recognition
tags:
  - tta
  - speech
  - translation
  - alignment
  - multilingual
  - retrieval
---

# TTA: Transcribe, Translate and Alignment for Cross-lingual Speech Representation

**TTA** is a multilingual model that jointly supports **transcribe**, **translate**, and **align**
tasks. It provides strong multilingual ASR/ST performance and cross-lingual speech retrieval
capability.

🔗 **Paper**: https://arxiv.org/abs/2511.14410  
🔗 **Model**: https://huggingface.co/AudenAI/auden-tta-m10  
🔗 **Encoder**: https://huggingface.co/AudenAI/auden-encoder-tta-m10  
🔗 **Code**: https://github.com/AudenAI/Auden/tree/main/examples/tta  

## 🔍 What Can This Model Do?

- 🎙️ **Multilingual ASR** (transcribe)
- 🌍 **Speech translation** (translate)
- 🧩 **Audio–text alignment** (align)
- 🔎 **Cross-lingual speech retrieval**

## Quick Start

### TTA model
```python
from auden.auto.auto_model import AutoModel

# 1) Load a model checkpoint directory (contains config.json + weights)
model_dir = "AudenAI/auden-tta-m10"  # or any exported directory / HF repo id
model = AutoModel.from_pretrained(model_dir)
model = model.to("cuda")
model.eval()

# 2) Prepare input features (x, x_lens). If you have raw audio, you can use
#    model.speech_encoder.extract_feature(wav) to get (x, x_lens).
x, x_lens = ...  # Tensor shapes: (B, T, F), (B,)

inputs = (x, x_lens)
# Alternatively, you can pass WAV inputs directly:
# - List of WAV paths (str):
#   inputs = ["/abs/a.wav", "/abs/b.wav"]
# - List of mono waveforms (Tensor/ndarray), 16 kHz:
#   inputs = [torch.randn(16000*5), torch.randn(16000*3)]

# 3a) Transcribe (RNNT greedy)
out = model.generate(inputs, task="transcribe", blank_penalty=0.0, return_timestamps=False)
print(out["hypotheses"])  # list[str]

# 3b) Translate (attention beam search). Language can be a single str or a list[str] per utterance
out = model.generate(
    inputs,
    task="translate",
    beam_size=5,
    source_language=["zh"] * x.size(0),
    target_language=["en"] * x.size(0),
)
print(out["hypotheses"])      # list[str]
print(out["source_language"]) # list[str], model-predicted or provided
print(out["target_language"]) # list[str], model-predicted or provided

# 3c) Align (audio-text similarity)
texts = ["hello world", "good morning"]
out = model.generate(inputs, task="align", texts=texts)
print(out["similarities"])  # (B, len(texts))
print(out["audio_emb"]) # (B, emb_dim)
print(out["text_emb"]) # (B, emb_dim)
```

### TTA encoder
```python
from auden.auto.auto_model import AutoModel
encoder = AutoModel.from_pretrained("AudenAI/auden-encoder-tta-m10")
encoder = encoder.to("cuda")

# 2) Prepare input features (x, x_lens). If you have raw audio, you can use
#    encoder.extract_feature(wav) to get (x, x_lens).
x, x_lens = ...  # Tensor shapes: (B, T, F), (B,)

encoder_output = encoder(x, x_lens)
print(encoder_output["encoder_out"]) # (B, T//4, D)
print(encoder_output["encoder_out_lens"]) # (B)
```

## 📌 Model Characteristics

- Input: Raw audio waveform (16 kHz recommended)
- Output: Transcription, translation, or alignment scores
- Encoder: TTA encoder (`AudenAI/auden-encoder-tta-m10`)
- Tasks: transcribe / translate / align

## 📊 Evaluation

### Multilingual ASR & ST

| Model | #Params | AISHELL1/2 (CER↓) | Wenet (CER↓) | LibriSpeech (WER↓) | CommonVoice (WER↓) | MLS (WER↓) | VoxPopuli (WER↓) | FLEURS (WER↓) | CoVoSTv2 (BLEU↑) |
|--------|----------|------------------|---------------|---------------------|--------------------|-------------|-------------------|----------------|-------------------|
| **Whisper Medium** | 762M | 6.74 / 6.23 | 11.00 / 22.68 | 2.88 / 6.08 | 11.86 | 7.27 | 12.08 | 6.62 | 35.12 |
| **Whisper Large-v2** | 1.54B | 5.90 / 5.24 | 9.47 / 22.77 | 2.64 / 5.14 | 9.70 | 5.65 | 11.90 | 5.20 | **38.80** |
| **Whisper Large-v3** | 1.54B | 5.33 / 4.76 | 9.00 / 15.68 | 2.01 / 3.89 | 8.30 | 4.48 | 13.78 | 4.51 | 37.60 |
| **ZT (ASR)** | 199M | 1.89 / 3.14 | 6.91 / 6.08 | 1.58 / 3.62 | 6.92 | 5.82 | 11.12 | 6.35 | – |
| **ZT-AED (ASR)** | 246M | 1.82 / 3.07 | 6.89 / 6.18 | 1.54 / 3.59 | 6.70 | 5.71 | 10.78 | 6.18 | – |
| **ZT-AED (Full)** | 246M | 1.80 / 3.03 | 6.96 / 5.94 | 1.56 / 3.76 | 6.69 | 5.72 | 10.88 | 6.17 | 34.72 |
| **🔥 TTA (Ours)** | **247M** | **1.85 / 3.09** | **7.06 / 6.44** | **1.58 / 3.85** | **6.76** | **5.74** | **10.87** | **6.19** | **35.28** |

### TTA Encoder (LLM-ASR Encoder Evaluation)

| Encoder | Aishell CER↓ | LibriSpeech WER↓ |
|----------|---------------|------------------|
| Whisper-Medium | 5.47 | 4.66 |
| Whisper-Large | 4.87 | 3.64 |
| ZT-AED | 2.92 | 2.30 |
| **TTA (Ours)** | **1.92** | **1.95** |

## Training Data

Full data composition (open-source links + in-house aggregation):

| Language | Data Source | Type | Hours | Total Hours | Share |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Chinese (Zh)** | [WenetSpeech](https://github.com/wenet-e2e/WenetSpeech) | Open Source | 10,005 | 129,265 | 37.1% |
| | [AISHELL-2](https://www.aishelltech.com/aishell_2) | Open Source | 1,000 |
| | [AISHELL-1](https://huggingface.co/datasets/AISHELL/AISHELL-1) | Open Source | 150 |
| | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 237 |
| | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 222 |
| | *In-house Data* | In-house | 117,651 |
| **Code-Switch** | [TALCS](https://github.com/SpeechClub/TALCS) | Open Source | 555 | 8,924 | 2.6% |
| | *In-house Data* | In-house | 8,369 |
| **English (En)** | [Libriheavy](https://huggingface.co/datasets/pkufool/libriheavy) | Open Source | 45,751 | 107,626 | 30.9% |
| | [Multilingual LibriSpeech (MLS)](https://huggingface.co/datasets/facebook/multilingual_librispeech) | Open Source | 44,659 |
| | [GigaSpeech](https://huggingface.co/datasets/speechcolab/gigaspeech) | Open Source | 10,000 |
| | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 3,426 |
| | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 1,778 |
| | [LibriSpeech](https://huggingface.co/datasets/librispeech_asr) | Open Source | 960 |
| | [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) | Open Source | 522 |
| | [TED-LIUM](https://huggingface.co/datasets/LIUM/tedlium) | Open Source | 453 |
| | [AMI Corpus](https://huggingface.co/datasets/edinburgh-cstr/ami) | Open Source | 77 |
| **Japanese (Ja)** | [ReazonSpeech](https://huggingface.co/datasets/reazon-research/reazonspeech) | Open Source | 35,389 | 40,426 | 11.6% |
| | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 499 |
| | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 19 |
| | *In-house Data* | In-house | 4,519 |
| **Korean (Ko)** | [KsponSpeech (AIHub)](https://huggingface.co/datasets/cheulyop/ksponspeech) | Open Source | 965 | 20,095 | 5.8% |
| | [KrespSpeech (AIHub)](https://aihub.or.kr/) | Open Source | 2,906 |
| | [KconfSpeech (AIHub)](https://aihub.or.kr/) | Open Source | 2,928 |
| | [MeetingSpeech (AIHub)](https://aihub.or.kr/) | Open Source | 4,962 |
| | [GyeongsangSpeech (AIHub)](https://aihub.or.kr/) | Open Source | 2,481 |
| | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 1,528 |
| | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 1 |
| | *In-house Data (Aggregated)* | In-house | 4,324 |
| **Russian (Ru)** | [Golos](https://huggingface.co/datasets/SberDevices/Golos) | Open Source | 1,221 | 15,246 | 4.4% |
| | [Public Speech & Radio](https://huggingface.co/datasets/bond005/sberdevices_golos_10h) | Open Source | 1,651 |
| | [Buriy Audiobook](https://huggingface.co/datasets/bond005/audio_books_russian) | Open Source | 874 |
| | Public Youtube Dataset | Open Source | 809 |
| | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 2,606 |
| | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 37 |
| | *In-house Data* | In-house | 8,048 |
| **Vietnamese (Vi)** | [GigaSpeech 2](https://huggingface.co/datasets/speechcolab/gigaspeech2) | Open Source | 6,048 | 8,390 | 2.4% |
| | [Bud500](https://huggingface.co/datasets/linhtran92/viet_bud500) | Open Source | 324 |
| | [VLSP 2020](https://vlsp.org.vn/vlsp2020) | Open Source | 101 |
| | [ViMD](https://github.com/NhutP/ViMD) | Open Source | 81 |
| | [LSVSC](https://huggingface.co/datasets/doof-ferb/LSVSC) | Open Source | 80 |
| | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 140 |
| | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 2 |
| | *In-house Data* | In-house | 1,614 |
| **Indonesian (Id)** | [GigaSpeech 2](https://huggingface.co/datasets/speechcolab/gigaspeech2) | Open Source | 6,352 | 8,238 | 2.4% |
| | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 442 |
| | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 7 |
| | *In-house Data* | In-house | 1,437 |
| **French (Fr)** | [Multilingual LibriSpeech (MLS)](https://huggingface.co/datasets/facebook/multilingual_librispeech) | Open Source | 1,076 | 4,124 | 1.2% |
| | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 1,423 |
| | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 831 |
| | [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) | Open Source | 205 |
| | *In-house Data* | In-house | 589 |
| **Spanish (Es)** | [Multilingual LibriSpeech (MLS)](https://huggingface.co/datasets/facebook/multilingual_librispeech) | Open Source | 917 | 4,596 | 1.3% |
| | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 2,399 |
| | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 502 |
| | [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) | Open Source | 151 |
| | *In-house Data* | In-house | 627 |
| **Portuguese (Pt)** | [Multilingual LibriSpeech (MLS)](https://huggingface.co/datasets/facebook/multilingual_librispeech) | Open Source | 160 | 1,602 | 0.5% |
| | [Yodas](https://huggingface.co/datasets/espnet/yodas) | Open Source | 852 |
| | [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | Open Source | 25 |
| | *In-house Data* | In-house | 565 |

Language totals from the same table:

| Language | Total Hours | Share |
| :--- | ---: | ---: |
| Chinese (Zh) | 129,265 | 37.1% |
| English (En) | 107,626 | 30.9% |
| Japanese (Ja) | 40,426 | 11.6% |
| Korean (Ko) | 20,095 | 5.8% |
| Russian (Ru) | 15,246 | 4.4% |
| Code-Switch | 8,924 | 2.6% |
| Vietnamese (Vi) | 8,390 | 2.4% |
| Indonesian (Id) | 8,238 | 2.4% |
| Spanish (Es) | 4,596 | 1.3% |
| French (Fr) | 4,124 | 1.2% |
| Portuguese (Pt) | 1,602 | 0.5% |

## ⚠️ Limitations

- Performance depends on audio quality and recording conditions.
- For long-form audio, chunking and post-processing might be required for optimal performance.
- Not designed for safety-critical applications.

## Citation

If you use this model in your research, please cite:

```bibtex
@article{liu2025tta,
  title={TTA: Transcribe, Translate and Alignment for Cross-lingual Speech Representation},
  author={Liu, Wei and Li, Jiahong and Shao, Yiwen and Yu, Dong},
  journal={arXiv preprint arXiv:2511.14410},
  year={2025}
}
```