CAFE_ASR / README.md
Add CAFE-25lang for anonymous review
f82a6c3 7 days ago
20.7 kB
---
license: other
license_name: anonymous-review-only
license_link: LICENSE
library_name: nemo
pipeline_tag: automatic-speech-recognition
language:
- bg
- hr
- cs
- da
- nl
- en
- et
- fi
- fr
- de
- el
- hu
- it
- lv
- lt
- mt
- pl
- pt
- ro
- ru
- sk
- sl
- es
- sv
- uk
tags:
- automatic-speech-recognition
- speech
- FastConformer
- TDT
- knowledge-distillation
- multilingual
- anonymous-review
---
# CAFE-25lang

> **This repository is for review only.** It is provided solely for reference by ARR reviewers and
> chairs, and no license is granted for any use of the released content, model, code, or results.
> It will be deleted after the review process ends and replaced by the official open-source Hugging
> Face model release under the Apache 2.0 license.

**Anonymous model repository for the ARR submission *"CAFE: Confining Feature Distillation to
Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition"*.**

CAFE-25lang is the 25-language CAFE model of the paper (*CAFE (ours)* in its tables). It is a
compact FastConformer-TDT transducer distilled from [Parakeet-TDT-0.6B-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3). CAFE
(**C**onfined-**A**dapter **Fe**ature distillation) sends the CKA feature-distillation gradient only
into language-routed adapters that English bypasses, so the shared encoder never receives it.

- **Size:** 254.45M parameters: a 253.95M shared model plus 0.495M of adapters. The optional
  language gate adds 0.495M (0.19%).
- **Architecture:** FastConformer encoder (24 layers, 640-dim) with a TDT decoder, initialized by
  structured pruning of the 627M teacher.
- **Adapters:** residual bottleneck adapters (rank 64) after encoder layers 3, 7, 11, 15, 19 and 23,
  applied to non-English utterances only.
- **Training data:** about 116.9M utterances (564k hours) in 25 languages from 10 corpora.
- **Checkpoint:** weight average of the last 7 checkpoints (*CA* in the paper).

**Languages (25):** Bulgarian (`bg`), Croatian (`hr`), Czech (`cs`), Danish (`da`), Dutch (`nl`),
English (`en`), Estonian (`et`), Finnish (`fi`), French (`fr`), German (`de`), Greek (`el`),
Hungarian (`hu`), Italian (`it`), Latvian (`lv`), Lithuanian (`lt`), Maltese (`mt`), Polish (`pl`),
Portuguese (`pt`), Romanian (`ro`), Russian (`ru`), Slovak (`sk`), Slovenian (`sl`), Spanish (`es`),
Swedish (`sv`), Ukrainian (`uk`).

## Files

| File | Contents |
| --- | --- |
| `CAFE-25lang.nemo` | NeMo checkpoint of the shared model (253.95M); on its own it is the English path |
| `adapters.safetensors` | The six language-routed adapters (0.495M), attached for non-English audio |
| `language_gate.safetensors` | Optional language gate (0.495M), for audio whose language is unknown |
| `inference.py` | Inference when the language is known (`--lang en` or `--lang other`) |
| `inference_gated.py` | Inference with the language gate; no language flag needed |
| `eval/` | Scripts for the Open ASR Leaderboard and Multilingual LibriSpeech numbers |
| `SHA256SUMS` | Checksums of all files above |

## Usage

Install the dependencies; audio must be 16 kHz mono WAV:

```bash
pip install "nemo_toolkit[asr]" torch safetensors soundfile
```

CAFE routes by language: English runs on the shared model alone, every other language also needs the
adapters. Routing the wrong way does not fail, it only costs accuracy (see
[Routing and the language gate](#routing-and-the-language-gate)). There are two ways to run it.

### 1. Without the gate: language known

The caller says whether the audio is English. This is the setting of the main results in the paper.

```bash
python inference.py --lang en    audio_en.wav                # shared model only
python inference.py --lang other audio_de.wav audio_pt.wav   # adapters attached
```

The same from Python:

```python
import torch
import nemo.collections.asr as nemo_asr
from inference import attach_adapters, safe_decoding
model = nemo_asr.models.ASRModel.restore_from("CAFE-25lang.nemo", map_location="cpu").eval()
safe_decoding(model)
for h in model.transcribe(["audio_en.wav"]):                  # English: no adapters
    print(h.text)
attach_adapters(model, "adapters.safetensors", torch.device("cpu"), torch.float32)
for h in model.transcribe(["audio_de.wav", "audio_pt.wav"]):  # other languages: adapters on
    print(h.text)
```

### 2. With the gate: language unknown

The gate decides per utterance. It reads encoder layer 11 with the adapters off and predicts whether
the utterance is non-English. Utterances with p(non-English) ≥ 0.8582 are then transcribed with the
adapters, the rest without:

```bash
python inference_gated.py audio_en.wav audio_de.wav audio_pt.wav
python inference_gated.py --json audio_en.wav audio_de.wav   # one JSON object per line
```

Each output line gives the gate probability, the chosen path (`EN` or `ML`) and the transcript.
`--threshold` overrides the threshold of 0.8582 stored in `language_gate.safetensors`. The gate uses
at most the first 20 s of each file.

Both scripts default to the files in this directory, use the GPU when one is available, and accept
`--model`, `--adapters`, `--device` and `--batch_size`.

## Results

All numbers are taken from the paper; accuracy is word error rate (WER, %, lower is better). Every system is
decoded and scored with the same pipeline and the official Open ASR Leaderboard text normalizer.
*EN-7* is the mean over the seven Open ASR Leaderboard English short-form sets and *ML-9* the mean of
its multilingual sets (FLEURS mean and MLS mean, five and four languages). *Avg* is the mean of EN-7
and ML-9. Unless stated otherwise, the language of each test set decides whether the adapters are
used.

### Comparison with external systems

Mean WER per test suite on 48 test sets in 24 languages, with model size and inference cost on one
NVIDIA H100 (efficiency protocol in the paper's appendix). Bold marks the best result and underline
the second best within each size group: the large group (Whisper-medium to omniASR-CTC 1B v2) and
the compact group (Whisper-small to CAFE-25lang). The teacher is shown for reference.

| Metric | Parakeet-TDT-0.6B-v3 (teacher) | Whisper-medium | omniASR-LLM 300M v2 | OWSM-CTC v4 | Qwen3-ASR-0.6B | omniASR-CTC 1B v2 | Whisper-small | omniASR-CTC 300M v2 | OWSM v4 small | **CAFE-25lang (this model)** |
|---|---|---|---|---|---|---|---|---|---|---|
| Params (M) | 627 | 764 | 1,632 | 1,011 | 782 | 976 | 242 | 326 | 370 | 254 |
| Number of languages | 25 | ~100 | >1600 | 75 | 30 | >1600 | ~100 | >1600 | 75 | 25 |
| RTFx bs1 ↑ | 198 | 21 | 10 | 590 | 18 | 513 | 35 | 848 | 39 | 213 |
| RTFx bs32 ↑ | 2,520 | 386 | 117 | 1,282 | 310 | 2,142 | 546 | 3,405 | 37 | 2,602 |
| P50 lat. bs1 (s) ↓ | 0.095 | 0.704 | 1.990 | 0.033 | 1.070 | 0.037 | 0.417 | 0.024 | 0.501 | 0.089 |
| Peak mem (MB) ↓ | 3,310 | 6,932 | 17,310 | 25,803 | 6,266 | 16,033 | 2,617 | 14,802 | 2,505 | 2,593 |
| EN short-form (7) | 6.83 | 11.21 | 11.28 | <u>7.90</u> | **6.50** | 12.21 | 13.91 | 17.94 | **8.73** | <u>9.03</u> |
| Multilingual (9) | 5.14 | **6.36** | 7.58 | 8.74 | 8.29 | <u>7.16</u> | 10.76 | 12.68 | <u>10.71</u> | **7.00** |
| EN long-form (4) | 10.77 | 16.28 | 23.50 | <u>12.22</u> | **10.91** | 54.82 | 16.71 | 42.45 | **13.01** | <u>13.35</u> |
| MLS-8 (8) | 6.66 | <u>7.76</u> | 7.79 | 10.65 | 12.60 | **7.71** | <u>11.51</u> | 11.89 | 11.84 | **6.58** |
| Common Voice v25 (24) | 16.48 | 26.94 | **17.72** | 32.87 | N/A | <u>20.07</u> | 38.17 | <u>32.23</u> | 37.74 | **21.56** |

Qwen3-ASR-0.6B has no 24-language Common Voice mean because some languages are outside its
supported set.

### Main comparison: Open ASR Leaderboard

All students share the same data, schedule, and model size, and differ only in the adapters and the
feature-distillation loss. *CAFE w/o sg* is CAFE without the stop-gradient. *mean ± std* is over
the last 7 checkpoints and *CA* is their weight average; this repository contains the CA model of
the *CAFE (ours)* row. The 8-language pair is trained on the 8 MLS languages only and is not part of
this repository.

| System | Languages | Adapters | Feature KD | EN-7 mean ± std | EN-7 CA | ML-9 mean ± std | ML-9 CA | Avg mean ± std | Avg CA |
|---|---|---|---|---|---|---|---|---|---|
| Baseline | 25 | – | – | 10.33 ± 0.82 | 9.85 | 7.59 ± 0.20 | 7.24 | 8.96 ± 0.39 | 8.55 |
| CKA | 25 | – | CKA | 11.42 ± 0.67 | 11.25 | **7.22 ± 0.12** | **6.83** | 9.32 ± 0.33 | 9.04 |
| Capacity | 25 | ✓ | – | 9.98 ± 0.53 | 9.67 | 7.77 ± 0.11 | 7.34 | 8.88 ± 0.27 | 8.51 |
| CAFE w/o sg | 25 | ✓ | CKA (no sg) | 10.75 ± 0.69 | 10.44 | 7.37 ± 0.12 | 6.95 | 9.06 ± 0.32 | 8.70 |
| **CAFE (ours)** | 25 | ✓ | CKA (confined) | **9.32 ± 0.32** | **9.03** | 7.43 ± 0.13 | 7.00 | **8.38 ± 0.19** | **8.02** |
| Baseline-8lang | 8 | – | – | 10.32 ± 0.54 | 10.11 | 7.08 ± 0.15 | 6.62 | 8.70 ± 0.22 | 8.36 |
| **CAFE-8lang (ours)** | 8 | ✓ | CKA (confined) | **8.93 ± 0.27** | **8.71** | **6.91 ± 0.12** | **6.53** | **7.92 ± 0.13** | **7.62** |

### Adapter overhead

|  | **CAFE-25lang** | Baseline (same model without adapters) |
|---|---|---|
| Params | 254.45M | 253.95M |
| RTFx bs1 ↑ | 212.9 ± 6.9 | 221.3 ± 5.2 |
| RTFx bs32 ↑ | 2602.3 ± 68.9 | 2656.6 ± 79.8 |
| P50 lat. (s) ↓ | 0.089 ± 0.001 | 0.088 ± 0.000 |
| Peak mem (MB) ↓ | 2593.4 | 2592.5 |

RTFx is the inverse real-time factor, P50 the median latency per utterance at batch size 1, and peak
memory is measured at batch size 32, all on one NVIDIA H100.

### Routing and the language gate

Gate accuracy on 7,692 utterances, 500 sampled from each of the 16 EN-7 and ML-9 test sets (all 192
for MLS pt):

| Overall | English | Non-English |
|---|---|---|
| 98.9% | 97.6% | 99.9% |

WER when the English/non-English decision comes from the test set (language given) or from the gate
(language unknown). The last two columns apply the adapters to every utterance or to none:

| Open ASR Leaderboard | Baseline (no adapters, no CKA) | CAFE, language given | CAFE + gate, language unknown | Adapters always on | Adapters never on |
|---|---|---|---|---|---|
| EN-7 mean | 9.85 | 9.03 | 9.11 | 15.95 | 9.03 |
| ML-9 mean | 7.24 | 7.00 | 7.03 | 7.00 | 8.04 |
| Avg (EN-7, ML-9) | 8.55 | 8.02 | 8.07 | 11.48 | 8.54 |

With the gate the average WER rises by 0.05 (8.02 to 8.07). Applying the adapters to English is
costly (EN-7 15.95 against 9.03), while never applying them loses the multilingual gain (ML-9 8.04
against 7.00).

<details>
<summary><b>Per-dataset WER with and without the gate</b></summary>

| Test set | Baseline (no adapters, no CKA) | CAFE, language given | CAFE + gate, language unknown | Adapters always on | Adapters never on |
|---|---|---|---|---|---|
| **EN-7 mean** | 9.85 | 9.03 | 9.11 | 15.95 | 9.03 |
| **FLEURS mean** | 7.22 | 7.00 | 7.04 | 7.00 | 8.46 |
| **MLS mean** | 7.27 | 7.01 | 7.01 | 7.01 | 7.62 |
| **Multilingual mean** | 7.24 | 7.00 | 7.03 | 7.00 | 8.04 |
| **Avg (EN, ML)** | 8.55 | 8.02 | 8.07 | 11.48 | 8.54 |

</details>

### Per-dataset results

Every test set behind the summary table, against the same external systems. Long-form results for
omniASR and Qwen3-ASR-0.6B use 40 s non-overlapping windows, the maximum input length of omniASR;
Qwen3-ASR-0.6B degenerated with its native 1200 s windows.

<details>
<summary><b>Model size and inference cost</b></summary>

| Metric | Parakeet-TDT-0.6B-v3 (teacher) | Whisper-medium | omniASR-LLM 300M v2 | OWSM-CTC v4 | Qwen3-ASR-0.6B | omniASR-CTC 1B v2 | Whisper-small | omniASR-CTC 300M v2 | OWSM v4 small | **CAFE-25lang (this model)** |
|---|---|---|---|---|---|---|---|---|---|---|
| Number of languages | 25 | ~100 | >1600 | 75 | 30 | >1600 | ~100 | >1600 | 75 | 25 |
| Params (M) ↓ | 627 | 764 | 1,632 | 1,011 | 782 | 976 | 242 | 326 | 370 | 254 |
| RTFx bs1 ↑ | 198 | 21 | 10 | 590 | 18 | 513 | 35 | 848 | 39 | 213 |
| RTFx bs32 ↑ | 2,520 | 386 | 117 | 1,282 | 310 | 2,142 | 546 | 3,405 | 37 | 2,602 |
| P50 lat. bs1 (s) ↓ | 0.095 | 0.704 | 1.990 | 0.033 | 1.070 | 0.037 | 0.417 | 0.024 | 0.501 | 0.089 |
| Peak mem (MB) ↓ | 3,310 | 6,932 | 17,310 | 25,803 | 6,266 | 16,033 | 2,617 | 14,802 | 2,505 | 2,593 |

</details>

<details>
<summary><b>Open ASR Leaderboard - English short-form (7)</b></summary>

| Test set | Parakeet-TDT-0.6B-v3 (teacher) | Whisper-medium | omniASR-LLM 300M v2 | OWSM-CTC v4 | Qwen3-ASR-0.6B | omniASR-CTC 1B v2 | Whisper-small | omniASR-CTC 300M v2 | OWSM v4 small | **CAFE-25lang (this model)** |
|---|---|---|---|---|---|---|---|---|---|---|
| ami | 11.44 | 38.96 | 26.03 | 13.12 | 10.96 | 24.24 | 53.35 | 32.51 | 15.47 | 13.91 |
| earnings22 | 11.19 | 11.49 | 16.90 | 13.89 | 10.76 | 19.79 | 13.21 | 26.33 | 14.48 | 13.55 |
| gigaspeech | 9.57 | 10.05 | 12.83 | 10.87 | 8.37 | 15.80 | 10.56 | 22.84 | 11.17 | 11.72 |
| LibriSpeech.clean | 1.93 | 2.58 | 2.16 | 2.60 | 1.75 | 2.83 | 2.59 | 4.71 | 2.56 | 2.48 |
| LibriSpeech.other | 3.59 | 4.87 | 6.54 | 4.88 | 4.08 | 6.67 | 6.60 | 11.86 | 5.98 | 5.81 |
| spgispeech | 3.98 | 2.92 | 5.82 | 2.57 | 2.79 | 7.37 | 3.24 | 14.05 | 2.79 | 7.98 |
| voxpopuli | 6.14 | 7.63 | 8.71 | 7.35 | 6.82 | 8.79 | 7.85 | 13.28 | 8.65 | 7.76 |
| **category mean (7)** | 6.83 | 11.21 | 11.28 | <u>7.90</u> | **6.50** | 12.21 | 13.91 | 17.94 | **8.73** | <u>9.03</u> |

</details>

<details>
<summary><b>Open ASR Leaderboard - English long-form (4 sets; CORAAL as 8 subsets)</b></summary>

| Test set | Parakeet-TDT-0.6B-v3 (teacher) | Whisper-medium | omniASR-LLM 300M v2 | OWSM-CTC v4 | Qwen3-ASR-0.6B | omniASR-CTC 1B v2 | Whisper-small | omniASR-CTC 300M v2 | OWSM v4 small | **CAFE-25lang (this model)** |
|---|---|---|---|---|---|---|---|---|---|---|
| earnings21 | 10.21 | 12.52 | 21.30 | 9.82 | 9.20 | 52.02 | 14.07 | 31.14 | 10.80 | 12.31 |
| earnings22-lf | 14.52 | 16.73 | 24.31 | 13.62 | 13.59 | 53.60 | 17.98 | 40.19 | 15.24 | 14.76 |
| tedlium | 2.74 | 3.10 | 11.20 | 3.38 | 2.89 | 52.95 | 3.52 | 31.48 | 4.06 | 4.08 |
| coraal avg (8) | 15.59 | 32.76 | 37.20 | 22.06 | 17.98 | 60.71 | 31.27 | 66.97 | 21.94 | 22.25 |
| &nbsp;&nbsp;CORAAL ATL | 16.69 | 39.45 | 40.75 | 24.11 | 18.20 | 55.15 | 34.57 | 72.58 | 23.63 | 22.77 |
| &nbsp;&nbsp;CORAAL DCA | 19.00 | 39.57 | 40.81 | 26.22 | 21.71 | 64.03 | 39.34 | 67.28 | 26.05 | 25.86 |
| &nbsp;&nbsp;CORAAL DCB | 11.72 | 23.40 | 34.54 | 16.93 | 13.70 | 60.98 | 25.31 | 63.50 | 17.74 | 18.35 |
| &nbsp;&nbsp;CORAAL DTA | 19.61 | 36.16 | 42.01 | 29.29 | 22.76 | 62.19 | 38.86 | 75.00 | 27.95 | 28.13 |
| &nbsp;&nbsp;CORAAL LES | 15.47 | 33.06 | 36.95 | 21.43 | 18.04 | 58.62 | 29.94 | 65.49 | 20.86 | 21.77 |
| &nbsp;&nbsp;CORAAL PRV | 22.36 | 43.39 | 46.82 | 31.41 | 26.92 | 72.71 | 41.13 | 79.68 | 30.64 | 31.01 |
| &nbsp;&nbsp;CORAAL ROC | 8.16 | 19.41 | 23.31 | 10.96 | 9.41 | 64.91 | 16.46 | 54.70 | 11.61 | 12.75 |
| &nbsp;&nbsp;CORAAL VLD | 11.71 | 27.64 | 32.38 | 16.11 | 13.11 | 47.10 | 24.51 | 57.57 | 17.06 | 17.38 |
| **category mean (4)** | 10.77 | 16.28 | 23.50 | <u>12.22</u> | **10.91** | 54.82 | 16.71 | 42.45 | **13.01** | <u>13.35</u> |

</details>

<details>
<summary><b>Open ASR Leaderboard - Multilingual: FLEURS (5)</b></summary>

| Test set | Parakeet-TDT-0.6B-v3 (teacher) | Whisper-medium | omniASR-LLM 300M v2 | OWSM-CTC v4 | Qwen3-ASR-0.6B | omniASR-CTC 1B v2 | Whisper-small | omniASR-CTC 300M v2 | OWSM v4 small | **CAFE-25lang (this model)** |
|---|---|---|---|---|---|---|---|---|---|---|
| de | 4.25 | 4.66 | 7.99 | 8.26 | 5.54 | 7.46 | 7.61 | 14.53 | 10.08 | 7.61 |
| es | 3.27 | 3.12 | 5.39 | 5.59 | 4.47 | 5.33 | 4.67 | 9.56 | 8.21 | 5.38 |
| fr | 4.72 | 6.89 | 11.25 | 9.79 | 7.07 | 10.84 | 11.52 | 21.23 | 13.16 | 9.83 |
| it | 2.45 | 3.56 | 5.47 | 5.76 | 4.62 | 4.87 | 16.02 | 9.46 | 6.87 | 4.43 |
| pt | 4.54 | 4.29 | 7.70 | 7.42 | 6.08 | 7.19 | 6.27 | 13.63 | 8.76 | 7.74 |
| **category mean (5)** | 3.85 | **4.50** | 7.56 | 7.36 | <u>5.56</u> | 7.14 | <u>9.22</u> | 13.68 | 9.42 | **7.00** |

</details>

<details>
<summary><b>Open ASR Leaderboard - Multilingual: MLS (4)</b></summary>

| Test set | Parakeet-TDT-0.6B-v3 (teacher) | Whisper-medium | omniASR-LLM 300M v2 | OWSM-CTC v4 | Qwen3-ASR-0.6B | omniASR-CTC 1B v2 | Whisper-small | omniASR-CTC 300M v2 | OWSM v4 small | **CAFE-25lang (this model)** |
|---|---|---|---|---|---|---|---|---|---|---|
| es | 4.40 | 4.77 | 4.80 | 6.44 | 6.80 | 4.20 | 6.89 | 7.25 | 7.61 | 4.74 |
| fr | 5.13 | 7.17 | 6.82 | 8.31 | 8.54 | 6.94 | 10.93 | 12.95 | 11.05 | 4.85 |
| it | 8.52 | 10.58 | 7.25 | 14.14 | 14.76 | 6.51 | 16.02 | 10.73 | 14.49 | 7.37 |
| pt | 7.67 | 10.33 | 11.50 | 11.55 | 14.03 | 11.05 | 15.37 | 15.81 | 14.83 | 11.07 |
| **category mean (4)** | 6.43 | 8.21 | <u>7.59</u> | 10.11 | 11.03 | **7.18** | 12.30 | <u>11.69</u> | 12.00 | **7.01** |
| **EN short-form & Multilingual mean** | 5.99 | 8.79 | 9.43 | <u>8.32</u> | **7.40** | 9.68 | 12.34 | 15.31 | <u>9.72</u> | **8.02** |

</details>

<details>
<summary><b>Multilingual LibriSpeech (MLS-8)</b></summary>

| Test set | Parakeet-TDT-0.6B-v3 (teacher) | Whisper-medium | omniASR-LLM 300M v2 | OWSM-CTC v4 | Qwen3-ASR-0.6B | omniASR-CTC 1B v2 | Whisper-small | omniASR-CTC 300M v2 | OWSM v4 small | **CAFE-25lang (this model)** |
|---|---|---|---|---|---|---|---|---|---|---|
| en | 4.82 | 6.21 | 8.68 | 6.75 | 5.77 | 9.20 | 7.66 | 15.10 | 7.96 | 6.48 |
| de | 5.79 | 5.75 | 7.02 | 10.36 | 8.18 | 6.69 | 9.35 | 11.02 | 10.43 | 6.59 |
| nl | 11.24 | 11.42 | 6.92 | 11.22 | 16.69 | 7.93 | 16.54 | 12.89 | 11.64 | 5.89 |
| pl | 5.74 | 5.89 | 9.32 | 16.39 | 26.02 | 9.18 | 9.26 | 9.39 | 16.72 | 5.63 |
| es | 4.39 | 4.76 | 4.80 | 6.44 | 6.80 | 4.20 | 6.90 | 7.25 | 7.61 | 4.74 |
| fr | 5.07 | 7.18 | 6.82 | 8.31 | 8.54 | 6.94 | 10.94 | 12.95 | 11.05 | 4.87 |
| it | 8.53 | 10.60 | 7.25 | 14.14 | 14.76 | 6.51 | 16.06 | 10.73 | 14.49 | 7.37 |
| pt | 7.69 | 10.28 | 11.50 | 11.55 | 14.03 | 11.05 | 15.37 | 15.81 | 14.83 | 11.04 |
| **mean (8)** | 6.66 | <u>7.76</u> | 7.79 | 10.65 | 12.60 | **7.71** | <u>11.51</u> | 11.89 | 11.84 | **6.58** |

</details>

<details>
<summary><b>Common Voice v25 (24)</b></summary>

| Test set | Parakeet-TDT-0.6B-v3 (teacher) | Whisper-medium | omniASR-LLM 300M v2 | OWSM-CTC v4 | Qwen3-ASR-0.6B | omniASR-CTC 1B v2 | Whisper-small | omniASR-CTC 300M v2 | OWSM v4 small | **CAFE-25lang (this model)** |
|---|---|---|---|---|---|---|---|---|---|---|
| bg | 18.21 | 23.76 | 12.87 | 46.54 | N/A | 16.64 | 41.00 | 30.08 | 52.54 | 18.20 |
| cs | 5.92 | 18.30 | 9.61 | 23.05 | 42.79 | 10.53 | 33.55 | 20.33 | 27.39 | 12.47 |
| da | 40.38 | 44.38 | 40.56 | 66.55 | 55.37 | 42.77 | 52.21 | 52.46 | 74.82 | 43.70 |
| de | 4.40 | 7.83 | 10.58 | 8.83 | 8.30 | 10.02 | 13.97 | 19.29 | 10.71 | 8.83 |
| el | 18.23 | 22.76 | 13.51 | 62.36 | 52.11 | 25.40 | 29.88 | 37.23 | 65.56 | 32.80 |
| en | 8.47 | 12.38 | 18.94 | 11.45 | 9.67 | 19.78 | 17.18 | 31.20 | 14.22 | 13.94 |
| es | 3.76 | 7.15 | 7.53 | 7.20 | 6.58 | 7.83 | 11.63 | 14.23 | 9.34 | 7.22 |
| et | 22.29 | 44.33 | 22.62 | 53.75 | N/A | 20.79 | 69.50 | 39.42 | 52.14 | 27.37 |
| fi | 48.76 | 53.14 | 51.97 | 64.14 | 72.67 | 52.67 | 58.76 | 58.49 | 67.07 | 54.29 |
| fr | 7.33 | 18.85 | 14.28 | 12.31 | 11.75 | 15.18 | 26.92 | 24.82 | 16.22 | 11.64 |
| hu | 6.55 | 18.90 | 11.02 | 35.12 | 55.36 | 13.44 | 36.10 | 26.10 | 36.78 | 11.79 |
| it | 3.59 | 8.62 | 8.97 | 8.18 | 8.70 | 8.78 | 16.81 | 16.28 | 10.24 | 7.15 |
| lt | 16.60 | 50.51 | 22.56 | 60.22 | N/A | 26.23 | 73.12 | 45.25 | 65.21 | 25.68 |
| lv | 36.08 | 33.94 | 18.87 | 55.13 | N/A | 21.05 | 55.98 | 36.61 | 58.51 | 17.79 |
| mt | 12.16 | 93.16 | 11.79 | 62.61 | N/A | 17.77 | 109.40 | 31.01 | 73.66 | 46.73 |
| nl | 4.21 | 6.53 | 6.98 | 8.07 | 10.11 | 7.70 | 12.66 | 16.74 | 12.11 | 6.04 |
| pl | 8.53 | 15.31 | 15.81 | 20.25 | 30.20 | 15.53 | 20.74 | 25.53 | 24.85 | 15.91 |
| pt | 8.04 | 9.85 | 12.16 | 8.91 | 10.92 | 12.87 | 22.75 | 23.69 | 13.01 | 10.94 |
| ro | 10.79 | 22.13 | 13.35 | 20.10 | 42.23 | 12.97 | 31.82 | 23.80 | 34.09 | 22.46 |
| ru | 5.54 | 9.44 | 12.93 | 9.06 | 12.96 | 14.86 | 16.42 | 32.45 | 14.68 | 14.62 |
| sk | 8.36 | 40.31 | 15.70 | 33.79 | N/A | 25.43 | 52.97 | 34.31 | 41.75 | 21.68 |
| sl | 49.60 | 33.42 | 17.71 | 34.86 | N/A | 25.05 | 45.02 | 48.08 | 42.90 | 26.51 |
| sv-SE | 17.59 | 12.13 | 17.90 | 35.41 | 37.47 | 17.22 | 21.15 | 33.36 | 41.65 | 17.99 |
| uk | 30.09 | 39.44 | 37.12 | 41.03 | N/A | 41.06 | 46.62 | 52.84 | 46.21 | 41.70 |
| **mean all (24)** | 16.48 | 26.94 | **17.72** | 32.87 | N/A | <u>20.07</u> | 38.17 | <u>32.23</u> | 37.74 | **21.56** |

</details>

## Evaluation

[`eval/README.md`](eval/README.md) shows how to reproduce the Open ASR Leaderboard and Multilingual
LibriSpeech numbers with the official leaderboard runners and normalizer. Decoding is greedy, with
the CUDA-graph decoder disabled.