anonymous1121/CAFE_ASR

CAFE-25lang

This repository is for review only. It is provided solely for reference by ARR reviewers and chairs, and no license is granted for any use of the released content, model, code, or results. It will be deleted after the review process ends and replaced by the official open-source Hugging Face model release under the Apache 2.0 license.

*Anonymous model repository for the ARR submission "CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition".*

CAFE-25lang is the 25-language CAFE model of the paper (CAFE (ours) in its tables). It is a compact FastConformer-TDT transducer distilled from Parakeet-TDT-0.6B-v3. CAFE (Confined-Adapter Feature distillation) sends the CKA feature-distillation gradient only into language-routed adapters that English bypasses, so the shared encoder never receives it.

• Size: 254.45M parameters: a 253.95M shared model plus 0.495M of adapters. The optional language gate adds 0.495M (0.19%).
• Architecture: FastConformer encoder (24 layers, 640-dim) with a TDT decoder, initialized by structured pruning of the 627M teacher.
• Adapters: residual bottleneck adapters (rank 64) after encoder layers 3, 7, 11, 15, 19 and 23, applied to non-English utterances only.
• Training data: about 116.9M utterances (564k hours) in 25 languages from 10 corpora.
• Checkpoint: weight average of the last 7 checkpoints (CA in the paper).

Languages (25): Bulgarian (bg), Croatian (hr), Czech (cs), Danish (da), Dutch (nl), English (en), Estonian (et), Finnish (fi), French (fr), German (de), Greek (el), Hungarian (hu), Italian (it), Latvian (lv), Lithuanian (lt), Maltese (mt), Polish (pl), Portuguese (pt), Romanian (ro), Russian (ru), Slovak (sk), Slovenian (sl), Spanish (es), Swedish (sv), Ukrainian (uk).

Files

File	Contents
CAFE-25lang.nemo	NeMo checkpoint of the shared model (253.95M); on its own it is the English path
adapters.safetensors	The six language-routed adapters (0.495M), attached for non-English audio
language_gate.safetensors	Optional language gate (0.495M), for audio whose language is unknown
inference.py	Inference when the language is known (--lang en or --lang other)
inference_gated.py	Inference with the language gate; no language flag needed
eval/	Scripts for the Open ASR Leaderboard and Multilingual LibriSpeech numbers
SHA256SUMS	Checksums of all files above

Usage

Install the dependencies; audio must be 16 kHz mono WAV:

pip install "nemo_toolkit[asr]" torch safetensors soundfile

CAFE routes by language: English runs on the shared model alone, every other language also needs the adapters. Routing the wrong way does not fail, it only costs accuracy (see Routing and the language gate). There are two ways to run it.

1. Without the gate: language known

The caller says whether the audio is English. This is the setting of the main results in the paper.

python inference.py --lang en    audio_en.wav                # shared model only
python inference.py --lang other audio_de.wav audio_pt.wav   # adapters attached

The same from Python:

import torch
import nemo.collections.asr as nemo_asr
from inference import attach_adapters, safe_decoding

model = nemo_asr.models.ASRModel.restore_from("CAFE-25lang.nemo", map_location="cpu").eval()
safe_decoding(model)

for h in model.transcribe(["audio_en.wav"]):                  # English: no adapters
    print(h.text)

attach_adapters(model, "adapters.safetensors", torch.device("cpu"), torch.float32)
for h in model.transcribe(["audio_de.wav", "audio_pt.wav"]):  # other languages: adapters on
    print(h.text)

2. With the gate: language unknown

The gate decides per utterance. It reads encoder layer 11 with the adapters off and predicts whether the utterance is non-English. Utterances with p(non-English) ≥ 0.8582 are then transcribed with the adapters, the rest without:

python inference_gated.py audio_en.wav audio_de.wav audio_pt.wav
python inference_gated.py --json audio_en.wav audio_de.wav   # one JSON object per line

Each output line gives the gate probability, the chosen path (EN or ML) and the transcript. --threshold overrides the threshold of 0.8582 stored in language_gate.safetensors. The gate uses at most the first 20 s of each file.

Both scripts default to the files in this directory, use the GPU when one is available, and accept --model, --adapters, --device and --batch_size.

Results

All numbers are taken from the paper; accuracy is word error rate (WER, %, lower is better). Every system is decoded and scored with the same pipeline and the official Open ASR Leaderboard text normalizer. EN-7 is the mean over the seven Open ASR Leaderboard English short-form sets and ML-9 the mean of its multilingual sets (FLEURS mean and MLS mean, five and four languages). Avg is the mean of EN-7 and ML-9. Unless stated otherwise, the language of each test set decides whether the adapters are used.

Comparison with external systems

Mean WER per test suite on 48 test sets in 24 languages, with model size and inference cost on one NVIDIA H100 (efficiency protocol in the paper's appendix). Bold marks the best result and underline the second best within each size group: the large group (Whisper-medium to omniASR-CTC 1B v2) and the compact group (Whisper-small to CAFE-25lang). The teacher is shown for reference.

Metric	Parakeet-TDT-0.6B-v3 (teacher)	Whisper-medium	omniASR-LLM 300M v2	OWSM-CTC v4	Qwen3-ASR-0.6B	omniASR-CTC 1B v2	Whisper-small	omniASR-CTC 300M v2	OWSM v4 small	CAFE-25lang (this model)
Params (M)	627	764	1,632	1,011	782	976	242	326	370	254
Number of languages	25	~100	>1600	75	30	>1600	~100	>1600	75	25
RTFx bs1 ↑	198	21	10	590	18	513	35	848	39	213
RTFx bs32 ↑	2,520	386	117	1,282	310	2,142	546	3,405	37	2,602
P50 lat. bs1 (s) ↓	0.095	0.704	1.990	0.033	1.070	0.037	0.417	0.024	0.501	0.089
Peak mem (MB) ↓	3,310	6,932	17,310	25,803	6,266	16,033	2,617	14,802	2,505	2,593
EN short-form (7)	6.83	11.21	11.28	7.90	6.50	12.21	13.91	17.94	8.73	9.03
Multilingual (9)	5.14	6.36	7.58	8.74	8.29	7.16	10.76	12.68	10.71	7.00
EN long-form (4)	10.77	16.28	23.50	12.22	10.91	54.82	16.71	42.45	13.01	13.35
MLS-8 (8)	6.66	7.76	7.79	10.65	12.60	7.71	11.51	11.89	11.84	6.58
Common Voice v25 (24)	16.48	26.94	17.72	32.87	N/A	20.07	38.17	32.23	37.74	21.56

Qwen3-ASR-0.6B has no 24-language Common Voice mean because some languages are outside its supported set.

Main comparison: Open ASR Leaderboard

All students share the same data, schedule, and model size, and differ only in the adapters and the feature-distillation loss. CAFE w/o sg is CAFE without the stop-gradient. mean ± std is over the last 7 checkpoints and CA is their weight average; this repository contains the CA model of the CAFE (ours) row. The 8-language pair is trained on the 8 MLS languages only and is not part of this repository.

System	Languages	Adapters	Feature KD	EN-7 mean ± std	EN-7 CA	ML-9 mean ± std	ML-9 CA	Avg mean ± std	Avg CA
Baseline	25	–	–	10.33 ± 0.82	9.85	7.59 ± 0.20	7.24	8.96 ± 0.39	8.55
CKA	25	–	CKA	11.42 ± 0.67	11.25	7.22 ± 0.12	6.83	9.32 ± 0.33	9.04
Capacity	25	✓	–	9.98 ± 0.53	9.67	7.77 ± 0.11	7.34	8.88 ± 0.27	8.51
CAFE w/o sg	25	✓	CKA (no sg)	10.75 ± 0.69	10.44	7.37 ± 0.12	6.95	9.06 ± 0.32	8.70
CAFE (ours)	25	✓	CKA (confined)	9.32 ± 0.32	9.03	7.43 ± 0.13	7.00	8.38 ± 0.19	8.02
Baseline-8lang	8	–	–	10.32 ± 0.54	10.11	7.08 ± 0.15	6.62	8.70 ± 0.22	8.36
CAFE-8lang (ours)	8	✓	CKA (confined)	8.93 ± 0.27	8.71	6.91 ± 0.12	6.53	7.92 ± 0.13	7.62

Adapter overhead

	CAFE-25lang	Baseline (same model without adapters)
Params	254.45M	253.95M
RTFx bs1 ↑	212.9 ± 6.9	221.3 ± 5.2
RTFx bs32 ↑	2602.3 ± 68.9	2656.6 ± 79.8
P50 lat. (s) ↓	0.089 ± 0.001	0.088 ± 0.000
Peak mem (MB) ↓	2593.4	2592.5

RTFx is the inverse real-time factor, P50 the median latency per utterance at batch size 1, and peak memory is measured at batch size 32, all on one NVIDIA H100.

Routing and the language gate

Gate accuracy on 7,692 utterances, 500 sampled from each of the 16 EN-7 and ML-9 test sets (all 192 for MLS pt):

Overall	English	Non-English
98.9%	97.6%	99.9%

WER when the English/non-English decision comes from the test set (language given) or from the gate (language unknown). The last two columns apply the adapters to every utterance or to none:

Open ASR Leaderboard	Baseline (no adapters, no CKA)	CAFE, language given	CAFE + gate, language unknown	Adapters always on	Adapters never on
EN-7 mean	9.85	9.03	9.11	15.95	9.03
ML-9 mean	7.24	7.00	7.03	7.00	8.04
Avg (EN-7, ML-9)	8.55	8.02	8.07	11.48	8.54

With the gate the average WER rises by 0.05 (8.02 to 8.07). Applying the adapters to English is costly (EN-7 15.95 against 9.03), while never applying them loses the multilingual gain (ML-9 8.04 against 7.00).

Per-dataset WER with and without the gate

Test set	Baseline (no adapters, no CKA)	CAFE, language given	CAFE + gate, language unknown	Adapters always on	Adapters never on
EN-7 mean	9.85	9.03	9.11	15.95	9.03
FLEURS mean	7.22	7.00	7.04	7.00	8.46
MLS mean	7.27	7.01	7.01	7.01	7.62
Multilingual mean	7.24	7.00	7.03	7.00	8.04
Avg (EN, ML)	8.55	8.02	8.07	11.48	8.54

Per-dataset results

Every test set behind the summary table, against the same external systems. Long-form results for omniASR and Qwen3-ASR-0.6B use 40 s non-overlapping windows, the maximum input length of omniASR; Qwen3-ASR-0.6B degenerated with its native 1200 s windows.

Model size and inference cost

Metric	Parakeet-TDT-0.6B-v3 (teacher)	Whisper-medium	omniASR-LLM 300M v2	OWSM-CTC v4	Qwen3-ASR-0.6B	omniASR-CTC 1B v2	Whisper-small	omniASR-CTC 300M v2	OWSM v4 small	CAFE-25lang (this model)
Number of languages	25	~100	>1600	75	30	>1600	~100	>1600	75	25
Params (M) ↓	627	764	1,632	1,011	782	976	242	326	370	254
RTFx bs1 ↑	198	21	10	590	18	513	35	848	39	213
RTFx bs32 ↑	2,520	386	117	1,282	310	2,142	546	3,405	37	2,602
P50 lat. bs1 (s) ↓	0.095	0.704	1.990	0.033	1.070	0.037	0.417	0.024	0.501	0.089
Peak mem (MB) ↓	3,310	6,932	17,310	25,803	6,266	16,033	2,617	14,802	2,505	2,593

Open ASR Leaderboard - English short-form (7)

Test set	Parakeet-TDT-0.6B-v3 (teacher)	Whisper-medium	omniASR-LLM 300M v2	OWSM-CTC v4	Qwen3-ASR-0.6B	omniASR-CTC 1B v2	Whisper-small	omniASR-CTC 300M v2	OWSM v4 small	CAFE-25lang (this model)
ami	11.44	38.96	26.03	13.12	10.96	24.24	53.35	32.51	15.47	13.91
earnings22	11.19	11.49	16.90	13.89	10.76	19.79	13.21	26.33	14.48	13.55
gigaspeech	9.57	10.05	12.83	10.87	8.37	15.80	10.56	22.84	11.17	11.72
LibriSpeech.clean	1.93	2.58	2.16	2.60	1.75	2.83	2.59	4.71	2.56	2.48
LibriSpeech.other	3.59	4.87	6.54	4.88	4.08	6.67	6.60	11.86	5.98	5.81
spgispeech	3.98	2.92	5.82	2.57	2.79	7.37	3.24	14.05	2.79	7.98
voxpopuli	6.14	7.63	8.71	7.35	6.82	8.79	7.85	13.28	8.65	7.76
category mean (7)	6.83	11.21	11.28	7.90	6.50	12.21	13.91	17.94	8.73	9.03

Open ASR Leaderboard - English long-form (4 sets; CORAAL as 8 subsets)

Test set	Parakeet-TDT-0.6B-v3 (teacher)	Whisper-medium	omniASR-LLM 300M v2	OWSM-CTC v4	Qwen3-ASR-0.6B	omniASR-CTC 1B v2	Whisper-small	omniASR-CTC 300M v2	OWSM v4 small	CAFE-25lang (this model)
earnings21	10.21	12.52	21.30	9.82	9.20	52.02	14.07	31.14	10.80	12.31
earnings22-lf	14.52	16.73	24.31	13.62	13.59	53.60	17.98	40.19	15.24	14.76
tedlium	2.74	3.10	11.20	3.38	2.89	52.95	3.52	31.48	4.06	4.08
coraal avg (8)	15.59	32.76	37.20	22.06	17.98	60.71	31.27	66.97	21.94	22.25
CORAAL ATL	16.69	39.45	40.75	24.11	18.20	55.15	34.57	72.58	23.63	22.77
CORAAL DCA	19.00	39.57	40.81	26.22	21.71	64.03	39.34	67.28	26.05	25.86
CORAAL DCB	11.72	23.40	34.54	16.93	13.70	60.98	25.31	63.50	17.74	18.35
CORAAL DTA	19.61	36.16	42.01	29.29	22.76	62.19	38.86	75.00	27.95	28.13
CORAAL LES	15.47	33.06	36.95	21.43	18.04	58.62	29.94	65.49	20.86	21.77
CORAAL PRV	22.36	43.39	46.82	31.41	26.92	72.71	41.13	79.68	30.64	31.01
CORAAL ROC	8.16	19.41	23.31	10.96	9.41	64.91	16.46	54.70	11.61	12.75
CORAAL VLD	11.71	27.64	32.38	16.11	13.11	47.10	24.51	57.57	17.06	17.38
category mean (4)	10.77	16.28	23.50	12.22	10.91	54.82	16.71	42.45	13.01	13.35

Open ASR Leaderboard - Multilingual: FLEURS (5)

Test set	Parakeet-TDT-0.6B-v3 (teacher)	Whisper-medium	omniASR-LLM 300M v2	OWSM-CTC v4	Qwen3-ASR-0.6B	omniASR-CTC 1B v2	Whisper-small	omniASR-CTC 300M v2	OWSM v4 small	CAFE-25lang (this model)
de	4.25	4.66	7.99	8.26	5.54	7.46	7.61	14.53	10.08	7.61
es	3.27	3.12	5.39	5.59	4.47	5.33	4.67	9.56	8.21	5.38
fr	4.72	6.89	11.25	9.79	7.07	10.84	11.52	21.23	13.16	9.83
it	2.45	3.56	5.47	5.76	4.62	4.87	16.02	9.46	6.87	4.43
pt	4.54	4.29	7.70	7.42	6.08	7.19	6.27	13.63	8.76	7.74
category mean (5)	3.85	4.50	7.56	7.36	5.56	7.14	9.22	13.68	9.42	7.00

Open ASR Leaderboard - Multilingual: MLS (4)

Test set	Parakeet-TDT-0.6B-v3 (teacher)	Whisper-medium	omniASR-LLM 300M v2	OWSM-CTC v4	Qwen3-ASR-0.6B	omniASR-CTC 1B v2	Whisper-small	omniASR-CTC 300M v2	OWSM v4 small	CAFE-25lang (this model)
es	4.40	4.77	4.80	6.44	6.80	4.20	6.89	7.25	7.61	4.74
fr	5.13	7.17	6.82	8.31	8.54	6.94	10.93	12.95	11.05	4.85
it	8.52	10.58	7.25	14.14	14.76	6.51	16.02	10.73	14.49	7.37
pt	7.67	10.33	11.50	11.55	14.03	11.05	15.37	15.81	14.83	11.07
category mean (4)	6.43	8.21	7.59	10.11	11.03	7.18	12.30	11.69	12.00	7.01
EN short-form & Multilingual mean	5.99	8.79	9.43	8.32	7.40	9.68	12.34	15.31	9.72	8.02

Multilingual LibriSpeech (MLS-8)

Test set	Parakeet-TDT-0.6B-v3 (teacher)	Whisper-medium	omniASR-LLM 300M v2	OWSM-CTC v4	Qwen3-ASR-0.6B	omniASR-CTC 1B v2	Whisper-small	omniASR-CTC 300M v2	OWSM v4 small	CAFE-25lang (this model)
en	4.82	6.21	8.68	6.75	5.77	9.20	7.66	15.10	7.96	6.48
de	5.79	5.75	7.02	10.36	8.18	6.69	9.35	11.02	10.43	6.59
nl	11.24	11.42	6.92	11.22	16.69	7.93	16.54	12.89	11.64	5.89
pl	5.74	5.89	9.32	16.39	26.02	9.18	9.26	9.39	16.72	5.63
es	4.39	4.76	4.80	6.44	6.80	4.20	6.90	7.25	7.61	4.74
fr	5.07	7.18	6.82	8.31	8.54	6.94	10.94	12.95	11.05	4.87
it	8.53	10.60	7.25	14.14	14.76	6.51	16.06	10.73	14.49	7.37
pt	7.69	10.28	11.50	11.55	14.03	11.05	15.37	15.81	14.83	11.04
mean (8)	6.66	7.76	7.79	10.65	12.60	7.71	11.51	11.89	11.84	6.58

Common Voice v25 (24)

Test set	Parakeet-TDT-0.6B-v3 (teacher)	Whisper-medium	omniASR-LLM 300M v2	OWSM-CTC v4	Qwen3-ASR-0.6B	omniASR-CTC 1B v2	Whisper-small	omniASR-CTC 300M v2	OWSM v4 small	CAFE-25lang (this model)
bg	18.21	23.76	12.87	46.54	N/A	16.64	41.00	30.08	52.54	18.20
cs	5.92	18.30	9.61	23.05	42.79	10.53	33.55	20.33	27.39	12.47
da	40.38	44.38	40.56	66.55	55.37	42.77	52.21	52.46	74.82	43.70
de	4.40	7.83	10.58	8.83	8.30	10.02	13.97	19.29	10.71	8.83
el	18.23	22.76	13.51	62.36	52.11	25.40	29.88	37.23	65.56	32.80
en	8.47	12.38	18.94	11.45	9.67	19.78	17.18	31.20	14.22	13.94
es	3.76	7.15	7.53	7.20	6.58	7.83	11.63	14.23	9.34	7.22
et	22.29	44.33	22.62	53.75	N/A	20.79	69.50	39.42	52.14	27.37
fi	48.76	53.14	51.97	64.14	72.67	52.67	58.76	58.49	67.07	54.29
fr	7.33	18.85	14.28	12.31	11.75	15.18	26.92	24.82	16.22	11.64
hu	6.55	18.90	11.02	35.12	55.36	13.44	36.10	26.10	36.78	11.79
it	3.59	8.62	8.97	8.18	8.70	8.78	16.81	16.28	10.24	7.15
lt	16.60	50.51	22.56	60.22	N/A	26.23	73.12	45.25	65.21	25.68
lv	36.08	33.94	18.87	55.13	N/A	21.05	55.98	36.61	58.51	17.79
mt	12.16	93.16	11.79	62.61	N/A	17.77	109.40	31.01	73.66	46.73
nl	4.21	6.53	6.98	8.07	10.11	7.70	12.66	16.74	12.11	6.04
pl	8.53	15.31	15.81	20.25	30.20	15.53	20.74	25.53	24.85	15.91
pt	8.04	9.85	12.16	8.91	10.92	12.87	22.75	23.69	13.01	10.94
ro	10.79	22.13	13.35	20.10	42.23	12.97	31.82	23.80	34.09	22.46
ru	5.54	9.44	12.93	9.06	12.96	14.86	16.42	32.45	14.68	14.62
sk	8.36	40.31	15.70	33.79	N/A	25.43	52.97	34.31	41.75	21.68
sl	49.60	33.42	17.71	34.86	N/A	25.05	45.02	48.08	42.90	26.51
sv-SE	17.59	12.13	17.90	35.41	37.47	17.22	21.15	33.36	41.65	17.99
uk	30.09	39.44	37.12	41.03	N/A	41.06	46.62	52.84	46.21	41.70
mean all (24)	16.48	26.94	17.72	32.87	N/A	20.07	38.17	32.23	37.74	21.56

Evaluation

eval/README.md shows how to reproduce the Open ASR Leaderboard and Multilingual LibriSpeech numbers with the official leaderboard runners and normalizer. Decoding is greedy, with the CUDA-graph decoder disabled.