Evaluation — CAFE-25lang

Scripts to reproduce the Open ASR Leaderboard and Multilingual LibriSpeech WER in the model card. Decoding and scoring are the Open ASR Leaderboard's own: greedy batched decoding with the CUDA-graph decoder disabled, the official English normalizer for English sets and ml_normalizer plus compound-word pairing for the multilingual ones. No training code or data is included.

The one thing to get right

This model routes by language. English uses the bare trunk; every other language needs adapters.safetensors attached. Evaluating a non-English set without the adapters silently measures the English path instead. adapter_wrapper.py does the attaching; run_leaderboard.py wires it into the official runners.

Files

File	Purpose
`eval_mls8.py`	Multilingual LibriSpeech, one language per run (the card's 8-language table)
`run_leaderboard.py`	runs an official Open ASR Leaderboard runner with the adapters attached
`adapter_wrapper.py`	attaches the adapters to a restored model via forward hooks

Setup

pip install "nemo_toolkit[asr]" datasets evaluate jiwer soundfile safetensors
git clone https://github.com/huggingface/open_asr_leaderboard
export PYTHONPATH=$PWD/open_asr_leaderboard:$PYTHONPATH   # provides the `normalizer` package

Multilingual LibriSpeech (8 languages)

English on the trunk, the other seven with adapters:

python eval_mls8.py --model_id ../CAFE-25lang.nemo --lang en \
    --result_json results/mls_en.json

for L in de nl pl es fr it pt; do
  python eval_mls8.py --model_id ../CAFE-25lang.nemo --lang $L \
      --adapters ../adapters.safetensors --result_json results/mls_$L.json
done

The card reports the unweighted mean of the eight per-language WERs.

Open ASR Leaderboard sets

cd open_asr_leaderboard/nemo_asr
EVAL=/path/to/CAFE_ASR/eval
MODEL=/path/to/CAFE_ASR/CAFE-25lang.nemo
ADAPTERS=/path/to/CAFE_ASR/adapters.safetensors

# English short-form, e.g. LibriSpeech test-clean — bare trunk
python $EVAL/run_leaderboard.py run_eval.py NONE -- \
    --model_id $MODEL --model_name CAFE-25lang \
    --dataset_path hf-audio/esb-datasets-test-only-sorted \
    --dataset librispeech --split test.clean \
    --batch_size 128 --device 0 --max_eval_samples -1

# a multilingual set, e.g. FLEURS German — adapters attached
python $EVAL/run_leaderboard.py run_eval_ml.py $ADAPTERS -- \
    --model_id $MODEL --model_name CAFE-25lang \
    --dataset nithinraok/asr-leaderboard-datasets --config_name fleurs_de --language de \
    --split test --batch_size 64 --device 0 --max_eval_samples -1

Each run prints WER: <x> % and writes a per-utterance manifest under results/ in the runner's working directory, which the leaderboard's normalizer/eval_utils.score_results can re-score.

Notes

A GPU with ~8 GB is enough at these batch sizes; drop --batch_size if memory is tight.
Audio is cached as wav under audio_cache/ on first use, so re-runs are much faster.
The numbers above assume the language of each set is known. For routing with the language gate instead (inference_gated.py in the repo root), see the model card and the language-gate appendix of the paper.