# Evaluation — CAFE-25lang

Scripts to reproduce the Open ASR Leaderboard and Multilingual LibriSpeech WER in the model card. Decoding and scoring are the
Open ASR Leaderboard's own: greedy batched decoding with the CUDA-graph decoder disabled, the
official English normalizer for English sets and `ml_normalizer` plus compound-word pairing for the
multilingual ones. No training code or data is included.

## The one thing to get right

This model routes by language. **English uses the bare trunk; every other language needs
`adapters.safetensors` attached.** Evaluating a non-English set without the adapters silently
measures the English path instead. `adapter_wrapper.py` does the attaching; `run_leaderboard.py`
wires it into the official runners.

## Files

| File | Purpose |
| --- | --- |
| `eval_mls8.py` | Multilingual LibriSpeech, one language per run (the card's 8-language table) |
| `run_leaderboard.py` | runs an official Open ASR Leaderboard runner with the adapters attached |
| `adapter_wrapper.py` | attaches the adapters to a restored model via forward hooks |

## Setup

```bash
pip install "nemo_toolkit[asr]" datasets evaluate jiwer soundfile safetensors
git clone https://github.com/huggingface/open_asr_leaderboard
export PYTHONPATH=$PWD/open_asr_leaderboard:$PYTHONPATH   # provides the `normalizer` package
```

## Multilingual LibriSpeech (8 languages)

English on the trunk, the other seven with adapters:

```bash
python eval_mls8.py --model_id ../CAFE-25lang.nemo --lang en \
    --result_json results/mls_en.json

for L in de nl pl es fr it pt; do
  python eval_mls8.py --model_id ../CAFE-25lang.nemo --lang $L \
      --adapters ../adapters.safetensors --result_json results/mls_$L.json
done
```

The card reports the unweighted mean of the eight per-language WERs.

## Open ASR Leaderboard sets

```bash
cd open_asr_leaderboard/nemo_asr
EVAL=/path/to/CAFE_ASR/eval
MODEL=/path/to/CAFE_ASR/CAFE-25lang.nemo
ADAPTERS=/path/to/CAFE_ASR/adapters.safetensors

# English short-form, e.g. LibriSpeech test-clean — bare trunk
python $EVAL/run_leaderboard.py run_eval.py NONE -- \
    --model_id $MODEL --model_name CAFE-25lang \
    --dataset_path hf-audio/esb-datasets-test-only-sorted \
    --dataset librispeech --split test.clean \
    --batch_size 128 --device 0 --max_eval_samples -1

# a multilingual set, e.g. FLEURS German — adapters attached
python $EVAL/run_leaderboard.py run_eval_ml.py $ADAPTERS -- \
    --model_id $MODEL --model_name CAFE-25lang \
    --dataset nithinraok/asr-leaderboard-datasets --config_name fleurs_de --language de \
    --split test --batch_size 64 --device 0 --max_eval_samples -1
```

Each run prints `WER: <x> %` and writes a per-utterance manifest under `results/` in the runner's
working directory, which the leaderboard's `normalizer/eval_utils.score_results` can re-score.

## Notes

- A GPU with ~8 GB is enough at these batch sizes; drop `--batch_size` if memory is tight.
- Audio is cached as wav under `audio_cache/` on first use, so re-runs are much faster.
- The numbers above assume the language of each set is known. For routing with the language gate
  instead (`inference_gated.py` in the repo root), see the model card and the
  language-gate appendix of the paper.