README.md

metadata

title: FonBench
emoji: 🗣️
colorFrom: indigo
colorTo: yellow
sdk: gradio
sdk_version: 6.24.0
app_file: app.py
pinned: true
license: apache-2.0
short_description: The public speech-recognition leaderboard for Fon

FonBench 🇧🇯

The public speech-recognition leaderboard for Fon, a tonal language spoken by over two million people in Benin.

Anyone can submit a model from the Hub: it is evaluated automatically on this Space's shared GPU (ZeroGPU) and added to the leaderboard. No account required.

Evaluation runs in slices, each fitting inside one GPU allocation. When the daily quota runs out the run pauses and resumes on its own at the next slice — and a Space restart loses nothing either.

Metrics

Fon is written with tones (á, ɔ́, ě…) that change the meaning of words, but corpora don't follow the same convention — some mark no tone at all. A raw WER is therefore not comparable from one corpus to the next. Hence:

• WER_seg — word errors with tones stripped. Phonetic accuracy, comparable everywhere. This is the default ranking.
• WER_ton — errors on tone marks alone. Not computed when the corpus doesn't annotate tones, so the figure is never misleading.
• T-WER = WER_seg + 2 × WER_ton — the headline metric, penalising tone mistakes twice over.
• RTFx — seconds of audio processed per second of compute. Higher is faster. Hardware-dependent: only compare at equal hardware.

Scoring is open and reproducible: see fonbench_eval.py.

Model types

• 🌍 Base — generic or multilingual systems evaluated as-is. A high error rate here measures an absence, not a weakness: they have never seen Fon.
• 🔧 FonBench — fine-tuned by this project on 30 hours of the JML corpus, under a documented protocol (see the Fine-tuning tab).
• 👥 Community — fine-tuned on Fon by third parties.

Test set

The main test set (2,555 utterances, 4.98 hours, 45 speakers) is not published. Public Fon corpora have been circulating since 2016 and several models were trained on them, which inflates their scores artificially. Its 45 speakers are strictly disjoint from the 471 training speakers — verified, zero in common. Only aggregate scores are made public — neither the audio nor the transcriptions are reachable from this Space.

Protocol

• Pinned model revision: every score is tied to the repository's commit hash.
• Identical text normalisation for every model.
• trust_remote_code=False — no arbitrary code is ever executed.
• A (model, revision, corpus) triple is never re-evaluated.

Submitting a model

The repository must be public on the Hub. Recognised architectures: wav2vec2, wav2vec2-BERT, HuBERT, MMS, Whisper.