README.md

---

title: FonBench
emoji: 🗣️
colorFrom: indigo
colorTo: yellow
sdk: gradio
sdk_version: 6.24.0
app_file: app.py
pinned: true
license: apache-2.0
short_description: The public speech-recognition leaderboard for Fon
---


# FonBench 🇧🇯

The public speech-recognition leaderboard for **Fon**, a tonal language
spoken by over two million people in Benin.

Anyone can submit a model from the Hub: it is evaluated automatically on
this Space's shared GPU (ZeroGPU) and added to the leaderboard. No account
required.

Evaluation runs in slices, each fitting inside one GPU allocation. When the
daily quota runs out the run pauses and resumes on its own at the next
slice — and a Space restart loses nothing either.

## Metrics

Fon is written with tones (á, ɔ́, ě…) that change the meaning of words, but
corpora don't follow the same convention — some mark no tone at all. A raw
WER is therefore not comparable from one corpus to the next. Hence:

- **WER_seg** — word errors with tones stripped. Phonetic accuracy,

  comparable everywhere. This is the default ranking.

- **WER_ton** — errors on tone marks alone. Not computed when the corpus
  doesn't annotate tones, so the figure is never misleading.
- **T-WER** = `WER_seg + 2 × WER_ton` — the headline metric, penalising
  tone mistakes twice over.
- **RTFx** — seconds of audio processed per second of compute. Higher is
  faster. Hardware-dependent: only compare at equal hardware.

Scoring is open and reproducible: see `fonbench_eval.py`.

## Model types

- 🌍 **Base** — generic or multilingual systems evaluated as-is. A high
  error rate here measures an absence, not a weakness: they have never
  seen Fon.
- 🔧 **FonBench** — fine-tuned by this project on 30 hours of the JML
  corpus, under a documented protocol (see the *Fine-tuning* tab).
- 👥 **Community** — fine-tuned on Fon by third parties.

## Test set

The main test set (2,555 utterances, 4.98 hours, 45 speakers) is **not

published**. Public Fon corpora have been circulating since 2016 and
several models were trained on them, which inflates their scores
artificially. Its 45 speakers are strictly disjoint from the 471 training
speakers — verified, zero in common. Only aggregate scores are made public — neither the audio nor the transcriptions
are reachable from this Space.

## Protocol

- Pinned model revision: every score is tied to the repository's commit hash.
- Identical text normalisation for every model.
- `trust_remote_code=False` — no arbitrary code is ever executed.
- A (model, revision, corpus) triple is never re-evaluated.

## Submitting a model

The repository must be **public** on the Hub. Recognised architectures:
wav2vec2, wav2vec2-BERT, HuBERT, MMS, Whisper.