| ---
|
| title: FonBench
|
| emoji: π£οΈ
|
| colorFrom: indigo
|
| colorTo: yellow
|
| sdk: gradio
|
| sdk_version: 6.24.0
|
| app_file: app.py
|
| pinned: true
|
| license: apache-2.0
|
| short_description: The public speech-recognition leaderboard for Fon
|
| ---
|
|
|
| # FonBench π§π―
|
|
|
| The public speech-recognition leaderboard for **Fon**, a tonal language
|
| spoken by over two million people in Benin.
|
|
|
| Anyone can submit a model from the Hub: it is evaluated automatically on
|
| this Space's shared GPU (ZeroGPU) and added to the leaderboard. No account
|
| required.
|
|
|
| Evaluation runs in slices, each fitting inside one GPU allocation. When the
|
| daily quota runs out the run pauses and resumes on its own at the next
|
| slice β and a Space restart loses nothing either.
|
|
|
| ## Metrics
|
|
|
| Fon is written with tones (Γ‘, ΙΜ, Δβ¦) that change the meaning of words, but
|
| corpora don't follow the same convention β some mark no tone at all. A raw
|
| WER is therefore not comparable from one corpus to the next. Hence:
|
|
|
| - **WER_seg** β word errors with tones stripped. Phonetic accuracy,
|
| comparable everywhere. This is the default ranking.
|
| - **WER_ton** β errors on tone marks alone. Not computed when the corpus
|
| doesn't annotate tones, so the figure is never misleading.
|
| - **T-WER** = `WER_seg + 2 Γ WER_ton` β the headline metric, penalising
|
| tone mistakes twice over.
|
| - **RTFx** β seconds of audio processed per second of compute. Higher is
|
| faster. Hardware-dependent: only compare at equal hardware.
|
|
|
| Scoring is open and reproducible: see `fonbench_eval.py`.
|
|
|
| ## Model types
|
|
|
| - π **Base** β generic or multilingual systems evaluated as-is. A high
|
| error rate here measures an absence, not a weakness: they have never
|
| seen Fon.
|
| - π§ **FonBench** β fine-tuned by this project on 30 hours of the JML
|
| corpus, under a documented protocol (see the *Fine-tuning* tab).
|
| - π₯ **Community** β fine-tuned on Fon by third parties.
|
|
|
| ## Test set
|
|
|
| The main test set (2,555 utterances, 4.98 hours, 45 speakers) is **not
|
| published**. Public Fon corpora have been circulating since 2016 and
|
| several models were trained on them, which inflates their scores
|
| artificially. Its 45 speakers are strictly disjoint from the 471 training
|
| speakers β verified, zero in common. Only aggregate scores are made public β neither the audio nor the transcriptions
|
| are reachable from this Space.
|
|
|
| ## Protocol
|
|
|
| - Pinned model revision: every score is tied to the repository's commit hash.
|
| - Identical text normalisation for every model.
|
| - `trust_remote_code=False` β no arbitrary code is ever executed.
|
| - A (model, revision, corpus) triple is never re-evaluated.
|
|
|
| ## Submitting a model
|
|
|
| The repository must be **public** on the Hub. Recognised architectures:
|
| wav2vec2, wav2vec2-BERT, HuBERT, MMS, Whisper.
|
|
|
