organisation/name, exactly as it appears in " +
"the model URL on Hugging Face.${esc(modele)} is queuedFon is a tonal language: tones are written with diacritics (á, ɔ́, ě…) and change the meaning of words. But Fon corpora don't follow the same convention — some mark no tone at all. A raw WER therefore isn't comparable from one corpus to the next. Hence three families of measures.
| Metric | What it tells you |
|---|---|
| WER_seg | Word errors with tones stripped. Measures phonetic accuracy and stays comparable across every corpus. |
| WER_ton | Errors on tone marks alone. Shown as “—” when the corpus doesn't annotate tones, so the figure is never misleading. |
| T-WER | WER_seg + 2 × WER_ton. The headline
metric: it penalises tone mistakes twice over. |
| WER, CER | Word and character errors, tones included. |
| MER, WIL | Match error rate and word information lost — more robust when a model produces many insertions. |
| RTFx | Seconds of audio per second of compute. Higher is faster. |
Model types. 🌍 General-purpose models were not built for Fon:
some are multilingual systems that do cover it among a thousand other languages
(the MMS family ships a Fon adapter), others are multilingual without Fon, and
others again are monolingual systems for English or French. They are not
base models in the pretrained sense — wav2vec2-large-960h-lv60-self
is a finished English recogniser, not a starting point. The Trained on
column says exactly what each one saw. 🔧 FonBench and 👥 Community
models were fine-tuned on Fon.
Truly base models — raw pretrained checkpoints such as
facebook/wav2vec2-large-xlsr-53 — cannot appear here at all: without
a CTC head or a vocabulary they transcribe nothing. They show up only in the
Base model column, as the starting point of the models fine-tuned from
them.
Speed. RTFx depends on the hardware, recorded with each score. Only compare speeds at equal hardware.
`; const FINE_TUNING = `Four pretrained models were fine-tuned on Fon under strictly identical conditions, so that any gap between them comes from the starting model alone.
The data. The training corpus holds 44,225 utterances, of which 13,716 were kept — 30.00 hours from 471 speakers. The sample is not drawn at random: it is built by taking turns across speakers, each contributing a little before any one of them dominates. For generalisation, the diversity of voices matters more than raw volume.
The 471 training speakers and the 45 test speakers are strictly disjoint — verified, zero in common. A temporal cutoff separates the transcripts; 20 sentences out of 2,555 (0.8%) do appear in both, too few to move a score but worth stating rather than claiming a perfect separation.
The protocol. Identical across all four: 3 epochs, i.e. 2,574 steps
with 257 of warmup. CTC decoding, character vocabulary built from the training
corpus, feature extractor taken from the base model. A single 24 GB L4,
fp16, gradient checkpointing.
| Resulting model | Base | LR | Batch | Wall time | Dev loss | WER_seg |
|---|---|---|---|---|---|---|
wav2vec2-large-xlsr-53-fon-30h | wav2vec2-large-xlsr-53 | 3·10⁻⁴ | 8×2 | 54 min | 0.468 | 38.6% |
mms-300m-fon-30h | mms-300m | 3·10⁻⁴ | 8×2 | not retained | not retained | 42.8% |
w2v-bert-2.0-fon-30h | w2v-bert-2.0 | 3·10⁻⁵ | 4×4 | 116 min | 3.102 | 96.9% |
AfriHuBERT-fon-30h | ajesujoba/AfriHuBERT | 3·10⁻⁵ | 8×2 | 24 min | 3.078 | 100.0% |
Fine-tuning makes the difference, not the base.
chrisjay/fonxlsr starts from exactly the same model as our best run
— facebook/wav2vec2-large-xlsr-53, confirmed in its configuration —
and reaches 69.3% where we reach 38.6%. The error is cut by a factor of 1.8 with
30 hours of well-chosen data.
Two runs out of four failed, and that deserves saying plainly.
w2v-bert-2.0 and AfriHuBERT do learn: their loss drops
clearly during training. But it plateaus around 3.1 against 0.47 for the run that
succeeds — the signature of settling into the trivial CTC solution, emitting the
blank symbol everywhere. This is not a learning-rate problem: both were first run
at 3·10⁻⁴, then rerun at 3·10⁻⁵ — the figures in the table — for exactly the same
outcome. The same collapse was later reproduced independently by three
afrihubert-fon-asr-vanilla runs from another team, all between 97.8%
and 99.7%; the same encoder preceded by continued pretraining on 960 h of Fon
reaches 18.3%. Four failures and one success point at the same cause.
Both failures stay in the leaderboard. Removing them would paint a flattering and false picture of what fine-tuning guarantees.
Reproducing. The training and evaluation code is published at github.com/Izzoudine/EvalScripts — one script per evaluated model.
`; const A_PROPOS = `The first public speech-recognition leaderboard for Fon, a tonal language spoken by over two million people in Benin. The project answers a concrete difficulty: until now, published Fon results were not comparable with one another, for lack of a shared protocol and a shared test set.
Public Fon corpora have been circulating since 2016, and several released models were trained on them. Their WER on that data then looks remarkable — while saying nothing about their real ability to transcribe an unseen voice. We measured it: one of the test sets we were using shared 83% of its utterances with the training split of a public dataset.
The main test set (2,555 utterances, 4.98 hours, 45 speakers) is
therefore not distributed. Only aggregate scores are made public. It is not
secret, though: request access to JMLdata/fon-test-v1 and you can
recompute any row yourself.
fonbench_eval.py.trust_remote_code=False.This page is a static Space: it renders the leaderboard and accepts submissions, but cannot execute Python. Evaluation runs on Kimyayd/FonBench, which holds the GPU and the read access to the private test set. Both read and write the same database, so what you see here is live, not a copy.
Scoring code, a standalone evaluator and one script per evaluated model: github.com/Izzoudine/EvalScripts. Expect agreement within ±0.0002 — CTC padding depends on batch composition, and we would rather document that than round the published figures to three decimals.
`; /* --- démarrage --------------------------------------------------------- */ function onglets() { const nav = document.getElementById("tabs"); nav.innerHTML = ONGLETS.map(([id, titre], i) => ``).join(""); nav.querySelectorAll("button").forEach((b) => { b.onclick = () => { nav.querySelectorAll("button").forEach((x) => x.setAttribute("aria-selected", x === b)); ONGLETS.forEach(([id]) => document.getElementById(`tab-${id}`).classList.toggle("hidden", id !== b.dataset.tab)); if (b.dataset.tab === "queue") chargerFile(); }; }); } async function demarrer() { onglets(); document.getElementById("metrics-note").innerHTML = NOTE_METRIQUES; document.getElementById("ft").innerHTML = FINE_TUNING; document.getElementById("about").innerHTML = A_PROPOS; document.getElementById("views").innerHTML = VUES.map(([t], i) => ``).join(""); document.querySelectorAll("#views button").forEach((b) => { b.onclick = () => { etat.vue = +b.dataset.vue; document.querySelectorAll("#views button").forEach((x) => x.setAttribute("aria-pressed", x === b)); dessinerClassement(); }; }); try { etat.benchmarks = await lire("benchmarks?is_active=eq.true&select=*&order=id"); } catch (e) { document.getElementById("meta").textContent = `Database unreachable: ${e.message}`; return; } const options = etat.benchmarks.map((b) => ``).join(""); document.getElementById("bench").innerHTML = options; document.getElementById("f-bench").innerHTML = options; ["bench", "sort", "arch", "dec"].forEach((id) => { document.getElementById(id).onchange = () => { etat.tri = null; if (id === "bench") chargerClassement(); else dessinerClassement(); }; }); document.getElementById("contam").onchange = dessinerClassement; document.getElementById("refresh").onclick = chargerClassement; document.getElementById("refresh-queue").onclick = chargerFile; document.getElementById("submit-form").onsubmit = soumettre; chargerClassement(); } demarrer();