Benin flag, plus four factual corrections after a full audit

README.md

@@ -1,72 +1,72 @@
----
-title: FonBench
-emoji: 🗣️
-colorFrom: indigo
-colorTo: yellow
-sdk: gradio
-sdk_version: 6.24.0
-app_file: app.py
-pinned: true
-license: apache-2.0
-short_description: The public speech-recognition leaderboard for Fon
----
-
-# FonBench
-
-The public speech-recognition leaderboard for **Fon**, a tonal language
-spoken by roughly 2.3 million people in Benin.
-
-Anyone can submit a model from the Hub: it is evaluated automatically on
-this Space's shared GPU (ZeroGPU) and added to the leaderboard. No account
-required.
-
-Evaluation runs in slices, each fitting inside one GPU allocation. When the
-daily quota runs out the run pauses and resumes on its own at the next
-slice — and a Space restart loses nothing either.
-
-## Metrics
-
-Fon is written with tones (á, ɔ́, ě…) that change the meaning of words, but
-corpora don't follow the same convention — some mark no tone at all. A raw
-WER is therefore not comparable from one corpus to the next. Hence:
-
-- **WER_seg** — word errors with tones stripped. Phonetic accuracy,
-  comparable everywhere. This is the default ranking.
-- **WER_ton** — errors on tone marks alone. Not computed when the corpus
-  doesn't annotate tones, so the figure is never misleading.
-- **T-WER** = `WER_seg + 2 × WER_ton` — the headline metric, penalising
-  tone mistakes twice over.
-- **RTFx** — seconds of audio processed per second of compute. Higher is
-  faster. Hardware-dependent: only compare at equal hardware.
-
-Scoring is open and reproducible: see `fonbench_eval.py`.
-
-## Model types
-
-- 🌍 **Base** — generic or multilingual systems evaluated as-is. A high
-  error rate here measures an absence, not a weakness: they have never
-  seen Fon.
-- 🔧 **FonBench** — fine-tuned by this project on 30 hours of the JML
-  corpus, under a documented protocol (see the *Fine-tuning* tab).
-- 👥 **Community** — fine-tuned on Fon by third parties.
-
-## Test set
-
-The main test set (2,555 utterances, 4.98 hours, 45 speakers) is **not
-published**. Public Fon corpora have been circulating since 2016 and
-several models were trained on them, which inflates their scores
-artificially. Its speakers are disjoint from training speakers. Only
-aggregate scores are made public — neither the audio nor the transcriptions
-are reachable from this Space.
-
-## Protocol
-
-- Pinned model revision: every score is tied to the repository's commit hash.
-- Identical text normalisation for every model.
-- `trust_remote_code=False` — no arbitrary code is ever executed.
-- A (model, revision, corpus) triple is never re-evaluated.
-
-## Submitting a model
-
-The repository must be **public** on the Hub. Recognised architectures:
-wav2vec2, wav2vec2-BERT, HuBERT, MMS, Whisper.
+---
+title: FonBench
+emoji: 🗣️
+colorFrom: indigo
+colorTo: yellow
+sdk: gradio
+sdk_version: 6.24.0
+app_file: app.py
+pinned: true
+license: apache-2.0
+short_description: The public speech-recognition leaderboard for Fon
+---
+
+# FonBench 🇧🇯
+
+The public speech-recognition leaderboard for **Fon**, a tonal language
+spoken by over two million people in Benin.
+
+Anyone can submit a model from the Hub: it is evaluated automatically on
+this Space's shared GPU (ZeroGPU) and added to the leaderboard. No account
+required.
+
+Evaluation runs in slices, each fitting inside one GPU allocation. When the
+daily quota runs out the run pauses and resumes on its own at the next
+slice — and a Space restart loses nothing either.
+
+## Metrics
+
+Fon is written with tones (á, ɔ́, ě…) that change the meaning of words, but
+corpora don't follow the same convention — some mark no tone at all. A raw
+WER is therefore not comparable from one corpus to the next. Hence:
+
+- **WER_seg** — word errors with tones stripped. Phonetic accuracy,
+  comparable everywhere. This is the default ranking.
+- **WER_ton** — errors on tone marks alone. Not computed when the corpus
+  doesn't annotate tones, so the figure is never misleading.
+- **T-WER** = `WER_seg + 2 × WER_ton` — the headline metric, penalising
+  tone mistakes twice over.
+- **RTFx** — seconds of audio processed per second of compute. Higher is
+  faster. Hardware-dependent: only compare at equal hardware.
+
+Scoring is open and reproducible: see `fonbench_eval.py`.
+
+## Model types
+
+- 🌍 **Base** — generic or multilingual systems evaluated as-is. A high
+  error rate here measures an absence, not a weakness: they have never
+  seen Fon.
+- 🔧 **FonBench** — fine-tuned by this project on 30 hours of the JML
+  corpus, under a documented protocol (see the *Fine-tuning* tab).
+- 👥 **Community** — fine-tuned on Fon by third parties.
+
+## Test set
+
+The main test set (2,555 utterances, 4.98 hours, 45 speakers) is **not
+published**. Public Fon corpora have been circulating since 2016 and
+several models were trained on them, which inflates their scores
+artificially. Its 45 speakers are strictly disjoint from the 471 training
+speakers — verified, zero in common. Only aggregate scores are made public — neither the audio nor the transcriptions
+are reachable from this Space.
+
+## Protocol
+
+- Pinned model revision: every score is tied to the repository's commit hash.
+- Identical text normalisation for every model.
+- `trust_remote_code=False` — no arbitrary code is ever executed.
+- A (model, revision, corpus) triple is never re-evaluated.
+
+## Submitting a model
+
+The repository must be **public** on the Hub. Recognised architectures:
+wav2vec2, wav2vec2-BERT, HuBERT, MMS, Whisper.

app.py

@@ -42,6 +42,7 @@ CSS = """
 .fb-title { font-size:2.4rem; font-weight:800; letter-spacing:-.03em;
             color:var(--fb-indigo); line-height:1.05; margin:0; }
 .fb-title span { color:var(--fb-ocre); }
+.fb-title .fb-flag { font-size:1.7rem; vertical-align:.18em; }
 .fb-sub { color:var(--fb-muted); margin:.35rem 0 0; font-size:1.02rem; }
 
 .fb-note { font-size:.9rem; line-height:1.6; color:var(--fb-ink); }
@@ -232,7 +233,7 @@ def build_table(benchmark_id: str, vue: str, archs: list[str],
               for c in ("maison", "tiers", "base")}
     meta = (
         "<div class='fb-meta'>"
-        f"<span><b>{bench.get('name', benchmark_id)}</b></span>"
+        f"<span>🇧🇯 <b>{bench.get('name', benchmark_id)}</b></span>"
         f"<span><b>{bench.get('num_utterances', '?')}</b> utterances</span>"
         f"<span><b>{bench.get('duration_hours', '?')}</b> hours</span>"
         + ("<span class='fb-badge'>private test set</span>"
@@ -390,9 +391,10 @@ it is built by taking turns across speakers, each contributing a little
 before any one of them dominates. For generalisation, the diversity of
 voices matters more than raw volume.
 
-Test-set speakers are **disjoint** from training speakers, and a temporal
-cutoff separates the transcripts. A model therefore cannot score well by
-having memorised a voice or a sentence.
+The 471 training speakers and the 45 test speakers are **strictly
+disjoint** — verified, zero in common. A temporal cutoff separates the
+transcripts; 20 sentences out of 2,555 (0.8%) do appear in both, too few to
+move a score but worth stating rather than claiming a perfect separation.
 
 ### The protocol
 
@@ -404,7 +406,7 @@ gradient checkpointing.
 | Resulting model | Base | Learning rate | Batch | Wall time | Final dev loss | WER_seg |
 |---|---|---|---|---|---|---|
 | `wav2vec2-large-xlsr-53-fon-30h` | facebook/wav2vec2-large-xlsr-53 | 3·10⁻⁴ | 8×2 | 54 min | **0.468** | **38.6%** |
-| `mms-300m-fon-30h` | facebook/mms-300m | 3·10⁻⁴ | 8×2 | ~50 min | not retained | **42.8%** |
+| `mms-300m-fon-30h` | facebook/mms-300m | 3·10⁻⁴ | 8×2 | not retained | not retained | **42.8%** |
 | `w2v-bert-2.0-fon-30h` | facebook/w2v-bert-2.0 | 3·10⁻⁵ | 4×4 | 116 min | 3.102 | 96.9% |
 | `AfriHuBERT-fon-30h` | ajesujoba/AfriHuBERT | 3·10⁻⁵ | 8×2 | 24 min | 3.078 | 100.0% |
 
@@ -420,9 +422,13 @@ with 30 hours of well-chosen data.
 `w2v-bert-2.0` and `AfriHuBERT` do learn: their loss drops clearly during
 training. But it plateaus around 3.1 against 0.47 for the run that
 succeeds — the signature of settling into CTC's trivial solution, emitting
-the blank symbol everywhere. This is not a learning-rate problem: retrying
-at 3·10⁻⁵ produced exactly the same outcome. They would need far more
-epochs, or more data.
+the blank symbol everywhere. This is not a learning-rate problem: both were
+first run at 3·10⁻⁴, then rerun at 3·10⁻⁵ — the figures in the table — for
+exactly the same outcome. They would need far more epochs, or more data.
+
+The `mms-300m` run predates per-model logging, so its wall time and final
+loss are genuinely lost. They are shown as *not retained* rather than
+reconstructed from memory.
 
 Both failures stay in the leaderboard. Removing them would paint a
 flattering and false picture of what fine-tuning guarantees.
@@ -438,7 +444,7 @@ A_PROPOS = """
 ## FonBench
 
 The first public speech-recognition leaderboard for **Fon**, a tonal
-language spoken by roughly 2.3 million people in Benin. The project answers
+language spoken by over two million people in Benin. The project answers
 a concrete difficulty: until now, published Fon results were not comparable
 with one another, for lack of a shared protocol and a shared test set.
 
@@ -491,7 +497,8 @@ def build_ui() -> gr.Blocks:
     with gr.Blocks(title="FonBench — Fon ASR leaderboard") as demo:
         gr.HTML(
             "<div class='fb-head'>"
-            "<p class='fb-title'>Fon<span>Bench</span></p>"
+            "<p class='fb-title'>Fon<span>Bench</span> "
+            "<span class='fb-flag'>🇧🇯</span></p>"
             "<p class='fb-sub'>The public speech-recognition leaderboard for "
             "Fon, a tonal language of Benin</p>"
             "</div>"