Public model card: CC-BY-4.0, attribution, 500-sentence single-shot vs gated benchmark

README.md

@@ -1,31 +1,47 @@
 ---
 language: sw
+license: cc-by-4.0
+base_model: nari-labs/Dia-1.6B
 tags:
 - text-to-speech
 - tts
 - swahili
+- kiswahili
 - dia
 - code-switching
+- african-languages
 library_name: dia
 pipeline_tag: text-to-speech
 ---
 
 # Sauti TTS — Dia-1.6B Swahili (full SFT)
 
-Swahili text-to-speech built by **Msingi-AI**, a full supervised fine-tune of
-[nari-labs/Dia-1.6B](https://huggingface.co/nari-labs/Dia-1.6B) on a pooled
-Swahili speech corpus. Unlike our VoxCPM2 model this is **not** a voice-cloning
-model — it has a single learned voice and takes text only.
+Open Swahili text-to-speech from **Msingi-AI**: a full supervised fine-tune of
+[nari-labs/Dia-1.6B](https://huggingface.co/nari-labs/Dia-1.6B) on a 500-hour
+Swahili speech corpus pooled from 15 openly-licensed public datasets.
 
-Pick this model when **code-switching matters** (English embedded in Swahili):
-it scores code-switch CER **0.050**, comfortably ahead of our VoxCPM2 full SFT
-(0.047 zero-shot judge / 0.090 Swahili judge) on the mixed-language path, and a
-native Swahili listener preferred it there. For plain Swahili, VoxCPM2 full SFT
-is still the stronger model (0.007 vs 0.011 CER) — see *Model selection* below.
+This is a **single-voice** model — it takes text and speaks it in one learned
+voice. It does not clone voices and needs no reference audio.
 
-> **Internal / private.** Weights derive from Dia-1.6B and training used a
-> corpus with mixed licences (see *Licensing*). Confirm licence compatibility
-> before any external release.
+**It handles code-switching**, English embedded in Swahili sentences, which is
+how a great deal of Swahili is actually spoken and where TTS models commonly
+break down: code-switch CER **0.050** against a multilingual ASR judge.
+
+| | Plain Swahili | Code-switched |
+|---|---:|---:|
+| CER | **0.011** | **0.050** |
+| WER | 0.054 | 0.286 |
+
+Plain Swahili is scored by a Swahili-tuned ASR judge, code-switch by a
+multilingual one — [for a reason](#read-code-switch-with-the-multilingual-judge-only).
+Full tables in [Evaluation](#evaluation).
+
+> **Read this before quoting the numbers.** The figures above come from a
+> curated 48-sentence set, generated with ASR-gated retries (up to 4 attempts).
+> On a harder 500-sentence benchmark of **unseen** news text, plain-Swahili CER
+> is **0.021 with retries and 0.060 single-shot**. Retries matter a lot — see
+> [Release benchmark](#release-benchmark-500-unseen-sentences-single-shot-vs-retried)
+> for the full picture and for why you should use the retry loop in production.
 
 ---
 
@@ -74,7 +90,7 @@ pip install descript-audio-codec soundfile
 python sauti-tts-v2/hpc/dia/patch_dia_fork.py --dia-dir dia-finetuning \
     --epochs 1 --eval-step 1 --save-step 1
 
-# 3. weights (private repo — needs an org token: `hf auth login`)
+# 3. weights
 hf download msingiai/dia --local-dir ./dia-sw
 ```
 
@@ -144,6 +160,43 @@ numbers-dates / code-switch. Two ASR judges: zero-shot
 | numbers_dates | 0.267 / 0.067 | 0.025 / 0.006 |
 | code_switch | **0.286 / 0.050** | 0.281 / 0.088 |
 
+### Release benchmark: 500 unseen sentences, single-shot vs retried
+
+The table above is our curated 48-clip set with ASR-gated retries. Because that
+is both small and curated, we also ran a 500-sentence benchmark on Swahili news
+text (MasakhaNEWS) that **neither this model nor its training corpus has seen** —
+each sentence checked against the actual training text and dropped on exact,
+near-duplicate or shared-5-gram match.
+
+We report it **both ways**, because the difference is large and you should plan
+for it:
+
+| Scope (Swahili judge, WER / CER) | Single-shot | ASR-gated (≤4 attempts) |
+|---|---|---|
+| **overall_plain_sw** | 0.154 / **0.060** | 0.095 / **0.021** |
+| general | 0.081 / 0.015 | 0.083 / 0.015 |
+| named_entities | 0.285 / **0.133** | 0.115 / **0.025** |
+| numbers_dates | 0.097 / 0.033 | 0.088 / 0.025 |
+| code_switch | 0.112 / 0.036 | 0.112 / 0.036 |
+
+**What this means in practice:**
+
+1. **Use the retry loop in production.** It cuts plain-Swahili CER by ~65%
+   (0.060 → 0.021). Generation is sampling-sensitive; a failed take is usually
+   fixed by re-rolling, and one check-ASR pass is far cheaper than shipping bad
+   audio.
+2. **Named entities are the weak spot.** Single-shot they degrade 5× (CER 0.133
+   vs 0.025 gated) — proper nouns are rare tokens and destabilise sampling. If
+   your text is name-heavy (news, directories, announcements), retries are not
+   optional.
+3. **Code-switch is stable either way** — identical with and without retries, so
+   its quality comes from the model rather than from re-rolling.
+4. Numbers on this set are higher than on the curated 48-clip set (0.021 vs
+   0.011 gated). News prose is harder and carries source typos and
+   quote-splitting artefacts that count against the model. Use the 48-clip
+   figure only for comparison against our other models on that same set; use
+   **this** table to predict real-world behaviour.
+
 ### Read code-switch with the multilingual judge only
 
 `Jacaranda-Health/ASR-STT` is Swahili-only and **cannot transcribe English
@@ -177,19 +230,23 @@ statistically indistinguishable plain-Swahili quality.
 UTMOS was deliberately **not** run: it is English-trained and unreliable for
 Swahili.
 
-### Model selection vs our other Swahili models
+### How it compares in our internal bake-off
+
+Same 48-clip set, same judges, so these are directly comparable. The other rows
+are internal reference points and are **not** part of this release.
 
 | Model | Plain SW CER | Code-switch CER (multilingual) | Voice cloning |
 |---|---:|---:|---|
-| VoxCPM2 full SFT | **0.007** | 0.047 | ✅ zero-shot |
+| VoxCPM2 full SFT *(not released)* | **0.007** | 0.047 | ✅ zero-shot |
 | **this model (Dia full SFT e10)** | 0.011 | **0.050** | ❌ single voice |
-| VoxCPM2 LoRA (300 steps) | 0.012 | — | ✅ |
-| Chatterbox LoRA e30 | 0.022 | — | prompt-based |
-| CosyVoice3 e1 | 0.029 | — | ✅ |
+| VoxCPM2 LoRA, 300 steps *(not released)* | 0.012 | — | ✅ |
+| Chatterbox LoRA e30 *(not released)* | 0.022 | — | prompt-based |
+| CosyVoice3 e1 *(not released)* | 0.029 | — | ✅ |
 
-**Choose VoxCPM2 full SFT** for plain Swahili or when you need a specific voice.
-**Choose this model** when you want a fixed, consistent voice with no reference
-audio to manage, or when code-switch robustness is the priority.
+We publish the comparison rather than only our best number: on plain Swahili a
+VoxCPM2 fine-tune scores better than this model. This model is competitive on
+plain Swahili, stronger on code-switching, and needs no reference audio to
+manage — which is why it is the one we release.
 
 ---
 
@@ -240,8 +297,10 @@ single biggest known lever for improving it.
 - **Plain Swahili is behind VoxCPM2** (0.011 vs 0.007 CER).
 - **Short-form only.** Eval is single sentences (~4–9 s); long-form and
   multi-sentence synthesis are unvalidated.
-- **Sampling-sensitive.** Occasional instant-EOS/babble takes need a retry;
-  budget for it in serving.
+- **Sampling-sensitive — budget for retries.** Single-shot plain-Swahili CER is
+  0.060 vs 0.021 with ASR-gated retries. **Named entities are worst affected**
+  (0.133 vs 0.025). Occasional takes come back as instant-EOS or babble and are
+  fixed by re-rolling.
 - **Interior transient sharpness rises with training** (median max sample-jump
   0.307 at epoch 2 → 0.415 at epoch 10). A jump-threshold detector counts these
   as "clicks", but it cannot distinguish a click from a plosive, and listening
@@ -250,11 +309,60 @@ single biggest known lever for improving it.
 - **No formal listening study.** Quality confirmed by one native listener, not
   a MOS panel.
 
-## Licensing
+## Responsible use
+
+Carried over from the [Dia-1.6B](https://huggingface.co/nari-labs/Dia-1.6B)
+disclaimer, and it applies here too:
 
-Derived from Dia-1.6B and trained on a mixed-licence corpus (components include
-MIT and CC-BY-4.0 material). **Verify base-model and dataset licence terms
-before any external distribution or commercial use.**
+- **Do not use this model to impersonate a real person** without their explicit
+  consent.
+- **Do not use it to generate deceptive or misleading content** (fake news,
+  fraudulent audio, misrepresentation).
+- Do not use it for illegal or harmful purposes.
+
+By using this model you accept responsibility for upholding the relevant legal
+and ethical standards in your jurisdiction. The voice in this model is learned
+from a pooled multi-speaker corpus and is not intended to represent any
+identifiable individual.
+
+## Licence and attribution
+
+**This model is released under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/).**
+You may use it commercially, modify it, and redistribute it, provided you give
+attribution.
+
+CC-BY-4.0 is chosen because the training data carries CC-BY attribution
+requirements which must be passed on; the base model is Apache-2.0, which is
+compatible.
+
+### Base model
+
+| Model | Licence |
+|---|---|
+| [nari-labs/Dia-1.6B](https://huggingface.co/nari-labs/Dia-1.6B) | Apache-2.0 |
+
+### Training data
+
+All sources are openly licensed. Attribution below satisfies CC-BY-4.0; please
+carry it forward if you redistribute derivatives.
+
+| Source | Dataset | Licence |
+|---|---|---|
+| WAXAL `swa_tts` | [google/WaxalNLP](https://huggingface.co/datasets/google/WaxalNLP) | CC-BY-4.0 |
+| FLEURS Swahili (KE) | [google/fleurs](https://huggingface.co/datasets/google/fleurs) | CC-BY-4.0 |
+| AfriVoice Swahili (agriculture, education, financial, government, health) | [DigitalUmuganda/Afrivoice_Swahili](https://huggingface.co/datasets/DigitalUmuganda/Afrivoice_Swahili) | CC-BY-4.0 |
+| Swahili Speech 400h | [badrex/swahili-speech-400hr](https://huggingface.co/datasets/badrex/swahili-speech-400hr) | CC-BY-4.0 |
+| YodaLingua Swahili | [Thomcles/YodaLingua-Swahili](https://huggingface.co/datasets/Thomcles/YodaLingua-Swahili) | CC-BY-4.0 |
+| Kiswahili TTS | [Bateesa/kiswahili-tts-dataset](https://huggingface.co/datasets/Bateesa/kiswahili-tts-dataset) | CC-BY-4.0 |
+| Swahili TTS | [jacksonwambali/swahili-tts-dataset](https://huggingface.co/datasets/jacksonwambali/swahili-tts-dataset) | CC-BY-4.0 |
+| Kenyan Swahili (non-standard) | [cdli/kenyan_swahili_nonstandard_speech_v1.0](https://huggingface.co/datasets/cdli/kenyan_swahili_nonstandard_speech_v1.0) | CC-BY-4.0 |
+| Swahili words parallel | [michsethowusu/swahili-words-speech-text-parallel](https://huggingface.co/datasets/michsethowusu/swahili-words-speech-text-parallel) | CC-BY-4.0 (audio originally published by the International Bible Association) |
+| STEM Swahili speech | [stem-content-ai-project/swahili-speech](https://huggingface.co/datasets/stem-content-ai-project/swahili-speech) | MIT |
+| OpenSLR-25 (ALFFA) | [openslr.org/25](https://www.openslr.org/25/) | MIT |
+| Common Voice 17 Swahili | [mozilla-foundation/common_voice_17_0](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | CC0-1.0 at time of corpus build; Mozilla moved Common Voice to the Mozilla Data Collective in Oct 2025 — check current terms for your use case |
+
+Our thanks to every dataset author above. Swahili speech technology exists
+because people chose to release this data openly.
 
 ## Files