F5-TTS Swahili (f5-tts-sw)

A Swahili text-to-speech voice, finetuned from F5-TTS (F5TTS_v1_Base, Emilia-pretrained). Flow-matching DiT (1024-dim, 22 layers) + Vocos vocoder, 24 kHz mono, character tokenizer. It is reference-conditioned (zero-shot): it speaks the target text in the voice of a short reference clip (3–10 s). An embedded reference (reference.wav) is included so it works text-only out of the box.

• Best in our bench: CER 0.029 / WER 0.202 (text-normalized; raw 0.039 / 0.228 — Whisper-large-v3 sw, n=25). See the benchmark below.
• Embedded voice: the Safari speaker — a clean studio voice from stem-content-ai-project/swahili-speech.

Files

File	Description
model.safetensors	Pruned EMA weights (finetune of F5TTS_v1_Base)
vocab.txt	Character vocab (reused from the Emilia base)
reference.wav / reference.txt	Embedded reference clip + transcript (Safari voice, ~6.9 s)

Benchmark — what each ingredient buys you

Apples-to-apples: every row was evaluated identically — the same 25 held-out Swahili sentences, the same Safari reference clip, transcribed with the same ASR (Whisper-large-v3 sw). Only the finetuning data changes (all share the F5TTS_v1_Base Emilia base). Lower is better.

#	Finetuning data	Eff. hours	CER (raw)	WER (raw)
1	none — F5TTS_v1_Base zero-shot (Emilia: EN/ZH only)	0	0.175 (0.175)	0.638 (0.639)
2	+ ~1.5 h studio (Safari/Toby)	~1.5	0.173 (0.174)	0.622 (0.624)
3	+ FLEURS-R sw	10.6	0.080 (0.087)	0.346 (0.369)
4	+ FLEURS + studio (overweighted 3×)	15.2	0.043 (0.051)	0.268 (0.285)
5	+ Common Voice + FLEURS + studio ⭐ (this model)	29.3	0.029 (0.039)	0.202 (0.228)

Headline numbers are text-normalized (punctuation stripped, digits expanded on both the reference and the ASR hypothesis — standard TTS-eval practice, cf. Seed-TTS eval / Whisper normalizers). Raw lowercase-only scores in parentheses: they count ASR-added punctuation and digit re-formatting as errors, inflating CER ~25% relative at this level.

Findings

• Zero-shot is a surprisingly strong baseline (1). With a Swahili reference, the EN/ZH base already reads Latin-script Swahili semi-intelligibly (CER 0.175) — but mispronounces native phonemes (ng', ny, dh, prenasalized stops).
• A small studio set does NOT teach the language (2). ~1.5 h of in-domain studio audio moves CER only 0.175→0.174 (noise) — it adjusts timbre, not pronunciation. Scale matters more than a tiny in-domain set.
• Scale of clean Swahili is the unlock (3). Adding ~10 h of FLEURS-R halves CER (0.174→0.087).
• Overweighting the target domain compounds it (4). Duplicating studio clips 3× on top of FLEURS reaches 0.051 — and delayed saturation, so the model kept improving with more training.
• Cleaning + Common Voice push the floor lower (5). Denoising/trimming the machine-fetched corpora, adding quality-filtered Common Voice (Tanzanian-accent breadth, countering FLEURS's Kenyan lean), and heavier overweighting (Safari ×6 / Toby ×12) give CER 0.029 / WER 0.202 — the best here.
• Trust CER over WER for Swahili: Whisper-sw mis-spells/segments, so WER over-states errors. CER and listening are the reliable signals.

Methodology note: the common set is the published model's held-out split; older models pre-date it, so any overlap with their training would flatter them — the swbase margin is therefore conservative. Numbers were produced with this repo's eval (synthesize → Whisper-large-v3 sw → CER/WER vs target).

How to use

from huggingface_hub import hf_hub_download
from f5_tts.api import F5TTS  # pip install f5-tts

repo  = "stem-content-ai-project/f5-tts-sw"
ckpt  = hf_hub_download(repo, "model.safetensors")
vocab = hf_hub_download(repo, "vocab.txt")
ref   = hf_hub_download(repo, "reference.wav")
ref_text = "Shughuli za mwovodhaji zilisaidia katika uokoaji wa samahani zilizozama."

tts = F5TTS(model="F5TTS_v1_Base", ckpt_file=ckpt, vocab_file=vocab)

# 1) PRE-PROCESS the text (see below), then 2) synthesize.
gen_text = preprocess("Piga *149*00# kulipa shilingi 2500.")   # -> Swahili words
wav, sr, _ = tts.infer(ref_file=ref, ref_text=ref_text, gen_text=gen_text)

To use a different voice, pass your own ref_file / ref_text (3–10 s of clean speech).

Pre-processor (run before synthesis)

The model was trained on text where numbers are spelled out as Swahili words and only , . ? ! ' - punctuation appears — it never saw digit/symbol glyphs. Real input has digits, %, USSD codes, etc., so normalize first. This is a sample, swappable heuristic — call it in sequence before infer, or drop in your own frontend:

import re

ONES = ["sifuri","moja","mbili","tatu","nne","tano","sita","saba","nane","tisa"]
TENS = {10:"kumi",20:"ishirini",30:"thelathini",40:"arobaini",50:"hamsini",
        60:"sitini",70:"sabini",80:"themanini",90:"tisini"}
SYM  = {"*":" nyota ","#":" alama ya reli ","/":" kwa ","+":" jumlisha ",
        "=":" sawa na ","&":" na ","@":" at ","_":" "}
_KEEP = re.compile(r"[^A-Za-zÀ-ſ .,?!'\-]")

def _two(n):  return ONES[n] if n<10 else TENS.get(n) or f"{TENS[n//10*10]} na {ONES[n%10]}"
def _three(n):
    h,r=divmod(n,100); p=[]
    if h: p+=["mia",ONES[h]]
    if r: p.append(("na "+_two(r)) if h else _two(r))
    return " ".join(p)
def cardinal(n):
    if n==0: return "sifuri"
    p=[]
    for v,name in [(10**9,"bilioni"),(10**6,"milioni"),(1000,"elfu")]:
        if n>=v: q,n=divmod(n,v); p.append(f"{name} {_three(q)}")
    if n: p.append(("na "+_three(n)) if p and n<100 else _three(n))
    return " ".join(p)
def digits(s): return " ".join(ONES[int(c)] for c in s if c.isdigit())
def _num(t):   return digits(t) if (len(t)>=5 or t.startswith("0")) else cardinal(int(t))

def preprocess(text: str) -> str:
    text = text.strip()
    text = re.sub(r"\*[\d*#]*#", lambda m: " "+re.sub(r"\d+",lambda d:digits(d.group()),m.group())+" ", text)  # USSD *149*00#
    text = re.sub(r"(\d+)\s*%", lambda m: " asilimia "+cardinal(int(m.group(1)))+" ", text)                     # 50% -> asilimia hamsini
    for s,w in SYM.items(): text = text.replace(s,w)                                                            # symbols -> words
    text = re.sub(r"\d+", lambda m: " "+_num(m.group())+" ", text)                                              # remaining numbers
    text = _KEEP.sub(" ", text)                                                                                 # drop unknown glyphs
    text = re.sub(r"\s+([,.?!])", r"\1", re.sub(r"\s+"," ",text)).strip()
    return text

Examples (call order: digits/codes → percent → symbols → number words → strip → collapse):

Input	preprocess(...)
Piga *606# kuangalia salio.	Piga nyota sita sifuri sita alama ya reli kuangalia salio.
Lipa shilingi 2500 kwa siku.	Lipa shilingi elfu mbili na mia tano kwa siku.
Punguzo la 50% leo.	Punguzo la asilimia hamsini leo.
Akaunti 0712345678.	Akaunti sifuri saba moja mbili tatu nne tano sita saba nane.

Heuristics it encodes (replace if your domain differs): ≥5-digit or leading-zero runs (phone/PIN/ account) are read digit-by-digit; shorter runs use cardinal words; % → asilimia N; USSD *…# → every digit spoken; unsupported glyphs are dropped.

Training

• Base: F5TTS_v1_Base (finetune), char tokenizer, Vocos @ 24 kHz, bf16 + TF32, batch 3200 frames.
• Data (29.3 h eff.): FLEURS-R sw (cleaned) + Common Voice sw v17 (cleaned + quality-filtered)
  • Safari ×6 + Toby ×12 (row-duplication overweighting). Audio cleaning = DeepFilterNet denoise + Silero-VAD trim (150 ms silence margin) + loudness norm; Common Voice was additionally filtered by DNSMOS (noise) and Whisper-CER (intelligibility).
• Checkpoint: CER bottomed in the 132k–168k-update plateau; published weights are the earliest-in-plateau checkpoint (update 132,000, lowest WER), pruned to EMA.
• Eval: synthesize n=25 held-out sentences from a fixed reference → transcribe (Whisper-large-v3 sw) → CER/WER vs the target text, with text normalization (punctuation strip + digit expansion) applied to both sides before scoring.
• Training metrics: TensorBoard logs (training loss + per-checkpoint eval CER/WER) are included under runs/ — see the Metrics tab on this page.

Training-data provenance

Source	License	Role
FLEURS-R sw	CC-BY-4.0	pronunciation breadth (Kenyan-leaning)
Common Voice sw v17	CC0	scale + Tanzanian-accent breadth
Safari (stem-content-ai-project/swahili-speech)	—	studio target; embedded voice
Toby	Vodacom (internal)	IVR-domain influence only

The Toby set is Vodacom-internal and used purely as a training-distribution influence; the model's output voice is the Safari speaker, not the Toby speaker.

Gotchas

• Always pre-process digits/symbols (above) — raw 2500, %, *…# are unseen glyphs and come out wrong or dropped.
• WER is misleading for Swahili (Whisper-sw mis-spells); use CER + listening.
• Reference quality drives output: use 3–10 s of clean, single-speaker audio; noisy/clipped refs degrade everything. Keep the reference's ref_text accurate.
• Sample rate / vocab must match: 24 kHz mono; pass this repo's vocab.txt (the char set the weights expect).
• torchaudio backend: on some setups (Windows / torch ≥2.9) F5's torchaudio.load dispatches to torchcodec and fails to find FFmpeg — read audio via soundfile instead (monkeypatch torchaudio.load), and for ASR use WhisperProcessor + model.generate rather than the transformers pipeline (which hard-imports torchcodec).
• Residual errors remain on native Swahili phoneme combinations (ng', prenasalized stops); a domain pronunciation lexicon helps for brand/technical terms.

Limitations & license

Non-commercial (CC-BY-NC-4.0) — inherits the NC terms of the F5TTS_v1_Base (Emilia) base model. Single embedded voice (Safari); for other voices supply your own reference.

Acknowledgements

Built on F5-TTS (SWivid et al.), FLEURS-R (Google), Mozilla Common Voice, Vocos, and the stem-content-ai-project/swahili-speech corpus.