README.md

---
language: sw
license: cc-by-4.0
base_model: nari-labs/Dia-1.6B
tags:
- text-to-speech
- tts
- swahili
- kiswahili
- dia
- code-switching
- african-languages
library_name: dia
pipeline_tag: text-to-speech
---

# Sauti TTS — Dia-1.6B Swahili (full SFT)

Open Swahili text-to-speech from **Msingi-AI**: a full supervised fine-tune of
[nari-labs/Dia-1.6B](https://huggingface.co/nari-labs/Dia-1.6B) on a 500-hour
Swahili speech corpus pooled from 15 openly-licensed public datasets.

This is a **single-voice** model — it takes text and speaks it in one learned
voice. It does not clone voices and needs no reference audio.

**It handles code-switching**, English embedded in Swahili sentences, which is
how a great deal of Swahili is actually spoken and where TTS models commonly
break down: code-switch CER **0.050** against a multilingual ASR judge.

| | Plain Swahili | Code-switched |
|---|---:|---:|
| CER | **0.011** | **0.050** |
| WER | 0.054 | 0.286 |

Plain Swahili is scored by a Swahili-tuned ASR judge, code-switch by a
multilingual one — [for a reason](#read-code-switch-with-the-multilingual-judge-only).
Full tables in [Evaluation](#evaluation).

> **Read this before quoting the numbers.** The figures above come from a
> curated 48-sentence set, generated with ASR-gated retries (up to 4 attempts).
> On a harder 500-sentence benchmark of **unseen** news text, plain-Swahili CER
> is **0.021 with retries and 0.060 single-shot**. Retries matter a lot — see
> [Release benchmark](#release-benchmark-500-unseen-sentences-single-shot-vs-retried)
> for the full picture and for why you should use the retry loop in production.

---

## ⚠️ Read this first: the `[sw]` language tag requires a source patch

Dia's tokenizer maps a language tag to a byte via a `LANG2BYTE` table that
**does not contain Swahili upstream**. This model was trained with `sw` mapped
to **byte 8**. If you load it with an unpatched checkout, `[sw]` is fed as
literal ASCII bytes and inference silently disagrees with training — output
degrades to babble or wrong-language prosody.

Patch **both** `dia/model.py` and the training file before use:

```python
LANG2BYTE = {"sw": 8, ...}   # add "sw": 8 as the first entry
```

Our patcher does this idempotently and fails loudly if upstream layout changes:
`hpc/dia/patch_dia_fork.py` in the `sauti-tts-v2` repo.

Text must be formatted exactly as trained — **bare text with the tag, no `[S1]`
dialogue tags**:

```python
text = f"[sw]{normalized_swahili_text}"     # ✅ matches training
text = f"[S1] {swahili_text}"               # ❌ never used in training
```

---

## Usage

Requires the [stlohrey/dia-finetuning](https://github.com/stlohrey/dia-finetuning)
fork (the upstream `dia` package lacks the multilingual tag path), patched as
above.

### Setup

```bash
# 1. environment (torch 2.5.1+cu121 is the combination we validated)
pip install torch==2.5.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
git clone https://github.com/stlohrey/dia-finetuning && pip install -e dia-finetuning
pip install descript-audio-codec soundfile

# 2. patch LANG2BYTE for Swahili (REQUIRED — see warning above)
python sauti-tts-v2/hpc/dia/patch_dia_fork.py --dia-dir dia-finetuning \
    --epochs 1 --eval-step 1 --save-step 1

# 3. weights
hf download msingiai/dia --local-dir ./dia-sw
```

```python
import dac, torch
from dia.config import DiaConfig
from dia.layers import DiaModel
from dia.model import Dia

device = torch.device("cuda")
cfg = DiaConfig.load("config.json")
model = DiaModel(cfg)
model.load_state_dict(torch.load("model.pth", map_location="cpu"))

# DiaModel builds layers in config.training.dtype (bfloat16) but this
# checkpoint is fp32 — cast to float or attention crashes on mixed dtypes.
model = model.float().to(device).eval()

engine = Dia(cfg, device)
engine.model = model
engine.dac_model = dac.DAC.load(dac.utils.download()).to(device)

with torch.inference_mode():
    wav = engine.generate(
        text="[sw]Habari za asubuhi, karibu katika matangazo yetu ya leo.",
        temperature=1.3,     # our eval default; see note below
    )
# -> float32 numpy array, 44.1 kHz mono
```

### Post-processing (recommended)

Raw takes contain occasional sample-level transients. Our eval harness applies
`declick()` plus a 12 ms raised-cosine fade at both ends
(`scripts/declick.py`, `scripts/synth_finetuned_dia.py`). After that repair,
**every delivered clip starts and ends at exactly zero** and near-silence tick
levels are flat across epochs — measured, not assumed
(`output/dia_samples/measure_artifacts.py`).

Do not skip the fade: it is what removes edge clicks.

### Temperature and ASR-gated retry

Generation is sampling-sensitive — failures show up as instant-EOS (empty
audio) or babble, and re-rolling fixes them. For batch work our harness retries
with a temperature ladder `[1.3, 1.0, 1.2, 0.9, ...]` until a check-ASR
(`openai/whisper-small`) agrees with the input at CER ≤ 0.30, max 4 attempts.
For low-latency serving, use a single attempt at `temperature=1.3` and handle
empty output by retrying.

---

## Evaluation

48-sentence held-out set, stratified across general / named-entities /
numbers-dates / code-switch. Two ASR judges: zero-shot
`openai/whisper-large-v3` and Swahili-tuned `Jacaranda-Health/ASR-STT`.

**Released checkpoint (epoch 10):**

| Scope | whisper-large-v3 WER / CER | Jacaranda ASR WER / CER |
|---|---|---|
| **overall_plain_sw** (headline) | 0.243 / 0.050 | **0.054 / 0.011** |
| overall (incl. code-switch) | 0.254 / 0.050 | 0.111 / 0.030 |
| general | 0.215 / 0.043 | 0.081 / 0.018 |
| named_entities | 0.249 / 0.040 | 0.056 / 0.010 |
| numbers_dates | 0.267 / 0.067 | 0.025 / 0.006 |
| code_switch | **0.286 / 0.050** | 0.281 / 0.088 |

### Release benchmark: 500 unseen sentences, single-shot vs retried

The table above is our curated 48-clip set with ASR-gated retries. Because that
is both small and curated, we also ran a 500-sentence benchmark on Swahili news
text (MasakhaNEWS) that **neither this model nor its training corpus has seen** —
each sentence checked against the actual training text and dropped on exact,
near-duplicate or shared-5-gram match.

We report it **both ways**, because the difference is large and you should plan
for it:

| Scope (Swahili judge, WER / CER) | Single-shot | ASR-gated (≤4 attempts) |
|---|---|---|
| **overall_plain_sw** | 0.154 / **0.060** | 0.095 / **0.021** |
| general | 0.081 / 0.015 | 0.083 / 0.015 |
| named_entities | 0.285 / **0.133** | 0.115 / **0.025** |
| numbers_dates | 0.097 / 0.033 | 0.088 / 0.025 |
| code_switch | 0.112 / 0.036 | 0.112 / 0.036 |

**What this means in practice:**

1. **Use the retry loop in production.** It cuts plain-Swahili CER by ~65%
   (0.060 → 0.021). Generation is sampling-sensitive; a failed take is usually
   fixed by re-rolling, and one check-ASR pass is far cheaper than shipping bad
   audio.
2. **Named entities are the weak spot.** Single-shot they degrade 5× (CER 0.133
   vs 0.025 gated) — proper nouns are rare tokens and destabilise sampling. If
   your text is name-heavy (news, directories, announcements), retries are not
   optional.
3. **Code-switch is stable either way** — identical with and without retries, so
   its quality comes from the model rather than from re-rolling.
4. Numbers on this set are higher than on the curated 48-clip set (0.021 vs
   0.011 gated). News prose is harder and carries source typos and
   quote-splitting artefacts that count against the model. Use the 48-clip
   figure only for comparison against our other models on that same set; use
   **this** table to predict real-world behaviour.

### Read code-switch with the multilingual judge only

`Jacaranda-Health/ASR-STT` is Swahili-only and **cannot transcribe English
words**, so it scores code-switch backwards. It rated epoch 9 (0.108) worse
than epoch 4 (0.101); the multilingual `whisper-large-v3` rated epoch 9 better
(0.067 vs 0.074) — and the native listener agreed with the multilingual judge.
For any code-switch decision, use whisper-large-v3.

### Epoch sweep (all 10 epochs)

Plain CER improves monotonically while code-switch does not — which is why the
released checkpoint is epoch 10, not the best-plain-CER epoch 9.

| Epoch | Plain CER (SW judge) | Code-switch CER (multilingual) |
|---:|---:|---:|
| 1 | 0.460 | 0.126 |
| 2 | 0.022 | 0.093 |
| 3 | 0.021 | 0.068 |
| 4 | 0.015 | 0.074 |
| 5 | 0.013 | 0.051 |
| 6 | 0.010 | 0.051 |
| 7 | 0.011 | 0.063 |
| 8 | 0.012 | **0.048** |
| 9 | **0.008** | 0.067 |
| **10 (released)** | 0.011 | 0.050 |

Epochs 8, 9 and 10 were auditioned by a native Swahili listener, who approved 9
and 10. Epoch 10 was released for its better code-switch score at
statistically indistinguishable plain-Swahili quality.

UTMOS was deliberately **not** run: it is English-trained and unreliable for
Swahili.

### How it compares in our internal bake-off

Same 48-clip set, same judges, so these are directly comparable. The other rows
are internal reference points and are **not** part of this release.

| Model | Plain SW CER | Code-switch CER (multilingual) | Voice cloning |
|---|---:|---:|---|
| VoxCPM2 full SFT *(not released)* | **0.007** | 0.047 | ✅ zero-shot |
| **this model (Dia full SFT e10)** | 0.011 | **0.050** | ❌ single voice |
| VoxCPM2 LoRA, 300 steps *(not released)* | 0.012 | — | ✅ |
| Chatterbox LoRA e30 *(not released)* | 0.022 | — | prompt-based |
| CosyVoice3 e1 *(not released)* | 0.029 | — | ✅ |

We publish the comparison rather than only our best number: on plain Swahili a
VoxCPM2 fine-tune scores better than this model. This model is competitive on
plain Swahili, stronger on code-switching, and needs no reference audio to
manage — which is why it is the one we release.

---

## Training

| | |
|---|---|
| Base model | nari-labs/Dia-1.6B |
| Method | Full supervised fine-tune (all parameters, fp32 + autocast, AdamW8bit) |
| Epochs | 10 (3 825 steps/epoch/rank), ~2 h/epoch, ~21 h total |
| Hardware | 4 × A100-64GB (CINECA Leonardo) |
| Optimiser | lr 1e-5, 500 warmup steps, grad-clip 1.0 |
| Batching | batch size 2 × grad-accum 4 × 4 GPUs = effective batch 32 |
| Audio | 44.1 kHz output (DAC codec) |

Effective batch and learning rate deliberately match our VoxCPM2 full SFT
(1 × 8 × 4 = 32 at lr 1e-5) so the two runs are comparable.

Training ran in two segments: epochs 1–4, then a warm-start resume for epochs
5–10 after a Leonardo filesystem incident killed the first job mid-epoch-5. The
resume restores weights only — optimiser state is not persisted by the trainer,
so the LR schedule restarted with a fresh 500-step warmup at epoch 5.

### Data

Same pooled corpus as our VoxCPM2 model (`sw_voxcpm_corpus_v1`: WAXAL `swa_tts`,
OpenSLR-25 Swahili, AfriVoice Swahili subsets, FLEURS), **but heavily filtered
by Dia's architecture**:

| | Clips | Hours |
|---|---:|---:|
| Full corpus | 99 495 | 500.00 |
| **Usable by Dia (≤ 17.5 s)** | **30 692** | **126.11** |
| Dropped (too long) | 68 803 | 373.89 |

Dia's `config.json` sets `data.audio_length = 1536` DAC frames ≈ 17.9 s;
longer clips would be truncated mid-utterance with the full transcript still
attached, destroying text/audio alignment. **This model therefore saw only 25 %
of the corpus that trained VoxCPM2** — re-segmenting the long clips is the
single biggest known lever for improving it.

---

## Limitations

- **Trained on 126 h, not 500 h** (see above). The most promising future work.
- **Single voice, no cloning.** If you need a specific speaker, use VoxCPM2.
- **Plain Swahili is behind VoxCPM2** (0.011 vs 0.007 CER).
- **Short-form only.** Eval is single sentences (~4–9 s); long-form and
  multi-sentence synthesis are unvalidated.
- **Sampling-sensitive — budget for retries.** Single-shot plain-Swahili CER is
  0.060 vs 0.021 with ASR-gated retries. **Named entities are worst affected**
  (0.133 vs 0.025). Occasional takes come back as instant-EOS or babble and are
  fixed by re-rolling.
- **Interior transient sharpness rises with training** (median max sample-jump
  0.307 at epoch 2 → 0.415 at epoch 10). A jump-threshold detector counts these
  as "clicks", but it cannot distinguish a click from a plosive, and listening
  did not confirm degradation. Treat raw click counts as a prompt to listen,
  not as evidence.
- **No formal listening study.** Quality confirmed by one native listener, not
  a MOS panel.

## Responsible use

Carried over from the [Dia-1.6B](https://huggingface.co/nari-labs/Dia-1.6B)
disclaimer, and it applies here too:

- **Do not use this model to impersonate a real person** without their explicit
  consent.
- **Do not use it to generate deceptive or misleading content** (fake news,
  fraudulent audio, misrepresentation).
- Do not use it for illegal or harmful purposes.

By using this model you accept responsibility for upholding the relevant legal
and ethical standards in your jurisdiction. The voice in this model is learned
from a pooled multi-speaker corpus and is not intended to represent any
identifiable individual.

## Licence and attribution

**This model is released under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/).**
You may use it commercially, modify it, and redistribute it, provided you give
attribution.

CC-BY-4.0 is chosen because the training data carries CC-BY attribution
requirements which must be passed on; the base model is Apache-2.0, which is
compatible.

### Base model

| Model | Licence |
|---|---|
| [nari-labs/Dia-1.6B](https://huggingface.co/nari-labs/Dia-1.6B) | Apache-2.0 |

### Training data

All sources are openly licensed. Attribution below satisfies CC-BY-4.0; please
carry it forward if you redistribute derivatives.

| Source | Dataset | Licence |
|---|---|---|
| WAXAL `swa_tts` | [google/WaxalNLP](https://huggingface.co/datasets/google/WaxalNLP) | CC-BY-4.0 |
| FLEURS Swahili (KE) | [google/fleurs](https://huggingface.co/datasets/google/fleurs) | CC-BY-4.0 |
| AfriVoice Swahili (agriculture, education, financial, government, health) | [DigitalUmuganda/Afrivoice_Swahili](https://huggingface.co/datasets/DigitalUmuganda/Afrivoice_Swahili) | CC-BY-4.0 |
| Swahili Speech 400h | [badrex/swahili-speech-400hr](https://huggingface.co/datasets/badrex/swahili-speech-400hr) | CC-BY-4.0 |
| YodaLingua Swahili | [Thomcles/YodaLingua-Swahili](https://huggingface.co/datasets/Thomcles/YodaLingua-Swahili) | CC-BY-4.0 |
| Kiswahili TTS | [Bateesa/kiswahili-tts-dataset](https://huggingface.co/datasets/Bateesa/kiswahili-tts-dataset) | CC-BY-4.0 |
| Swahili TTS | [jacksonwambali/swahili-tts-dataset](https://huggingface.co/datasets/jacksonwambali/swahili-tts-dataset) | CC-BY-4.0 |
| Kenyan Swahili (non-standard) | [cdli/kenyan_swahili_nonstandard_speech_v1.0](https://huggingface.co/datasets/cdli/kenyan_swahili_nonstandard_speech_v1.0) | CC-BY-4.0 |
| Swahili words parallel | [michsethowusu/swahili-words-speech-text-parallel](https://huggingface.co/datasets/michsethowusu/swahili-words-speech-text-parallel) | CC-BY-4.0 (audio originally published by the International Bible Association) |
| STEM Swahili speech | [stem-content-ai-project/swahili-speech](https://huggingface.co/datasets/stem-content-ai-project/swahili-speech) | MIT |
| OpenSLR-25 (ALFFA) | [openslr.org/25](https://www.openslr.org/25/) | MIT |
| Common Voice 17 Swahili | [mozilla-foundation/common_voice_17_0](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | CC0-1.0 at time of corpus build; Mozilla moved Common Voice to the Mozilla Data Collective in Oct 2025 — check current terms for your use case |

Our thanks to every dataset author above. Swahili speech technology exists
because people chose to release this data openly.

## Files

| File | Purpose |
|---|---|
| `model.pth` | fine-tuned Dia-1.6B weights, fp32 (6.0 GB) |
| `config.json` | Dia model config (must match the training config) |
| `samples/` | 48 eval-set generations from this checkpoint |
| `eval/results.json` | full per-clip eval output |
| `eval/results.md` | eval summary tables |

Optimiser/scheduler state is intentionally excluded (not needed for inference).