README.md

---
language: sw
license: cc-by-4.0
base_model: nari-labs/Dia-1.6B
tags:
- text-to-speech
- tts
- swahili
- kiswahili
- dia
- code-switching
- african-languages
library_name: dia
pipeline_tag: text-to-speech
datasets:
- google/WaxalNLP
- google/fleurs
- DigitalUmuganda/Afrivoice_Swahili
- badrex/swahili-speech-400hr
- Thomcles/YodaLingua-Swahili
- Bateesa/kiswahili-tts-dataset
- jacksonwambali/swahili-tts-dataset
- cdli/kenyan_swahili_nonstandard_speech_v1.0
- michsethowusu/swahili-words-speech-text-parallel
- stem-content-ai-project/swahili-speech
- mozilla-foundation/common_voice_17_0
metrics:
- wer
- cer
model-index:
- name: msingiai/dia
  results:
    - task:
        type: text-to-speech
        name: Text-to-speech
      metrics:
      - type: cer
        value: 1.1
        name: Test CER
---

# Sauti TTS — Dia-1.6B Swahili (full SFT)

Open Swahili text-to-speech from **Msingi-AI**: a full supervised fine-tune of
[nari-labs/Dia-1.6B](https://huggingface.co/nari-labs/Dia-1.6B) on a 500-hour
Swahili speech corpus pooled from 15 openly-licensed public datasets.

This is a **single-voice** model — it takes text and speaks it in one learned
voice. It does not clone voices and needs no reference audio.

**It handles code-switching**, English embedded in Swahili sentences, which is
how a great deal of Swahili is actually spoken and where TTS models commonly
break down: code-switch CER **0.050** against a multilingual ASR judge.

| | Plain Swahili | Code-switched |
|---|---:|---:|
| CER | **0.011** | **0.050** |
| WER | 0.054 | 0.286 |

Plain Swahili is scored by a Swahili-tuned ASR judge, code-switch by a
multilingual one — [for a reason](#read-code-switch-with-the-multilingual-judge-only).
Full tables in [Evaluation](#evaluation).

> **Read this before quoting the numbers.** The figures above come from a
> curated 48-sentence set, generated with ASR-gated retries (up to 4 attempts).
> On a harder 500-sentence benchmark of **unseen** news text, plain-Swahili CER
> is **0.021 with retries and 0.060 single-shot**. Retries matter a lot — see
> [Release benchmark](#release-benchmark-500-unseen-sentences-single-shot-vs-retried)
> for the full picture and for why you should use the retry loop in production.

---

## Inference and usage

Use the maintained [Msingi-AI/dia-infer](https://github.com/Msingi-AI/dia-infer)
runtime. It vendors the compatible Dia inference code and includes the required
Swahili mapping: `[sw]` is mapped to **byte 8**, exactly as it was during
training. No `stlohrey/dia-finetuning` checkout, source patch, manual model
construction, or audio post-processing is required.

Pass **bare Swahili text** to the runtime. It normalizes the text and adds the
`[sw]` marker itself. Do not use Dia dialogue markers such as `[S1]` or `[S2]`;
they were not used during training.

### Modal (recommended)

An A10G is sufficient for realtime, streaming use. Modal avoids requiring a
local GPU and keeps downloaded weights in a persistent Volume:

```bash
git clone https://github.com/Msingi-AI/dia-infer.git
cd dia-infer
uv sync --python 3.11 --extra modal
uv run modal setup
uv run modal serve modal_app.py
```

Modal prints an HTTPS endpoint. Generate a WAV with:

```bash
export DIA_URL='https://your-modal-endpoint.modal.run'

curl -fsS "$DIA_URL/generate" \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Habari za asubuhi, karibu katika matangazo yetu ya leo.",
    "temperature": 1.3,
    "seed": 37
  }' \
  -o output.wav
```

The same endpoint streams 44.1 kHz mono PCM16 over WebSocket at
`wss://your-modal-endpoint.modal.run/tts`. Send the same JSON request; binary
audio chunks are followed by an `end` JSON event. Run `uv run modal deploy
modal_app.py` when you need a persistent endpoint.

The first container startup downloads the checkpoint and compiles Dia. Keep a
container warm while iterating; later requests reuse both the model Volume and
the compiled runtime.

### Local generation

Local generation is useful for smoke tests; an A10-class CUDA GPU is recommended
for realtime performance.

```bash
git clone https://github.com/Msingi-AI/dia-infer.git
cd dia-infer
uv sync --python 3.11
uv run python -m dia_infer.download

uv run python infer.py \
  "Habari za asubuhi, karibu katika matangazo yetu ya leo." \
  --output output.wav \
  --no-compile
```

`output.wav` is 44.1 kHz mono. Remove `--no-compile` after the first local
test to enable the warmed CUDA path.

### Python API

```python
from dia_infer import DiaEngine, GenerationConfig

engine = DiaEngine.load("models/dia", compile=True)
generation = GenerationConfig(temperature=1.3, seed=37)

with open("output.pcm", "wb") as audio:
    for pcm16 in engine.stream_pcm16(
        "Habari za asubuhi, karibu katika matangazo yetu ya leo.",
        generation_config=generation,
    ):
        audio.write(pcm16)
```

`stream_pcm16()` yields little-endian signed 16-bit audio; `stream()` yields
NumPy `float32` waveform chunks.

### Voice reference and generation settings

The runtime loads its repository-managed reference by default. The selected
voice, its audio, and its exact transcript live together in
`references/default.json`; select another bundled voice by changing only the
top-level `id`. The transcript must match the reference audio exactly.

Reference conditioning keeps the selected house voice more consistent across
requests, but it is not a guaranteed zero-shot voice-cloning solution. It can
also make some short outputs sound hurried or unstable. The runtime applies a
small pitch-preserving tempo adjustment to reference-conditioned audio by
default (`conditioned_tempo=0.9`); set it to `1.0` to disable it. Treat this as
a limited mitigation, not a replacement for a model trained with explicit
speaker conditioning.

Start with the validated defaults: `temperature=1.3`, `cfg_scale=3.0`,
`top_p=0.95`, and `cfg_filter_top_k=45`. A fixed `seed` makes sampling
repeatable but does not itself lock a voice. Sampling remains sensitive:
occasional empty or poor takes should be retried. For batch-quality workflows,
ASR-gated retries are a useful optional check; realtime serving normally uses
a single attempt and retries only an empty result.

Long input is segmented automatically while retaining the selected reference.
For conversational responses, keep individual generations short; Dia's
upstream guidance notes that very short and very long utterances may sound less
natural.

---



## Evaluation

48-sentence held-out set, stratified across general / named-entities /
numbers-dates / code-switch. Two ASR judges: zero-shot
`openai/whisper-large-v3` and Swahili-tuned `Jacaranda-Health/ASR-STT`.

**Released checkpoint (epoch 10):**

| Scope | whisper-large-v3 WER / CER | Jacaranda ASR WER / CER |
|---|---|---|
| **overall_plain_sw** (headline) | 0.243 / 0.050 | **0.054 / 0.011** |
| overall (incl. code-switch) | 0.254 / 0.050 | 0.111 / 0.030 |
| general | 0.215 / 0.043 | 0.081 / 0.018 |
| named_entities | 0.249 / 0.040 | 0.056 / 0.010 |
| numbers_dates | 0.267 / 0.067 | 0.025 / 0.006 |
| code_switch | **0.286 / 0.050** | 0.281 / 0.088 |

### Release benchmark: 500 unseen sentences, single-shot vs retried

The table above is our curated 48-clip set with ASR-gated retries. Because that
is both small and curated, we also ran a 500-sentence benchmark on Swahili news
text (MasakhaNEWS) that **neither this model nor its training corpus has seen** —
each sentence checked against the actual training text and dropped on exact,
near-duplicate or shared-5-gram match.

We report it **both ways**, because the difference is large and you should plan
for it:

| Scope (Swahili judge, WER / CER) | Single-shot | ASR-gated (≤4 attempts) |
|---|---|---|
| **overall_plain_sw** | 0.154 / **0.060** | 0.095 / **0.021** |
| general | 0.081 / 0.015 | 0.083 / 0.015 |
| named_entities | 0.285 / **0.133** | 0.115 / **0.025** |
| numbers_dates | 0.097 / 0.033 | 0.088 / 0.025 |
| code_switch | 0.112 / 0.036 | 0.112 / 0.036 |

**What this means in practice:**

1. **Use the retry loop in production.** It cuts plain-Swahili CER by ~65%
   (0.060 → 0.021). Generation is sampling-sensitive; a failed take is usually
   fixed by re-rolling, and one check-ASR pass is far cheaper than shipping bad
   audio.
2. **Named entities are the weak spot.** Single-shot they degrade 5× (CER 0.133
   vs 0.025 gated) — proper nouns are rare tokens and destabilise sampling. If
   your text is name-heavy (news, directories, announcements), retries are not
   optional.
3. **Code-switch is stable either way** — identical with and without retries, so
   its quality comes from the model rather than from re-rolling.
4. Numbers on this set are higher than on the curated 48-clip set (0.021 vs
   0.011 gated). News prose is harder and carries source typos and
   quote-splitting artefacts that count against the model. Use the 48-clip
   figure only for comparison against our other models on that same set; use
   **this** table to predict real-world behaviour.

### Read code-switch with the multilingual judge only

`Jacaranda-Health/ASR-STT` is Swahili-only and **cannot transcribe English
words**, so it scores code-switch backwards. It rated epoch 9 (0.108) worse
than epoch 4 (0.101); the multilingual `whisper-large-v3` rated epoch 9 better
(0.067 vs 0.074) — and the native listener agreed with the multilingual judge.
For any code-switch decision, use whisper-large-v3.

### Epoch sweep (all 10 epochs)

Plain CER improves monotonically while code-switch does not — which is why the
released checkpoint is epoch 10, not the best-plain-CER epoch 9.

| Epoch | Plain CER (SW judge) | Code-switch CER (multilingual) |
|---:|---:|---:|
| 1 | 0.460 | 0.126 |
| 2 | 0.022 | 0.093 |
| 3 | 0.021 | 0.068 |
| 4 | 0.015 | 0.074 |
| 5 | 0.013 | 0.051 |
| 6 | 0.010 | 0.051 |
| 7 | 0.011 | 0.063 |
| 8 | 0.012 | **0.048** |
| 9 | **0.008** | 0.067 |
| **10 (released)** | 0.011 | 0.050 |

Epochs 8, 9 and 10 were auditioned by a native Swahili listener, who approved 9
and 10. Epoch 10 was released for its better code-switch score at
statistically indistinguishable plain-Swahili quality.

UTMOS was deliberately **not** run: it is English-trained and unreliable for
Swahili.

### How it compares in our internal bake-off

Same 48-clip set, same judges, so these are directly comparable. The other rows
are internal reference points and are **not** part of this release.

| Model | Plain SW CER | Code-switch CER (multilingual) | Voice cloning |
|---|---:|---:|---|
| VoxCPM2 full SFT *(not released)* | **0.007** | 0.047 | ✅ zero-shot |
| **this model (Dia full SFT e10)** | 0.011 | **0.050** | ❌ single voice |
| VoxCPM2 LoRA, 300 steps *(not released)* | 0.012 | — | ✅ |
| Chatterbox LoRA e30 *(not released)* | 0.022 | — | prompt-based |
| CosyVoice3 e1 *(not released)* | 0.029 | — | ✅ |

We publish the comparison rather than only our best number: on plain Swahili a
VoxCPM2 fine-tune scores better than this model. This model is competitive on
plain Swahili, stronger on code-switching, and needs no reference audio to
manage — which is why it is the one we release.

---

## Training

| | |
|---|---|
| Base model | nari-labs/Dia-1.6B |
| Method | Full supervised fine-tune (all parameters, fp32 + autocast, AdamW8bit) |
| Epochs | 10 (3 825 steps/epoch/rank), ~2 h/epoch, ~21 h total |
| Hardware | 4 × A100-64GB (CINECA Leonardo) |
| Optimiser | lr 1e-5, 500 warmup steps, grad-clip 1.0 |
| Batching | batch size 2 × grad-accum 4 × 4 GPUs = effective batch 32 |
| Audio | 44.1 kHz output (DAC codec) |

Effective batch and learning rate deliberately match our VoxCPM2 full SFT
(1 × 8 × 4 = 32 at lr 1e-5) so the two runs are comparable.

Training ran in two segments: epochs 1–4, then a warm-start resume for epochs
5–10 after a Leonardo filesystem incident killed the first job mid-epoch-5. The
resume restores weights only — optimiser state is not persisted by the trainer,
so the LR schedule restarted with a fresh 500-step warmup at epoch 5.

### Data

Same pooled corpus as our VoxCPM2 model (`sw_voxcpm_corpus_v1`: WAXAL `swa_tts`,
OpenSLR-25 Swahili, AfriVoice Swahili subsets, FLEURS), **but heavily filtered
by Dia's architecture**:

| | Clips | Hours |
|---|---:|---:|
| Full corpus | 99 495 | 500.00 |
| **Usable by Dia (≤ 17.5 s)** | **30 692** | **126.11** |
| Dropped (too long) | 68 803 | 373.89 |

Dia's `config.json` sets `data.audio_length = 1536` DAC frames ≈ 17.9 s;
longer clips would be truncated mid-utterance with the full transcript still
attached, destroying text/audio alignment. **This model therefore saw only 25 %
of the corpus that trained VoxCPM2** — re-segmenting the long clips is the
single biggest known lever for improving it.

---

## Limitations

- **Trained on 126 h, not 500 h** (see above). The most promising future work.
- **Single voice, no cloning.** If you need a specific speaker, use VoxCPM2.
- **Plain Swahili is behind VoxCPM2** (0.011 vs 0.007 CER).
- **Short-form only.** Eval is single sentences (~4–9 s); long-form and
  multi-sentence synthesis are unvalidated.
- **Sampling-sensitive — budget for retries.** Single-shot plain-Swahili CER is
  0.060 vs 0.021 with ASR-gated retries. **Named entities are worst affected**
  (0.133 vs 0.025). Occasional takes come back as instant-EOS or babble and are
  fixed by re-rolling.
- **Interior transient sharpness rises with training** (median max sample-jump
  0.307 at epoch 2 → 0.415 at epoch 10). A jump-threshold detector counts these
  as "clicks", but it cannot distinguish a click from a plosive, and listening
  did not confirm degradation. Treat raw click counts as a prompt to listen,
  not as evidence.
- **No formal listening study.** Quality confirmed by one native listener, not
  a MOS panel.

## Responsible use

Carried over from the [Dia-1.6B](https://huggingface.co/nari-labs/Dia-1.6B)
disclaimer, and it applies here too:

- **Do not use this model to impersonate a real person** without their explicit
  consent.
- **Do not use it to generate deceptive or misleading content** (fake news,
  fraudulent audio, misrepresentation).
- Do not use it for illegal or harmful purposes.

By using this model you accept responsibility for upholding the relevant legal
and ethical standards in your jurisdiction. The voice in this model is learned
from a pooled multi-speaker corpus and is not intended to represent any
identifiable individual.

## Licence and attribution

**This model is released under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/).**
You may use it commercially, modify it, and redistribute it, provided you give
attribution.

CC-BY-4.0 is chosen because the training data carries CC-BY attribution
requirements which must be passed on; the base model is Apache-2.0, which is
compatible.

### Base model

| Model | Licence |
|---|---|
| [nari-labs/Dia-1.6B](https://huggingface.co/nari-labs/Dia-1.6B) | Apache-2.0 |

### Training data

All sources are openly licensed. Attribution below satisfies CC-BY-4.0; please
carry it forward if you redistribute derivatives.

| Source | Dataset | Licence |
|---|---|---|
| WAXAL `swa_tts` | [google/WaxalNLP](https://huggingface.co/datasets/google/WaxalNLP) | CC-BY-4.0 |
| FLEURS Swahili (KE) | [google/fleurs](https://huggingface.co/datasets/google/fleurs) | CC-BY-4.0 |
| AfriVoice Swahili (agriculture, education, financial, government, health) | [DigitalUmuganda/Afrivoice_Swahili](https://huggingface.co/datasets/DigitalUmuganda/Afrivoice_Swahili) | CC-BY-4.0 |
| Swahili Speech 400h | [badrex/swahili-speech-400hr](https://huggingface.co/datasets/badrex/swahili-speech-400hr) | CC-BY-4.0 |
| YodaLingua Swahili | [Thomcles/YodaLingua-Swahili](https://huggingface.co/datasets/Thomcles/YodaLingua-Swahili) | CC-BY-4.0 |
| Kiswahili TTS | [Bateesa/kiswahili-tts-dataset](https://huggingface.co/datasets/Bateesa/kiswahili-tts-dataset) | CC-BY-4.0 |
| Swahili TTS | [jacksonwambali/swahili-tts-dataset](https://huggingface.co/datasets/jacksonwambali/swahili-tts-dataset) | CC-BY-4.0 |
| Kenyan Swahili (non-standard) | [cdli/kenyan_swahili_nonstandard_speech_v1.0](https://huggingface.co/datasets/cdli/kenyan_swahili_nonstandard_speech_v1.0) | CC-BY-4.0 |
| Swahili words parallel | [michsethowusu/swahili-words-speech-text-parallel](https://huggingface.co/datasets/michsethowusu/swahili-words-speech-text-parallel) | CC-BY-4.0 (audio originally published by the International Bible Association) |
| STEM Swahili speech | [stem-content-ai-project/swahili-speech](https://huggingface.co/datasets/stem-content-ai-project/swahili-speech) | MIT |
| OpenSLR-25 (ALFFA) | [openslr.org/25](https://www.openslr.org/25/) | MIT |
| Common Voice 17 Swahili | [mozilla-foundation/common_voice_17_0](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | CC0-1.0 at time of corpus build; Mozilla moved Common Voice to the Mozilla Data Collective in Oct 2025 — check current terms for your use case |

Our thanks to every dataset author above. Swahili speech technology exists
because people chose to release this data openly.

## Files

| File | Purpose |
|---|---|
| `model.pth` | fine-tuned Dia-1.6B weights, fp32 (6.0 GB) |
| `config.json` | Dia model config (must match the training config) |
| `samples/` | 48 eval-set generations from this checkpoint |
| `eval/results.json` | full per-clip eval output |
| `eval/results.md` | eval summary tables |

Optimiser/scheduler state is intentionally excluded (not needed for inference).