Hugging Face's logo

Modularcomputing
/
dynamic-accent-pro-v1.1

Text-to-Speech
PyTorch
Safetensors
dynamic-accent
code-switching
nigerian-languages
igbo
yoruba
hausa
nigerian-pidgin
cosyvoice3
instruction-tts
dpo
experimental
Model card Files
xet
 Community
dynamic-accent-pro-v1.1 / README.md
UncleanCode's picture
UncleanCode
Dynamic Accent Pro v1.1 (experimental prompt-controlled Nigerian TTS); Flash-DPO default, Pro via --instruct
582aab9 verified
preview code
|
Raw History Blame Contribute Delete
19.1 kB
---
license: cc-by-sa-4.0
language:
- en
- pcm
- yo
- ig
- ha
base_model:
- FunAudioLLM/Fun-CosyVoice3-0.5B-2512
pipeline_tag: text-to-speech
library_name: pytorch
tags:
- text-to-speech
- code-switching
- nigerian-languages
- igbo
- yoruba
- hausa
- nigerian-pidgin
- cosyvoice3
- instruction-tts
- dpo
- experimental
datasets:
- google/WaxalNLP
- McGill-NLP/NaijaS2ST
- AfriSpeech/open-bible-speech-african
- nkowaokwu/ibo-dict
- google/fleurs
---
<p align="center"><img src="assets/dynamic_accent_banner.jpg" width="600"/></p>
# Dynamic Accent Pro v1.1
**Experimental prompt-controlled Nigerian TTS and code-switching technology** by Modular. English, Nigerian Pidgin, Igbo, Yoruba and Hausa, mixed inside one sentence and spoken by one consistent voice, with a plain-English **style instruction** (`--instruct`) that steers pace and pausing.
Pro v1.1 is a research release, not a production system. It is the Flash recipe (CosyVoice3 LM fine-tuned on licensed Nigerian-language speech, then DPO) with a new instruction layer on top. Read **What works and what does not** before using it.
## What works and what does not
| control | status | how |
|---|---|---|
| **Pace** ("speak slowly", "speak fast") | **works** | read by the LM from the instruction, sharpened by classifier-free guidance (`--guidance 2`, the default). Slow vs fast is ordered correctly on 18/18 held-out items; slow ≈ 0.89× and fast ≈ 1.15× the default articulation rate |
| **Pauses** ("plenty of pauses", "hardly any pauses") | **works through a text step** | the instruction also edits the text before synthesis (`pause_text.py`: inserts the model's learned `[pause]` tag for "many", strips commas and pause tags for "few"). Many ≈ 1.28× default pause rate; few ≈ 0.91×. The LM alone barely moves pauses, so `--no_text_pauses` mostly turns this off |
| Calm / lively tone | **not advertised** | trained, but indistinguishable from chance in our measurements. The words are accepted and ignored in effect |
| Emotion (happy, sad, angry, …) | **no** | not trained, no data |
| Code-switching from **tags** (`<ibo>…</ibo> <eng>…</eng>`) | works (the core Dynamic Accent capability) | language embeddings + a speaker token keep one voice across switches |
| Code-switching from a **request** (`--codeswitch_prompt "mostly English, Igbo greetings"`) | **experimental, separate step** | a text LLM ([Qwen/Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B), Apache-2.0, not bundled) rewrites plain text into tagged code-switched text *before* synthesis. It is not a TTS capability. Its Pidgin and Hausa are usable; **its Igbo and Yoruba beyond simple greetings are weak and often wrong**, so always read the printed tagged text and fix it |
Without `--instruct`, `infer.py` uses Flash-DPO (the default version). Passing `--instruct` switches to Pro automatically unless you set `--version`; `--version pro` without an instruction uses Pro's default prompt, trained on native-like clips.
## Versions and voices
| version | LM | notes |
|---|---|---|
| `pro` (used automatically with `--instruct`) | `checkpoints/pro/flash_instruct` | Pro v1.1: Flash-DPO → instruction SFT → DPO polish. Instruction control, CFG 2.0 on the instruction, text pause step |
| `flash` (**default**) | `checkpoints/flash/flash_dpo` | Flash-DPO (no instruction control), the default when no `--instruct` is given: it sounds more natural on plain code-switched text. A/B: `--compare` renders the same text with both |
Both use the stock Fun-CosyVoice3-0.5B-2512 flow and vocoder, zero-shot from the voice clip, and the same text front-end (numbers verbalised; missing Yoruba/Igbo diacritics restored by `checkpoints/flash/diacritics_byt5`).
| voice (`--voice`) | who |
|---|---|
| `anchor_male` (default) | WAXAL Media Trust Yoruba speaker 4 (male), the voice both models were trained around |
| `waxal_ibo_5_female` | WAXAL Igbo speaker 5 (female), zero-shot from a 6.6 s clip |
| `waxal_yor_5_female` | WAXAL Yoruba speaker 5 (female), zero-shot; short prompt, so short pauses and more clipped endings |
## Setup (single CUDA GPU)
```bash
git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git
cd CosyVoice && git checkout 074ca6dc9e80a2f424f1f74b48bdd7d3fea531cc && git submodule update --init --recursive && cd ..
huggingface-cli download Modularcomputing/dynamic-accent-pro-v1.1 --local-dir dynamic-accent-pro
pip install -r dynamic-accent-pro/requirements.txt
```
Fun-CosyVoice3-0.5B-2512 downloads on first run (`--cv3_dir` / `CV3_DIR` to reuse a copy). Tested on an H100 in bf16; a 16 GB GPU is enough for TTS. The optional code-switch step needs a GPU that holds Qwen3-32B in bf16 (~65 GB); `--cs_llm Qwen/Qwen3-14B` is lighter and worse.
## Usage
```bash
cd dynamic-accent-pro
# default prompt (native-like pace and pauses)
python infer.py --cosyvoice_dir ../CosyVoice --text "<ibo>Ndewo nụ, ụmụnna m.</ibo> <eng>Tonight we will talk about the new market road.</eng>" --out out.wav
# pace
python infer.py --cosyvoice_dir ../CosyVoice --instruct "Please speak slowly and take your time." --text "..." --out slow.wav
python infer.py --cosyvoice_dir ../CosyVoice --instruct "Speak fast, like a quick everyday conversation." --text "..." --out fast.wav
# pauses (text step + instruction)
python infer.py --cosyvoice_dir ../CosyVoice --instruct "Speak with plenty of pauses between phrases." --text "..." --out many.wav
python infer.py --cosyvoice_dir ../CosyVoice --instruct "Say it in one smooth flow with hardly any pauses." --text "..." --out few.wav
# female Igbo voice
python infer.py --cosyvoice_dir ../CosyVoice --voice waxal_ibo_5_female --text "<ibo>Kedu ka ị mere?</ibo> <pcm>Abeg no vex, I go call you for evening.</pcm>" --out ibo_f.wav
# experimental: plain text + a code-switch request -> Qwen3-32B writes the tagged text (check it!) -> TTS
python infer.py --cosyvoice_dir ../CosyVoice --codeswitch_prompt "Nigerian Pidgin with an Igbo greeting" \
--text "My brother, the work is too much today, but we will finish it." --out cs.wav
# A/B the same text, seed and voice with Flash and Pro -> cmp_flash.wav, cmp_pro.wav
python infer.py --cosyvoice_dir ../CosyVoice --compare --compare_versions flash,pro --text "..." --out cmp.wav
```
| flag | what it does |
|---|---|
| `--instruct "..."` | `pro`: natural-language style instruction. Reliable: slow / fast; plenty of / hardly any pauses. Phrasings close to the examples above work best |
| `--guidance G` | `pro`: classifier-free guidance strength on the instruction (default 2.0; 0 = off, weaker pace control) |
| `--no_text_pauses` | `pro`: do not edit the text for pause instructions |
| `--codeswitch_prompt "..."` | optional text-LLM step before synthesis (see above); output is validated (tags, length, loops, back-translation meaning check) and falls back to the input text |
| `--cs_llm` | the LLM for that step (default `Qwen/Qwen3-32B`) |
| `--version pro\|flash`, `--compare`, `--compare_versions` | model version / A/B rendering |
| `--voice`, `--list_voices` | voice selection |
| `--auto_tag`, `--tag_only`, `--force_lang`, `--lang_map` | language tagging of untagged text (see `tagger.py`) |
| `--frontend codeswitch\|native\|accent`, `--frontend_lang`, `--no_diacritics` | text front-end |
| `--seed N` | sampling seed (default 0). Try 2–3 seeds and keep the best take |
| `--tail_pad N` | append N silence speech tokens before the vocoder so the last syllable is not clipped (opt-in; 6 tested) |
Text: wrap each language span in `<eng>`, `<pcm>`, `<yor>`, `<ibo>` or `<hau>`; untagged text is read as English unless `--auto_tag` / `--force_lang` is given. Write Yoruba and Igbo with diacritics where you can. GPU generation is not bit-reproducible (bf16 near-ties).
## Samples
`samples/` holds MP3 renders for all five languages and a code-switched sentence, both main voices, with the default prompt and the slow / fast / few-pauses / many-pauses instructions, plus a Flash-DPO render of the same text and seed for A/B (`samples/meta.json` lists text, voice and instruction per file). `samples/codeswitch_frontend/` shows the experimental Qwen3-32B step: input text, request, the tagged text it produced, and the audio. The Igbo and Yoruba demo sentences have **not yet been checked by a native speaker**.
## Results
Automatic regression set: 197 held-out sentences per voice (30 each of Igbo, Yoruba, Hausa, Pidgin, English; 24 code-switched; 23 paragraphs), default prompt, seed 0. CER from automatic ASR judges (per language, same judges for every system), pace and pause rate relative to native recordings of the same languages, clipped ending = less than 0.1 s of trailing silence, SIM = speaker similarity to the prompt clip, UTMOSv2 = predicted naturalness.
| voice | group | system | CER | CER no-tone | pace × native | pauses × native | clipped endings | SIM | UTMOSv2 |
|---|---|---|---|---|---|---|---|---|---|
| anchor_male | all | Flash-DPO | 0.162 | 0.135 | 0.99 | 0.75 | 7% | 0.784 | 3.17 |
| anchor_male | all | **Pro v1.1** | 0.167 | 0.141 | 0.99 | 1.02 | 8% | 0.789 | 3.16 |
| anchor_male | ibo | Flash-DPO | 0.189 | 0.158 | 1.06 | 0.40 | 7% | 0.762 | 3.06 |
| anchor_male | ibo | **Pro v1.1** | 0.216 | 0.181 | 1.02 | 0.62 | 3% | 0.779 | 3.01 |
| anchor_male | yor | Flash-DPO | 0.230 | 0.144 | 1.08 | 0.30 | 3% | 0.762 | 2.99 |
| anchor_male | yor | **Pro v1.1** | 0.218 | 0.141 | 1.06 | 0.68 | 3% | 0.770 | 3.02 |
| anchor_male | hau | Flash-DPO | 0.123 | 0.123 | 0.83 | 0.77 | 0% | 0.788 | 3.26 |
| anchor_male | hau | **Pro v1.1** | 0.147 | 0.147 | 0.81 | 1.19 | 3% | 0.796 | 3.19 |
| anchor_male | pcm | Flash-DPO | 0.227 | 0.227 | 0.97 | 0.72 | 10% | 0.782 | 3.26 |
| anchor_male | pcm | **Pro v1.1** | 0.199 | 0.199 | 1.01 | 0.97 | 3% | 0.779 | 3.25 |
| anchor_male | eng | Flash-DPO | 0.026 | 0.026 | 0.94 | 1.85 | 17% | 0.790 | 3.26 |
| anchor_male | eng | **Pro v1.1** | 0.031 | 0.031 | 0.94 | 1.89 | 7% | 0.796 | 3.26 |
| anchor_male | cs | Flash-DPO | 0.208 | 0.159 | 1.02 | 1.00 | 12% | 0.773 | 3.09 |
| anchor_male | cs | **Pro v1.1** | 0.213 | 0.162 | 1.05 | 1.27 | 25% | 0.766 | 3.22 |
| anchor_male | para | Flash-DPO | 0.131 | 0.106 | 1.06 | 1.08 | 0% | 0.843 | 3.28 |
| anchor_male | para | **Pro v1.1** | 0.149 | 0.126 | 1.05 | 1.29 | 13% | 0.843 | 3.16 |
| waxal_ibo_5_female | all | Flash-DPO | 0.165 | 0.134 | 1.03 | 0.95 | 11% | 0.700 | 3.51 |
| waxal_ibo_5_female | all | **Pro v1.1** | 0.152 | 0.123 | 1.01 | 0.84 | 12% | 0.698 | 3.45 |
| waxal_ibo_5_female | ibo | Flash-DPO | 0.215 | 0.171 | 1.09 | 0.52 | 7% | 0.697 | 3.41 |
| waxal_ibo_5_female | ibo | **Pro v1.1** | 0.178 | 0.145 | 1.07 | 0.58 | 0% | 0.682 | 3.40 |
| waxal_ibo_5_female | yor | Flash-DPO | 0.251 | 0.150 | 1.18 | 0.86 | 7% | 0.661 | 3.45 |
| waxal_ibo_5_female | yor | **Pro v1.1** | 0.224 | 0.118 | 1.11 | 0.38 | 10% | 0.671 | 3.36 |
| waxal_ibo_5_female | hau | Flash-DPO | 0.117 | 0.117 | 0.86 | 1.20 | 7% | 0.700 | 3.51 |
| waxal_ibo_5_female | hau | **Pro v1.1** | 0.132 | 0.132 | 0.85 | 0.74 | 13% | 0.676 | 3.47 |
| waxal_ibo_5_female | pcm | Flash-DPO | 0.184 | 0.183 | 1.02 | 0.92 | 13% | 0.674 | 3.63 |
| waxal_ibo_5_female | pcm | **Pro v1.1** | 0.152 | 0.151 | 1.02 | 0.83 | 20% | 0.690 | 3.48 |
| waxal_ibo_5_female | eng | Flash-DPO | 0.031 | 0.031 | 0.99 | 1.34 | 10% | 0.726 | 3.60 |
| waxal_ibo_5_female | eng | **Pro v1.1** | 0.031 | 0.031 | 0.93 | 1.62 | 13% | 0.745 | 3.52 |
| waxal_ibo_5_female | cs | Flash-DPO | 0.230 | 0.179 | 1.01 | 1.32 | 29% | 0.682 | 3.50 |
| waxal_ibo_5_female | cs | **Pro v1.1** | 0.220 | 0.174 | 1.07 | 1.22 | 17% | 0.660 | 3.40 |
| waxal_ibo_5_female | para | Flash-DPO | 0.133 | 0.106 | 1.07 | 0.99 | 4% | 0.773 | 3.44 |
| waxal_ibo_5_female | para | **Pro v1.1** | 0.141 | 0.114 | 1.05 | 1.20 | 9% | 0.772 | 3.50 |
Native-speaker reference: pace 1.00 / pauses 1.00 by definition. Summary: Pro v1.1 matches Flash-DPO on overall CER for the anchor voice (0.167 vs 0.162) and beats it for the female Igbo voice (0.152 vs 0.165); it pauses much closer to native in Igbo and Yoruba on the anchor voice (0.62× / 0.68× vs 0.40× / 0.30×); it is worse than Flash-DPO on anchor Igbo CER (0.216 vs 0.189) and on Hausa for both voices.
Instruction adherence (anchor voice, 18 held-out Igbo/Yoruba/Hausa/Pidgin/English/code-switch items, CFG 2, text pause step on; "direction" = share of items moving the requested way vs the default prompt):
| instruction | articulation rate × default | pause rate × default | direction correct |
|---|---|---|---|
| Please speak slowly and take your time. | 0.89 | 1.16 | 89% |
| Speak fast, like a quick everyday conversation. | 1.15 | 0.87 | 94% |
| Say it in one smooth flow with hardly any pauses. | 0.92 | 0.91 | 56% |
| Speak with plenty of pauses between phrases. | 0.90 | 1.28 | 78% |
| Use a calm, gentle, even tone. *(not advertised)* | 0.98 | 1.02 | 50% |
| Sound lively and expressive. *(not advertised)* | 0.96 | 0.94 | 44% |
Slow vs fast on the same item: correct order on 100% of items; many vs few pauses: 67%; lively vs calm: 50% (chance). Note that "slow" also adds pauses (1.16× pause rate). 18 items is a small set; treat these as indicative.
## Limitations
- **Experimental.** Pace control is real; pause control depends on the text step; calm/lively and emotion are not available.
- **Anchor Igbo** CER is still higher than Flash-DPO (0.216 vs 0.189); **Hausa** CER is worse than Flash-DPO for both voices. Flash-DPO remains available as `--version flash`.
- English stays over-paused relative to native English (about 1.9× native pause rate on the anchor voice, the same as Flash-DPO).
- Clipped endings remain on code-switched sentences and paragraphs (13–25% on the anchor voice). `--tail_pad 6` (six silence tokens appended before the vocoder, inference-only) removed every measured clipped ending in testing with no consistent CER change (+0.005 overall, within seed noise); it is opt-in until it has been checked by ear.
- The code-switch request step (Qwen3-32B) is a separate text model: weak Igbo/Yoruba beyond greetings, sometimes refuses to switch and falls back to the input text.
- Pidgin has little training data and no reliable ASR judge; Yoruba tone quality needs native listening.
- Female voices are zero-shot. Not evaluated for numbers, abbreviations or URLs.
- Do not use it to impersonate real people. The voices are WAXAL Media Trust speakers, released under CC-BY-SA-4.0 for speech technology.
## Training
1. **Flash-DPO** (the `flash` version): CosyVoice3 LM fine-tuned on the licensed set below (language embeddings, speaker token, adversarial speaker classifier, rate embedding), then DPO on 1,206 preference pairs.
2. **Instruction SFT** (3,000 steps): ~100.9k licensed clips labelled from the audio (articulation rate, pause rate, energy/F0 variation); each training item gets a natural-language instruction built from its labels in the CosyVoice3 system-prompt slot, plus explicit pause/energy control embeddings; 18% of items keep the default prompt, drawn from native-like clips.
3. **DPO polish** (134 steps, β 0.1, lr 1e-6, SFT weight 0.1): 700 prompts × 4 samples from the SFT model, 564 preference pairs; reward = CER + pace and pause-rate match to native + a clipped-ending penalty + a small UTMOS guard.
Inference: classifier-free guidance on the instruction (conditional = instruction + controls, unconditional = default prompt), g = 2.
## What is in the repo
| path | what it is | licence |
|---|---|---|
| `checkpoints/pro/flash_instruct/` | Pro v1.1 LM (`best.pt`, sha256 `87c3a7e9…4cab3db`), `config.json`, `spk2id.json`, DPO `args.json`, `log.jsonl` | CC-BY-SA-4.0 |
| `checkpoints/flash/flash_dpo/` | Flash-DPO LM (`best.pt`, sha256 `f59a6fc0…7e3a307`) | CC-BY-SA-4.0 |
| `checkpoints/flash/diacritics_byt5/` | Yoruba/Igbo diacritic restoration (fine-tune of google/byt5-small) | CC-BY-SA-4.0 |
| `prompts/` | voice clips and transcripts (WAXAL) | CC-BY-SA-4.0 |
| `samples/` | demo renders | CC-BY-SA-4.0 |
| `config.json` | repo-level manifest (also lets the Hub count downloads) | Apache-2.0 |
| `settings.json` | versions, voices, defaults | Apache-2.0 |
| `infer.py`, `flash_lm.py`, `flash_instruct.py`, `instruct_text.py`, `pause_text.py`, `cs_frontend.py`, `tagger.py`, `frontend/`, `code/` | inference, model, instruction, text pause step, code-switch step, tagger, front-end, reference training code | Apache-2.0 |
## Training data and attribution
Licensed data only. The weights are trained on, and the voice prompts taken from:
| dataset | licence | attribution |
|---|---|---|
| [WAXAL](https://huggingface.co/datasets/google/WaxalNLP), Media Trust TTS subsets (Igbo, Yoruba, Hausa, Nigerian Pidgin); all voice prompts | CC-BY-SA-4.0 | Google and Media Trust, WAXAL (arXiv:2602.02734) |
| [Open.Bible Igbo](https://huggingface.co/datasets/AfriSpeech/open-bible-speech-african) | CC-BY-SA-4.0 | Biblica, Inc. via open.bible |
| [NaijaS2ST](https://huggingface.co/datasets/McGill-NLP/NaijaS2ST) (Igbo) | CC-BY-4.0 | McGill-NLP (arXiv:2604.16287) |
| [IgboAPI](https://huggingface.co/datasets/nkowaokwu/ibo-dict) | CC-BY-4.0 | Nkọwa okwu / IgboAPI contributors |
| [IgboSynCorp](https://doi.org/10.7910/dvn/yb9fwk) (+ pseudo-transcripts) | CC0-1.0 | University of Ibadan and Afe Babalola University (Lacuna Fund) |
| [FLEURS](https://huggingface.co/datasets/google/fleurs) | CC-BY-4.0 | Google (Conneau et al., 2022) |
| [BibleTTS](https://masakhane-io.github.io/bibleTTS/) (Yoruba, Hausa) | CC-BY-SA-4.0 | Meyer et al. (2022) |
| [OpenSLR SLR70](https://openslr.org/70/), [SLR86](https://openslr.org/86/) | CC-BY-SA-4.0 | Google |
| synthetic code-switch pairs (concatenated clips from the sets above) | follows the sources | after Zhang et al. (2019) |
| base model [Fun-CosyVoice3-0.5B-2512](https://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512) | Apache-2.0 | FunAudioLLM / Alibaba Tongyi Lab |
| optional [Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) (not bundled; code-switch step only) | Apache-2.0 | Qwen team, Alibaba |
DPO preference pairs are the model's own renders, scored automatically. Excluded on licence grounds: African Voices / WaZoBiaSpeech (cards ban TTS use), NaijaVoices (non-commercial), and non-commercial TTS weights.
## Licence
- **Model weights, voice prompts and samples** (`checkpoints/`, `prompts/`, `samples/`): **[CC-BY-SA-4.0](LICENSE)**. Commercial use is allowed; credit Modular and the datasets above; derivatives of the weights must be shared under CC-BY-SA-4.0.
- **Code** (`*.py`, `frontend/`, `code/`, `settings.json`, `config.json`): **[Apache-2.0](LICENSE-CODE)**.
- Fun-CosyVoice3-0.5B-2512, byt5-small and Qwen3 are Apache-2.0; their notices apply to the parts they contribute.
## Citation
```bibtex
@misc{dynamicaccentpro2026,
title = {Dynamic Accent Pro v1.1: prompt-controlled code-switching TTS for Nigerian languages},
author = {Modular},
year = {2026},
url = {https://huggingface.co/Modularcomputing/dynamic-accent-pro-v1.1}
}
```