Dynamic Accent Flash: add Flash-DPO (CosyVoice3) as --version flash (default), diacritic front-end, Flash results, samples

.gitattributes

@@ -51,3 +51,13 @@ samples/phase3/phase3/para_ibo_pcm_eng.mp3 filter=lfs diff=lfs merge=lfs -text
 samples/phase3/phase3/para_yor_eng.mp3 filter=lfs diff=lfs merge=lfs -text
 samples/phase3/phase3_female/para_ibo_pcm_eng__waxal_ibo_5.mp3 filter=lfs diff=lfs merge=lfs -text
 samples/phase3/phase3_female/para_yor_eng__waxal_yor_5.mp3 filter=lfs diff=lfs merge=lfs -text
+samples/flash/codeswitch_ibo_pcm_eng.mp3 filter=lfs diff=lfs merge=lfs -text
+samples/flash/codeswitch_yor_eng.mp3 filter=lfs diff=lfs merge=lfs -text
+samples/flash/diacritics_yoruba_from_plain.mp3 filter=lfs diff=lfs merge=lfs -text
+samples/flash/hausa_male.mp3 filter=lfs diff=lfs merge=lfs -text
+samples/flash/igbo_para_female.mp3 filter=lfs diff=lfs merge=lfs -text
+samples/flash/igbo_pure_female.mp3 filter=lfs diff=lfs merge=lfs -text
+samples/flash/numbers_igbo.mp3 filter=lfs diff=lfs merge=lfs -text
+samples/flash/pidgin_female.mp3 filter=lfs diff=lfs merge=lfs -text
+samples/flash/yoruba_female.mp3 filter=lfs diff=lfs merge=lfs -text
+samples/flash/yoruba_male.mp3 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -6,7 +6,9 @@ language:
   - yo
   - ig
   - ha
-base_model: FunAudioLLM/CosyVoice2-0.5B
+base_model:
+  - FunAudioLLM/Fun-CosyVoice3-0.5B-2512
+  - FunAudioLLM/CosyVoice2-0.5B
 pipeline_tag: text-to-speech
 library_name: pytorch
 tags:
@@ -18,6 +20,8 @@ tags:
   - hausa
   - nigerian-pidgin
   - cosyvoice2
+  - cosyvoice3
+  - dpo
 datasets:
   - google/WaxalNLP
   - McGill-NLP/NaijaS2ST
@@ -32,7 +36,9 @@ datasets:
 
 **Code-switching TTS for Nigerian languages.** English, Nigerian Pidgin, Igbo, Yoruba and Hausa, mixed freely inside one sentence and spoken by one consistent voice, with no model swapping between languages.
 
-Dynamic Accent is a fine-tune of [FunAudioLLM/CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B) by Modular. It adds:
+**New: Dynamic Accent Flash** (`--version flash`, now the default) is the same recipe ported to [FunAudioLLM/Fun-CosyVoice3-0.5B-2512](https://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512), then preference-tuned with DPO for naturalness, intelligibility, pace and pauses, with a text front-end that restores Yoruba/Igbo diacritics. It is trained on licensed data only. See **Flash results** below.
+
+Dynamic Accent (Phase 2/3) is a fine-tune of [FunAudioLLM/CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B) by Modular. It adds:
 - per-token **language embeddings**, a **speaker token** and an adversarial (gradient-reversal) speaker classifier on the LM, so the voice stays the same across a language switch (after Zhang et al., Google, 2019, *Learning to Speak Fluently in a Foreign Language*);
 - training on about 178 h of Nigerian-language and code-switched speech, including about 102 h of Igbo;
 - learned `[pause]`/`[breath]` events and a speaking-rate instruction (Phase 3);
@@ -42,11 +48,12 @@ It is a research release, not a production system. Read **Limitations** before y
 
 ## Versions and voices
 
-Two model versions ship side by side. Pick one with `--version`, or render both with `--compare` and A/B before use.
+Three model versions ship side by side. Pick one with `--version`, or render several with `--compare` and A/B before use.
 
 | version | LM | flow | decoding | what changed |
 |---|---|---|---|---|
-| `phase3` (**default**) | `checkpoints/phase3/p3_lm_full` (Phase 2 LM + 5,000 steps on a 178 h re-cut with ~102 h Igbo, `[pause]`/`[breath]` tags and a speed instruction, rate `normal`) | shared `checkpoints/phase2/p2_flow_sft` | one pass per paragraph, silence-aware repetition check, no audio post-processing | lower CER in Igbo/Hausa/Yoruba/code-switch/paragraphs, comma pauses at the anchor's length, clipped endings 63% → 11% |
+| `flash` (**default**) | `checkpoints/flash/flash_dpo` on **Fun-CosyVoice3-0.5B-2512**: Flash SFT (CV3 LM, the Phase 3 data re-weighted towards natural speech, plus IgboSynCorp pseudo-transcripts) + DPO (1,206 preference pairs; reward = naturalness + CER + articulation rate + pause rate) | stock CV3 flow + vocoder, zero-shot from the voice clip | text front-end (numbers verbalised, missing Yoruba/Igbo diacritics restored by `checkpoints/flash/diacritics_byt5`), one pass per paragraph, silence-aware repetition check, no audio post-processing | pace about native, fewest clipped endings for every voice, lower anchor CER (see **Flash results**) |
+| `phase3` | `checkpoints/phase3/p3_lm_full` (Phase 2 LM + 5,000 steps on a 178 h re-cut with ~102 h Igbo, `[pause]`/`[breath]` tags and a speed instruction, rate `normal`) | shared `checkpoints/phase2/p2_flow_sft` | one pass per paragraph, silence-aware repetition check, no audio post-processing | lower CER in Igbo/Hausa/Yoruba/code-switch/paragraphs, comma pauses at the anchor's length, clipped endings 63% → 11% |
 | `phase2` | `checkpoints/phase2/p2_lm_B` (~120 h, pause profile B) | `checkpoints/phase2/p2_flow_sft` | sentence by sentence, internal silences capped at 0.6 s | the Phase 2 final system |
 
 | voice (`--voice`) | who | notes |
@@ -55,7 +62,7 @@ Two model versions ship side by side. Pick one with `--version`, or render both
 | `waxal_ibo_5_female` | WAXAL Igbo speaker 5, female, native Igbo | zero-shot (prompt clip, no fine-tune); best CER of the three on the female test subset |
 | `waxal_yor_5_female` | WAXAL Yoruba speaker 5, female, native Yoruba | zero-shot; strongest identity match, but its 3.8 s prompt has no pauses, so pauses are short and endings often clipped |
 
-**Recommended pairing** (author's listening verdict): **Phase 3 sounds best with the female voices; Phase 2 is fine for the male anchor.** `infer.py` prints a note when you pick another pairing. The female voices were evaluated with `phase3` only; they run with `phase2` but are untested there. Settings live in `settings.json`.
+**Recommended:** `flash` with any of the three voices; it scores best overall (see **Flash results**), with one known weakness: with `anchor_male` it under-pauses in Igbo and Yoruba. Earlier listening verdict (before Flash): Phase 3 sounds best with the female voices, Phase 2 is fine for the male anchor. `infer.py` prints a note when you pick a pairing outside these. The female voices run with `phase2` but are untested there. Settings live in `settings.json`.
 
 ## Setup (single GPU)
 
@@ -71,21 +78,25 @@ huggingface-cli download Modularcomputing/dynamic-accent-preview --local-dir dyn
 pip install -r dynamic-accent-preview/requirements.txt
 ```
 
-The stock base model `FunAudioLLM/CosyVoice2-0.5B` (tokenizer, vocoder, speech tokenizer, speaker encoder) downloads to `pretrained/CosyVoice2-0.5B` on first run; nothing in it needs changing. Point `--model_dir` (or `COSYVOICE2_DIR`) at an existing copy, and `--cosyvoice_dir` (or `COSYVOICE_REPO`) at your CosyVoice clone. A 16 GB GPU is enough for the CLI; the Gradio app keeps both versions loaded. Tested on an H100 (bf16). `infer.py` also runs on CPU in fp32, which is slow. T4-class GPUs fall back to fp16, which has not been tested.
+The stock base models download on first run and need no changes: `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` to `pretrained/Fun-CosyVoice3-0.5B-2512` for `flash` (`--cv3_dir` / `CV3_DIR` to reuse a copy), and `FunAudioLLM/CosyVoice2-0.5B` to `pretrained/CosyVoice2-0.5B` for `phase2`/`phase3`. The same CosyVoice commit runs both (it has `CosyVoice3`). **`flash` needs a CUDA GPU** (tested on an H100, bf16). Point `--model_dir` (or `COSYVOICE2_DIR`) at an existing copy, and `--cosyvoice_dir` (or `COSYVOICE_REPO`) at your CosyVoice clone. A 16 GB GPU is enough for the CLI; the Gradio app keeps both versions loaded. Tested on an H100 (bf16). `infer.py` also runs on CPU in fp32, which is slow. T4-class GPUs fall back to fp16, which has not been tested.
 
 ## Usage
 
 ```bash
 cd dynamic-accent-preview
-# default: --version phase3 --voice anchor_male
+# default: --version flash --voice anchor_male
 python infer.py --cosyvoice_dir ../CosyVoice --text "<eng>this is dynamic accent,</eng> <pcm>e dey sound nice abi?</pcm>" --out out.wav
 
-# recommended pairing
-python infer.py --cosyvoice_dir ../CosyVoice --version phase3 --voice waxal_ibo_5_female \
+# Flash with the female Igbo voice
+python infer.py --cosyvoice_dir ../CosyVoice --version flash --voice waxal_ibo_5_female \
   --text "<ibo>Kedu ka ị mere?</ibo> <pcm>Abeg no vex, I go call you for evening.</pcm>" --out ibo_f.wav
 
-# A/B: the same text, seed and voice through both versions -> cmp_phase2.wav, cmp_phase3.wav
-python infer.py --cosyvoice_dir ../CosyVoice --compare --voice anchor_male --text "..." --out cmp.wav
+# plain Yoruba without tone marks: Flash's front-end restores them before synthesis
+python infer.py --cosyvoice_dir ../CosyVoice --version flash --voice waxal_yor_5_female --frontend native --frontend_lang yor \
+  --text "E kaaaro, se daadaa ni gbogbo yin wa?" --out yor.wav
+
+# A/B: the same text, seed and voice through several versions -> cmp_phase2.wav, cmp_phase3.wav, cmp_flash.wav
+python infer.py --cosyvoice_dir ../CosyVoice --compare --compare_versions phase2,phase3,flash --voice anchor_male --text "..." --out cmp.wav
 
 python infer.py --list_voices     # voices, versions, recommended pairing
 ```
@@ -93,11 +104,16 @@ python infer.py --list_voices     # voices, versions, recommended pairing
 | flag | what it does |
 |---|---|
 | `--text` | the text: tagged (below) or plain with `--auto_tag` |
-| `--version phase2\|phase3` | model version (default `phase3`) |
+| `--version flash\|phase3\|phase2` | model version (default `flash`) |
 | `--voice anchor_male\|waxal_ibo_5_female\|waxal_yor_5_female` | voice (default `anchor_male`); old ids `anchor`, `waxal_ibo_5`, `waxal_yor_5` still work |
 | `--list_voices` | list voices, versions and the recommended pairing |
-| `--compare` | render with both versions → `<out>_phase2.wav`, `<out>_phase3.wav` |
-| `--rate default\|none\|very slow\|slow\|normal\|fast\|very fast` | Phase 3 speaking-rate instruction to the LM (default: the version's setting, `normal`). It is a learned instruction, not time-stretching. `phase2` ignores it |
+| `--compare`, `--compare_versions phase2,phase3,flash` | render with several versions → `<out>_<version>.wav` |
+| `--frontend codeswitch\|native\|accent` | `flash` text front-end: `codeswitch` (default; tagged text, or untagged text in `--frontend_lang`), `native` (whole text in `--frontend_lang`, diacritics restored), `accent` (words as written, read with the `--frontend_lang` accent) |
+| `--frontend_lang eng\|pcm\|ibo\|yor\|hau` | language for untagged text / `native` / `accent` |
+| `--no_diacritics` | `flash`: do not restore missing Yoruba/Igbo tone marks and underdots |
+| `--frontend translate --experimental_translate` | **experimental, off by default**: translate English/Pidgin into `--frontend_lang` (ibo/yor/hau) with MADLAD-400-3B-MT (Apache-2.0, ~12 GB download) before synthesis. MADLAD often loops (repeated words or dots) on longer inputs and translates Pidgin poorly, so check the printed text |
+| `--cv3_dir` | `flash`: path of the Fun-CosyVoice3-0.5B-2512 snapshot |
+| `--rate default\|none\|very slow\|slow\|normal\|fast\|very fast` | Phase 3 / Flash speaking-rate instruction to the LM (default: the version's setting, `normal`). It is a learned instruction, not time-stretching. `phase2` ignores it |
 | `--auto_tag` | tag untagged mixed text automatically (see below), print the tagged text, then synthesise |
 | `--tag_only` | with `--auto_tag`: print the tagged text and stop, so you can edit it and pass it back as `--text` |
 | `--force_lang eng\|pcm\|ibo\|yor\|hau` | read the whole text in one language (tags are ignored) |
@@ -149,7 +165,7 @@ Write Yoruba and Igbo with their diacritics where you can; text without diacriti
 python app.py --cosyvoice_dir ../CosyVoice      # http://127.0.0.1:7860  (--share for a public link)
 ```
 
-It has a text box, an auto-tag toggle with an editable tagged-text preview, word overrides, a force-language option, version, voice and rate dropdowns, a seed, and a **Compare** button that plays Phase 2 and Phase 3 side by side.
+It has a text box, an auto-tag toggle with an editable tagged-text preview, word overrides, a force-language option, version, voice and rate dropdowns, a seed, and a **Compare** button that plays Phase 3 and Flash side by side. The app launch has not been smoke-tested yet.
 
 ### From Python
 
@@ -158,13 +174,33 @@ import types, soundfile as sf, infer
 a = types.SimpleNamespace(model_dir="pretrained/CosyVoice2-0.5B", cosyvoice_dir="../CosyVoice", lm="", flow="",
                           prompts="prompts/prompts.json", voices="prompts/voices.json", dtype="auto", stitch=False,
                           rate="default", voice="waxal_ibo_5_female")
-tts = infer.build("phase3", a)
+tts = infer.build("phase3", a)      # for "flash" also set frontend="codeswitch", frontend_lang="eng", no_diacritics=False,
+                                     # experimental_translate=False, cv3_dir="pretrained/Fun-CosyVoice3-0.5B-2512"
 wav = tts.say("<ibo>Kedu ka ị mere?</ibo> <eng>The meeting has been moved to Friday morning.</eng>", seed=0)
 sf.write("out.wav", wav, tts.sr)
 ```
 
 ## Samples
 
+### Flash (`--version flash`, Flash-DPO)
+
+Seed 0, model pauses, speed 1.0, no post-processing. Inputs are written with diacritics unless noted.
+
+| sample | voice | text |
+|---|---|---|
+| [igbo_pure_female](samples/flash/igbo_pure_female.mp3) | waxal_ibo_5_female | Chei, obi dị m ụtọ nke ukwuu ịkpọsa ụdị ọhụrụ a, ọ na-eme m obi ụtọ, agaghị m agha gị ụgha. |
+| [igbo_para_female](samples/flash/igbo_para_female.mp3) | waxal_ibo_5_female | Ndewo nu, ụmụnna m. Taa bụ ụbọchị pụrụ iche n'obodo anyị. Anyị ga-ezukọ n'ama egwuregwu, rie nri, ma kparịta ụka maka ọdịnihu ụmụaka anyị. |
+| [yoruba_female](samples/flash/yoruba_female.mp3) / [yoruba_male](samples/flash/yoruba_male.mp3) | waxal_yor_5_female / anchor_male | Ẹ káàsán o, ẹ̀yin ará. Inú mi dùn láti rí gbogbo yín lónìí. Ẹ jẹ́ ká jọ ṣiṣẹ́ pọ̀ fún ìlọsíwájú ìlú wa. |
+| [hausa_male](samples/flash/hausa_male.mp3) | anchor_male | Barka da yamma, 'yan uwa. Muna farin cikin ganin ku a yau. Za mu fara taron nan ba da daɗewa ba, don haka ku zauna lafiya. |
+| [pidgin_female](samples/flash/pidgin_female.mp3) | waxal_ibo_5_female | Omo, this traffic no be here o. Abeg make una no vex, I go reach there small time. Wetin dey happen for your side? |
+| [codeswitch_ibo_pcm_eng](samples/flash/codeswitch_ibo_pcm_eng.mp3) | waxal_ibo_5_female | `<ibo>Nne m, ọ dị mma, ka anyị gaa ahịa taa.</ibo> <pcm>Abeg make we no late o,</pcm> <eng>the shop closes at 6pm.</eng>` |
+| [codeswitch_yor_eng](samples/flash/codeswitch_yor_eng.mp3) | waxal_yor_5_female | `<yor>Ẹ jọ̀ọ́, ẹ dúró díẹ̀,</yor> <eng>I will send you the report this afternoon.</eng> <yor>Ẹ ṣé púpọ̀.</yor>` |
+| [codeswitch_dynacc](samples/flash/codeswitch_dynacc.mp3) | waxal_ibo_5_female | `<eng>this is dynamic accent,</eng> <pcm>e dey sound nice abi?</pcm>` |
+| [numbers_igbo](samples/flash/numbers_igbo.mp3) | waxal_ibo_5_female | Ọnụ ahịa ya bụ ₦2,500, anyị ga-ahụ gị na 21/10/2026 n'elekere 4:30. (numbers verbalised by the front-end) |
+| [diacritics_yoruba_from_plain](samples/flash/diacritics_yoruba_from_plain.mp3) | waxal_yor_5_female | input without marks: "E kaaaro, se daadaa ni gbogbo yin wa? Oju ojo dara pupo loni." → front-end: Ẹ káàárọ̀, ṣé dáadáa ni gbogbo yín wà? Ojú ọjọ́ dára púpọ̀ lónì. |
+
+### Phase 2 and Phase 3
+
 Seed 0, rendered with `infer.py`. Inputs: `para_ibo_pcm_eng`, `para_yor_eng`, `para_hau_eng_pcm` (code-switched paragraphs), `ibo_000` (Igbo), `custom_dynacc` ("this is dynamic accent, e dey sound nice abi?") and `omo_announce` ("omo, I'm very excited to announce this new model, ọ na-eme m obi ụtọ, I no go lie give you").
 
 | input | Phase 2, anchor_male | Phase 3, anchor_male | Phase 3, female |
@@ -178,6 +214,24 @@ Seed 0, rendered with `infer.py`. Inputs: `para_ibo_pcm_eng`, `para_yor_eng`, `p
 
 Older renders: `samples/pcm_eng_dynamic_accent_seed{0,1,2}.mp3` (Phase 2, three seeds) and `samples/prosody_v2/` (the rejected prosody-v2 post-processing, kept for reference).
 
+## Flash results
+
+Same eval set and judges as the Phase 2 vs Phase 3 tables below (183 items plus code-switched and paragraph groups), seed 0, all groups pooled. **Pace** and **pauses** are ratios to native speakers reading the same texts (1.00 = native). **Clipped endings** = share of renders whose trailing silence is under 0.1 s (native recordings: 23.8%). SIM is ECAPA cosine to the voice's own real clips.
+
+| voice | system | CER | CER no tones | SIM | pace × native | pauses/s × native | clipped endings |
+|---|---|---|---|---|---|---|---|
+| anchor_male | Phase 3 | 0.174 | 0.146 | 0.815 | 0.92 | 1.11 | 9.1% |
+| anchor_male | **Flash** | **0.162** | **0.135** | 0.784 | **0.99** | 0.75 | **7.1%** |
+| waxal_ibo_5_female | Phase 3 | 0.164 | 0.132 | 0.683 | 1.00 | 0.74 | 22.8% |
+| waxal_ibo_5_female | **Flash** | 0.165 | 0.134 | **0.700** | 1.03 | **0.95** | **10.7%** |
+
+- Flash lowers the anchor's CER (0.174 → 0.162) and clipped endings (9.1% → 7.1%), brings its pace to about native (0.92× → 0.99×), and more than halves clipped endings on the female Igbo voice (22.8% → 10.7%) at about native pace and pause rate.
+- Per language with the anchor, Flash CER: Igbo 0.189 (Phase 3: 0.210), Yoruba 0.230 (0.243), Hausa 0.123 (0.146), English 0.026 (0.039), paragraphs 0.131 (0.168), code-switch 0.208 (0.212); Pidgin 0.227 (0.209) is worse, and the Pidgin judge is weak anyway.
+- **Known weakness:** with `anchor_male`, Flash **under-pauses in Igbo (0.40× native pauses per second) and Yoruba (0.30×)**. Long Igbo or Yoruba passages with the male voice can sound rushed; use a female voice or `--compare` with `phase3` there.
+- Speaker similarity of the anchor drops slightly (0.815 → 0.784), because Flash uses the stock CV3 flow zero-shot instead of the anchor-SFT flow.
+- **UTMOS caveat:** UTMOS22 rates real native speech (3.53) below every synthetic system (3.8–4.3), so it rewards clean studio-like synthesis rather than nativeness. It was not used to choose the model (Phase 3 has the highest UTMOS22: 4.15 anchor / 4.27 female; Flash 3.89 / 4.16; Flash UTMOSv2 3.17 / 3.51). The choice rests on CER, pace and pauses relative to native speech, and clipped endings. Native-speaker MOS / blind A/B is still pending.
+- Text front-end: diacritic restoration word accuracy Yoruba 0.862 (stripped input: 0.171), Igbo 0.968 (0.640).
+
 ## Evaluation: Phase 2 vs Phase 3
 
 183-item eval set (30 per language, 10 code-switched, 3 hand-written + 20 native long-form paragraphs), anchor voice, seed 0. CER is ASR round-trip: MMS-1b-all for Igbo/Hausa/Yoruba, Whisper-large-v3-turbo for English and Pidgin, the lower of the two for code-switched items and paragraphs. "No tones" (in brackets) strips diacritics. SIM is ECAPA cosine to the anchor (ceiling about 0.84 on real held-out anchor clips). The Pidgin judge is weak: real Pidgin recordings score about 0.43.
@@ -221,7 +275,8 @@ On the held-out single-language set, per language (rule-based / + GlotLID): Engl
 
 ## Limitations
 
-- **Pace**: speech is still 13–28% slower than native speakers (Igbo 4.60 vs 5.30 syl/s, Hausa 3.77 vs 5.26 syl/s), and there are about half as many pauses per second as in real speech. It can sound careful rather than fully native.
+- **Flash** (`flash`): the male anchor under-pauses in Igbo and Yoruba (0.40× / 0.30× native pause rate); Pidgin CER is slightly worse than Phase 3; CUDA only; the translate-first mode is experimental and off by default (MADLAD-400 loops on long inputs and is weak on Pidgin); diacritic restoration can still pick the wrong tone on ambiguous words, so write marks yourself where you can. The limitations below for Phase 3 (no laughter, Pidgin data, accent after a switch) apply to Flash too; the pace gap is largely closed for Flash.
+- **Pace (`phase2`/`phase3`)**: speech is still 13–28% slower than native speakers (Igbo 4.60 vs 5.30 syl/s, Hausa 3.77 vs 5.26 syl/s), and there are about half as many pauses per second as in real speech. It can sound careful rather than fully native.
 - **Pidgin is the weakest language**: only about 5 h of training data, no reliable ASR judge, and the auto-tagger's Pidgin/English split is its weakest point (~0.63 word accuracy on plain Pidgin).
 - **No laughter or other expressive events**: the training data has effectively no laughter, so tags like `[laughter]` do nothing useful. `[pause]`/`[breath]` were learned from data, but there is no reliable user control over them yet.
 - English after a Nigerian-language span keeps a strong local accent.
@@ -234,13 +289,17 @@ On the held-out single-language set, per language (rule-based / + GlotLID): Engl
 
 | path | what it is | licence |
 |---|---|---|
+| `checkpoints/flash/flash_dpo/` | Flash LM (`best.pt`, sha256 `f59a6fc0…7e3a307`, CosyVoice3 LM + language/speaker/rate embeddings), `config.json`, `spk2id.json`, DPO `args.json`, `log.jsonl` | CC-BY-SA-4.0 |
+| `checkpoints/flash/diacritics_byt5/` | Yoruba/Igbo diacritic-restoration model (fine-tune of [google/byt5-small](https://huggingface.co/google/byt5-small), Apache-2.0) used by the Flash front-end | CC-BY-SA-4.0 |
 | `checkpoints/phase3/p3_lm_full/` | Phase 3 LM (`best.pt`, val step 4,500), `config.json` (rate, sampler, one-pass, points at the shared Phase 2 flow), `spk2id.json`, `args.json`, `log.jsonl` | CC-BY-SA-4.0 |
 | `checkpoints/phase2/p2_lm_B/` | Phase 2 LM (`best.pt`) and `spk2id.json` | CC-BY-SA-4.0 |
 | `checkpoints/phase2/p2_flow_sft/best.pt` | anchor-SFT flow-matching decoder, used by both versions | CC-BY-SA-4.0 |
 | `prompts/` | anchor prompt clips (`clip0..3`, `prompts.json` with transcripts and S3 tokens) and female voice prompts (`voice_*.wav`, `voices.json`), all from WAXAL | CC-BY-SA-4.0 |
-| `samples/` | example renders | CC-BY-SA-4.0 |
+| `samples/` | example renders (`samples/flash/` for Flash) | CC-BY-SA-4.0 |
 | `settings.json` | versions, voices, recommended pairing | Apache-2.0 |
 | `infer.py`, `tagger.py`, `app.py` | inference CLI, auto-tagger, Gradio app | Apache-2.0 |
+| `flash_lm.py`, `frontend/` | Flash model and synthesis code (training code for reference; the data manifest is not shipped), Flash text front-end (normalisation, diacritics, experimental MT) | Apache-2.0 |
+| `prompts/flash_voices_text.json` | transcripts of the voice clips, used as Flash LM prompts | CC-BY-SA-4.0 |
 | `code/` | the original training and synthesis code, for reference | Apache-2.0 |
 | `requirements.txt`, `cosyvoice_requirements.txt` | dependencies (`cosyvoice_requirements.txt` is CosyVoice's own file at the commit used) | – |
 | `assets/` | banner | – |
@@ -255,22 +314,24 @@ The model weights are trained on, and the voice prompts are taken from, the data
 | [Open.Bible Igbo](https://huggingface.co/datasets/AfriSpeech/open-bible-speech-african) (Biblica® Open Igbo Contemporary Bible™, Audio Edition) | Igbo (Phase 3) | CC-BY-SA-4.0 | Biblica, Inc. via open.bible; Biblica® and Open Igbo Contemporary Bible™ are trademarks of Biblica, Inc. |
 | [NaijaS2ST](https://huggingface.co/datasets/McGill-NLP/NaijaS2ST) (Igbo) | Igbo, many native speakers (Phase 3) | CC-BY-4.0 | McGill-NLP, NaijaS2ST (arXiv:2604.16287) |
 | [IgboAPI](https://huggingface.co/datasets/nkowaokwu/ibo-dict) (ibo-dict, ibo-dict-expansion) | Igbo sentences (Phase 3) | CC-BY-4.0 | Nkọwa okwu / IgboAPI contributors |
-| [IgboSynCorp](https://doi.org/10.7910/dvn/yb9fwk) | natural Igbo speech (Phase 3) | CC0-1.0 | University of Ibadan and Afe Babalola University (Lacuna Fund) |
+| [IgboSynCorp](https://doi.org/10.7910/dvn/yb9fwk) | natural Igbo speech (Phase 3); Flash adds pseudo-transcripts of its untranscribed audio | CC0-1.0 | University of Ibadan and Afe Babalola University (Lacuna Fund) |
 | [FLEURS](https://huggingface.co/datasets/google/fleurs) (English, Hausa, Igbo, Yoruba) | multi-speaker data | CC-BY-4.0 | Google, FLEURS (Conneau et al., 2022) |
 | [BibleTTS](https://masakhane-io.github.io/bibleTTS/) (Yoruba, Hausa) | Yoruba and Hausa | CC-BY-SA-4.0 | Meyer et al., BibleTTS (2022); Biblica / open.bible audio |
 | [OpenSLR SLR70](https://openslr.org/70/) (Nigerian English) | English | CC-BY-SA-4.0 | Google, crowdsourced Nigerian English |
 | [OpenSLR SLR86](https://openslr.org/86/) (Yoruba) | Yoruba | CC-BY-SA-4.0 | Google, crowdsourced Yoruba |
 | code-switched pairs | about 9 h of synthetic code-switch training pairs | follows the source clips (CC-BY-SA-4.0 / CC-BY-4.0) | built by concatenating clips from the sets above, following the Google 2019 code-switching recipe (Zhang et al., 2019); no separate Google dataset |
+| base model [FunAudioLLM/Fun-CosyVoice3-0.5B-2512](https://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512) | Flash: LM initialisation, flow, vocoder, tokenizers | Apache-2.0 | FunAudioLLM / Alibaba Tongyi Lab, CosyVoice 3 |
+| [google/byt5-small](https://huggingface.co/google/byt5-small) | base of the Flash diacritic-restoration model (fine-tuned on transcripts of the datasets above) | Apache-2.0 | Google, ByT5 (Xue et al., 2022) |
 | base model [FunAudioLLM/CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B) | LM initialisation, flow, vocoder, tokenizers | Apache-2.0 | FunAudioLLM / Alibaba Tongyi Lab, CosyVoice 2 (Du et al., 2024) |
 | optional [GlotLID](https://huggingface.co/cis-lmu/glotlid) (not bundled) | `--lid glotlid` in the tagger | Apache-2.0 plus notices | Kargaran et al., GlotLID (EMNLP 2023) |
 
-Excluded on licence grounds: WaZoBiaSpeech (bans TTS use), NaijaVoices (non-commercial), and non-commercial TTS weights.
+Flash's training data is the licensed set above (the same sources as Phase 3, re-weighted with less Bible narration and slow read speech, plus IgboSynCorp pseudo-transcripts); its DPO pairs are the model's own renders of eval-style prompts, scored automatically. Excluded on licence grounds from every released checkpoint, Flash included: African Voices / WaZoBiaSpeech (its cards ban TTS use), NaijaVoices (non-commercial), and non-commercial TTS weights. The optional MADLAD-400-3B-MT translator (Apache-2.0) is not bundled; NLLB was avoided because it is non-commercial.
 
 ## Licence
 
 - **Model weights, voice prompts and samples** (`checkpoints/`, `prompts/`, `samples/`): **[CC-BY-SA-4.0](LICENSE)**. You may use them commercially. You must credit Modular and the datasets listed above, and anything you derive from the weights (fine-tunes, merges) must be shared under CC-BY-SA-4.0 too.
-- **Code** (`infer.py`, `tagger.py`, `app.py`, `code/`, `settings.json`): **[Apache-2.0](LICENSE-CODE)**.
-- The base model, CosyVoice2-0.5B, is Apache-2.0; its notices still apply to the parts it contributes.
+- **Code** (`infer.py`, `tagger.py`, `app.py`, `flash_lm.py`, `frontend/`, `code/`, `settings.json`): **[Apache-2.0](LICENSE-CODE)**.
+- The base models, CosyVoice2-0.5B and Fun-CosyVoice3-0.5B-2512 (and byt5-small for the diacritic model), are Apache-2.0; their notices still apply to the parts they contribute.
 
 ## Citation
 

app.py

@@ -5,8 +5,8 @@
     python app.py --cosyvoice_dir ../CosyVoice  # then open http://127.0.0.1:7860
 
 Text box -> optional auto-tag (editable preview of the tagged text) -> version + voice + rate -> Synthesize, or
-Compare to hear Phase 2 and Phase 3 on the same text, seed and voice. Both versions stay loaded (one shared CosyVoice2
-base; each version has its own LM copy), so a GPU with >= 16 GB is recommended. CPU works but is slow.
+Compare to hear Phase 3 and Flash on the same text, seed and voice. Loaded versions stay resident (Phase 2/3 share one
+CosyVoice2 base; Flash has its own CosyVoice3 base), so a GPU with >= 24 GB is recommended. Flash needs CUDA.
 """
 import argparse, json, os, sys, types
 
@@ -35,7 +35,8 @@ class Engine:
             ns = types.SimpleNamespace(**vars(self.a))
             ns.voice = voice
             S = infer.build(version, ns, self.cosy, self.tok, copy_llm=self.cosy is not None)
-            self.cosy, self.tok = S.cosy, S.tok
+            if S.cosy is not None:                 # flash (CosyVoice3) keeps its own base
+                self.cosy, self.tok = S.cosy, S.tok
             S._rate0 = S.rate
             self.models[version] = S
         S = self.models[version]
@@ -65,7 +66,9 @@ def main():
     ap.add_argument("--share", action="store_true")
     a = ap.parse_args()
     for k, v in dict(lm="", flow="", prompts=os.path.join(HERE, "prompts", "prompts.json"),
-                     voices=os.path.join(HERE, "prompts", "voices.json"), dtype="auto", stitch=False, rate="default").items():
+                     voices=os.path.join(HERE, "prompts", "voices.json"), dtype="auto", stitch=False, rate="default",
+                     frontend="codeswitch", frontend_lang="eng", no_diacritics=False, experimental_translate=False,
+                     cv3_dir=os.environ.get("CV3_DIR", os.path.join(HERE, "pretrained", "Fun-CosyVoice3-0.5B-2512"))).items():
         setattr(a, k, v)
     if not os.path.isdir(a.model_dir):
         from huggingface_hub import snapshot_download
@@ -98,9 +101,9 @@ def main():
 
     def compare(text, tagged, auto, lang_map, use_lid, force, voice, rate, seed):
         t = tagged.strip() if tagged.strip() else prepare(text, auto, lang_map, use_lid, force)
-        a2 = E.say(t, "phase2", voice, rate, seed)
         a3 = E.say(t, "phase3", voice, rate, seed)
-        return t, a2, a3
+        af = E.say(t, "flash", voice, rate, seed)
+        return t, a3, af
 
     with gr.Blocks(title="Dynamic Accent: code-switching TTS") as demo:
         gr.Markdown("## Dynamic Accent: code-switching TTS\nEnglish, Nigerian Pidgin, Igbo, Yoruba and Hausa, mixed in one "
@@ -114,18 +117,18 @@ def main():
         tag_btn = gr.Button("Preview tags")
         tagged = gr.Textbox(label="Tagged text (edit before synthesis; leave empty to tag automatically)", lines=3)
         with gr.Row():
-            version = gr.Dropdown(sorted(infer.VERSIONS), value=S.get("default_version", "phase3"), label="Version")
+            version = gr.Dropdown(sorted(infer.VERSIONS), value=S.get("default_version", "flash"), label="Version")
             voice = gr.Dropdown(voices, value=S.get("default_voice", "anchor_male"), label="Voice")
-            rate = gr.Dropdown(rates, value="default", label="Rate (phase3)")
+            rate = gr.Dropdown(rates, value="default", label="Rate (phase3/flash)")
             seed = gr.Number(value=0, precision=0, label="Seed")
         tip = gr.Markdown()
         with gr.Row():
             go = gr.Button("Synthesize", variant="primary")
-            cmp_btn = gr.Button("Compare Phase 2 vs Phase 3")
+            cmp_btn = gr.Button("Compare Phase 3 vs Flash")
         out = gr.Audio(label="Output", type="numpy")
         with gr.Row():
-            out2 = gr.Audio(label="Phase 2", type="numpy")
-            out3 = gr.Audio(label="Phase 3", type="numpy")
+            out2 = gr.Audio(label="Phase 3", type="numpy")
+            out3 = gr.Audio(label="Flash", type="numpy")
         gr.Examples(EXAMPLES, inputs=text)
 
         tag_btn.click(prepare, [text, auto, lang_map, use_lid, force], tagged)

checkpoints/flash/diacritics_byt5/added_tokens.json

@@ -0,0 +1,127 @@
+{
+  "<extra_id_0>": 259,
+  "<extra_id_100>": 359,
+  "<extra_id_101>": 360,
+  "<extra_id_102>": 361,
+  "<extra_id_103>": 362,
+  "<extra_id_104>": 363,
+  "<extra_id_105>": 364,
+  "<extra_id_106>": 365,
+  "<extra_id_107>": 366,
+  "<extra_id_108>": 367,
+  "<extra_id_109>": 368,
+  "<extra_id_10>": 269,
+  "<extra_id_110>": 369,
+  "<extra_id_111>": 370,
+  "<extra_id_112>": 371,
+  "<extra_id_113>": 372,
+  "<extra_id_114>": 373,
+  "<extra_id_115>": 374,
+  "<extra_id_116>": 375,
+  "<extra_id_117>": 376,
+  "<extra_id_118>": 377,
+  "<extra_id_119>": 378,
+  "<extra_id_11>": 270,
+  "<extra_id_120>": 379,
+  "<extra_id_121>": 380,
+  "<extra_id_122>": 381,
+  "<extra_id_123>": 382,
+  "<extra_id_124>": 383,
+  "<extra_id_12>": 271,
+  "<extra_id_13>": 272,
+  "<extra_id_14>": 273,
+  "<extra_id_15>": 274,
+  "<extra_id_16>": 275,
+  "<extra_id_17>": 276,
+  "<extra_id_18>": 277,
+  "<extra_id_19>": 278,
+  "<extra_id_1>": 260,
+  "<extra_id_20>": 279,
+  "<extra_id_21>": 280,
+  "<extra_id_22>": 281,
+  "<extra_id_23>": 282,
+  "<extra_id_24>": 283,
+  "<extra_id_25>": 284,
+  "<extra_id_26>": 285,
+  "<extra_id_27>": 286,
+  "<extra_id_28>": 287,
+  "<extra_id_29>": 288,
+  "<extra_id_2>": 261,
+  "<extra_id_30>": 289,
+  "<extra_id_31>": 290,
+  "<extra_id_32>": 291,
+  "<extra_id_33>": 292,
+  "<extra_id_34>": 293,
+  "<extra_id_35>": 294,
+  "<extra_id_36>": 295,
+  "<extra_id_37>": 296,
+  "<extra_id_38>": 297,
+  "<extra_id_39>": 298,
+  "<extra_id_3>": 262,
+  "<extra_id_40>": 299,
+  "<extra_id_41>": 300,
+  "<extra_id_42>": 301,
+  "<extra_id_43>": 302,
+  "<extra_id_44>": 303,
+  "<extra_id_45>": 304,
+  "<extra_id_46>": 305,
+  "<extra_id_47>": 306,
+  "<extra_id_48>": 307,
+  "<extra_id_49>": 308,
+  "<extra_id_4>": 263,
+  "<extra_id_50>": 309,
+  "<extra_id_51>": 310,
+  "<extra_id_52>": 311,
+  "<extra_id_53>": 312,
+  "<extra_id_54>": 313,
+  "<extra_id_55>": 314,
+  "<extra_id_56>": 315,
+  "<extra_id_57>": 316,
+  "<extra_id_58>": 317,
+  "<extra_id_59>": 318,
+  "<extra_id_5>": 264,
+  "<extra_id_60>": 319,
+  "<extra_id_61>": 320,
+  "<extra_id_62>": 321,
+  "<extra_id_63>": 322,
+  "<extra_id_64>": 323,
+  "<extra_id_65>": 324,
+  "<extra_id_66>": 325,
+  "<extra_id_67>": 326,
+  "<extra_id_68>": 327,
+  "<extra_id_69>": 328,
+  "<extra_id_6>": 265,
+  "<extra_id_70>": 329,
+  "<extra_id_71>": 330,
+  "<extra_id_72>": 331,
+  "<extra_id_73>": 332,
+  "<extra_id_74>": 333,
+  "<extra_id_75>": 334,
+  "<extra_id_76>": 335,
+  "<extra_id_77>": 336,
+  "<extra_id_78>": 337,
+  "<extra_id_79>": 338,
+  "<extra_id_7>": 266,
+  "<extra_id_80>": 339,
+  "<extra_id_81>": 340,
+  "<extra_id_82>": 341,
+  "<extra_id_83>": 342,
+  "<extra_id_84>": 343,
+  "<extra_id_85>": 344,
+  "<extra_id_86>": 345,
+  "<extra_id_87>": 346,
+  "<extra_id_88>": 347,
+  "<extra_id_89>": 348,
+  "<extra_id_8>": 267,
+  "<extra_id_90>": 349,
+  "<extra_id_91>": 350,
+  "<extra_id_92>": 351,
+  "<extra_id_93>": 352,
+  "<extra_id_94>": 353,
+  "<extra_id_95>": 354,
+  "<extra_id_96>": 355,
+  "<extra_id_97>": 356,
+  "<extra_id_98>": 357,
+  "<extra_id_99>": 358,
+  "<extra_id_9>": 268
+}

checkpoints/flash/diacritics_byt5/config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "T5ForConditionalGeneration"
+  ],
+  "classifier_dropout": 0.0,
+  "d_ff": 3584,
+  "d_kv": 64,
+  "d_model": 1472,
+  "decoder_start_token_id": 0,
+  "dense_act_fn": "gelu_new",
+  "dropout_rate": 0.1,
+  "eos_token_id": 1,
+  "feed_forward_proj": "gated-gelu",
+  "gradient_checkpointing": false,
+  "initializer_factor": 1.0,
+  "is_encoder_decoder": true,
+  "is_gated_act": true,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "t5",
+  "num_decoder_layers": 4,
+  "num_heads": 6,
+  "num_layers": 12,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "tie_word_embeddings": false,
+  "tokenizer_class": "ByT5Tokenizer",
+  "torch_dtype": "float32",
+  "transformers_version": "4.51.3",
+  "use_cache": true,
+  "vocab_size": 384
+}

checkpoints/flash/diacritics_byt5/generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "decoder_start_token_id": 0,
+  "eos_token_id": 1,
+  "pad_token_id": 0,
+  "transformers_version": "4.51.3"
+}

checkpoints/flash/diacritics_byt5/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27c2f5f06306a51af9f6107c38aebff2606d7589cd4f870a4478c7b7047f789e
+size 1198571496

checkpoints/flash/diacritics_byt5/special_tokens_map.json

@@ -0,0 +1,150 @@
+{
+  "additional_special_tokens": [
+    "<extra_id_0>",
+    "<extra_id_1>",
+    "<extra_id_2>",
+    "<extra_id_3>",
+    "<extra_id_4>",
+    "<extra_id_5>",
+    "<extra_id_6>",
+    "<extra_id_7>",
+    "<extra_id_8>",
+    "<extra_id_9>",
+    "<extra_id_10>",
+    "<extra_id_11>",
+    "<extra_id_12>",
+    "<extra_id_13>",
+    "<extra_id_14>",
+    "<extra_id_15>",
+    "<extra_id_16>",
+    "<extra_id_17>",
+    "<extra_id_18>",
+    "<extra_id_19>",
+    "<extra_id_20>",
+    "<extra_id_21>",
+    "<extra_id_22>",
+    "<extra_id_23>",
+    "<extra_id_24>",
+    "<extra_id_25>",
+    "<extra_id_26>",
+    "<extra_id_27>",
+    "<extra_id_28>",
+    "<extra_id_29>",
+    "<extra_id_30>",
+    "<extra_id_31>",
+    "<extra_id_32>",
+    "<extra_id_33>",
+    "<extra_id_34>",
+    "<extra_id_35>",
+    "<extra_id_36>",
+    "<extra_id_37>",
+    "<extra_id_38>",
+    "<extra_id_39>",
+    "<extra_id_40>",
+    "<extra_id_41>",
+    "<extra_id_42>",
+    "<extra_id_43>",
+    "<extra_id_44>",
+    "<extra_id_45>",
+    "<extra_id_46>",
+    "<extra_id_47>",
+    "<extra_id_48>",
+    "<extra_id_49>",
+    "<extra_id_50>",
+    "<extra_id_51>",
+    "<extra_id_52>",
+    "<extra_id_53>",
+    "<extra_id_54>",
+    "<extra_id_55>",
+    "<extra_id_56>",
+    "<extra_id_57>",
+    "<extra_id_58>",
+    "<extra_id_59>",
+    "<extra_id_60>",
+    "<extra_id_61>",
+    "<extra_id_62>",
+    "<extra_id_63>",
+    "<extra_id_64>",
+    "<extra_id_65>",
+    "<extra_id_66>",
+    "<extra_id_67>",
+    "<extra_id_68>",
+    "<extra_id_69>",
+    "<extra_id_70>",
+    "<extra_id_71>",
+    "<extra_id_72>",
+    "<extra_id_73>",
+    "<extra_id_74>",
+    "<extra_id_75>",
+    "<extra_id_76>",
+    "<extra_id_77>",
+    "<extra_id_78>",
+    "<extra_id_79>",
+    "<extra_id_80>",
+    "<extra_id_81>",
+    "<extra_id_82>",
+    "<extra_id_83>",
+    "<extra_id_84>",
+    "<extra_id_85>",
+    "<extra_id_86>",
+    "<extra_id_87>",
+    "<extra_id_88>",
+    "<extra_id_89>",
+    "<extra_id_90>",
+    "<extra_id_91>",
+    "<extra_id_92>",
+    "<extra_id_93>",
+    "<extra_id_94>",
+    "<extra_id_95>",
+    "<extra_id_96>",
+    "<extra_id_97>",
+    "<extra_id_98>",
+    "<extra_id_99>",
+    "<extra_id_100>",
+    "<extra_id_101>",
+    "<extra_id_102>",
+    "<extra_id_103>",
+    "<extra_id_104>",
+    "<extra_id_105>",
+    "<extra_id_106>",
+    "<extra_id_107>",
+    "<extra_id_108>",
+    "<extra_id_109>",
+    "<extra_id_110>",
+    "<extra_id_111>",
+    "<extra_id_112>",
+    "<extra_id_113>",
+    "<extra_id_114>",
+    "<extra_id_115>",
+    "<extra_id_116>",
+    "<extra_id_117>",
+    "<extra_id_118>",
+    "<extra_id_119>",
+    "<extra_id_120>",
+    "<extra_id_121>",
+    "<extra_id_122>",
+    "<extra_id_123>",
+    "<extra_id_124>"
+  ],
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoints/flash/diacritics_byt5/tokenizer_config.json

@@ -0,0 +1,1163 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "259": {
+      "content": "<extra_id_0>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "260": {
+      "content": "<extra_id_1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261": {
+      "content": "<extra_id_2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "262": {
+      "content": "<extra_id_3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "263": {
+      "content": "<extra_id_4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "264": {
+      "content": "<extra_id_5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "265": {
+      "content": "<extra_id_6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "266": {
+      "content": "<extra_id_7>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "267": {
+      "content": "<extra_id_8>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "268": {
+      "content": "<extra_id_9>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "269": {
+      "content": "<extra_id_10>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "270": {
+      "content": "<extra_id_11>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "271": {
+      "content": "<extra_id_12>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "272": {
+      "content": "<extra_id_13>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "273": {
+      "content": "<extra_id_14>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "274": {
+      "content": "<extra_id_15>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "275": {
+      "content": "<extra_id_16>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "276": {
+      "content": "<extra_id_17>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "277": {
+      "content": "<extra_id_18>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "278": {
+      "content": "<extra_id_19>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "279": {
+      "content": "<extra_id_20>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "280": {
+      "content": "<extra_id_21>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "281": {
+      "content": "<extra_id_22>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "282": {
+      "content": "<extra_id_23>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "283": {
+      "content": "<extra_id_24>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "284": {
+      "content": "<extra_id_25>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "285": {
+      "content": "<extra_id_26>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "286": {
+      "content": "<extra_id_27>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "287": {
+      "content": "<extra_id_28>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "288": {
+      "content": "<extra_id_29>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "289": {
+      "content": "<extra_id_30>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "290": {
+      "content": "<extra_id_31>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "291": {
+      "content": "<extra_id_32>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "292": {
+      "content": "<extra_id_33>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "293": {
+      "content": "<extra_id_34>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "294": {
+      "content": "<extra_id_35>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "295": {
+      "content": "<extra_id_36>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "296": {
+      "content": "<extra_id_37>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "297": {
+      "content": "<extra_id_38>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "298": {
+      "content": "<extra_id_39>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "299": {
+      "content": "<extra_id_40>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "300": {
+      "content": "<extra_id_41>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "301": {
+      "content": "<extra_id_42>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "302": {
+      "content": "<extra_id_43>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "303": {
+      "content": "<extra_id_44>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "304": {
+      "content": "<extra_id_45>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "305": {
+      "content": "<extra_id_46>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "306": {
+      "content": "<extra_id_47>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "307": {
+      "content": "<extra_id_48>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "308": {
+      "content": "<extra_id_49>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "309": {
+      "content": "<extra_id_50>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "310": {
+      "content": "<extra_id_51>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "311": {
+      "content": "<extra_id_52>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "312": {
+      "content": "<extra_id_53>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "313": {
+      "content": "<extra_id_54>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "314": {
+      "content": "<extra_id_55>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "315": {
+      "content": "<extra_id_56>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "316": {
+      "content": "<extra_id_57>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "317": {
+      "content": "<extra_id_58>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "318": {
+      "content": "<extra_id_59>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "319": {
+      "content": "<extra_id_60>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "320": {
+      "content": "<extra_id_61>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "321": {
+      "content": "<extra_id_62>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "322": {
+      "content": "<extra_id_63>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "323": {
+      "content": "<extra_id_64>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "324": {
+      "content": "<extra_id_65>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "325": {
+      "content": "<extra_id_66>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "326": {
+      "content": "<extra_id_67>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "327": {
+      "content": "<extra_id_68>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "328": {
+      "content": "<extra_id_69>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "329": {
+      "content": "<extra_id_70>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "330": {
+      "content": "<extra_id_71>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "331": {
+      "content": "<extra_id_72>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "332": {
+      "content": "<extra_id_73>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "333": {
+      "content": "<extra_id_74>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "334": {
+      "content": "<extra_id_75>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "335": {
+      "content": "<extra_id_76>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "336": {
+      "content": "<extra_id_77>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "337": {
+      "content": "<extra_id_78>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "338": {
+      "content": "<extra_id_79>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "339": {
+      "content": "<extra_id_80>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "340": {
+      "content": "<extra_id_81>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "341": {
+      "content": "<extra_id_82>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "342": {
+      "content": "<extra_id_83>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "343": {
+      "content": "<extra_id_84>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "344": {
+      "content": "<extra_id_85>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "345": {
+      "content": "<extra_id_86>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "346": {
+      "content": "<extra_id_87>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "347": {
+      "content": "<extra_id_88>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "348": {
+      "content": "<extra_id_89>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "349": {
+      "content": "<extra_id_90>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "350": {
+      "content": "<extra_id_91>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "351": {
+      "content": "<extra_id_92>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "352": {
+      "content": "<extra_id_93>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "353": {
+      "content": "<extra_id_94>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "354": {
+      "content": "<extra_id_95>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "355": {
+      "content": "<extra_id_96>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "356": {
+      "content": "<extra_id_97>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "357": {
+      "content": "<extra_id_98>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "358": {
+      "content": "<extra_id_99>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "359": {
+      "content": "<extra_id_100>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "360": {
+      "content": "<extra_id_101>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "361": {
+      "content": "<extra_id_102>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "362": {
+      "content": "<extra_id_103>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "363": {
+      "content": "<extra_id_104>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "364": {
+      "content": "<extra_id_105>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "365": {
+      "content": "<extra_id_106>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "366": {
+      "content": "<extra_id_107>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "367": {
+      "content": "<extra_id_108>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "368": {
+      "content": "<extra_id_109>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "369": {
+      "content": "<extra_id_110>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "370": {
+      "content": "<extra_id_111>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "371": {
+      "content": "<extra_id_112>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "372": {
+      "content": "<extra_id_113>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "373": {
+      "content": "<extra_id_114>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "374": {
+      "content": "<extra_id_115>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "375": {
+      "content": "<extra_id_116>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "376": {
+      "content": "<extra_id_117>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "377": {
+      "content": "<extra_id_118>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "378": {
+      "content": "<extra_id_119>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "379": {
+      "content": "<extra_id_120>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "380": {
+      "content": "<extra_id_121>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "381": {
+      "content": "<extra_id_122>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "382": {
+      "content": "<extra_id_123>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "383": {
+      "content": "<extra_id_124>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<extra_id_0>",
+    "<extra_id_1>",
+    "<extra_id_2>",
+    "<extra_id_3>",
+    "<extra_id_4>",
+    "<extra_id_5>",
+    "<extra_id_6>",
+    "<extra_id_7>",
+    "<extra_id_8>",
+    "<extra_id_9>",
+    "<extra_id_10>",
+    "<extra_id_11>",
+    "<extra_id_12>",
+    "<extra_id_13>",
+    "<extra_id_14>",
+    "<extra_id_15>",
+    "<extra_id_16>",
+    "<extra_id_17>",
+    "<extra_id_18>",
+    "<extra_id_19>",
+    "<extra_id_20>",
+    "<extra_id_21>",
+    "<extra_id_22>",
+    "<extra_id_23>",
+    "<extra_id_24>",
+    "<extra_id_25>",
+    "<extra_id_26>",
+    "<extra_id_27>",
+    "<extra_id_28>",
+    "<extra_id_29>",
+    "<extra_id_30>",
+    "<extra_id_31>",
+    "<extra_id_32>",
+    "<extra_id_33>",
+    "<extra_id_34>",
+    "<extra_id_35>",
+    "<extra_id_36>",
+    "<extra_id_37>",
+    "<extra_id_38>",
+    "<extra_id_39>",
+    "<extra_id_40>",
+    "<extra_id_41>",
+    "<extra_id_42>",
+    "<extra_id_43>",
+    "<extra_id_44>",
+    "<extra_id_45>",
+    "<extra_id_46>",
+    "<extra_id_47>",
+    "<extra_id_48>",
+    "<extra_id_49>",
+    "<extra_id_50>",
+    "<extra_id_51>",
+    "<extra_id_52>",
+    "<extra_id_53>",
+    "<extra_id_54>",
+    "<extra_id_55>",
+    "<extra_id_56>",
+    "<extra_id_57>",
+    "<extra_id_58>",
+    "<extra_id_59>",
+    "<extra_id_60>",
+    "<extra_id_61>",
+    "<extra_id_62>",
+    "<extra_id_63>",
+    "<extra_id_64>",
+    "<extra_id_65>",
+    "<extra_id_66>",
+    "<extra_id_67>",
+    "<extra_id_68>",
+    "<extra_id_69>",
+    "<extra_id_70>",
+    "<extra_id_71>",
+    "<extra_id_72>",
+    "<extra_id_73>",
+    "<extra_id_74>",
+    "<extra_id_75>",
+    "<extra_id_76>",
+    "<extra_id_77>",
+    "<extra_id_78>",
+    "<extra_id_79>",
+    "<extra_id_80>",
+    "<extra_id_81>",
+    "<extra_id_82>",
+    "<extra_id_83>",
+    "<extra_id_84>",
+    "<extra_id_85>",
+    "<extra_id_86>",
+    "<extra_id_87>",
+    "<extra_id_88>",
+    "<extra_id_89>",
+    "<extra_id_90>",
+    "<extra_id_91>",
+    "<extra_id_92>",
+    "<extra_id_93>",
+    "<extra_id_94>",
+    "<extra_id_95>",
+    "<extra_id_96>",
+    "<extra_id_97>",
+    "<extra_id_98>",
+    "<extra_id_99>",
+    "<extra_id_100>",
+    "<extra_id_101>",
+    "<extra_id_102>",
+    "<extra_id_103>",
+    "<extra_id_104>",
+    "<extra_id_105>",
+    "<extra_id_106>",
+    "<extra_id_107>",
+    "<extra_id_108>",
+    "<extra_id_109>",
+    "<extra_id_110>",
+    "<extra_id_111>",
+    "<extra_id_112>",
+    "<extra_id_113>",
+    "<extra_id_114>",
+    "<extra_id_115>",
+    "<extra_id_116>",
+    "<extra_id_117>",
+    "<extra_id_118>",
+    "<extra_id_119>",
+    "<extra_id_120>",
+    "<extra_id_121>",
+    "<extra_id_122>",
+    "<extra_id_123>",
+    "<extra_id_124>"
+  ],
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_ids": 0,
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "ByT5Tokenizer",
+  "unk_token": "<unk>"
+}

checkpoints/flash/flash_dpo/args.json

@@ -0,0 +1 @@
+{"cmd": "train", "lm": "/home/dev/flash/runs/flash_lm/best.pt", "out": "/home/dev/flash/runs/flash_dpo", "shard": "0/1", "k": 4, "lr": 1e-06, "beta": 0.1, "sft": 0.1, "bs": 8, "epochs": 2}

checkpoints/flash/flash_dpo/best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f59a6fc0dacc331844cf8b513ec99e4d5c75f2583115037c8d38ba9ff7e3a307
+size 2047760470

checkpoints/flash/flash_dpo/config.json

@@ -0,0 +1,9 @@
+{
+ "engine": "cosyvoice3",
+ "base": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
+ "rate": "normal",
+ "sampler": "silexcap",
+ "one_pass": true,
+ "sha256_best_pt": "f59a6fc0dacc331844cf8b513ec99e4d5c75f2583115037c8d38ba9ff7e3a307",
+ "training": "Flash SFT (CV3 LM, Phase 3 licensed manifest re-weighted + IgboSynCorp pseudo-transcripts) then DPO: 1,206 preference pairs, 286 steps, beta 0.1, lr 1e-6, val pref-acc 0.65"
+}

checkpoints/flash/flash_dpo/log.jsonl

@@ -0,0 +1,31 @@
+{"step": 0, "val": {"acc": 0.0, "margin": 0.0}}
+{"step": 10, "of": 286, "loss": 0.9068, "acc": 0.625, "margin": 0.046, "gn": 57.21, "min": 0.1}
+{"step": 20, "of": 286, "loss": 0.9388, "acc": 0.5, "margin": -0.002, "gn": 66.55, "min": 0.1}
+{"step": 30, "of": 286, "loss": 0.9699, "acc": 0.375, "margin": -0.035, "gn": 61.96, "min": 0.2}
+{"step": 40, "of": 286, "loss": 0.938, "acc": 0.375, "margin": 0.011, "gn": 63.67, "min": 0.3}
+{"step": 50, "of": 286, "loss": 0.8633, "acc": 0.875, "margin": 0.15, "gn": 52.25, "min": 0.3}
+{"step": 60, "of": 286, "loss": 0.939, "acc": 0.625, "margin": 0.002, "gn": 63.83, "min": 0.4}
+{"step": 70, "of": 286, "loss": 0.8483, "acc": 0.75, "margin": 0.189, "gn": 69.48, "min": 0.4}
+{"step": 80, "of": 286, "loss": 1.0202, "acc": 0.375, "margin": -0.139, "gn": 80.15, "min": 0.5}
+{"step": 90, "of": 286, "loss": 0.8866, "acc": 0.625, "margin": 0.142, "gn": 59.47, "min": 0.5}
+{"step": 100, "of": 286, "loss": 0.8947, "acc": 0.625, "margin": 0.134, "gn": 60.78, "min": 0.6}
+{"step": 110, "of": 286, "loss": 0.9459, "acc": 0.5, "margin": 0.027, "gn": 66.75, "min": 0.6}
+{"step": 120, "of": 286, "loss": 1.0125, "acc": 0.375, "margin": -0.11, "gn": 65.48, "min": 0.7}
+{"step": 130, "of": 286, "loss": 0.8009, "acc": 0.75, "margin": 0.365, "gn": 50.92, "min": 0.8}
+{"step": 140, "of": 286, "loss": 0.8328, "acc": 0.75, "margin": 0.277, "gn": 60.21, "min": 0.8}
+{"epoch": 1, "step": 143, "val": {"acc": 0.6, "margin": 0.15548671048794252}}
+{"step": 150, "of": 286, "loss": 0.5242, "acc": 1.0, "margin": 1.345, "gn": 31.71, "min": 0.9}
+{"step": 160, "of": 286, "loss": 0.476, "acc": 1.0, "margin": 1.509, "gn": 28.95, "min": 0.9}
+{"step": 170, "of": 286, "loss": 0.5501, "acc": 1.0, "margin": 1.171, "gn": 29.98, "min": 1.0}
+{"step": 180, "of": 286, "loss": 0.474, "acc": 1.0, "margin": 1.492, "gn": 26.95, "min": 1.0}
+{"step": 190, "of": 286, "loss": 0.5711, "acc": 1.0, "margin": 1.032, "gn": 34.13, "min": 1.1}
+{"step": 200, "of": 286, "loss": 0.5283, "acc": 1.0, "margin": 1.181, "gn": 32.39, "min": 1.1}
+{"step": 210, "of": 286, "loss": 0.6212, "acc": 0.875, "margin": 0.86, "gn": 36.1, "min": 1.2}
+{"step": 220, "of": 286, "loss": 0.5229, "acc": 1.0, "margin": 1.227, "gn": 32.65, "min": 1.2}
+{"step": 230, "of": 286, "loss": 0.5555, "acc": 1.0, "margin": 1.367, "gn": 35.05, "min": 1.3}
+{"step": 240, "of": 286, "loss": 0.5312, "acc": 1.0, "margin": 1.314, "gn": 36.6, "min": 1.3}
+{"step": 250, "of": 286, "loss": 0.493, "acc": 1.0, "margin": 1.513, "gn": 28.32, "min": 1.3}
+{"step": 260, "of": 286, "loss": 0.5076, "acc": 1.0, "margin": 1.31, "gn": 25.17, "min": 1.4}
+{"step": 270, "of": 286, "loss": 0.434, "acc": 1.0, "margin": 1.739, "gn": 23.64, "min": 1.4}
+{"step": 280, "of": 286, "loss": 0.4263, "acc": 1.0, "margin": 1.907, "gn": 24.99, "min": 1.5}
+{"epoch": 2, "step": 286, "val": {"acc": 0.65, "margin": 0.17853533582141}}

flash_lm.py

@@ -0,0 +1,532 @@
+"""Dynamic Accent Flash: CosyVoice3 (Fun-CosyVoice3-0.5B-2512, Apache-2.0) LM fine-tune + synthesis.
+
+Same additions as Phase 2/3 (per-token language embedding, learned speaker token after the text, rate embedding added to
+the speaker vector, GRL speaker classifier on text positions, in-context same-speaker prompt p=0.5), ported to the CV3 LM:
+  seq = [sos, "You are a helpful assistant.<|endofprompt|>", (prompt text) text, spk(+rate), task, (prompt speech) speech, eos]
+sos/task live in speech_embedding (CV3), eos = speech_token_size+1. Targets are CV3 speech tokens (s3v3.*).
+Flow (DiT) + HiFT = CV3 base, zero-shot prompted with the voice clip (no flow fine-tune).
+
+  python flash_lm.py lm  --out ~/flash/runs/flash_lm --max_steps 12000 --minutes 75
+  python flash_lm.py gen --lm CKPT --out DIR [--voice anchor|waxal_ibo_5|waxal_yor_5] [--rate normal] [--rows ROWS]
+
+Release note: in this repo use `infer.py --version flash` (it wraps Synth below). Training needs the Phase 3 data
+manifest, which is not shipped; the training code is included for reference. Paths come from COSYVOICE_REPO (CosyVoice
+clone with CosyVoice3 support) and CV3_DIR (Fun-CosyVoice3-0.5B-2512 snapshot).
+"""
+import argparse, json, math, os, random, sys, time, inspect
+import numpy as np, torch, torch.nn as nn, torch.nn.functional as F, soundfile as sf, librosa
+
+HOME = os.path.expanduser("~")
+_HERE = os.path.dirname(os.path.abspath(__file__))
+_CV = os.environ.get("COSYVOICE_REPO") or (f"{_HERE}/CosyVoice" if os.path.isdir(f"{_HERE}/CosyVoice") else f"{HOME}/CosyVoice")
+sys.path[:0] = [_CV, f"{_CV}/third_party/Matcha-TTS", f"{_HERE}/code", f"{HOME}/da"]
+from da.text import segments  # noqa: E402
+from da.model import grad_reverse  # noqa: E402
+
+CV3 = os.environ.get("CV3_DIR") or (f"{_HERE}/pretrained/Fun-CosyVoice3-0.5B-2512" if os.path.isdir(f"{_HERE}/pretrained/Fun-CosyVoice3-0.5B-2512") else f"{HOME}/models/Fun-CosyVoice3-0.5B-2512")
+LANGS = ["eng", "ibo", "hau", "yor", "pcm"]
+L2I = {l: i for i, l in enumerate(LANGS)}
+RATES = [None, "very slow", "slow", "normal", "fast", "very fast"]
+R2I = {r: i for i, r in enumerate(RATES)}
+TAGS = ("[pause]", "[breath]", "[quick_breath]", "[laughter]")
+SIL = {1, 2, 28, 29, 55, 248, 494, 2241, 2242, 2322, 2323}   # CV3 FSQ silent/breath tokens (cosyvoice/cli/model.py)
+ANCHOR = "waxal_yor_4"
+NEW = ["waxal", "igboapi", "open_bible_igbo", "naijas2st_igbo", "igbosyncorp"]
+MAN = f"{HOME}/p3t/manifest_p3.jsonl"
+PDIR = os.environ.get("DA_PROMPTS") or (f"{_HERE}/prompts" if os.path.isdir(f"{_HERE}/prompts") else f"{HOME}/hf_test/prompts")
+VOICES = {
+    "anchor": dict(wav=f"{PDIR}/clip0_waxal_yor_yor_1258.wav" if os.path.exists(f"{PDIR}/clip0_waxal_yor_yor_1258.wav")
+                   else f"{PDIR}/waxal_yor_yor_1258.wav", spk=ANCHOR),
+    "waxal_ibo_5": dict(wav=f"{PDIR}/voice_waxal_ibo_5.wav", spk="waxal_ibo_5"),
+    "waxal_yor_5": dict(wav=f"{PDIR}/voice_waxal_yor_5.wav", spk="waxal_yor_5"),
+}
+# Flash source re-weighting on top of the Phase 3 manifest weight: less Bible narration and slow phone/dictionary reads
+FLASH_W = {"open_bible_igbo": 0.4, "bible_hau": 0.5, "bible_yor": 0.5, "igboapi": 0.6, "naijas2st_igbo": 0.6,
+           "igbosyn_pseudo": 1.0}
+
+
+def strip_tags(t):
+    for g in TAGS:
+        t = t.replace(g, " ")
+    return " ".join(t.split()).replace(" ,", ",").replace(" .", ".")
+
+
+def _sf_load_wav(wav, target_sr, min_sr=16000):
+    y, sr = sf.read(wav, dtype="float32", always_2d=True)
+    y = y.mean(1)
+    if sr != target_sr:
+        y = librosa.resample(y, orig_sr=sr, target_sr=target_sr)
+    return torch.from_numpy(y)[None]
+
+
+def load_cv3():
+    import cosyvoice.utils.file_utils as fu
+    fu.load_wav = _sf_load_wav
+    import cosyvoice.cli.frontend as cf
+    cf.load_wav = _sf_load_wav
+    from cosyvoice.cli.cosyvoice import CosyVoice3
+    kw = dict(load_jit=False, load_trt=False, fp16=False)
+    sig = inspect.signature(CosyVoice3.__init__).parameters
+    return CosyVoice3(CV3, **{k: v for k, v in kw.items() if k in sig})
+
+
+class Tok:
+    def __init__(self, cosy):
+        self.t = cosy.frontend.tokenizer
+        self.prefix = self.t.encode("You are a helpful assistant.") + [151646]
+        self.cache = {}
+
+    def __call__(self, text, default="eng"):
+        if text in self.cache:
+            return self.cache[text]
+        ids, langs = [], []
+        for l, s in segments(text, default):
+            t = self.t.encode(s)
+            ids += t; langs += [L2I[l]] * len(t)
+        self.cache[text] = (ids, langs)
+        return ids, langs
+
+
+class FLM(nn.Module):
+    def __init__(self, q, n_spk, prefix, d=896):
+        super().__init__()
+        self.q = q
+        self.lang_emb = nn.Embedding(len(LANGS), d); nn.init.zeros_(self.lang_emb.weight)
+        self.spk_emb = nn.Embedding(n_spk, d); nn.init.normal_(self.spk_emb.weight, std=0.02)
+        self.rate_emb = nn.Embedding(len(RATES), d, padding_idx=0); nn.init.zeros_(self.rate_emb.weight)
+        self.adv = nn.Sequential(nn.Linear(d, 256), nn.ReLU(), nn.Linear(256, n_spk))
+        self.V = q.speech_token_size
+        self.EOS = q.eos_token
+        self.register_buffer("prefix", torch.tensor(prefix, dtype=torch.long), persistent=False)
+
+    def embed_text(self, ids, langs):
+        return self.q.llm.model.model.embed_tokens(ids) + self.lang_emb(langs)
+
+    def spkvec(self, spk):
+        return self.spk_emb(spk[0]) + self.rate_emb(spk[1])
+
+    def build(self, ids, langs, spk, sp=None, pids=None, plangs=None, psp=None):
+        E = self.q.speech_embedding.weight
+        if pids is not None:
+            ids = torch.cat([pids, ids]); langs = torch.cat([plangs, langs])
+        pre = self.q.llm.model.model.embed_tokens(self.prefix.to(ids.device))
+        parts = [E[self.q.sos][None], pre, self.embed_text(ids, langs), self.spkvec(spk)[None], E[self.q.task_id][None]]
+        npsp = 0
+        if psp is not None:
+            parts.append(self.q.speech_embedding(psp)); npsp = len(psp)
+        if sp is not None:
+            parts.append(self.q.speech_embedding(sp))
+        return torch.cat(parts, 0), len(pre), len(ids), npsp
+
+    def hidden(self, x, mask):
+        return self.q.llm.model.model(inputs_embeds=x, attention_mask=mask, return_dict=True).last_hidden_state
+
+    def pack(self, batch):
+        seqs, tgts, tpos, spks = [], [], [], []
+        for b in batch:
+            ids, langs, spk, sp = b[:4]
+            pr = b[4] if len(b) > 4 else None
+            x, P, nt, npsp = self.build(ids, langs, spk, sp, *(pr if pr else (None, None, None)))
+            tgt = torch.full((x.shape[0] + 1,), -100, device=x.device, dtype=torch.long)
+            start = P + nt + 2 + npsp
+            tgt[start: start + len(sp)] = sp
+            tgt[start + len(sp)] = self.EOS
+            seqs.append(x); tgts.append(tgt[:x.shape[0]]); tpos.append((1 + P, 1 + P + nt)); spks.append(int(spk[0]))
+        Tm = max(s.shape[0] for s in seqs); D = seqs[0].shape[1]
+        X = seqs[0].new_zeros(len(seqs), Tm, D)
+        Y = torch.full((len(seqs), Tm), -100, device=X.device, dtype=torch.long)
+        M = torch.zeros(len(seqs), Tm, device=X.device, dtype=torch.long)
+        for i, (s, t) in enumerate(zip(seqs, tgts)):
+            X[i, :len(s)] = s; Y[i, :len(t)] = t; M[i, :len(s)] = 1
+        return X, Y, M, tpos, spks
+
+    def forward2(self, batch, adv_lambda=1.0):
+        X, Y, M, tpos, spks = self.pack(batch)
+        H = self.hidden(X, M)
+        logits = self.q.llm_decoder(H)
+        ce = F.cross_entropy(logits.float().reshape(-1, logits.shape[-1]), Y.reshape(-1), ignore_index=-100)
+        valid = Y != -100
+        acc = (logits.argmax(-1)[valid] == Y[valid]).float().mean()
+        hs = torch.cat([H[i, a:b] for i, (a, b) in enumerate(tpos)])
+        ys = torch.cat([torch.full((b - a,), spks[i], device=X.device) for i, (a, b) in enumerate(tpos)])
+        al = self.adv(grad_reverse(hs.float(), adv_lambda))
+        return ce, acc, F.cross_entropy(al, ys), (al.argmax(-1) == ys).float().mean()
+
+    def seq_logp(self, batch):
+        """sum log p(speech tokens + eos) per item (for DPO)."""
+        X, Y, M, _, _ = self.pack(batch)
+        logits = self.q.llm_decoder(self.hidden(X, M)).float()
+        lp = logits.log_softmax(-1)
+        valid = Y != -100
+        g = lp.gather(-1, Y.clamp(min=0)[..., None])[..., 0] * valid
+        return g.sum(-1), valid.sum(-1)
+
+    @torch.no_grad()
+    def generate(self, ids, langs, spk, prompt=None, B=1, top_p=0.8, top_k=25, win=10, tau_r=0.1, seed=0,
+                 min_ratio=2.0, max_ratio=30.0, sampler="silexcap", max_tok=900):
+        g = torch.Generator(device=ids.device).manual_seed(seed)
+        x, _, _, _ = self.build(ids, langs, spk, None, *(prompt if prompt else (None, None, None)))
+        inp = x[None].repeat(B, 1, 1)
+        n = len(ids)
+        min_len, max_len = int(n * min_ratio), min(int(n * max_ratio), max_tok)
+        past, outs, done = None, [[] for _ in range(B)], [False] * B
+        E = self.q.speech_embedding.weight
+        for i in range(max_len):
+            o = self.q.llm.model.model(inputs_embeds=inp, past_key_values=past, use_cache=True, return_dict=True)
+            past = o.past_key_values
+            logp = self.q.llm_decoder(o.last_hidden_state[:, -1]).float().log_softmax(-1)
+            eos_lp = logp[:, self.EOS].clone()
+            logp[:, self.V:] = -float("inf")
+            if i >= min_len:
+                logp[:, self.EOS] = eos_lp
+            P = logp.exp()
+            nxt = []
+            for b in range(B):
+                if done[b]:
+                    nxt.append(0); continue
+                p = P[b]; out = outs[b]
+                sp_, si = p.sort(descending=True)
+                keep = (sp_.cumsum(0) - sp_) < top_p
+                keep[top_k:] = False
+                cp, ci = sp_[keep], si[keep]
+                t = int(ci[torch.multinomial(cp / cp.sum(), 1, generator=g)])
+                exempt = False
+                if sampler == "silexcap" and t in SIL:
+                    run = 0
+                    for u in reversed(out):
+                        if u not in SIL:
+                            break
+                        run += 1
+                    exempt = run < 25
+                if not exempt and out and t != self.EOS and sum(1 for u in out[-win:] if u == t) >= win * tau_r:
+                    t = int(torch.multinomial(p, 1, generator=g))
+                if t >= self.V:
+                    done[b] = True; nxt.append(0); continue
+                out.append(t); nxt.append(t)
+            if all(done):
+                break
+            inp = E[torch.tensor(nxt, device=ids.device)][:, None]
+        return outs
+
+
+def load_model(cosy, ckpt, tok):
+    ck = torch.load(ckpt, map_location="cuda", weights_only=False)
+    m = FLM(cosy.model.llm, ck["n_spk"], tok.prefix).cuda().eval()
+    print("load", ckpt, m.load_state_dict(ck["model"], strict=False), flush=True)
+    return m, json.load(open(os.path.join(os.path.dirname(ckpt), "spk2id.json")))
+
+
+# ------------------------------------------------------------------ data
+def load_data(splits, max_tok=750, extra_man=None, qual=None, qual_drop=0.1):
+    rows = [json.loads(l) for l in open(MAN, encoding="utf-8")]
+    if extra_man and os.path.exists(extra_man):
+        rows += [json.loads(l) for l in open(extra_man, encoding="utf-8")]
+    want = {r["id"] for r in rows if r["split"] in splits}
+    drop = set()
+    if qual and os.path.exists(qual) and "train" in splits:       # drop the bottom qual_drop of UTMOS per source
+        Q = [json.loads(l) for l in open(qual)]
+        by = {}
+        for q in Q:
+            by.setdefault(q["src"], []).append(q)
+        for src, L in by.items():
+            if src in ("cs_pairs",):
+                continue
+            thr = float(np.quantile([q["u22"] for q in L], qual_drop))
+            drop |= {q["id"] for q in L if q["u22"] < thr}
+        print("quality filter drops", len(drop), flush=True)
+    src_idx, mm = {}, {}
+    for src in sorted({r["src"] for r in rows}):
+        if src in NEW:
+            d = f"{HOME}/p3t/{src}"
+        elif src == "igbosyn_pseudo":
+            d = f"{HOME}/flash/data/igbosyn_pseudo"
+        else:
+            d = f"{HOME}/data3/{src}"
+        if not os.path.exists(f"{d}/s3v3.done"):
+            print("missing v3 tokens", d, flush=True); continue
+        meta = [json.loads(l) for l in open(f"{d}/meta.jsonl", encoding="utf-8")]
+        tk = [json.loads(l) for l in open(f"{d}/s3v3.jsonl")]
+        ix = {}
+        for k, (m, t) in enumerate(zip(meta, tk)):
+            assert m["id"] == t["id"]
+            if m["id"] in want:
+                ix[m["id"]] = (t["toff"], t["ntok"])
+        src_idx[src] = ix
+        mm[src] = np.memmap(f"{d}/s3v3.i32", dtype=np.int32, mode="r")
+    items = []
+    for r in rows:
+        if r["split"] not in splits or r["src"] not in src_idx or r["id"] in drop and r["speaker"] != ANCHOR:
+            continue
+        ix = src_idx[r["src"]]
+        if r["id"] not in ix:
+            continue
+        toff, ntok = ix[r["id"]]
+        if not (5 < ntok <= max_tok):
+            continue
+        w = float(r.get("weight", 1.0)) * FLASH_W.get(r["src"], 1.0)
+        items.append(dict(id=r["id"], ds=r["src"], src=r["src"], lang=r["lang"], spk=r["speaker"], split=r["split"],
+                          dur=r["dur"], text=r["text_plain"], tags=r.get("text_tags"), rate=R2I.get(r.get("rate_bin"), 0),
+                          w=w, toff=toff, ntok=ntok))
+    return items, mm
+
+
+def tagged(text, lang):
+    return text if lang == "cs" else f"<{lang}>{text}</{lang}>"
+
+
+def batches(items, budget, rng, wfn, cost):
+    idx = []
+    for i, m in enumerate(items):
+        w = wfn(m)
+        idx += [i] * int(w) + ([i] if rng.random() < w - int(w) else [])
+    idx.sort(key=lambda i: cost(items[i]) + rng.random() * 10)
+    out, cur, mx = [], [], 0
+    for i in idx:
+        L = cost(items[i])
+        if cur and max(mx, L) * (len(cur) + 1) > budget:
+            out.append(cur); cur, mx = [], 0
+        cur.append(i); mx = max(mx, L)
+    if cur:
+        out.append(cur)
+    rng.shuffle(out)
+    return out
+
+
+VAL_SPLITS = {"val", "validation", "dev"}
+
+
+def train_lm(a):
+    cosy = load_cv3(); tok = Tok(cosy)
+    items, mm = load_data({"train"}, a.max_tok, a.extra_man, a.qual, a.qual_drop)
+    val_all, mmv = load_data(VAL_SPLITS, a.max_tok)
+    mm.update(mmv)
+    spk2id = {}
+    for s in sorted({m["spk"] for m in items}):
+        spk2id[s] = len(spk2id)
+    n_spk = len(spk2id)
+    rng = random.Random(1); rng.shuffle(val_all)
+    val, per_l, per_s = [], {}, {}
+    for m in val_all:
+        if m["spk"] not in spk2id or m["lang"] == "cs":
+            continue
+        L = "anchor" if m["spk"] == ANCHOR else m["lang"]
+        if per_l.get(L, 0) >= 60 or per_s.get(m["spk"], 0) >= 15:
+            continue
+        per_l[L] = per_l.get(L, 0) + 1; per_s[m["spk"]] = per_s.get(m["spk"], 0) + 1; val.append(m)
+    by_spk = {}
+    for m in items:
+        if m["ntok"] <= 200 and m["lang"] != "cs":
+            by_spk.setdefault(m["spk"], []).append(m)
+    os.makedirs(a.out, exist_ok=True)
+    json.dump(spk2id, open(f"{a.out}/spk2id.json", "w")); json.dump(vars(a), open(f"{a.out}/args.json", "w"))
+    hrs = {}
+    for m in items:
+        hrs[m["src"]] = hrs.get(m["src"], 0) + m["dur"] / 3600
+    print("lm items", len(items), "hours", round(sum(hrs.values()), 1), {k: round(v, 1) for k, v in hrs.items()},
+          "spk", n_spk, "val", len(val), per_l, flush=True)
+    model = FLM(cosy.model.llm, n_spk, tok.prefix).cuda().train()
+    for p in list(cosy.model.flow.parameters()) + list(cosy.model.hift.parameters()):
+        p.requires_grad = False
+    new = list(model.lang_emb.parameters()) + list(model.spk_emb.parameters()) + list(model.adv.parameters()) + list(model.rate_emb.parameters())
+    newids = {id(p) for p in new}
+    old = [p for p in model.parameters() if id(p) not in newids and p.requires_grad]
+    opt = torch.optim.AdamW([{"params": old, "lr": a.lr}, {"params": new, "lr": a.lr_new}], weight_decay=0.01)
+    base = [a.lr, a.lr_new]
+
+    def text_of(m, r):
+        if m["tags"] and r is not None and r.random() >= a.tag_drop:
+            return m["tags"]
+        return strip_tags(m["text"])
+
+    def spk_t(m, r):
+        rate = m["rate"] if (r is not None and r.random() >= a.rate_drop) else 0
+        return torch.tensor([spk2id[m["spk"]], rate]).cuda()
+
+    def enc(m, prng=None, force_prompt=None, val_mode=False):
+        r = None if val_mode else prng
+        ids, langs = tok(tagged(text_of(m, r), m["lang"]))
+        sp = np.asarray(mm[m["ds"]][m["toff"]: m["toff"] + m["ntok"]], dtype=np.int64)
+        out = [torch.tensor(ids).cuda(), torch.tensor(langs).cuda(), spk_t(m, r), torch.from_numpy(sp).cuda()]
+        use = force_prompt if force_prompt is not None else (prng is not None and prng.random() < a.p_prompt)
+        if use:
+            c = by_spk.get(m["spk"], [])
+            c = [x for x in (prng or random).sample(c, min(6, len(c))) if x["id"] != m["id"]] if c else []
+            if c and c[0]["ntok"] + m["ntok"] <= a.max_tok + 150:
+                p = c[0]
+                pids, plangs = tok(tagged(strip_tags(p["text"]), p["lang"]))
+                psp = np.asarray(mm[p["ds"]][p["toff"]: p["toff"] + p["ntok"]], dtype=np.int64)
+                out.append((torch.tensor(pids).cuda(), torch.tensor(plangs).cuda(), torch.from_numpy(psp).cuda()))
+        return out
+
+    def validate():
+        model.eval(); res = {}
+        with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
+            for mode in (False, True):
+                vr = random.Random(7); L, W = [], []
+                for i in range(0, len(val), 16):
+                    bt = val[i:i + 16]
+                    ce, _, _, _ = model.forward2([enc(m, vr, force_prompt=mode, val_mode=True) for m in bt])
+                    L.append(float(ce) * len(bt)); W.append(len(bt))
+                res["p" if mode else "np"] = sum(L) / sum(W)
+        model.train()
+        res["mean"] = (res["np"] + res["p"]) / 2
+        return res
+
+    wfn = lambda m: m["w"] * (a.anchor_w if m["spk"] == ANCHOR else 1.0)  # noqa
+    cost = lambda m: (lambda L: int(L * (1 + L / a.quad)))(m["ntok"] + len(m["text"]) // 2 + 20 + int(a.p_prompt * 200))  # noqa
+    rng = random.Random(0); prng = random.Random(3)
+    step, nskip, noom, t0 = 0, 0, 0, time.time()
+    logf = open(f"{a.out}/log.jsonl", "a")
+    v = validate(); best = v["mean"]; bad = 0
+    print(json.dumps({"step": 0, "val": v}), flush=True); logf.write(json.dumps({"step": 0, "val": v}) + "\n")
+    total, done = a.max_steps, False
+    while not done:
+        for bi in batches(items, a.tokens, rng, wfn, cost):
+            lrm = min(1.0, (step + 1) / a.warmup) * (0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * min(1.0, step / total))))
+            for gp, b in zip(opt.param_groups, base):
+                gp["lr"] = b * lrm
+            batch = [enc(items[i], prng) for i in bi]
+            opt.zero_grad(set_to_none=True)
+            oom = False
+            try:
+                with torch.autocast("cuda", dtype=torch.bfloat16):
+                    ce, acc, adv, adv_acc = model.forward2(batch, adv_lambda=min(1.0, step / 1000))
+                loss = ce + a.adv_w * adv
+                if not torch.isfinite(loss):
+                    nskip += 1; continue
+                loss.backward()
+            except torch.cuda.OutOfMemoryError:
+                oom = True
+            if oom:
+                batch = ce = acc = adv = adv_acc = loss = None
+                opt.zero_grad(set_to_none=True); torch.cuda.empty_cache(); noom += 1; continue
+            gn = torch.nn.utils.clip_grad_norm_([p for p in model.parameters() if p.requires_grad], 1.0)
+            if not torch.isfinite(gn):
+                nskip += 1; opt.zero_grad(set_to_none=True); continue
+            opt.step(); step += 1
+            if step % 50 == 0:
+                r = dict(step=step, ce=round(float(ce), 4), acc=round(float(acc), 4), adv=round(float(adv), 3),
+                         adv_acc=round(float(adv_acc), 3), gn=round(float(gn), 2), B=len(bi), lr=round(lrm * a.lr, 9),
+                         skips=nskip, oom=noom, min=round((time.time() - t0) / 60, 2))
+                print(json.dumps(r), flush=True); logf.write(json.dumps(r) + "\n"); logf.flush()
+            if step % a.val_every == 0:
+                v = validate(); imp = v["mean"] < best - 1e-4
+                rec = {"step": step, "val": v, "best": imp}
+                print(json.dumps(rec), flush=True); logf.write(json.dumps(rec) + "\n"); logf.flush()
+                if imp:
+                    best, bad = v["mean"], 0
+                    torch.save(dict(model={k: t for k, t in model.state_dict().items()}, step=step, n_spk=n_spk, val=v), f"{a.out}/best.pt")
+                else:
+                    bad += 1
+                    if bad >= a.patience:
+                        print("EARLY_STOP", step, flush=True); done = True; break
+            if (time.time() - t0) / 60 > a.minutes or step >= total:
+                done = True; break
+    print("TRAIN_LM_DONE", step, "best", best, "skips", nskip, "oom", noom, flush=True)
+
+
+# ------------------------------------------------------------------ synthesis
+def prompt_feats(cosy, wav):
+    fe = cosy.frontend
+    tokn, _ = fe._extract_speech_token(wav)
+    feat, _ = fe._extract_speech_feat(wav)
+    n = min(feat.shape[1] // 2, tokn.shape[1])
+    return tokn[:, :n].cuda(), feat[:, :2 * n].cuda(), fe._extract_spk_embedding(wav).cuda()
+
+
+@torch.no_grad()
+def token2wav(cosy, tokens, ptok, pfeat, emb):
+    flow = cosy.model.flow
+    t = torch.tensor([tokens], dtype=torch.int32).cuda()
+    mel, _ = flow.inference(token=t, token_len=torch.tensor([t.shape[1]], dtype=torch.int32).cuda(),
+                            prompt_token=ptok.int(), prompt_token_len=torch.tensor([ptok.shape[1]], dtype=torch.int32).cuda(),
+                            prompt_feat=pfeat, prompt_feat_len=torch.tensor([pfeat.shape[1]], dtype=torch.int32).cuda(),
+                            embedding=emb, streaming=False, finalize=True)
+    w, _ = cosy.model.hift.inference(speech_feat=mel, finalize=True)
+    return w.squeeze().float().cpu().numpy()
+
+
+def item_text(r):
+    return r["text"] if r["lang"] in ("cs", "para") else f"<{r['lang']}>{r['text']}</{r['lang']}>"
+
+
+class Synth:
+    def __init__(self, lm_ckpt):
+        self.cosy = load_cv3(); self.tok = Tok(self.cosy)
+        self.m, self.spk2id = load_model(self.cosy, lm_ckpt, self.tok)
+        self.vcache = {}
+        vt = f"{PDIR}/flash_voices_text.json"
+        self.vtext = json.load(open(vt if os.path.exists(vt) else f"{_HERE}/voices_text.json", encoding="utf-8"))
+
+    def voice(self, name):
+        if name not in self.vcache:
+            v = VOICES[name]
+            ptok, pfeat, emb = prompt_feats(self.cosy, v["wav"])
+            ids, langs = self.tok(self.vtext[name])
+            lmp = (torch.tensor(ids).cuda(), torch.tensor(langs).cuda(), ptok[0].long())
+            self.vcache[name] = (ptok, pfeat, emb, lmp, self.spk2id.get(v["spk"], self.spk2id.get(ANCHOR)))
+        return self.vcache[name]
+
+    def tokens(self, text, voice="anchor", rate="normal", seed=0, B=1, sampler="silexcap"):
+        ptok, pfeat, emb, lmp, sid = self.voice(voice)
+        ids, langs = self.tok(text)
+        spk = torch.tensor([sid, R2I.get(None if rate == "none" else rate, 0)]).cuda()
+        with torch.autocast("cuda", dtype=torch.bfloat16):
+            return self.m.generate(torch.tensor(ids).cuda(), torch.tensor(langs).cuda(), spk, prompt=lmp, B=B, seed=seed,
+                                   sampler=sampler)
+
+    def wav(self, toks, voice="anchor"):
+        ptok, pfeat, emb, _, _ = self.voice(voice)
+        if len(toks) < 3:
+            toks = list(toks) + [1, 1, 1]
+        return token2wav(self.cosy, toks, ptok, pfeat, emb)
+
+
+def gen(a):
+    S = Synth(a.lm)
+    rows = [json.loads(l) for l in open(a.rows, encoding="utf-8")]
+    if a.shard:
+        i, n = map(int, a.shard.split("/")); rows = rows[i::n]
+    os.makedirs(a.out, exist_ok=True)
+    json.dump(vars(a), open(f"{a.out}/variant.json", "w"))
+    for r in rows:
+        if os.path.exists(f"{a.out}/{r['id']}.wav"):
+            continue
+        text = r.get("tts_text") or item_text(r)
+        toks = S.tokens(text, a.voice, a.rate, a.seed)[0]
+        w = S.wav(toks, a.voice)
+        sf.write(f"{a.out}/{r['id']}.wav", w, S.cosy.sample_rate)
+        json.dump(dict(tokens=[toks], ntok=[len(toks)]), open(f"{a.out}/{r['id']}.tok.json", "w"))
+    print("GEN_DONE", a.out, len(rows), flush=True)
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser()
+    ap.add_argument("cmd")
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--lm")
+    ap.add_argument("--rows", default=f"{HOME}/flash/score/rows_flash.jsonl")
+    ap.add_argument("--voice", default="anchor")
+    ap.add_argument("--rate", default="normal")
+    ap.add_argument("--seed", type=int, default=0)
+    ap.add_argument("--shard")
+    ap.add_argument("--minutes", type=float, default=75)
+    ap.add_argument("--max_steps", type=int, default=12000)
+    ap.add_argument("--lr", type=float, default=2e-5)
+    ap.add_argument("--lr_new", type=float, default=1e-3)
+    ap.add_argument("--warmup", type=int, default=300)
+    ap.add_argument("--tokens", type=int, default=16000)
+    ap.add_argument("--max_tok", type=int, default=750)
+    ap.add_argument("--adv_w", type=float, default=0.02)
+    ap.add_argument("--p_prompt", type=float, default=0.5)
+    ap.add_argument("--tag_drop", type=float, default=0.5)
+    ap.add_argument("--rate_drop", type=float, default=0.5)
+    ap.add_argument("--anchor_w", type=float, default=1.0)
+    ap.add_argument("--val_every", type=int, default=1000)
+    ap.add_argument("--patience", type=int, default=4)
+    ap.add_argument("--quad", type=float, default=600.0)
+    ap.add_argument("--extra_man", default=f"{HOME}/flash/data/igbosyn_pseudo/manifest.jsonl")
+    ap.add_argument("--qual", default=f"{HOME}/flash/qual/utmos_train.jsonl")
+    ap.add_argument("--qual_drop", type=float, default=0.1)
+    a = ap.parse_args()
+    {"lm": train_lm, "gen": gen}[a.cmd](a)

frontend/__init__.py

@@ -0,0 +1,2 @@
+"""Dynamic Accent Flash text front-end (normalisation, Yoruba/Igbo diacritic restoration, experimental MT)."""
+from .frontend import FrontEnd  # noqa: F401

frontend/frontend.py

@@ -0,0 +1,122 @@
+"""Dynamic Accent Flash text front-end.
+
+    from frontend import FrontEnd
+    fe = FrontEnd(diac_model="models/diacritics", mt=True)
+    fe("<eng>The meeting is at 3:30 on 21/10/2026.</eng>", mode="codeswitch")          # tags kept, numbers verbalised
+    fe("Ojo naa dara pupo", mode="native", lang="yor")                                # adds Yorùbá tone marks/underdots
+    fe("Please come home early today.", mode="translate", lang="ibo")                 # MADLAD-400 eng/pcm -> ibo
+    fe("I go call you tomorrow", mode="accent", lang="ibo")                           # English/Pidgin words, Igbo accent
+
+Modes
+  codeswitch : input already tagged <ibo>…</ibo> <eng>…</eng> (or untagged = one language `lang`); each segment is
+               normalised in its own language; Yorùbá/Igbo segments that carry no marks are diacritised.
+  native     : the whole text is one language (`lang`); normalise + diacritise.
+  translate  : `lang` is the target (ibo|yor|hau); the input (English or Pidgin) is translated first with
+               MADLAD-400-3B-MT (Apache-2.0), then normalised + diacritised (MADLAD Yorùbá is often unmarked).
+  accent     : keep the words as written but read them with the `lang` accent (whole text tagged <lang>), i.e.
+               "English words with a native accent". Numbers are verbalised in English.
+Output: a tagged string for infer.py / flash_lm.py.
+"""
+import re
+import unicodedata
+
+try:
+    from .normalize import normalize
+except ImportError:  # run as a script
+    from normalize import normalize
+
+SEG = re.compile(r"<(\w{3})>(.*?)</\1>", re.S)
+
+
+def _strip(s):
+    return "".join(c for c in unicodedata.normalize("NFD", s) if unicodedata.category(c) != "Mn")
+
+
+def unmarked(text, thresh=0.05):
+    w = re.findall(r"\w+", unicodedata.normalize("NFC", text))
+    if not w:
+        return False
+    return sum(_strip(x) != unicodedata.normalize("NFC", x) for x in w) / len(w) < thresh
+
+
+class FrontEnd:
+    def __init__(self, diac_model=None, mt=False, device="cuda"):
+        self.diac_dir, self.device = diac_model, device
+        self._d = self._mt = None
+        self.use_mt = mt
+
+    # ---- diacritics (ByT5-small fine-tune, word-guarded)
+    def diacritise(self, text, lang):
+        if lang not in ("yor", "ibo") or not self.diac_dir:
+            return text
+        import torch
+        from transformers import AutoTokenizer, T5ForConditionalGeneration
+        if self._d is None:
+            self._d = (AutoTokenizer.from_pretrained(self.diac_dir),
+                       T5ForConditionalGeneration.from_pretrained(self.diac_dir).to(self.device).eval())
+        tok, m = self._d
+        src = unicodedata.normalize("NFC", _strip(text))
+        x = tok([f"{lang}: {src}"], return_tensors="pt").to(self.device)
+        with torch.no_grad():
+            y = m.generate(**x, max_length=int(x.input_ids.shape[1] * 1.6) + 16)
+        hyp = unicodedata.normalize("NFC", tok.decode(y[0], skip_special_tokens=True))
+        sw, hw = src.split(), hyp.split()
+        if len(sw) != len(hw):
+            return text
+        return " ".join(h if unicodedata.normalize("NFC", _strip(h)) == s else s for s, h in zip(sw, hw))
+
+    # ---- MT (MADLAD-400-3B-MT)
+    def translate(self, text, tgt):
+        import torch
+        from transformers import T5ForConditionalGeneration, T5Tokenizer
+        if self._mt is None:
+            self._mt = (T5Tokenizer.from_pretrained("google/madlad400-3b-mt"),
+                        T5ForConditionalGeneration.from_pretrained("google/madlad400-3b-mt", torch_dtype=torch.bfloat16).to(self.device).eval())
+        tok, m = self._mt
+        code = {"ibo": "ig", "yor": "yo", "hau": "ha", "eng": "en"}[tgt]
+        out = []
+        for sent in re.split(r"(?<=[.!?])\s+", text.strip()):
+            if not sent:
+                continue
+            x = tok([f"<2{code}> {sent}"], return_tensors="pt").to(self.device)
+            with torch.no_grad():
+                y = m.generate(**x, max_new_tokens=256, num_beams=4)
+            out.append(tok.decode(y[0], skip_special_tokens=True))
+        return " ".join(out)
+
+    def _seg(self, lang, s):
+        s = normalize(s, lang)
+        if lang in ("yor", "ibo") and unmarked(s):
+            s = self.diacritise(s, lang)
+        return f"<{lang}>{s}</{lang}>"
+
+    def __call__(self, text, mode="codeswitch", lang="eng"):
+        text = unicodedata.normalize("NFC", text).strip()
+        if mode == "translate":
+            if lang not in ("ibo", "yor", "hau"):
+                raise ValueError("translate mode needs lang in ibo|yor|hau")
+            plain = re.sub(r"</?\w{3}>", "", text)
+            return self._seg(lang, self.translate(normalize(plain, "eng"), lang))
+        if mode == "accent":
+            plain = re.sub(r"</?\w{3}>", "", text)
+            return f"<{lang}>{normalize(plain, 'eng')}</{lang}>"
+        if mode == "native" or not SEG.search(text):
+            return self._seg(lang, re.sub(r"</?\w{3}>", "", text))
+        parts, pos = [], 0
+        for m in SEG.finditer(text):
+            if text[pos:m.start()].strip():
+                parts.append(self._seg(lang, text[pos:m.start()]))
+            parts.append(self._seg(m.group(1), m.group(2)))
+            pos = m.end()
+        if text[pos:].strip():
+            parts.append(self._seg(lang, text[pos:]))
+        return " ".join(parts)
+
+
+if __name__ == "__main__":
+    import argparse
+    ap = argparse.ArgumentParser()
+    ap.add_argument("text"); ap.add_argument("--mode", default="codeswitch"); ap.add_argument("--lang", default="eng")
+    ap.add_argument("--diac_model", default=None); ap.add_argument("--cpu", action="store_true")
+    a = ap.parse_args()
+    print(FrontEnd(a.diac_model, device="cpu" if a.cpu else "cuda")(a.text, a.mode, a.lang))

frontend/normalize.py

@@ -0,0 +1,308 @@
+"""Dynamic Accent Flash text normaliser: numbers, ordinals-as-cardinals, years, dates, times, currency and percent
+-> words, per language (eng, pcm, ibo, hau, yor). Pure Python, no dependencies.
+
+    from normalize import normalize
+    normalize("Ọ bụ ₦5,000 n'ụbọchị 21/10/2026", "ibo")
+
+Coverage: integers 0 .. 999,999,999 (larger stay as digits), decimals (point-by-digit), ₦/N/NGN/naira, $/USD,
+£, kobo, %, dd/mm/yyyy and "21 October 2026" dates, hh:mm times, 1st/2nd... in eng/pcm.
+The Igbo and Hausa systems are decimal and regular. Yorùbá uses the traditional vigesimal/subtractive system for
+1-99 (with tone marks) and decimal composition above 100 (ọgọ́rùn-ún, ẹgbẹ̀rún, mílíọ̀nù + "ó lé"); the Yorùbá forms and
+all date/currency phrasings need a native-speaker check before release (see TEXT_FRONTEND.md).
+"""
+import re
+import unicodedata
+
+# ------------------------------------------------------------------ English / Pidgin
+_E1 = "zero one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen".split()
+_E10 = "_ _ twenty thirty forty fifty sixty seventy eighty ninety".split()
+
+
+def eng(n):
+    if n < 20:
+        return _E1[n]
+    if n < 100:
+        return _E10[n // 10] + ("" if n % 10 == 0 else "-" + _E1[n % 10])
+    if n < 1000:
+        return _E1[n // 100] + " hundred" + ("" if n % 100 == 0 else " and " + eng(n % 100))
+    for v, w in ((10 ** 9, "billion"), (10 ** 6, "million"), (1000, "thousand")):
+        if n >= v:
+            r = n % v
+            tail = "" if r == 0 else ((" and " if r < 100 else " ") + eng(r))
+            return eng(n // v) + " " + w + tail
+    return str(n)
+
+
+def eng_year(n):
+    if 2000 <= n <= 2009:
+        return eng(n)
+    if 1100 <= n <= 2099 and n % 100 != 0:
+        return eng(n // 100) + " " + (("oh " + _E1[n % 100]) if n % 100 < 10 else eng(n % 100))
+    return eng(n)
+
+
+_ORD = {"one": "first", "two": "second", "three": "third", "five": "fifth", "eight": "eighth", "nine": "ninth", "twelve": "twelfth"}
+
+
+def eng_ord(n):
+    w = eng(n)
+    head, _, last = w.rpartition(" ") if " " in w else ("", "", w)
+    h2, dash, l2 = last.rpartition("-")
+    base = l2 if dash else last
+    o = _ORD.get(base) or (base[:-1] + "ieth" if base.endswith("y") else base + "th")
+    last = (h2 + "-" + o) if dash else o
+    return (head + " " + last).strip()
+
+
+# ------------------------------------------------------------------ Igbo (decimal)
+_I1 = ["efu", "otu", "abụọ", "atọ", "anọ", "ise", "isii", "asaa", "asatọ", "itoolu"]
+
+
+def ibo(n):
+    if n < 10:
+        return _I1[n]
+    if n < 100:
+        t, u = divmod(n, 10)
+        s = "iri" if t == 1 else "iri " + _I1[t]
+        return s if u == 0 else s + " na " + _I1[u]
+    for v, w in ((10 ** 9, "ijeri"), (10 ** 6, "nde"), (1000, "puku"), (100, "narị")):
+        if n >= v:
+            q, r = divmod(n, v)
+            s = w if q == 1 else w + " " + ibo(q)
+            return s if r == 0 else s + " na " + ibo(r)
+    return str(n)
+
+
+# ------------------------------------------------------------------ Hausa (decimal)
+_H1 = ["sifili", "ɗaya", "biyu", "uku", "huɗu", "biyar", "shida", "bakwai", "takwas", "tara"]
+_H10 = [None, "goma", "ashirin", "talatin", "arba'in", "hamsin", "sittin", "saba'in", "tamanin", "casa'in"]
+
+
+def hau(n):
+    if n < 10:
+        return _H1[n]
+    if n < 20:
+        return "goma" if n == 10 else "goma sha " + _H1[n - 10]
+    if n < 100:
+        t, u = divmod(n, 10)
+        return _H10[t] if u == 0 else _H10[t] + " da " + _H1[u]
+    for v, w in ((10 ** 9, "biliyan"), (10 ** 6, "miliyan"), (1000, "dubu"), (100, "ɗari")):
+        if n >= v:
+            q, r = divmod(n, v)
+            s = w if q == 1 else w + " " + hau(q)
+            return s if r == 0 else s + " da " + hau(r)
+    return str(n)
+
+
+# ------------------------------------------------------------------ Yorùbá
+# counting forms 1-10, then vigesimal/subtractive 11-99 (standard grammar tables); decimal composition >= 100
+_Y1 = ["òdo", "oókan", "eéjì", "ẹẹ́ta", "ẹẹ́rin", "aárùn-ún", "ẹẹ́fà", "eéje", "ẹẹ́jọ", "ẹẹ́sàn-án", "ẹẹ́wàá"]
+_Y11 = {11: "ọ̀kanlá", 12: "èjìlá", 13: "ẹ̀tàlá", 14: "ẹ̀rìnlá"}
+# tens: name, locative form used after lé/dín ("added to / less than")
+_YT = {20: ("ogún", "lógún"), 30: ("ọgbọ̀n", "lọ́gbọ̀n"), 40: ("ogójì", "lógójì"), 50: ("àádọ́ta", "láàádọ́ta"),
+       60: ("ọgọ́ta", "lọ́gọ́ta"), 70: ("àádọ́rin", "láàádọ́rin"), 80: ("ọgọ́rin", "lọ́gọ́rin"),
+       90: ("àádọ́rùn-ún", "láàádọ́rùn-ún"), 100: ("ọgọ́rùn-ún", "lọ́gọ́rùn-ún")}
+_YLE = {1: "ọ̀kàn", 2: "éjì", 3: "ẹ́tà", 4: "ẹ́rìn"}            # +1..+4
+_YDIN = {1: "ọ̀kàn", 2: "éjì", 3: "ẹẹ́tà", 4: "ẹẹ́rìn"}          # -1..-4
+_YFIVE = {20: "ẹ̀ẹ́dógún", 30: "ẹ̀ẹ́dọ́gbọ̀n", 40: "ẹ̀ẹ́dógójì", 50: "ẹ̀ẹ́dáàádọ́ta", 60: "ẹ̀ẹ́dọ́gọ́ta", 70: "ẹ̀ẹ́dáàádọ́rin",
+          80: "ẹ̀ẹ́dọ́gọ́rin", 90: "ẹ̀ẹ́dáàádọ́rùn-ún", 100: "ẹ̀ẹ́dọ́gọ́rùn-ún"}   # (T - 5)
+_YADJ = {1: "kan", 2: "méjì", 3: "mẹ́ta", 4: "mẹ́rin", 5: "márùn-ún", 6: "mẹ́fà", 7: "méje", 8: "mẹ́jọ", 9: "mẹ́sàn-án", 10: "mẹ́wàá"}
+_YH = {200: "igba", 300: "ọ̀ọ́dúnrún", 400: "irinwó", 500: "ẹ̀ẹ́dẹ́gbẹ̀ta", 600: "ẹgbẹ̀ta", 700: "ẹ̀ẹ́dẹ́gbẹ̀rin",
+       800: "ẹgbẹ̀rin", 900: "ẹ̀ẹ́dẹ́gbẹ̀rún"}
+
+
+def _y99(n):
+    if n <= 10:
+        return _Y1[n]
+    if n in _Y11:
+        return _Y11[n]
+    if n in _YT:
+        return _YT[n][0]
+    T = (n // 10) * 10
+    u = n - T
+    if u <= 4:                       # T + u  (u-lé-T)
+        return _YLE[u] + "lé" + _YT[T][1]
+    if u == 5:                       # (T+10) - 5
+        return _YFIVE[T + 10]
+    return _YDIN[10 - u] + "dín" + _YT[T + 10][1]      # (T+10) - (10-u)
+
+
+def yor(n):
+    if n < 100:
+        return _y99(n)
+    if n < 1000:
+        h, r = divmod(n, 100)
+        s = _YT[100][0] if h == 1 else _YH[h * 100]
+        return s if r == 0 else s + " ó lé " + _y99(r)
+    for v, w in ((10 ** 9, "bílíọ̀nù"), (10 ** 6, "mílíọ̀nù"), (1000, "ẹgbẹ̀rún")):
+        if n >= v:
+            q, r = divmod(n, v)
+            s = w + " " + (_YADJ[q] if q in _YADJ else yor(q))
+            return s if r == 0 else s + " ó lé " + yor(r)
+    return str(n)
+
+
+NUM = {"eng": eng, "pcm": eng, "ibo": ibo, "hau": hau, "yor": yor}
+
+MONTHS = {
+    "eng": "January February March April May June July August September October November December".split(),
+    "ibo": "Jenụwarị Febụwarị Maachị Epreel Mee Juun Julaị Ọgọọst Septemba Ọktoba Novemba Disemba".split(),
+    "hau": "Janairu Fabrairu Maris Afrilu Mayu Yuni Yuli Agusta Satumba Oktoba Nuwamba Disamba".split(),
+    "yor": "Ṣẹ́rẹ́ Èrèlé Ẹrẹ̀nà Ìgbé Ẹ̀bìbí Òkúdu Agẹmọ Ògún Owewe Ọ̀wàrà Bélú Ọ̀pẹ̀".split(),
+}
+MONTHS["pcm"] = MONTHS["eng"]
+_MIDX = {m.lower(): i for i, m in enumerate(MONTHS["eng"])}
+_MIDX.update({m[:3].lower(): i for i, m in enumerate(MONTHS["eng"])})
+
+CUR = {  # (major, minor) per language; phrase order handled in money()
+    "NGN": {"eng": ("naira", "kobo"), "pcm": ("naira", "kobo"), "ibo": ("naịra", "kobo"), "hau": ("naira", "kobo"), "yor": ("náírà", "kọ́bọ̀")},
+    "USD": {"eng": ("dollars", "cents"), "pcm": ("dollars", "cents"), "ibo": ("dọla", "senti"), "hau": ("dala", "santi"), "yor": ("dọ́là", "sẹ́ǹtì")},
+    "GBP": {"eng": ("pounds", "pence"), "pcm": ("pounds", "pence"), "ibo": ("paụnd", "pens"), "hau": ("fam", "fens"), "yor": ("pọ́n-ùn", "pẹ́ńsì")},
+}
+
+
+def num(n, lang):
+    return NUM[lang](n) if n < 10 ** 9 else str(n)
+
+
+def money(amount, code, lang):
+    whole, _, frac = amount.partition(".")
+    whole = int(whole.replace(",", "") or 0)
+    major, minor = CUR[code][lang]
+    if lang in ("eng", "pcm"):
+        s = f"{num(whole, lang)} {major}"
+        if code == "USD" and whole == 1:
+            s = "one dollar"
+        if frac and int(frac[:2].ljust(2, "0")):
+            s += f" {num(int(frac[:2].ljust(2, '0')), lang)} {minor}"
+        return s
+    s = f"{major} {num(whole, lang)}"            # ibo/hau/yor: currency word first ("naịra puku ise")
+    if lang == "yor":
+        s = f"{major} {yor_adj(whole)}"
+    if frac and int(frac[:2].ljust(2, "0")):
+        k = int(frac[:2].ljust(2, "0"))
+        s += {"ibo": " na ", "hau": " da ", "yor": " ó lé "}[lang] + f"{minor} {yor_adj(k) if lang == 'yor' else num(k, lang)}"
+    return s
+
+
+def yor_adj(n):
+    """Yorùbá number after a noun (adjectival form for 1-10, counting form otherwise)."""
+    return _YADJ.get(n) or yor(n)
+
+
+def date(d, m, y, lang):
+    M = MONTHS[lang][m - 1]
+    if lang in ("eng", "pcm"):
+        return f"the {eng_ord(d)} of {M}" + (f", {eng_year(y)}" if y else "")
+    if lang == "ibo":
+        return f"ụbọchị {ibo(d)} n'ọnwa {M}" + (f", afọ {ibo(y)}" if y else "")
+    if lang == "hau":
+        return f"ranar {hau(d)} ga watan {M}" + (f", shekarar {hau(y)}" if y else "")
+    return f"ọjọ́ {yor(d)} oṣù {M}" + (f", ọdún {yor(y)}" if y else "")
+
+
+def clock(h, mi, lang):
+    if lang in ("eng", "pcm"):
+        return eng(h) + ("" if mi == 0 else (" oh " + eng(mi) if mi < 10 else " " + eng(mi))) + (" o'clock" if mi == 0 else "")
+    if lang == "ibo":
+        return f"elekere {ibo(h)}" + ("" if mi == 0 else f" na nkeji {ibo(mi)}")
+    if lang == "hau":
+        return f"ƙarfe {hau(h)}" + ("" if mi == 0 else f" da minti {hau(mi)}")
+    return f"agogo {yor_adj(h)}" + ("" if mi == 0 else f" ó lé ìṣẹ́jú {yor_adj(mi)}")
+
+
+def percent(x, lang):
+    w = decimal(x, lang)
+    return {"eng": f"{w} percent", "pcm": f"{w} percent", "ibo": f"pasentị {w}", "hau": f"kashi {w} cikin ɗari",
+            "yor": f"ìdá {w} nínú ọgọ́rùn-ún"}[lang]
+
+
+_POINT = {"eng": "point", "pcm": "point", "ibo": "ntụpọ", "hau": "digo", "yor": "ààmì"}
+
+
+def decimal(x, lang):
+    a, _, b = x.replace(",", "").partition(".")
+    s = num(int(a or 0), lang)
+    if b:
+        s += f" {_POINT[lang]} " + " ".join(num(int(c), lang) for c in b)
+    return s
+
+
+_AMT = r"\d{1,3}(?:,\d{3})+(?:\.\d+)?|\d+(?:\.\d+)?"
+_RE_MONEY = re.compile(r"(?:(₦|NGN\s?|N(?=\d)|\$|USD\s?|£)\s?(" + _AMT + r")(\s?(?:k|m|bn)\b)?)|(?:(" + _AMT + r")\s?(naira|kobo|dollars?)\b)", re.I)
+_RE_DATE = re.compile(r"\b(\d{1,2})[/.-](\d{1,2})[/.-](\d{4})\b")
+_RE_DATE2 = re.compile(r"\b(\d{1,2})(?:st|nd|rd|th)?\s+(January|February|March|April|May|June|July|August|September|October|November|December|Jan|Feb|Mar|Apr|Jun|Jul|Aug|Sep|Sept|Oct|Nov|Dec)\.?,?\s*(\d{4})?\b", re.I)
+_RE_TIME = re.compile(r"\b(\d{1,2}):(\d{2})(?:\s?(am|pm)\b)?", re.I)
+_RE_TIME2 = re.compile(r"\b(\d{1,2})\s?(am|pm)\b", re.I)
+_RE_PCT = re.compile(r"(" + _AMT + r")\s?%")
+_RE_ORD = re.compile(r"\b(\d+)(st|nd|rd|th)\b", re.I)
+_RE_YEAR = re.compile(r"\b(1[5-9]\d\d|20\d\d)\b")
+_RE_NUM = re.compile(_AMT)
+_DUP = re.compile(r"\b(n'elekere|elekere|karfe|ƙarfe|ranar|agogo|ọjọ́|ụbọchị)\s+(elekere|ƙarfe|ranar|agogo|ọjọ́|ụbọchị)\b", re.I)
+_SCALE = {"k": 1000, "m": 10 ** 6, "bn": 10 ** 9}
+
+
+def normalize(text, lang):
+    """Verbalise numbers etc. for one language segment (no <lang> tags inside)."""
+    lang = lang if lang in NUM else "eng"
+    t = unicodedata.normalize("NFC", text)
+
+    def _money(m):
+        if m.group(1):
+            sym, amt, sc = m.group(1).strip().upper(), m.group(2), (m.group(3) or "").strip().lower()
+            code = {"₦": "NGN", "N": "NGN", "NGN": "NGN", "$": "USD", "USD": "USD", "£": "GBP"}[sym]
+        else:
+            amt, word, sc = m.group(4), m.group(5).lower(), ""
+            if word == "kobo":
+                k = int(amt.replace(",", ""))
+                return f"{num(k, lang)} kobo" if lang in ("eng", "pcm") else f"{CUR['NGN'][lang][1]} {yor_adj(k) if lang == 'yor' else num(k, lang)}"
+            code = "NGN" if word == "naira" else "USD"
+        if sc:
+            amt = str(int(round(float(amt.replace(",", "")) * _SCALE[sc])))
+        return money(amt, code, lang)
+
+    t = _RE_MONEY.sub(_money, t)
+    t = _RE_PCT.sub(lambda m: percent(m.group(1), lang), t)
+    t = _RE_DATE.sub(lambda m: date(int(m.group(1)), int(m.group(2)), int(m.group(3)), lang)
+                     if 1 <= int(m.group(1)) <= 31 and 1 <= int(m.group(2)) <= 12 else m.group(0), t)
+    t = _RE_DATE2.sub(lambda m: date(int(m.group(1)), _MIDX[m.group(2).lower()[:3]] + 1,
+                                     int(m.group(3)) if m.group(3) else None, lang), t)
+
+    def _time(m):
+        h, mi, ap = int(m.group(1)), int(m.group(2)), (m.group(3) or "").lower()
+        s = clock(h, mi, lang)
+        if ap and lang in ("eng", "pcm"):
+            s += " " + {"am": "a m", "pm": "p m"}[ap]
+        return s
+
+    t = _RE_TIME.sub(_time, t)
+    t = _RE_TIME2.sub(lambda m: (eng(int(m.group(1))) + " " + {"am": "a m", "pm": "p m"}[m.group(2).lower()])
+                      if lang in ("eng", "pcm") else clock(int(m.group(1)), 0, lang), t)
+    if lang in ("eng", "pcm"):
+        t = _RE_ORD.sub(lambda m: eng_ord(int(m.group(1))), t)
+        t = _RE_YEAR.sub(lambda m: eng_year(int(m.group(1))), t)
+    else:
+        t = _RE_ORD.sub(lambda m: num(int(m.group(1)), lang), t)
+    t = _RE_NUM.sub(lambda m: decimal(m.group(0), lang) if "." in m.group(0) else num(int(m.group(0).replace(",", "")), lang), t)
+    t = _DUP.sub(lambda m: m.group(1) if m.group(1).lower().endswith(m.group(2).lower()) or m.group(1).lower() == "karfe" else m.group(0), t)
+    return re.sub(r"\s+", " ", t).strip()
+
+
+def normalize_tagged(text, default="eng"):
+    """Apply normalize() to every <lang>…</lang> segment of a tagged string (untagged text uses `default`)."""
+    out, pos = [], 0
+    for m in re.finditer(r"<(\w{3})>(.*?)</\1>", text, re.S):
+        if m.start() > pos:
+            out.append(normalize(text[pos:m.start()], default) if text[pos:m.start()].strip() else text[pos:m.start()])
+        out.append(f"<{m.group(1)}>{normalize(m.group(2), m.group(1))}</{m.group(1)}>")
+        pos = m.end()
+    if pos < len(text):
+        rest = text[pos:]
+        out.append(normalize(rest, default) if rest.strip() else rest)
+    return "".join(out)
+
+
+if __name__ == "__main__":
+    import sys
+    for L in ["eng", "pcm", "ibo", "hau", "yor"]:
+        print(L, "|", normalize("₦5,000 on 21/10/2026 at 7:30, 45% of 1,235 people, 3.5 km", L))

infer.py

@@ -1,7 +1,12 @@
 #!/usr/bin/env python
-"""Dynamic Accent: Nigerian multilingual, code-switching TTS on CosyVoice2-0.5B, with selectable model versions.
+"""Dynamic Accent: Nigerian multilingual, code-switching TTS, with selectable model versions.
 
 Versions (`--version`):
+  flash   (default) LM `checkpoints/flash/flash_dpo` on Fun-CosyVoice3-0.5B-2512 (Apache-2.0): Flash SFT + DPO preference
+          tuning (naturalness + CER + pace + pause reward), licensed data only. CV3 base flow/vocoder, zero-shot from the
+          voice clip. Text front-end: numbers verbalised, Yoruba/Igbo diacritics restored when missing (ByT5 model in
+          checkpoints/flash/diacritics_byt5). One pass per paragraph (<= ~60 words), model pauses, no post-processing.
+          Needs a CUDA GPU and a CosyVoice clone with CosyVoice3 support.
   phase2  LM `p2_lm_B` + anchor-SFT flow `p2_flow_sft`. Phase 2 sampler, sentence stitching, edge trim, internal
           silences capped at 0.6 s, 0.6 s between sentences (exactly the Phase 2 final samples).
   phase3  LM `p3_lm_full` (re-cut long-form data with natural pauses, ~102 h of Igbo) + the shared Phase 2 flow (see
@@ -14,7 +19,7 @@ Voices (`--voice`): `anchor_male` (default; WAXAL Yoruba speaker 4), `waxal_ibo_
 speaker token is used when the LM was trained on that speaker.
 
   python infer.py --text "<eng>this is dynamic accent,</eng> <pcm>e dey sound nice abi?</pcm>" --version phase3 --out out.wav
-  python infer.py --text "..." --compare --out cmp.wav       # -> cmp_phase2.wav and cmp_phase3.wav, same text, seed, voice
+  python infer.py --text "..." --compare --out cmp.wav       # -> cmp_phase2/phase3/flash.wav, same text, seed, voice
   python infer.py --list_voices
   python infer.py --auto_tag --tag_only --text "Omo, this traffic no be here o. Ẹ ṣé púpọ̀."   # inspect / edit the tags
 
@@ -382,10 +387,75 @@ def version_cfg(name):
     cj = os.path.join(HERE, os.path.dirname(cfg["lm"]), "config.json")
     if os.path.exists(cj):                      # a checkpoint may override flow / rate / sampler
         cfg.update({k: v for k, v in json.load(open(cj)).items() if k in cfg})
-    cfg["lm"] = os.path.join(HERE, cfg["lm"]); cfg["flow"] = os.path.join(HERE, cfg["flow"])
+    for k in ("lm", "flow", "diacritics"):
+        if cfg.get(k):
+            cfg[k] = os.path.join(HERE, cfg[k])
     return cfg
 
 
+# ------------------------------------------------------------------ Flash (CosyVoice3 engine)
+FLASH_VOICE = {"anchor_male": "anchor", "waxal_ibo_5_female": "waxal_ibo_5", "waxal_yor_5_female": "waxal_yor_5"}
+
+
+def canonical_voice(voice, voices_json=None):
+    try:
+        al = json.load(open(voices_json, encoding="utf-8")).get("aliases", {}) if voices_json else {}
+    except OSError:
+        al = {}
+    return al.get(voice, voice)
+
+
+class FlashTTS:
+    """--version flash: CosyVoice3 base + Flash-DPO LM (flash_lm.Synth) behind the same say()/sr interface."""
+
+    def __init__(self, a, cfg):
+        if not torch.cuda.is_available():
+            raise SystemExit("--version flash needs a CUDA GPU (phase2/phase3 also run on CPU)")
+        cv3 = getattr(a, "cv3_dir", "") or os.environ.get("CV3_DIR") or os.path.join(HERE, "pretrained", "Fun-CosyVoice3-0.5B-2512")
+        if not os.path.isdir(cv3):
+            from huggingface_hub import snapshot_download
+            snapshot_download("FunAudioLLM/Fun-CosyVoice3-0.5B-2512", local_dir=cv3)
+        os.environ["CV3_DIR"] = cv3
+        os.environ.setdefault("COSYVOICE_REPO", a.cosyvoice_dir)
+        os.environ.setdefault("DA_PROMPTS", os.path.join(HERE, "prompts"))
+        import flash_lm
+        self.S = flash_lm.Synth(a.lm or cfg["lm"])
+        self.cosy = self.tok = None                  # not shareable with the CosyVoice2 versions
+        self.m = self.S.m
+        self.sr = self.S.cosy.sample_rate
+        r = a.rate if a.rate != "default" else cfg.get("rate", "normal")
+        self.rate = RATES.index(None if r == "none" else r) if (None if r == "none" else r) in RATES else 0
+        self.fe_mode = getattr(a, "frontend", "codeswitch")
+        self.fe_lang = getattr(a, "frontend_lang", "eng")
+        if self.fe_mode == "translate" and not getattr(a, "experimental_translate", False):
+            raise SystemExit("--frontend translate is EXPERIMENTAL and off by default (MADLAD-400 loops on long inputs and is "
+                             "weak on Pidgin). Add --experimental_translate to use it anyway.")
+        from frontend import FrontEnd
+        diac = cfg.get("diacritics") if not getattr(a, "no_diacritics", False) else None
+        self.fe = FrontEnd(diac if diac and os.path.isdir(diac) else None, mt=self.fe_mode == "translate")
+        self.set_voice(a.voice, a.voices)
+
+    def set_voice(self, voice, voices_json=None):
+        v = canonical_voice(voice, voices_json)
+        if v not in FLASH_VOICE:
+            raise SystemExit(f"unknown voice '{voice}' for flash; available: {', '.join(FLASH_VOICE)}")
+        self.voice = FLASH_VOICE[v]
+
+    def frontend(self, text):
+        return self.fe(text, self.fe_mode, self.fe_lang)
+
+    def say(self, tagged_text, seed=0, return_info=False, **_):
+        t = self.frontend(tagged_text)
+        print(f"[flash front-end] {t}")
+        rate = RATES[self.rate]
+        ws = []
+        for i, ch in enumerate(word_chunks(t)):
+            toks = self.S.tokens(ch, self.voice, rate or "none", seed + 1000 * i)[0]
+            ws.append(self.S.wav(toks, self.voice))
+        w = np.concatenate(ws).astype(np.float32)
+        return (w, None) if return_info else w
+
+
 def word_chunks(text, max_words=60):
     """one pass per <= max_words, split only at sentence ends (a whole paragraph usually fits in one pass)."""
     sents = split_sentences(text)
@@ -522,6 +592,8 @@ class DynamicAccent:
 
 def build(version, a, cosy=None, tok=None, copy_llm=False):
     cfg = version_cfg(version)
+    if cfg.get("engine") == "cosyvoice3":
+        return FlashTTS(a, cfg)
     lm = a.lm or cfg["lm"]; flow = a.flow or cfg["flow"]
     S = DynamicAccent(a.model_dir, a.cosyvoice_dir, lm, flow, a.prompts, a.dtype, sampler=cfg["sampler"],
                       one_pass=cfg["one_pass"] and not a.stitch, postprocess=cfg["postprocess"],
@@ -539,8 +611,7 @@ def list_voices(voices_json):
     for k, v in SETTINGS.get("versions", {}).items():
         print(f"  {k:20s} {v.get('desc', '')}")
     if RECOMMENDED:
-        print("\nrecommended pairing: phase3 -> " + ", ".join(RECOMMENDED.get("phase3", [])) +
-              "; phase2 -> " + ", ".join(RECOMMENDED.get("phase2", [])))
+        print("\nrecommended pairing: " + "; ".join(f"{k} -> {', '.join(v)}" for k, v in RECOMMENDED.items() if isinstance(v, list)))
         print("  " + RECOMMENDED.get("note", ""))
 
 
@@ -564,14 +635,25 @@ def main():
     ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
     ap.add_argument("--text", help="text; tagged ('<eng>Good morning,</eng> <pcm>how body?</pcm>') or plain with --auto_tag")
     ap.add_argument("--version", default=SETTINGS.get("default_version", "phase3"), choices=sorted(VERSIONS),
-                    help="model version (default phase3)")
+                    help=f"model version (default {SETTINGS.get('default_version', 'phase3')})")
     ap.add_argument("--compare", action="store_true",
-                    help="render the same text, seed and voice with phase2 AND phase3 -> <out>_phase2.wav, <out>_phase3.wav")
+                    help="render the same text, seed and voice with every version in --compare_versions -> <out>_<version>.wav")
+    ap.add_argument("--compare_versions", default="phase2,phase3,flash", help="versions for --compare (comma-separated)")
     ap.add_argument("--voice", default=SETTINGS.get("default_voice", "anchor_male"),
                     help="anchor_male | waxal_ibo_5_female | waxal_yor_5_female (see --list_voices)")
     ap.add_argument("--list_voices", action="store_true", help="list voices, versions and the recommended pairing")
     ap.add_argument("--rate", default="default", choices=["default", "none", "very slow", "slow", "normal", "fast", "very fast"],
-                    help="phase3 speed instruction to the LM (default: the version's setting, 'normal'; 'none' = no instruction)")
+                    help="phase3/flash speed instruction to the LM (default: the version's setting, 'normal'; 'none' = no instruction)")
+    # flash text front-end
+    ap.add_argument("--frontend", default="codeswitch", choices=["codeswitch", "native", "accent", "translate"],
+                    help="flash only: codeswitch (tagged/one-language text), native (whole text in --frontend_lang), accent "
+                         "(words as written, read with the --frontend_lang accent), translate (EXPERIMENTAL, off by default)")
+    ap.add_argument("--frontend_lang", default="eng", choices=LANGS, help="flash front-end language for untagged text / native / accent / translate target")
+    ap.add_argument("--no_diacritics", action="store_true", help="flash: do not restore missing Yoruba/Igbo diacritics")
+    ap.add_argument("--experimental_translate", action="store_true",
+                    help="flash: allow --frontend translate (MADLAD-400-3B-MT, ~12 GB download; loops on long inputs, weak on Pidgin)")
+    ap.add_argument("--cv3_dir", default=os.environ.get("CV3_DIR", os.path.join(HERE, "pretrained", "Fun-CosyVoice3-0.5B-2512")),
+                    help="flash: stock FunAudioLLM/Fun-CosyVoice3-0.5B-2512 snapshot (downloaded on first run)")
     # code-switching control
     ap.add_argument("--auto_tag", action="store_true",
                     help="tag untagged mixed text automatically (word/span language ID), print it, then synthesise")
@@ -607,10 +689,13 @@ def main():
         print(f"[tagged] {text}")
     if a.tag_only:
         return
-    if not os.path.isdir(a.model_dir):
+    versions = [v.strip() for v in a.compare_versions.split(",") if v.strip()] if a.compare else [a.version]
+    bad = [v for v in versions if v not in VERSIONS]
+    if bad:
+        ap.error(f"unknown version(s) {bad}; available: {sorted(VERSIONS)}")
+    if any(VERSIONS[v].get("engine") != "cosyvoice3" for v in versions) and not os.path.isdir(a.model_dir):
         from huggingface_hub import snapshot_download
         snapshot_download("FunAudioLLM/CosyVoice2-0.5B", local_dir=a.model_dir)
-    versions = ["phase2", "phase3"] if a.compare else [a.version]
     cosy = tok = None
     base, ext = os.path.splitext(a.out)
     for v in versions:
@@ -618,13 +703,17 @@ def main():
         if hint:
             print(hint)
         S = build(v, a, cosy, tok)
-        cosy, tok = S.cosy, S.tok
+        if S.cosy is not None:
+            cosy, tok = S.cosy, S.tok
         w = S.say(text, seed=a.seed, prompt_k=a.prompt_clip, stitch=not a.no_stitch, pause_mode=a.pause_mode,
                   speed=a.speed if v == "phase2" else 1.0)
         out = f"{base}_{v}{ext or '.wav'}" if a.compare else a.out
         sf.write(out, w, S.sr)
         print(f"[{v}] wrote {out} ({len(w) / S.sr:.2f} s, {S.sr} Hz, voice={a.voice}, device={DEV.type})")
-        del S.m; torch.cuda.empty_cache()
+        del S.m
+        if isinstance(S, FlashTTS):
+            del S.S
+        del S; torch.cuda.empty_cache()
 
 
 if __name__ == "__main__":

prompts/flash_voices_text.json

@@ -0,0 +1,3 @@
+{"anchor": "<yor>Púpọ̀ jùlọ àwọn olókìkí ní ìlú Èkó ń gbé ní erékùṣù náà.</yor>",
+ "waxal_ibo_5": "<ibo>I nwere ike itinye mmemme ọbụla n'izu a ma gị atụla ụjọ maka oké mmiri ozuzo.</ibo>",
+ "waxal_yor_5": "<yor>Nínú oṣù Ọpẹ́ ni ó ra mọ́tò tí wọ́n jí gbé.</yor>"}

samples/flash/codeswitch_ibo_pcm_eng.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26d4ac3d1fcae8484641e547e455420f54dd987b9b1184624ef40de80c3d3d8a
+size 175245

samples/flash/codeswitch_yor_eng.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c09b28d483a2f7c740d9b05438767f2f457cb1bd2bd49029d31760a8a0fc7ad
+size 104205

samples/flash/diacritics_yoruba_from_plain.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ce4b97b0225a62356b2996e9d551ff74e2a2b16ca34b2707873095772995889
+size 107085

samples/flash/hausa_male.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21354db9f258956e861b5526289f2b49dc908d2760de89913a65af8e74ac95eb
+size 208845

samples/flash/igbo_para_female.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c24da6ec6eaec6b332f1a9b0dac2eb944e08a13187e9c07dd5d1b0577ef8c0f5
+size 245805

samples/flash/igbo_pure_female.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39abdd290724dcc05cd7d5f928cde2f178044115b26795bf16da48963cfe2ff9
+size 164205

samples/flash/numbers_igbo.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daa1ea59e60f9c1b3fefde255ce5c98ec16513d64fef2aebbd1f035ae0227283
+size 262605

samples/flash/pidgin_female.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:255ae0e9a883f93b5bc48c2eeedac3138bd7304ee01bfd675e9c29ff99957189
+size 181005

samples/flash/yoruba_female.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c0011b2c16edbdd88e91511ce8d455e358dee285084a7bffc4f6211e6ec6888
+size 162285

samples/flash/yoruba_male.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:550aa664f313b854594ebfd263b015224a1fce1a0abae2c8ce6215541121092f
+size 181005

settings.json

@@ -1,7 +1,20 @@
 {
- "default_version": "phase3",
+ "default_version": "flash",
  "default_voice": "anchor_male",
  "versions": {
+  "flash": {
+   "engine": "cosyvoice3",
+   "base": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
+   "lm": "checkpoints/flash/flash_dpo/best.pt",
+   "diacritics": "checkpoints/flash/diacritics_byt5",
+   "sampler": "silexcap",
+   "one_pass": true,
+   "postprocess": false,
+   "rate": "normal",
+   "frontend": "codeswitch",
+   "translate": "experimental, off by default (--frontend translate --experimental_translate)",
+   "desc": "Flash (default): Flash-DPO LM on Fun-CosyVoice3-0.5B-2512 (SFT on licensed data + DPO with a naturalness/CER/pace/pause reward), CV3 flow+vocoder zero-shot from the voice clip; text front-end restores Yoruba/Igbo diacritics; one pass per paragraph, model pauses, no post-processing"
+  },
   "phase2": {
    "lm": "checkpoints/phase2/p2_lm_B/best.pt",
    "flow": "checkpoints/phase2/p2_flow_sft/best.pt",
@@ -27,6 +40,11 @@
   "waxal_yor_5_female"
  ],
  "recommended": {
+  "flash": [
+   "anchor_male",
+   "waxal_ibo_5_female",
+   "waxal_yor_5_female"
+  ],
   "phase3": [
    "waxal_ibo_5_female",
    "waxal_yor_5_female"
@@ -34,7 +52,7 @@
   "phase2": [
    "anchor_male"
   ],
-  "note": "Listening verdict (model author, 2026-10-21): Phase 3 sounds best with the female voices; Phase 2 is fine for the male anchor. Use --compare to A/B before use."
+  "note": "Flash is the default and the best-scoring version for every voice (clipped endings anchor 9.1%->7.1%, female Igbo 22.8%->10.7%; pace ~native). Known weakness: with anchor_male, Flash under-pauses in Igbo and Yoruba. Earlier listening verdict (2026-10-21): Phase 3 with the female voices, Phase 2 with the male anchor. Use --compare to A/B before use."
  },
  "rates": [
   "none",