zeroweight-ai
/

ZeroTTS

@@ -12,6 +12,84 @@ tags:
   - onnxruntime
   - zero-shot
   - speech-synthesis
 ---
 # ZeroTTS
@@ -43,44 +121,61 @@ for chunk in tts.synthesize_stream("Một đoạn văn bản dài hơn…", voic
 ## Benchmarks
-[ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS) —
-137 items, 59 held-out reference voices × 4 subsets, against the two public
-Vietnamese XTTS finetunes. 137/137 scored, 0 empty generations.
-WER uses [`vinai/PhoWhisper-large`](https://huggingface.co/vinai/PhoWhisper-large)
-(a Vietnamese-specialized Whisper finetune — scoring Vietnamese TTS with raw
-`whisper-large-v3` measures the ASR's weakness as much as the TTS system's).
-Every model is reported twice: `as-is`, and `+vinorm` which adds
-[`soe-vinorm`](https://pypi.org/project/soe-vinorm/)'s automatic spoken-out
-normalization as an extra accepted reference.
-| Model | WER as-is | WER +vinorm | SSIM | UTMOS | Excess silence |
-|---|---|---|---|---|---|
-| **ZeroTTS** | **5.09%** | **4.51%** | 0.936 | **2.95** | **0.029 s** |
-| XTTS-v2-vietnamse | 21.49% | 21.16% | **0.940** | 2.36 | 0.532 s |
-| viXTTS | 25.13% | 24.92% | 0.935 | 2.35 | 0.233 s |
-Per subset (WER, as-is → +vinorm):
-| Subset | n | ZeroTTS | XTTS-v2-vietnamse | viXTTS |
-|---|---|---|---|---|
-| `vietnamese` | 39 | **0.16% → 0.16%** | 8.18% → 8.18% | 11.25% → 11.25% |
-| `code_switch` | 39 | **5.74% → 5.72%** | 16.16% → 16.15% | 17.66% → 17.64% |
-| `cross_lingual` (foreign voice, vi text) | 20 | **6.05% → 5.98%** | 30.49% → 30.37% | 34.51% → 34.51% |
-| `challenging` (acronyms, dates, %) | 39 | **8.86% → 6.90%** | 35.50% → 34.43% | 41.68% → 40.93% |
 Reading these fairly:
-* **The `+vinorm` column tests an objection rather than flattering us.** XTTS's
-  stock tokenizer has no Vietnamese number/symbol expansion, so one could argue
-  the `challenging` gap is a missing text frontend. Supplying that normalization
-  on the scoring side moves XTTS 35.50% → 34.43% and viXTTS 41.68% → 40.93% —
-  about a point each. It does not close a 26-point gap.
-* **Voice similarity is a tie, not a win** (0.936 / 0.940 / 0.935, within noise),
-  and on `cross_lingual` ZeroTTS is genuinely behind (0.911 vs ~0.935).
-Full tables, ablations and reproduction commands:
-[docs/BENCHMARKS.md](https://github.com/zeroweight-ai/ZeroTTS/blob/main/docs/BENCHMARKS.md).
 ## Voices, and voice cloning

   - onnxruntime
   - zero-shot
   - speech-synthesis
+  - voice-cloning
+  - vietnamese-tts
+  - tieng-viet
+datasets:
+  - zeroweight-ai/ZeroBench-TTS
+metrics:
+  - wer
+model-index:
+  - name: ZeroTTS
+    results:
+      - task:
+          type: text-to-speech
+          name: Zero-Shot Text-to-Speech
+        dataset:
+          type: zeroweight-ai/ZeroBench-TTS
+          name: ZeroBench-TTS
+          split: test
+        metrics:
+          - type: wer
+            value: 1.03
+            name: WER (%)
+          - type: utmos
+            value: 2.91
+            name: UTMOSv2 naturalness MOS
+          - type: speaker_similarity
+            value: 0.936
+            name: Speaker similarity (WavLM-SV cosine)
+          - type: excess_silence
+            value: 0.029
+            name: Excess silence (s)
+      - task:
+          type: text-to-speech
+          name: Zero-Shot TTS — monolingual Vietnamese
+        dataset:
+          type: zeroweight-ai/ZeroBench-TTS
+          name: ZeroBench-TTS (vietnamese)
+          config: vietnamese
+          split: test
+        metrics:
+          - type: wer
+            value: 0.16
+            name: WER (%)
+      - task:
+          type: text-to-speech
+          name: Zero-Shot TTS — Vietnamese/English code-switching
+        dataset:
+          type: zeroweight-ai/ZeroBench-TTS
+          name: ZeroBench-TTS (code_switch)
+          config: code_switch
+          split: test
+        metrics:
+          - type: wer
+            value: 0.97
+            name: WER (%)
+      - task:
+          type: text-to-speech
+          name: Zero-Shot TTS — cross-lingual voice prompt
+        dataset:
+          type: zeroweight-ai/ZeroBench-TTS
+          name: ZeroBench-TTS (cross_lingual)
+          config: cross_lingual
+          split: test
+        metrics:
+          - type: wer
+            value: 1.42
+            name: WER (%)
+      - task:
+          type: text-to-speech
+          name: Zero-Shot TTS — acronyms, dates, numbers
+        dataset:
+          type: zeroweight-ai/ZeroBench-TTS
+          name: ZeroBench-TTS (challenging)
+          config: challenging
+          split: test
+        metrics:
+          - type: wer
+            value: 1.75
+            name: WER (%)
 ---
 # ZeroTTS
 ## Benchmarks
+Measured on **[ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)**
+— 137 items, 59 held-out reference voices, 4 subsets — against the two public
+Vietnamese XTTS-v2 finetunes. 137/137 scored, 0 empty generations.
+Scored by the benchmark's own published scorer (`zerobench_eval`), not by us.
+WER is the minimum over two ASRs (`whisper-large-v3` + `PhoWhisper-large`) and
+over every acceptable reading of the target text — see the
+[benchmark README](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)
+for why both matter.
+| | **ZeroTTS** | XTTS-v2-vietnamse | viXTTS |
+|---|:-:|:-:|:-:|
+| **WER** ↓ | **1.03%** | 16.42% | 18.40% |
+| **Naturalness** (UTMOS) ↑ | **2.91** | 2.43 | 2.35 |
+| **Voice similarity** (SSIM) ↑ | 0.936 | **0.940** | 0.935 |
+| **Dead air** (excess silence) ↓ | **0.029 s** | 0.532 s | 0.233 s |
+**16× fewer word errors**, ~0.5 MOS more natural, an order of magnitude less
+dead air. Median WER is **0.00%** on all four subsets.
+WER by subset:
+| Subset | what it tests | **ZeroTTS** | XTTS-v2-vietnamse | viXTTS |
+|---|---|:-:|:-:|:-:|
+| `vietnamese` | plain Vietnamese | **0.16%** | 7.92% | 9.56% |
+| `code_switch` | Vietnamese + embedded English | **0.97%** | 10.94% | 9.25% |
+| `cross_lingual` | foreign voice prompt → Vietnamese | **1.42%** | 21.37% | 27.27% |
+| `challenging` | acronyms, dates, %, currency | **1.75%** | 27.86% | 31.85% |
 Reading these fairly:
+* **Voice similarity is a tie, not a win** — 0.936 / 0.939 / 0.935 is within
+  noise. On `cross_lingual` ZeroTTS is genuinely behind (0.911 vs ~0.935): it
+  carries a foreign speaker's timbre into Vietnamese less faithfully than the
+  XTTS backbone, while winning that subset's WER by 15×.
+* **"The baselines just need text normalization" is testable, and fails.**
+  Given the spoken-out text instead of raw orthography, XTTS improves to 7.27%
+  and viXTTS to 8.61% — big gains, confirming their tokenizers lack Vietnamese
+  number expansion — but ZeroTTS improves to 0.56%, so the gap stays 13–15×.
+* **ZeroTTS's own remaining errors are known and listed**, not hidden:
+  [evaluation/HIGH_WER_ANALYSIS.md](https://github.com/zeroweight-ai/ZeroTTS/blob/main/evaluation/HIGH_WER_ANALYSIS.md)
+  audits every item above 0.00 — mostly voiced leading zeros in dates and
+  `W`/`H` acronym letter names.
+Score your own model on the same benchmark:
+```bash
+huggingface-cli download zeroweight-ai/ZeroBench-TTS --repo-type dataset --local-dir ZeroBench-TTS
+cd ZeroBench-TTS && pip install -r zerobench_eval/requirements.txt
+python -m zerobench_eval manifest --out manifest.jsonl    # what to synthesize
+python -m zerobench_eval score --wav_dir my_wavs/ --name MyModel
+```
+Full tables and reproduction commands:
+[evaluation/RESULTS.md](https://github.com/zeroweight-ai/ZeroTTS/blob/main/evaluation/RESULTS.md).
 ## Voices, and voice cloning