Files changed (1) hide show
  1. README.md +2 -101
README.md CHANGED
@@ -15,12 +15,6 @@ tags:
15
  - voice-cloning
16
  - vietnamese-tts
17
  - tieng-viet
18
- # NOTE deliberately NO `datasets:` field. It is the only thing that populates
19
- # the Hub's cross-link, but the Hub renders it as "Models trained or fine-tuned
20
- # on <dataset>" — which for our own held-out benchmark reads as train/test
21
- # contamination. ZeroBench-TTS is EVALUATION data; every voice in it is held
22
- # out of training. The `model-index` block below states that correctly, and the
23
- # body links the benchmark in prose.
24
  metrics:
25
  - wer
26
  model-index:
@@ -141,29 +135,10 @@ for chunk in tts.synthesize_stream("Một đoạn văn bản dài hơn…", voic
141
 
142
  ## Benchmarks
143
 
144
- Measured on **[ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)**
145
- 137 items, 59 held-out reference voices × 4 subsets — against
146
- [OmniVoice](https://huggingface.co/k2-fsa/OmniVoice) and the two public
147
- Vietnamese XTTS-v2 finetunes. 137/137 scored for every system, 0 empty
148
- generations. OmniVoice is given its optional `language="vi"` hint, which its
149
- model card recommends and which measurably helps it.
150
-
151
- **Scored by the benchmark, not by us.** ZeroTTS synthesizes the clips and hands
152
- them to `zerobench_eval`, the official scorer published inside the benchmark
153
- dataset repo. Nothing in this repo computes a metric.
154
-
155
- ### Headline
156
 
157
  Every system reads **normalized text** — dates, numbers and acronyms already
158
- spoken out, from the benchmark's own curated reading. Every system gets exactly
159
- the same input, so the comparison is like-for-like.
160
-
161
- This is the condition a Vietnamese TTS system meets in production, where a text
162
- frontend runs ahead of the model. ZeroTTS ships one — `normalize_vi_text`,
163
- applied by default (see the [GitHub README](https://github.com/zeroweight-ai/ZeroTTS#usage)) — which reproduces the benchmark's
164
- reading on 34 of the 35 items that need normalization. Neither baseline ships a
165
- Vietnamese frontend at all, which is why the raw-text table below is so much
166
- harsher on them.
167
 
168
  | | **ZeroTTS** | OmniVoice | XTTS-v2-vietnamse | viXTTS |
169
  |---|:-:|:-:|:-:|:-:|
@@ -206,60 +181,6 @@ with no Vietnamese text frontend faces.
206
  | `challenging` | acronyms, dates, %, currency | **1.75 %** | 4.46 % | 27.86 % | 31.85 % |
207
  | **overall** | | **1.03 %** | **4.13 %** | **16.42 %** | **18.40 %** |
208
 
209
- **Reading these fairly:**
210
-
211
- * **OmniVoice beats us on two things, and they are worth naming.** Its speaker
212
- similarity is the best of the four (0.951 vs our 0.938), and on `code_switch`
213
- it is roughly half our error rate (0.46 % vs 0.95 %). If cloning fidelity or
214
- English-in-Vietnamese is your priority, it is a genuinely strong option — at
215
- 3.1 GB on a GPU.
216
- * **OmniVoice's overall figure is dominated by one subset.** `cross_lingual`
217
- (foreign voice prompt, Vietnamese text) costs it 17.71 % raw against our
218
- 1.42 %, and it is language-dependent — German 0.00 %, Korean 0.13 %, Japanese
219
- 0.41 %. Excluding that subset it lands near 1.7 % raw. Both ASRs agree the
220
- audio genuinely degrades there, so it is the model, not the scorer.
221
- * **Normalization is where the weakest systems gain most, and the order does not
222
- change.** The XTTS tokenizers have no Vietnamese number expansion, so raw text
223
- punishes them hard (`challenging` 27.86 %) and the normalized column is the
224
- fairest comparison available — it improves XTTS 2.3× and viXTTS 2.1×, against
225
- 1.8× for us. What remains is the acoustic model.
226
- * **`vietnamese` barely moves for anyone** (0.16 % → 0.21 % for ZeroTTS). It has
227
- no digits or acronyms, so there is nothing to normalize — which is the control
228
- showing the other subsets' gains are real and not a scoring artifact.
229
- * **On `cross_lingual` our voice similarity is the weak spot** (0.911 vs
230
- ~0.935 for the others): ZeroTTS carries a foreign speaker's timbre into
231
- Vietnamese slightly less faithfully, while winning that subset's WER by 12×.
232
- * **The WER definition matters more than the WER.** ZeroBench scores every clip
233
- with **two ASRs** (`whisper-large-v3` + `PhoWhisper-large`, min taken — neither
234
- can judge Vietnamese code-switch TTS alone) against **every acceptable
235
- reading** of the target text, so a system is never charged for an ASR's
236
- choice between "31/12/2025" and "ba mươi mốt tháng mười hai". Its test suite
237
- pins that in both directions: format differences must score 0, real
238
- mispronunciations must still cost.
239
- * **Our remaining errors are published, not hidden.** Every item scoring above
240
- 0.00 is audited in [docs/BENCHMARKS.md](https://github.com/zeroweight-ai/ZeroTTS/blob/main/docs/BENCHMARKS.md). The two recurring
241
- ones: a leading zero read aloud (`18/04` → "tháng **không** tư"), and the
242
- letters `W` and `H` coming out wrong when an acronym has to be spelled —
243
- `WHO` should be spelled out letter by letter, and instead comes out as
244
- something like "Hall".
245
-
246
- Reproduce, or score your own system:
247
-
248
- ```bash
249
- pip install "zerotts[eval]"
250
- SYNTH_FROM=text_normalized OUT_DIR=./eval/norm ./evaluation/run_benchmark.sh
251
- ./evaluation/run_benchmark.sh # raw text
252
- ```
253
-
254
- Not using ZeroTTS? The scorer stands alone — bring wavs from any system:
255
-
256
- ```bash
257
- huggingface-cli download zeroweight-ai/ZeroBench-TTS --repo-type dataset --local-dir ZeroBench-TTS
258
- cd ZeroBench-TTS && pip install -r zerobench_eval/requirements.txt
259
- python -m zerobench_eval manifest --out manifest.jsonl # what to synthesize
260
- python -m zerobench_eval score --wav_dir my_wavs/ --name MyModel
261
- ```
262
-
263
  ## Voices, and voice cloning
264
 
265
  A voice is a small array of speaker latents, `(1, n_voice_queries, d_model)`,
@@ -277,25 +198,6 @@ conditioning — no reference transcript, no audio prompt.
277
  Because a voice is just an array, latents obtained that way drop into
278
  `voices/<name>/voice.npz` and work with no code change.
279
 
280
- ## Repository layout
281
-
282
- ```
283
- config.json runtime config
284
- tokenizer.json BPE tokenizer
285
- null_voice_emb.npy learned unconditional voice prefix
286
- onnx/text_encoder.onnx text → encoder states (once per utterance)
287
- onnx/prefix_step.onnx global transformer step (once per frame)
288
- onnx/local_frame_decode.onnx frame decode + sampling (once per frame)
289
- onnx/codec/ MOSS-Audio-Tokenizer-Nano decoder (Apache-2.0)
290
- voices/<name>/voice.npz speaker latents
291
- ```
292
-
293
- fp32, not quantized: ~900 MB total. Two ONNX Runtime calls per audio frame;
294
- frames are produced at 12.5 Hz and decoded to 48 kHz.
295
-
296
- The model architecture, training code, and the ONNX export script are not
297
- published, and the voice encoder is not included.
298
-
299
  ## Intended use and limitations
300
 
301
  Built for **Vietnamese**. It handles English words embedded in Vietnamese text
@@ -328,7 +230,6 @@ no external runtime dependency; the encoder is not included. See
328
  ## License
329
 
330
  ZeroTTS weights and code: **MIT**.
331
- Bundled MOSS codec decoder: **Apache-2.0**.
332
 
333
  The ZeroBench-TTS *dataset* is CC-BY-NC-4.0 because it redistributes reference
334
  audio from VIVOS, viVoice, phoaudiobook and Emilia. That license applies to the
 
15
  - voice-cloning
16
  - vietnamese-tts
17
  - tieng-viet
 
 
 
 
 
 
18
  metrics:
19
  - wer
20
  model-index:
 
135
 
136
  ## Benchmarks
137
 
138
+ Measured on **[ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)**
 
 
 
 
 
 
 
 
 
 
 
139
 
140
  Every system reads **normalized text** — dates, numbers and acronyms already
141
+ spoken out, from the benchmark's own curated reading.
 
 
 
 
 
 
 
 
142
 
143
  | | **ZeroTTS** | OmniVoice | XTTS-v2-vietnamse | viXTTS |
144
  |---|:-:|:-:|:-:|:-:|
 
181
  | `challenging` | acronyms, dates, %, currency | **1.75 %** | 4.46 % | 27.86 % | 31.85 % |
182
  | **overall** | | **1.03 %** | **4.13 %** | **16.42 %** | **18.40 %** |
183
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
184
  ## Voices, and voice cloning
185
 
186
  A voice is a small array of speaker latents, `(1, n_voice_queries, d_model)`,
 
198
  Because a voice is just an array, latents obtained that way drop into
199
  `voices/<name>/voice.npz` and work with no code change.
200
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
201
  ## Intended use and limitations
202
 
203
  Built for **Vietnamese**. It handles English words embedded in Vietnamese text
 
230
  ## License
231
 
232
  ZeroTTS weights and code: **MIT**.
 
233
 
234
  The ZeroBench-TTS *dataset* is CC-BY-NC-4.0 because it redistributes reference
235
  audio from VIVOS, viVoice, phoaudiobook and Emilia. That license applies to the