zeroweightai commited on
Commit
aba7690
·
verified ·
1 Parent(s): 6f064d2

Update README (intro bullets, trimmed benchmark section, voice example) and banner image

Browse files
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +16 -152
  3. banner.png +2 -2
.gitattributes CHANGED
@@ -4,3 +4,4 @@
4
  *.npz filter=lfs diff=lfs merge=lfs -text
5
  *.bin filter=lfs diff=lfs merge=lfs -text
6
  *.wav filter=lfs diff=lfs merge=lfs -text
 
 
4
  *.npz filter=lfs diff=lfs merge=lfs -text
5
  *.bin filter=lfs diff=lfs merge=lfs -text
6
  *.wav filter=lfs diff=lfs merge=lfs -text
7
+ banner.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -15,12 +15,6 @@ tags:
15
  - voice-cloning
16
  - vietnamese-tts
17
  - tieng-viet
18
- # NOTE deliberately NO `datasets:` field. It is the only thing that populates
19
- # the Hub's cross-link, but the Hub renders it as "Models trained or fine-tuned
20
- # on <dataset>" — which for our own held-out benchmark reads as train/test
21
- # contamination. ZeroBench-TTS is EVALUATION data; every voice in it is held
22
- # out of training. The `model-index` block below states that correctly, and the
23
- # body links the benchmark in prose.
24
  metrics:
25
  - wer
26
  model-index:
@@ -108,14 +102,12 @@ the next open model**, and it runs faster than real time on a laptop CPU.
108
  * 🎯 **Ultra-natural** — 2.91 UTMOS, ~0.5 MOS above every other open Vietnamese
109
  system, with near-zero dead air (0.029 s vs 0.23–0.53 s).
110
  * 🗣️ **Zero-shot voice cloning** — a voice is a small latent array; drop it in
111
- and the model speaks in it. No fine-tuning, no per-speaker training.
112
- * **Real-time on CPU, streaming** — first audio chunk in ~100 ms, then chunks
113
- ramp up. No GPU required.
 
114
  * 🇻🇳 **Built for Vietnamese** — tones, code-switched English, and a built-in
115
  normalizer that reads `31/12/2025` and `1.250 tỷ` the way a person would.
116
- * 📊 **Measured, not asserted** — every number below comes from
117
- [ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)'s
118
- own public scorer, on 59 held-out voices.
119
 
120
  * Code, examples, browser demo: **https://github.com/zeroweight-ai/ZeroTTS**
121
  * Benchmark dataset: **https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS**
@@ -128,137 +120,29 @@ pip install zerotts
128
  from zerotts import ZeroTTS
129
 
130
  tts = ZeroTTS.from_pretrained("zeroweight-ai/ZeroTTS")
131
- audio = tts.synthesize("Xin chào các bạn, mình là ZeroTTS.", voice="arya")
132
  tts.save_audio(audio, "out.wav")
133
  ```
134
 
135
- Streaming, with first audio in roughly 100 ms:
136
 
137
  ```python
138
- for chunk in tts.synthesize_stream("Một đoạn văn bản dài hơn…", voice="arya"):
139
  play(chunk) # (1, n) float32 at 48 kHz
140
  ```
141
 
142
  ## Benchmarks
143
 
144
- Measured on **[ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)**
145
- 137 items, 59 held-out reference voices × 4 subsets — against
146
- [OmniVoice](https://huggingface.co/k2-fsa/OmniVoice) and the two public
147
- Vietnamese XTTS-v2 finetunes. 137/137 scored for every system, 0 empty
148
- generations. OmniVoice is given its optional `language="vi"` hint, which its
149
- model card recommends and which measurably helps it.
150
 
151
- **Scored by the benchmark, not by us.** ZeroTTS synthesizes the clips and hands
152
- them to `zerobench_eval`, the official scorer published inside the benchmark
153
- dataset repo. Nothing in this repo computes a metric.
154
-
155
- ### Headline
156
-
157
- Every system reads **normalized text** — dates, numbers and acronyms already
158
- spoken out, from the benchmark's own curated reading. Every system gets exactly
159
- the same input, so the comparison is like-for-like.
160
-
161
- This is the condition a Vietnamese TTS system meets in production, where a text
162
- frontend runs ahead of the model. ZeroTTS ships one — `normalize_vi_text`,
163
- applied by default (see the [GitHub README](https://github.com/zeroweight-ai/ZeroTTS#usage)) — which reproduces the benchmark's
164
- reading on 34 of the 35 items that need normalization. Neither baseline ships a
165
- Vietnamese frontend at all, which is why the raw-text table below is so much
166
- harsher on them.
167
-
168
- | | **ZeroTTS** | OmniVoice | XTTS-v2-vietnamse | viXTTS |
169
- |---|:-:|:-:|:-:|:-:|
170
- | **WER** ↓ | **0.56 %** | 2.12 % | 7.27 % | 8.61 % |
171
- | **Naturalness** (UTMOS) ↑ | **2.91** | 2.75 | 2.49 | 2.34 |
172
- | **Voice similarity** (SSIM) ↑ | 0.938 | **0.951** | 0.941 | 0.935 |
173
- | **Dead air** (excess silence) ↓ | **0.029 s** | 0.386 s | 0.568 s | 0.215 s |
174
- | Size | **81 M**, CPU | 3.1 GB, GPU | 1.9 GB, GPU | 1.9 GB, GPU |
175
-
176
- **4× fewer word errors than the next-best system**, ~0.2 MOS more natural, an
177
- order of magnitude less dead air — from a model small enough to run real-time
178
- on a laptop CPU. Median WER is **0.00 %** on all four subsets: the typical
179
- generation is transcribed exactly. (Every figure is from the same
180
- normalized-text runs, so the rows are mutually consistent.)
181
-
182
- ### WER — normalized text
183
-
184
- The headline condition: numbers and dates already spoken out, as the shipped
185
- normalizer produces.
186
-
187
- | Subset | what it tests | **ZeroTTS** | OmniVoice | XTTS-v2-vietnamse | viXTTS |
188
- |---|---|:-:|:-:|:-:|:-:|
189
- | `vietnamese` | monolingual Vietnamese | **0.21 %** | 0.50 % | 7.21 % | 7.54 % |
190
- | `code_switch` | Vietnamese + embedded English | 0.95 % | **0.46 %** | 10.14 % | 5.86 % |
191
- | `cross_lingual` | foreign voice prompt → Vietnamese | **0.38 %** | 9.60 % | 4.94 % | 6.61 % |
192
- | `challenging` | acronyms, dates, %, currency | **0.61 %** | 1.56 % | 5.63 % | 13.44 % |
193
- | **overall** | | **0.56 %** | **2.12 %** | **7.27 %** | **8.61 %** |
194
-
195
- ### WER — raw text
196
-
197
- The harder condition: the model is handed `31/12/2025` and `ChatGPT` verbatim
198
- and has to read them itself, with no normalizer in front. This is what a system
199
- with no Vietnamese text frontend faces.
200
-
201
- | Subset | what it tests | **ZeroTTS** | OmniVoice | XTTS-v2-vietnamse | viXTTS |
202
- |---|---|:-:|:-:|:-:|:-:|
203
- | `vietnamese` | monolingual Vietnamese | **0.16 %** | 0.50 % | 7.92 % | 9.56 % |
204
- | `code_switch` | Vietnamese + embedded English | 0.97 % | **0.46 %** | 10.94 % | 9.25 % |
205
- | `cross_lingual` | foreign voice prompt → Vietnamese | **1.42 %** | 17.71 % | 21.37 % | 27.27 % |
206
- | `challenging` | acronyms, dates, %, currency | **1.75 %** | 4.46 % | 27.86 % | 31.85 % |
207
- | **overall** | | **1.03 %** | **4.13 %** | **16.42 %** | **18.40 %** |
208
-
209
- **Reading these fairly:**
210
-
211
- * **OmniVoice beats us on two things, and they are worth naming.** Its speaker
212
- similarity is the best of the four (0.951 vs our 0.938), and on `code_switch`
213
- it is roughly half our error rate (0.46 % vs 0.95 %). If cloning fidelity or
214
- English-in-Vietnamese is your priority, it is a genuinely strong option — at
215
- 3.1 GB on a GPU.
216
- * **OmniVoice's overall figure is dominated by one subset.** `cross_lingual`
217
- (foreign voice prompt, Vietnamese text) costs it 17.71 % raw against our
218
- 1.42 %, and it is language-dependent — German 0.00 %, Korean 0.13 %, Japanese
219
- 0.41 %. Excluding that subset it lands near 1.7 % raw. Both ASRs agree the
220
- audio genuinely degrades there, so it is the model, not the scorer.
221
- * **Normalization is where the weakest systems gain most, and the order does not
222
- change.** The XTTS tokenizers have no Vietnamese number expansion, so raw text
223
- punishes them hard (`challenging` 27.86 %) and the normalized column is the
224
- fairest comparison available — it improves XTTS 2.3× and viXTTS 2.1×, against
225
- 1.8× for us. What remains is the acoustic model.
226
- * **`vietnamese` barely moves for anyone** (0.16 % → 0.21 % for ZeroTTS). It has
227
- no digits or acronyms, so there is nothing to normalize — which is the control
228
- showing the other subsets' gains are real and not a scoring artifact.
229
- * **On `cross_lingual` our voice similarity is the weak spot** (0.911 vs
230
- ~0.935 for the others): ZeroTTS carries a foreign speaker's timbre into
231
- Vietnamese slightly less faithfully, while winning that subset's WER by 12×.
232
- * **The WER definition matters more than the WER.** ZeroBench scores every clip
233
- with **two ASRs** (`whisper-large-v3` + `PhoWhisper-large`, min taken — neither
234
- can judge Vietnamese code-switch TTS alone) against **every acceptable
235
- reading** of the target text, so a system is never charged for an ASR's
236
- choice between "31/12/2025" and "ba mươi mốt tháng mười hai". Its test suite
237
- pins that in both directions: format differences must score 0, real
238
- mispronunciations must still cost.
239
- * **Our remaining errors are published, not hidden.** Every item scoring above
240
- 0.00 is audited in [docs/BENCHMARKS.md](https://github.com/zeroweight-ai/ZeroTTS/blob/main/docs/BENCHMARKS.md). The two recurring
241
- ones: a leading zero read aloud (`18/04` → "tháng **không** tư"), and the
242
- letters `W` and `H` coming out wrong when an acronym has to be spelled —
243
- `WHO` should be spelled out letter by letter, and instead comes out as
244
- something like "Hall".
245
-
246
- Reproduce, or score your own system:
247
-
248
- ```bash
249
- pip install "zerotts[eval]"
250
- SYNTH_FROM=text_normalized OUT_DIR=./eval/norm ./evaluation/run_benchmark.sh
251
- ./evaluation/run_benchmark.sh # raw text
252
- ```
253
-
254
- Not using ZeroTTS? The scorer stands alone — bring wavs from any system:
255
-
256
- ```bash
257
- huggingface-cli download zeroweight-ai/ZeroBench-TTS --repo-type dataset --local-dir ZeroBench-TTS
258
- cd ZeroBench-TTS && pip install -r zerobench_eval/requirements.txt
259
- python -m zerobench_eval manifest --out manifest.jsonl # what to synthesize
260
- python -m zerobench_eval score --wav_dir my_wavs/ --name MyModel
261
- ```
262
 
263
  ## Voices, and voice cloning
264
 
@@ -277,25 +161,6 @@ conditioning — no reference transcript, no audio prompt.
277
  Because a voice is just an array, latents obtained that way drop into
278
  `voices/<name>/voice.npz` and work with no code change.
279
 
280
- ## Repository layout
281
-
282
- ```
283
- config.json runtime config
284
- tokenizer.json BPE tokenizer
285
- null_voice_emb.npy learned unconditional voice prefix
286
- onnx/text_encoder.onnx text → encoder states (once per utterance)
287
- onnx/prefix_step.onnx global transformer step (once per frame)
288
- onnx/local_frame_decode.onnx frame decode + sampling (once per frame)
289
- onnx/codec/ MOSS-Audio-Tokenizer-Nano decoder (Apache-2.0)
290
- voices/<name>/voice.npz speaker latents
291
- ```
292
-
293
- fp32, not quantized: ~900 MB total. Two ONNX Runtime calls per audio frame;
294
- frames are produced at 12.5 Hz and decoded to 48 kHz.
295
-
296
- The model architecture, training code, and the ONNX export script are not
297
- published, and the voice encoder is not included.
298
-
299
  ## Intended use and limitations
300
 
301
  Built for **Vietnamese**. It handles English words embedded in Vietnamese text
@@ -328,7 +193,6 @@ no external runtime dependency; the encoder is not included. See
328
  ## License
329
 
330
  ZeroTTS weights and code: **MIT**.
331
- Bundled MOSS codec decoder: **Apache-2.0**.
332
 
333
  The ZeroBench-TTS *dataset* is CC-BY-NC-4.0 because it redistributes reference
334
  audio from VIVOS, viVoice, phoaudiobook and Emilia. That license applies to the
 
15
  - voice-cloning
16
  - vietnamese-tts
17
  - tieng-viet
 
 
 
 
 
 
18
  metrics:
19
  - wer
20
  model-index:
 
102
  * 🎯 **Ultra-natural** — 2.91 UTMOS, ~0.5 MOS above every other open Vietnamese
103
  system, with near-zero dead air (0.029 s vs 0.23–0.53 s).
104
  * 🗣️ **Zero-shot voice cloning** — a voice is a small latent array; drop it in
105
+ and the model speaks in it, cloned from as little as 3 seconds of reference
106
+ audio (up to 30 seconds). No fine-tuning, no per-speaker training.
107
+ * **Real-time on CPU, streaming** — ~2× faster than real time (RTF 0.5×),
108
+ first audio chunk in ~70 ms. No GPU required.
109
  * 🇻🇳 **Built for Vietnamese** — tones, code-switched English, and a built-in
110
  normalizer that reads `31/12/2025` and `1.250 tỷ` the way a person would.
 
 
 
111
 
112
  * Code, examples, browser demo: **https://github.com/zeroweight-ai/ZeroTTS**
113
  * Benchmark dataset: **https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS**
 
120
  from zerotts import ZeroTTS
121
 
122
  tts = ZeroTTS.from_pretrained("zeroweight-ai/ZeroTTS")
123
+ audio = tts.synthesize("Xin chào các bạn, mình là ZeroTTS.", voice="maichi")
124
  tts.save_audio(audio, "out.wav")
125
  ```
126
 
127
+ Streaming, with first audio in roughly 70 ms:
128
 
129
  ```python
130
+ for chunk in tts.synthesize_stream("Một đoạn văn bản dài hơn…", voice="maichi"):
131
  play(chunk) # (1, n) float32 at 48 kHz
132
  ```
133
 
134
  ## Benchmarks
135
 
136
+ Measured on **[ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)**
137
+ against OmniVoice, XTTS-v2-vietnamse and viXTTS, on normalized text (dates,
138
+ numbers and acronyms already spoken out): **0.56 % WER** (4× fewer errors than
139
+ the next-best system, median 0.00 % on all four subsets), **2.91 UTMOS**, and
140
+ **0.029 s** of dead air. On CPU it runs ~2× faster than real time (RTF 0.5×)
141
+ with first audio in ~70 ms of streaming.
142
 
143
+ Full comparison tables, per-subset breakdowns, and CPU speed methodology:
144
+ **[docs/BENCHMARKS.md](https://github.com/zeroweight-ai/ZeroTTS/blob/main/docs/BENCHMARKS.md)**
145
+ on GitHub.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
146
 
147
  ## Voices, and voice cloning
148
 
 
161
  Because a voice is just an array, latents obtained that way drop into
162
  `voices/<name>/voice.npz` and work with no code change.
163
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
164
  ## Intended use and limitations
165
 
166
  Built for **Vietnamese**. It handles English words embedded in Vietnamese text
 
193
  ## License
194
 
195
  ZeroTTS weights and code: **MIT**.
 
196
 
197
  The ZeroBench-TTS *dataset* is CC-BY-NC-4.0 because it redistributes reference
198
  audio from VIVOS, viVoice, phoaudiobook and Emilia. That license applies to the
banner.png CHANGED

Git LFS Details

  • SHA256: 755cc2e9d50b70eef853bfd207ed6ea86326c128f73647af0af01753933d1ca0
  • Pointer size: 131 Bytes
  • Size of remote file: 346 kB

Git LFS Details

  • SHA256: 0dd95a69fce14c796d020d4d8bb743857bd30b7c54646d362ee6d3002181139b
  • Pointer size: 131 Bytes
  • Size of remote file: 248 kB