Update README (intro bullets, trimmed benchmark section, voice example) and banner image
Browse files- .gitattributes +1 -0
- README.md +16 -152
- banner.png +2 -2
.gitattributes
CHANGED
|
@@ -4,3 +4,4 @@
|
|
| 4 |
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 5 |
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 6 |
*.wav filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 4 |
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 5 |
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 6 |
*.wav filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
banner.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -15,12 +15,6 @@ tags:
|
|
| 15 |
- voice-cloning
|
| 16 |
- vietnamese-tts
|
| 17 |
- tieng-viet
|
| 18 |
-
# NOTE deliberately NO `datasets:` field. It is the only thing that populates
|
| 19 |
-
# the Hub's cross-link, but the Hub renders it as "Models trained or fine-tuned
|
| 20 |
-
# on <dataset>" — which for our own held-out benchmark reads as train/test
|
| 21 |
-
# contamination. ZeroBench-TTS is EVALUATION data; every voice in it is held
|
| 22 |
-
# out of training. The `model-index` block below states that correctly, and the
|
| 23 |
-
# body links the benchmark in prose.
|
| 24 |
metrics:
|
| 25 |
- wer
|
| 26 |
model-index:
|
|
@@ -108,14 +102,12 @@ the next open model**, and it runs faster than real time on a laptop CPU.
|
|
| 108 |
* 🎯 **Ultra-natural** — 2.91 UTMOS, ~0.5 MOS above every other open Vietnamese
|
| 109 |
system, with near-zero dead air (0.029 s vs 0.23–0.53 s).
|
| 110 |
* 🗣️ **Zero-shot voice cloning** — a voice is a small latent array; drop it in
|
| 111 |
-
and the model speaks in it
|
| 112 |
-
|
| 113 |
-
|
|
|
|
| 114 |
* 🇻🇳 **Built for Vietnamese** — tones, code-switched English, and a built-in
|
| 115 |
normalizer that reads `31/12/2025` and `1.250 tỷ` the way a person would.
|
| 116 |
-
* 📊 **Measured, not asserted** — every number below comes from
|
| 117 |
-
[ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)'s
|
| 118 |
-
own public scorer, on 59 held-out voices.
|
| 119 |
|
| 120 |
* Code, examples, browser demo: **https://github.com/zeroweight-ai/ZeroTTS**
|
| 121 |
* Benchmark dataset: **https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS**
|
|
@@ -128,137 +120,29 @@ pip install zerotts
|
|
| 128 |
from zerotts import ZeroTTS
|
| 129 |
|
| 130 |
tts = ZeroTTS.from_pretrained("zeroweight-ai/ZeroTTS")
|
| 131 |
-
audio = tts.synthesize("Xin chào các bạn, mình là ZeroTTS.", voice="
|
| 132 |
tts.save_audio(audio, "out.wav")
|
| 133 |
```
|
| 134 |
|
| 135 |
-
Streaming, with first audio in roughly
|
| 136 |
|
| 137 |
```python
|
| 138 |
-
for chunk in tts.synthesize_stream("Một đoạn văn bản dài hơn…", voice="
|
| 139 |
play(chunk) # (1, n) float32 at 48 kHz
|
| 140 |
```
|
| 141 |
|
| 142 |
## Benchmarks
|
| 143 |
|
| 144 |
-
Measured on **[ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)**
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
|
| 150 |
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
|
| 154 |
-
|
| 155 |
-
### Headline
|
| 156 |
-
|
| 157 |
-
Every system reads **normalized text** — dates, numbers and acronyms already
|
| 158 |
-
spoken out, from the benchmark's own curated reading. Every system gets exactly
|
| 159 |
-
the same input, so the comparison is like-for-like.
|
| 160 |
-
|
| 161 |
-
This is the condition a Vietnamese TTS system meets in production, where a text
|
| 162 |
-
frontend runs ahead of the model. ZeroTTS ships one — `normalize_vi_text`,
|
| 163 |
-
applied by default (see the [GitHub README](https://github.com/zeroweight-ai/ZeroTTS#usage)) — which reproduces the benchmark's
|
| 164 |
-
reading on 34 of the 35 items that need normalization. Neither baseline ships a
|
| 165 |
-
Vietnamese frontend at all, which is why the raw-text table below is so much
|
| 166 |
-
harsher on them.
|
| 167 |
-
|
| 168 |
-
| | **ZeroTTS** | OmniVoice | XTTS-v2-vietnamse | viXTTS |
|
| 169 |
-
|---|:-:|:-:|:-:|:-:|
|
| 170 |
-
| **WER** ↓ | **0.56 %** | 2.12 % | 7.27 % | 8.61 % |
|
| 171 |
-
| **Naturalness** (UTMOS) ↑ | **2.91** | 2.75 | 2.49 | 2.34 |
|
| 172 |
-
| **Voice similarity** (SSIM) ↑ | 0.938 | **0.951** | 0.941 | 0.935 |
|
| 173 |
-
| **Dead air** (excess silence) ↓ | **0.029 s** | 0.386 s | 0.568 s | 0.215 s |
|
| 174 |
-
| Size | **81 M**, CPU | 3.1 GB, GPU | 1.9 GB, GPU | 1.9 GB, GPU |
|
| 175 |
-
|
| 176 |
-
**4× fewer word errors than the next-best system**, ~0.2 MOS more natural, an
|
| 177 |
-
order of magnitude less dead air — from a model small enough to run real-time
|
| 178 |
-
on a laptop CPU. Median WER is **0.00 %** on all four subsets: the typical
|
| 179 |
-
generation is transcribed exactly. (Every figure is from the same
|
| 180 |
-
normalized-text runs, so the rows are mutually consistent.)
|
| 181 |
-
|
| 182 |
-
### WER — normalized text
|
| 183 |
-
|
| 184 |
-
The headline condition: numbers and dates already spoken out, as the shipped
|
| 185 |
-
normalizer produces.
|
| 186 |
-
|
| 187 |
-
| Subset | what it tests | **ZeroTTS** | OmniVoice | XTTS-v2-vietnamse | viXTTS |
|
| 188 |
-
|---|---|:-:|:-:|:-:|:-:|
|
| 189 |
-
| `vietnamese` | monolingual Vietnamese | **0.21 %** | 0.50 % | 7.21 % | 7.54 % |
|
| 190 |
-
| `code_switch` | Vietnamese + embedded English | 0.95 % | **0.46 %** | 10.14 % | 5.86 % |
|
| 191 |
-
| `cross_lingual` | foreign voice prompt → Vietnamese | **0.38 %** | 9.60 % | 4.94 % | 6.61 % |
|
| 192 |
-
| `challenging` | acronyms, dates, %, currency | **0.61 %** | 1.56 % | 5.63 % | 13.44 % |
|
| 193 |
-
| **overall** | | **0.56 %** | **2.12 %** | **7.27 %** | **8.61 %** |
|
| 194 |
-
|
| 195 |
-
### WER — raw text
|
| 196 |
-
|
| 197 |
-
The harder condition: the model is handed `31/12/2025` and `ChatGPT` verbatim
|
| 198 |
-
and has to read them itself, with no normalizer in front. This is what a system
|
| 199 |
-
with no Vietnamese text frontend faces.
|
| 200 |
-
|
| 201 |
-
| Subset | what it tests | **ZeroTTS** | OmniVoice | XTTS-v2-vietnamse | viXTTS |
|
| 202 |
-
|---|---|:-:|:-:|:-:|:-:|
|
| 203 |
-
| `vietnamese` | monolingual Vietnamese | **0.16 %** | 0.50 % | 7.92 % | 9.56 % |
|
| 204 |
-
| `code_switch` | Vietnamese + embedded English | 0.97 % | **0.46 %** | 10.94 % | 9.25 % |
|
| 205 |
-
| `cross_lingual` | foreign voice prompt → Vietnamese | **1.42 %** | 17.71 % | 21.37 % | 27.27 % |
|
| 206 |
-
| `challenging` | acronyms, dates, %, currency | **1.75 %** | 4.46 % | 27.86 % | 31.85 % |
|
| 207 |
-
| **overall** | | **1.03 %** | **4.13 %** | **16.42 %** | **18.40 %** |
|
| 208 |
-
|
| 209 |
-
**Reading these fairly:**
|
| 210 |
-
|
| 211 |
-
* **OmniVoice beats us on two things, and they are worth naming.** Its speaker
|
| 212 |
-
similarity is the best of the four (0.951 vs our 0.938), and on `code_switch`
|
| 213 |
-
it is roughly half our error rate (0.46 % vs 0.95 %). If cloning fidelity or
|
| 214 |
-
English-in-Vietnamese is your priority, it is a genuinely strong option — at
|
| 215 |
-
3.1 GB on a GPU.
|
| 216 |
-
* **OmniVoice's overall figure is dominated by one subset.** `cross_lingual`
|
| 217 |
-
(foreign voice prompt, Vietnamese text) costs it 17.71 % raw against our
|
| 218 |
-
1.42 %, and it is language-dependent — German 0.00 %, Korean 0.13 %, Japanese
|
| 219 |
-
0.41 %. Excluding that subset it lands near 1.7 % raw. Both ASRs agree the
|
| 220 |
-
audio genuinely degrades there, so it is the model, not the scorer.
|
| 221 |
-
* **Normalization is where the weakest systems gain most, and the order does not
|
| 222 |
-
change.** The XTTS tokenizers have no Vietnamese number expansion, so raw text
|
| 223 |
-
punishes them hard (`challenging` 27.86 %) and the normalized column is the
|
| 224 |
-
fairest comparison available — it improves XTTS 2.3× and viXTTS 2.1×, against
|
| 225 |
-
1.8× for us. What remains is the acoustic model.
|
| 226 |
-
* **`vietnamese` barely moves for anyone** (0.16 % → 0.21 % for ZeroTTS). It has
|
| 227 |
-
no digits or acronyms, so there is nothing to normalize — which is the control
|
| 228 |
-
showing the other subsets' gains are real and not a scoring artifact.
|
| 229 |
-
* **On `cross_lingual` our voice similarity is the weak spot** (0.911 vs
|
| 230 |
-
~0.935 for the others): ZeroTTS carries a foreign speaker's timbre into
|
| 231 |
-
Vietnamese slightly less faithfully, while winning that subset's WER by 12×.
|
| 232 |
-
* **The WER definition matters more than the WER.** ZeroBench scores every clip
|
| 233 |
-
with **two ASRs** (`whisper-large-v3` + `PhoWhisper-large`, min taken — neither
|
| 234 |
-
can judge Vietnamese code-switch TTS alone) against **every acceptable
|
| 235 |
-
reading** of the target text, so a system is never charged for an ASR's
|
| 236 |
-
choice between "31/12/2025" and "ba mươi mốt tháng mười hai". Its test suite
|
| 237 |
-
pins that in both directions: format differences must score 0, real
|
| 238 |
-
mispronunciations must still cost.
|
| 239 |
-
* **Our remaining errors are published, not hidden.** Every item scoring above
|
| 240 |
-
0.00 is audited in [docs/BENCHMARKS.md](https://github.com/zeroweight-ai/ZeroTTS/blob/main/docs/BENCHMARKS.md). The two recurring
|
| 241 |
-
ones: a leading zero read aloud (`18/04` → "tháng **không** tư"), and the
|
| 242 |
-
letters `W` and `H` coming out wrong when an acronym has to be spelled —
|
| 243 |
-
`WHO` should be spelled out letter by letter, and instead comes out as
|
| 244 |
-
something like "Hall".
|
| 245 |
-
|
| 246 |
-
Reproduce, or score your own system:
|
| 247 |
-
|
| 248 |
-
```bash
|
| 249 |
-
pip install "zerotts[eval]"
|
| 250 |
-
SYNTH_FROM=text_normalized OUT_DIR=./eval/norm ./evaluation/run_benchmark.sh
|
| 251 |
-
./evaluation/run_benchmark.sh # raw text
|
| 252 |
-
```
|
| 253 |
-
|
| 254 |
-
Not using ZeroTTS? The scorer stands alone — bring wavs from any system:
|
| 255 |
-
|
| 256 |
-
```bash
|
| 257 |
-
huggingface-cli download zeroweight-ai/ZeroBench-TTS --repo-type dataset --local-dir ZeroBench-TTS
|
| 258 |
-
cd ZeroBench-TTS && pip install -r zerobench_eval/requirements.txt
|
| 259 |
-
python -m zerobench_eval manifest --out manifest.jsonl # what to synthesize
|
| 260 |
-
python -m zerobench_eval score --wav_dir my_wavs/ --name MyModel
|
| 261 |
-
```
|
| 262 |
|
| 263 |
## Voices, and voice cloning
|
| 264 |
|
|
@@ -277,25 +161,6 @@ conditioning — no reference transcript, no audio prompt.
|
|
| 277 |
Because a voice is just an array, latents obtained that way drop into
|
| 278 |
`voices/<name>/voice.npz` and work with no code change.
|
| 279 |
|
| 280 |
-
## Repository layout
|
| 281 |
-
|
| 282 |
-
```
|
| 283 |
-
config.json runtime config
|
| 284 |
-
tokenizer.json BPE tokenizer
|
| 285 |
-
null_voice_emb.npy learned unconditional voice prefix
|
| 286 |
-
onnx/text_encoder.onnx text → encoder states (once per utterance)
|
| 287 |
-
onnx/prefix_step.onnx global transformer step (once per frame)
|
| 288 |
-
onnx/local_frame_decode.onnx frame decode + sampling (once per frame)
|
| 289 |
-
onnx/codec/ MOSS-Audio-Tokenizer-Nano decoder (Apache-2.0)
|
| 290 |
-
voices/<name>/voice.npz speaker latents
|
| 291 |
-
```
|
| 292 |
-
|
| 293 |
-
fp32, not quantized: ~900 MB total. Two ONNX Runtime calls per audio frame;
|
| 294 |
-
frames are produced at 12.5 Hz and decoded to 48 kHz.
|
| 295 |
-
|
| 296 |
-
The model architecture, training code, and the ONNX export script are not
|
| 297 |
-
published, and the voice encoder is not included.
|
| 298 |
-
|
| 299 |
## Intended use and limitations
|
| 300 |
|
| 301 |
Built for **Vietnamese**. It handles English words embedded in Vietnamese text
|
|
@@ -328,7 +193,6 @@ no external runtime dependency; the encoder is not included. See
|
|
| 328 |
## License
|
| 329 |
|
| 330 |
ZeroTTS weights and code: **MIT**.
|
| 331 |
-
Bundled MOSS codec decoder: **Apache-2.0**.
|
| 332 |
|
| 333 |
The ZeroBench-TTS *dataset* is CC-BY-NC-4.0 because it redistributes reference
|
| 334 |
audio from VIVOS, viVoice, phoaudiobook and Emilia. That license applies to the
|
|
|
|
| 15 |
- voice-cloning
|
| 16 |
- vietnamese-tts
|
| 17 |
- tieng-viet
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
metrics:
|
| 19 |
- wer
|
| 20 |
model-index:
|
|
|
|
| 102 |
* 🎯 **Ultra-natural** — 2.91 UTMOS, ~0.5 MOS above every other open Vietnamese
|
| 103 |
system, with near-zero dead air (0.029 s vs 0.23–0.53 s).
|
| 104 |
* 🗣️ **Zero-shot voice cloning** — a voice is a small latent array; drop it in
|
| 105 |
+
and the model speaks in it, cloned from as little as 3 seconds of reference
|
| 106 |
+
audio (up to 30 seconds). No fine-tuning, no per-speaker training.
|
| 107 |
+
* ⚡ **Real-time on CPU, streaming** — ~2× faster than real time (RTF 0.5×),
|
| 108 |
+
first audio chunk in ~70 ms. No GPU required.
|
| 109 |
* 🇻🇳 **Built for Vietnamese** — tones, code-switched English, and a built-in
|
| 110 |
normalizer that reads `31/12/2025` and `1.250 tỷ` the way a person would.
|
|
|
|
|
|
|
|
|
|
| 111 |
|
| 112 |
* Code, examples, browser demo: **https://github.com/zeroweight-ai/ZeroTTS**
|
| 113 |
* Benchmark dataset: **https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS**
|
|
|
|
| 120 |
from zerotts import ZeroTTS
|
| 121 |
|
| 122 |
tts = ZeroTTS.from_pretrained("zeroweight-ai/ZeroTTS")
|
| 123 |
+
audio = tts.synthesize("Xin chào các bạn, mình là ZeroTTS.", voice="maichi")
|
| 124 |
tts.save_audio(audio, "out.wav")
|
| 125 |
```
|
| 126 |
|
| 127 |
+
Streaming, with first audio in roughly 70 ms:
|
| 128 |
|
| 129 |
```python
|
| 130 |
+
for chunk in tts.synthesize_stream("Một đoạn văn bản dài hơn…", voice="maichi"):
|
| 131 |
play(chunk) # (1, n) float32 at 48 kHz
|
| 132 |
```
|
| 133 |
|
| 134 |
## Benchmarks
|
| 135 |
|
| 136 |
+
Measured on **[ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)**
|
| 137 |
+
against OmniVoice, XTTS-v2-vietnamse and viXTTS, on normalized text (dates,
|
| 138 |
+
numbers and acronyms already spoken out): **0.56 % WER** (4× fewer errors than
|
| 139 |
+
the next-best system, median 0.00 % on all four subsets), **2.91 UTMOS**, and
|
| 140 |
+
**0.029 s** of dead air. On CPU it runs ~2× faster than real time (RTF 0.5×)
|
| 141 |
+
with first audio in ~70 ms of streaming.
|
| 142 |
|
| 143 |
+
Full comparison tables, per-subset breakdowns, and CPU speed methodology:
|
| 144 |
+
**[docs/BENCHMARKS.md](https://github.com/zeroweight-ai/ZeroTTS/blob/main/docs/BENCHMARKS.md)**
|
| 145 |
+
on GitHub.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 146 |
|
| 147 |
## Voices, and voice cloning
|
| 148 |
|
|
|
|
| 161 |
Because a voice is just an array, latents obtained that way drop into
|
| 162 |
`voices/<name>/voice.npz` and work with no code change.
|
| 163 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
## Intended use and limitations
|
| 165 |
|
| 166 |
Built for **Vietnamese**. It handles English words embedded in Vietnamese text
|
|
|
|
| 193 |
## License
|
| 194 |
|
| 195 |
ZeroTTS weights and code: **MIT**.
|
|
|
|
| 196 |
|
| 197 |
The ZeroBench-TTS *dataset* is CC-BY-NC-4.0 because it redistributes reference
|
| 198 |
audio from VIVOS, viVoice, phoaudiobook and Emilia. That license applies to the
|
banner.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|