Update README benchmarks section with full comparison and CPU speed tables
#8
by zeroweightai - opened
README.md
CHANGED
|
@@ -134,15 +134,49 @@ for chunk in tts.synthesize_stream("Một đoạn văn bản dài hơn…", voic
|
|
| 134 |
## Benchmarks
|
| 135 |
|
| 136 |
Measured on **[ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)**
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 142 |
|
| 143 |
Full comparison tables, per-subset breakdowns, and CPU speed methodology:
|
| 144 |
**[docs/BENCHMARKS.md](https://github.com/zeroweight-ai/ZeroTTS/blob/main/docs/BENCHMARKS.md)**
|
| 145 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 146 |
|
| 147 |
## Voices, and voice cloning
|
| 148 |
|
|
|
|
| 134 |
## Benchmarks
|
| 135 |
|
| 136 |
Measured on **[ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)**
|
| 137 |
+
|
| 138 |
+
Every system reads **normalized text** — dates, numbers and acronyms already
|
| 139 |
+
spoken out, from the benchmark's own curated reading.
|
| 140 |
+
|
| 141 |
+
| | **ZeroTTS** | OmniVoice | XTTS-v2-vietnamse | viXTTS |
|
| 142 |
+
|---|:-:|:-:|:-:|:-:|
|
| 143 |
+
| **WER** ↓ | **0.56 %** | 2.12 % | 7.27 % | 8.61 % |
|
| 144 |
+
| **Naturalness** (UTMOS) ↑ | **2.91** | 2.75 | 2.49 | 2.34 |
|
| 145 |
+
| **Voice similarity** (SSIM) ↑ | 0.938 | **0.951** | 0.941 | 0.935 |
|
| 146 |
+
| **Dead air** (excess silence) ↓ | **0.029 s** | 0.386 s | 0.568 s | 0.215 s |
|
| 147 |
+
| **RTF, CPU** ↓ | **0.50×** | 6.12× | 0.71× | 0.73× |
|
| 148 |
+
| **Time to first audio, CPU** ↓ | **~70 ms** | ~34 s | ~6.1 s | ~5.1 s |
|
| 149 |
+
| Size | **81 M**, CPU | 3.1 GB, GPU | 1.9 GB, GPU | 1.9 GB, GPU |
|
| 150 |
+
|
| 151 |
+
**4× fewer word errors than the next-best system**, and the only one of the four
|
| 152 |
+
that's actually fast on CPU — the others are sized and tuned for a GPU, and it
|
| 153 |
+
shows.
|
| 154 |
|
| 155 |
Full comparison tables, per-subset breakdowns, and CPU speed methodology:
|
| 156 |
**[docs/BENCHMARKS.md](https://github.com/zeroweight-ai/ZeroTTS/blob/main/docs/BENCHMARKS.md)**
|
| 157 |
+
|
| 158 |
+
### Speed — CPU
|
| 159 |
+
|
| 160 |
+
RTF (realtime factor, wall-clock synthesis time ÷ output audio duration — lower
|
| 161 |
+
is faster; below 1× is faster than real time) and time-to-first-audio, all
|
| 162 |
+
measured **on CPU**, single request, 8 inference threads pinned to a dedicated
|
| 163 |
+
core pool (no other synthesis running concurrently). Three Vietnamese samples —
|
| 164 |
+
short (26 chars), medium (77 chars), long (227 chars) — each run 6 times with
|
| 165 |
+
the first 2 (cold-cache) discarded; figures below are the mean of the
|
| 166 |
+
remaining 4.
|
| 167 |
+
|
| 168 |
+
| | **ZeroTTS** | OmniVoice | XTTS-v2-vietnamse | viXTTS |
|
| 169 |
+
|---|:-:|:-:|:-:|:-:|
|
| 170 |
+
| RTF — short | **0.51×** | 10.87× | 0.70× | 0.71× |
|
| 171 |
+
| RTF — medium | **0.47×** | 4.82× | 0.70× | 0.70× |
|
| 172 |
+
| RTF — long | **0.53×** | 2.67× | 0.71× | 0.78× |
|
| 173 |
+
| TTFA — short | **53 ms** | 21.7 s | 4.02 s | 2.45 s |
|
| 174 |
+
| TTFA — medium | **66 ms** | 28.9 s | 4.02 s | 3.72 s |
|
| 175 |
+
| TTFA — long | **89 ms** | 52.3 s | 10.3 s | 9.22 s |
|
| 176 |
+
|
| 177 |
+
ZeroTTS's time-to-first-audio comes from its real streaming path — first audio frame,
|
| 178 |
+
not first full utterance. The three baselines have no working CPU streaming
|
| 179 |
+
path, so their TTFA is the time to the complete utterance.
|
| 180 |
|
| 181 |
## Voices, and voice cloning
|
| 182 |
|