update benchmark

README.md

@@ -29,8 +29,8 @@ model-index:
           split: test
         metrics:
           - type: wer
-            value: 0.56
-            name: WER (%) — normalized text
+            value: 1.03
+            name: WER (%) — raw text
           - type: utmos
             value: 2.91
             name: UTMOSv2 naturalness MOS
@@ -50,8 +50,8 @@ model-index:
           split: test
         metrics:
           - type: wer
-            value: 0.21
-            name: WER (%) — normalized text
+            value: 0.16
+            name: WER (%) — raw text
       - task:
           type: text-to-speech
           name: Zero-Shot TTS — Vietnamese/English code-switching
@@ -62,8 +62,8 @@ model-index:
           split: test
         metrics:
           - type: wer
-            value: 0.95
-            name: WER (%) — normalized text
+            value: 0.97
+            name: WER (%) — raw text
       - task:
           type: text-to-speech
           name: Zero-Shot TTS — cross-lingual voice prompt
@@ -74,8 +74,8 @@ model-index:
           split: test
         metrics:
           - type: wer
-            value: 0.38
-            name: WER (%) — normalized text
+            value: 1.42
+            name: WER (%) — raw text
       - task:
           type: text-to-speech
           name: Zero-Shot TTS — acronyms, dates, numbers
@@ -86,8 +86,8 @@ model-index:
           split: test
         metrics:
           - type: wer
-            value: 0.61
-            name: WER (%) — normalized text
+            value: 1.75
+            name: WER (%) — raw text
 ---
 
 <img src="banner.png" alt="ZeroTTS — Vietnamese zero-shot text-to-speech" width="100%">
@@ -185,15 +185,15 @@ with sd.OutputStream(samplerate=tts.sample_rate, channels=1,
 
 Measured on **[ZeroBench-TTS](https://huggingface.co/datasets/zeroweight-ai/ZeroBench-TTS)**
 
-Every system reads **normalized text** — dates, numbers and acronyms already
-spoken out, from the benchmark's own curated reading. 
+Every system reads **raw text** — dates, numbers and acronyms verbatim, exactly
+as they appear in the wild, with no text frontend in front of the model. 
 
 | | **ZeroTTS** | OmniVoice | XTTS-v2-vietnamse | viXTTS |
 |---|:-:|:-:|:-:|:-:|
-| **WER** ↓ | **0.56 %** | 2.12 % | 7.27 % | 8.61 % |
-| **Naturalness** (UTMOS) ↑ | **2.91** | 2.75 | 2.49 | 2.34 |
-| **Voice similarity** (SSIM) ↑ | 0.938 | **0.951** | 0.941 | 0.935 |
-| **Dead air** (excess silence) ↓ | **0.029 s** | 0.386 s | 0.568 s | 0.215 s |
+| **WER** ↓ | **1.03 %** | 4.13 % | 16.42 % | 18.40 % |
+| **Naturalness** (UTMOS) ↑ | **2.91** | 2.76 | 2.43 | 2.35 |
+| **Voice similarity** (SSIM) ↑ | 0.936 | **0.950** | 0.940 | 0.935 |
+| **Dead air** (excess silence) ↓ | **0.029 s** | 0.340 s | 0.532 s | 0.233 s |
 | **RTF, CPU** ↓ | **0.50×** | 6.12× | 0.71× | 0.73× |
 | **Time to first audio, CPU** ↓ | **~70 ms** | ~34 s | ~6.1 s | ~5.1 s |
 | Size | **202 M** params, 0.86 GB fp32, CPU | 3.1 GB, GPU | 1.9 GB, GPU | 1.9 GB, GPU |