Refine evaluation comparisons

README.md

@@ -167,23 +167,38 @@ Lower WER/CER is better; higher SIM and ASV similarity are better.
 
 ### CV3 zero-shot
 
-| Test set | CER/WER | L50 | Bad samples (>50%) | SIM |
-|---|---:|---:|---:|---:|
-| zh (500) | 3.619 | 3.503 | 1 | 62.41 |
-| en (500) | 3.307 | 3.180 | 1 | 53.99 |
-| hard_zh (60) | 17.261 | 8.357 | 6 | 57.60 |
-| hard_en (64) | 8.267 | 6.271 | 2 | 53.56 |
+| Test set | CER/WER | L50 | Bad samples (>50%) |
+|---|---:|---:|---:|
+| zh (500) | 3.619 | 3.503 | 1 |
+| en (500) | 3.307 | 3.180 | 1 |
+| hard_zh (60) | 17.261 | 8.357 | 6 |
+| hard_en (64) | 8.267 | 6.271 | 2 |
 
 ### Multilingual CV3 (500 samples per language)
 
-| Language | WER/CER | L50 | Bad samples | SIM |
-|---|---:|---:|---:|---:|
-| de | 5.292 | 4.746 | 5 | 59.07 |
-| es | 8.548 | 7.636 | 8 | 58.86 |
-| fr | 12.349 | 9.032 | 25 | 54.78 |
-| it | 14.480 | 10.395 | 12 | 55.61 |
-| ja | 12.322 | 10.111 | 15 | 60.06 |
-| ko | 7.653 | 6.535 | 8 | 63.01 |
+| Language | WER/CER | L50 | Bad samples |
+|---|---:|---:|---:|
+| de | 5.292 | 4.746 | 5 |
+| es | 8.548 | 7.636 | 8 |
+| fr | 12.349 | 9.032 | 25 |
+| it | 14.480 | 10.395 | 12 |
+| ja | 12.322 | 10.111 | 15 |
+| ko | 7.653 | 6.535 | 8 |
+
+### CV3 error-rate comparison
+
+Lower is better. These comparison values follow the evaluation table published
+for Audio8 TTS Preview 0.6B; they are reference comparisons rather than a
+strictly matched re-evaluation.
+
+| Model | Parameters | zh | en | hard-zh | hard-en | ja | ko | de | es | fr | it |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| **Audio8 TTS Preview 0.1B** | **~0.17B** | 3.619 | 3.307 | 17.261 | 8.267 | 12.322 | 7.653 | 5.292 | 8.548 | 12.349 | 14.480 |
+| Audio8 TTS Preview 0.6B | 0.6B | **3.205** | **3.128** | 10.535 | 5.997 | 7.205 | 4.223 | 3.447 | 3.641 | 8.790 | 4.790 |
+| Fish S2 Pro | 4.6B | 3.600 | 3.493 | 10.588 | 7.349 | 5.139 | **4.111** | 3.605 | 2.972 | **8.600** | 4.229 |
+| Higgs Audio v2 | 4.7B | 3.378 | 3.404 | 10.424 | **5.754** | **4.742** | 4.260 | **3.300** | **2.929** | 9.425 | **3.555** |
+| CosyVoice3-1.5B | 1.5B | 3.91 | 4.99 | 9.77 | 10.55 | 7.57 | 5.69 | 6.43 | 4.47 | 11.8 | 10.5 |
+| VoxCPM2 | 2.3B | 3.65 | 5.00 | **8.55** | 8.48 | 5.96 | 5.69 | 4.77 | 3.80 | 9.85 | 4.25 |
 
 ### Seed-TTS test set (temperature = 0.1)
 
@@ -193,6 +208,26 @@ Lower WER/CER is better; higher SIM and ASV similarity are better.
 | zh_meta | 1.13% | 0.682 | 2019/2020 |
 | zh_hardcase | 17.504% | 0.63 | 399/400 |
 
+### Seed-TTS comparison
+
+Similarity values are shown as percentages in this comparison table. Lower
+WER/CER is better; higher similarity is better.
+
+| Model | Parameters | EN WER / SIM | ZH CER / SIM | Hard ZH CER / SIM |
+|---|---:|---:|---:|---:|
+| **Audio8 TTS Preview 0.1B** | **~0.17B** | **1.662 / 56.7** | 1.13 / 68.2 | 17.504 / 63.0 |
+| Audio8 TTS Preview 0.6B | 0.6B | 1.506 / 63.2 | 0.950 / 73.1 | 11.510 / 68.7 |
+| Fish S2 Pro | 4.6B | 1.607 / 64.6 | 1.038 / 73.8 | 10.149 / 70.1 |
+| Higgs Audio v2 | 4.7B | 1.524 / 66.4 | **0.806** / 72.1 | 10.622 / 69.3 |
+| CosyVoice3-1.5B | 1.5B | 2.22 / 72.0 | 1.12 / 78.1 | **5.83** / **75.8** |
+| MOSS-TTS | 8.5B | 1.85 / 73.4 | 1.20 / 78.8 | - |
+| VoxCPM2 | 2.3B | 1.84 / **75.3** | 0.97 / **79.5** | 8.13 / 75.3 |
+| IndexTTS2.5 | 0.8B | 5.12 / 68.06 | 4.36 / 77.10 | - |
+
+The IndexTTS2.5 row is transcribed from Table 2 of the
+[IndexTTS 2.5 technical report](https://arxiv.org/abs/2601.03888). Its paper
+does not report the Hard ZH value, so that cell is left as `-`.
+
 The source evaluation image was truncated after the `zh_hardcase` row; only
 the results visible in the source have been included here.