Align speaker similarity results with the paper's WavLM model
Browse files- README.md +18 -18
- assets/english_reconstruction.png +2 -2
- assets/multilingual_reconstruction.png +2 -2
README.md
CHANGED
|
@@ -121,7 +121,7 @@ comparators achieve higher PESQ-WB and STOI.
|
|
| 121 |
|
| 122 |
dWER/dCER compare ASR transcripts of the reconstructed and original audio
|
| 123 |
(Whisper-large-v3 for English and Japanese, Paraformer-zh for Mandarin). SIM is
|
| 124 |
-
|
| 125 |
|
| 126 |
The tables below show a subset of the compared codecs. **Bold** and
|
| 127 |
<ins>underline</ins> mark the best and second-best scores within each bitrate
|
|
@@ -132,16 +132,16 @@ band.
|
|
| 132 |
| Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 β | DNSMOS β | PESQ-WB β | STOI β | dWER (%) β | SIM β |
|
| 133 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 134 |
| *~1.2 kbit/s* | | | | | | | | |
|
| 135 |
-
| FlexiCodec, Ο=1, depth 8 | 12.5 | 1.24 | 3.194 | 3.284 | 2.813 | 0.940 | <ins>0.83</ins> | 0.
|
| 136 |
-
| MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | 3.183 | 3.291 | 2.677 | 0.927 | 0.86 |
|
| 137 |
-
| MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 3.230 | 3.217 | **3.052** | **0.949** | 0.89 | **0.
|
| 138 |
-
| DTM-Codec, 80-Hz target | 81.5 | 1.28 | <ins>3.421</ins> | **3.334** | <ins>2.971</ins> | <ins>0.947</ins> | 1.41 | 0.
|
| 139 |
-
| **RubatoCodec, depth 8** | 12.5 | 1.23 | **3.478** | <ins>3.327</ins> | 2.684 | 0.931 | **0.75** | 0.
|
| 140 |
| *~2.4 kbit/s* | | | | | | | | |
|
| 141 |
-
| Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 3.230 | 3.273 | <ins>3.177</ins> | <ins>0.958</ins> | <ins>0.53</ins> | 0.
|
| 142 |
-
| MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | 3.210 | <ins>3.288</ins> | 3.094 | 0.947 | 0.67 |
|
| 143 |
-
| MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | <ins>3.239</ins> | 3.203 | **3.536** | **0.968** | 0.62 | **0.
|
| 144 |
-
| **RubatoCodec, depth 16** | 12.5 | 2.43 | **3.486** | **3.314** | 3.029 | 0.950 | **0.50** | 0.
|
| 145 |
|
| 146 |
**JVNV (Japanese emotional speech with nonverbal vocalizations)**
|
| 147 |
|
|
@@ -151,16 +151,16 @@ every other compared codec exceeds 3%.
|
|
| 151 |
| Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 β | DNSMOS β | PESQ-WB β | STOI β | dCER (%) β | SIM β |
|
| 152 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 153 |
| *~1.2 kbit/s* | | | | | | | | |
|
| 154 |
-
| FlexiCodec, Ο=1, depth 8 | 12.5 | 1.24 | 1.811 | 3.128 | 2.062 | 0.862 | 5.52 | 0.
|
| 155 |
-
| MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | **2.681** | **3.173** | 2.295 | 0.872 | 7.88 | 0.
|
| 156 |
-
| MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 2.396 | 3.089 | **3.032** | **0.917** | <ins>5.44</ins> | **0.
|
| 157 |
| DTM-Codec, 80-Hz target | 46.1 | 0.77 | 2.523 | 3.154 | 2.081 | 0.862 | 14.82 | 0.783 |
|
| 158 |
-
| **RubatoCodec, depth 8** | 12.5 | 1.23 | <ins>2.628</ins> | <ins>3.164</ins> | <ins>2.650</ins> | <ins>0.907</ins> | **1.92** | <ins>0.
|
| 159 |
| *~2.4 kbit/s* | | | | | | | | |
|
| 160 |
-
| Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 2.638 | 3.097 | <ins>3.041</ins> | <ins>0.931</ins> | <ins>3.01</ins> | <ins>0.
|
| 161 |
-
| MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | **2.708** | **3.176** | 2.949 | 0.911 | 4.21 | 0.
|
| 162 |
-
| MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | 2.383 | 3.057 | **3.496** | **0.941** | 3.96 | **0.
|
| 163 |
-
| **RubatoCodec, depth 16** | 12.5 | 2.43 | <ins>2.651</ins> | <ins>3.138</ins> | 3.032 | 0.929 | **1.54** | 0.
|
| 164 |
|
| 165 |
**Quality versus bitrate** (RubatoCodec is marked with a red star)
|
| 166 |
|
|
|
|
| 121 |
|
| 122 |
dWER/dCER compare ASR transcripts of the reconstructed and original audio
|
| 123 |
(Whisper-large-v3 for English and Japanese, Paraformer-zh for Mandarin). SIM is
|
| 124 |
+
cosine similarity of speaker embeddings from a fine-tuned WavLM-Large speaker verification model. Bitrates include duration information.
|
| 125 |
|
| 126 |
The tables below show a subset of the compared codecs. **Bold** and
|
| 127 |
<ins>underline</ins> mark the best and second-best scores within each bitrate
|
|
|
|
| 132 |
| Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 β | DNSMOS β | PESQ-WB β | STOI β | dWER (%) β | SIM β |
|
| 133 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 134 |
| *~1.2 kbit/s* | | | | | | | | |
|
| 135 |
+
| FlexiCodec, Ο=1, depth 8 | 12.5 | 1.24 | 3.194 | 3.284 | 2.813 | 0.940 | <ins>0.83</ins> | 0.846 |
|
| 136 |
+
| MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | 3.183 | 3.291 | 2.677 | 0.927 | 0.86 | 0.848 |
|
| 137 |
+
| MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 3.230 | 3.217 | **3.052** | **0.949** | 0.89 | **0.904** |
|
| 138 |
+
| DTM-Codec, 80-Hz target | 81.5 | 1.28 | <ins>3.421</ins> | **3.334** | <ins>2.971</ins> | <ins>0.947</ins> | 1.41 | <ins>0.871</ins> |
|
| 139 |
+
| **RubatoCodec, depth 8** | 12.5 | 1.23 | **3.478** | <ins>3.327</ins> | 2.684 | 0.931 | **0.75** | <ins>0.871</ins> |
|
| 140 |
| *~2.4 kbit/s* | | | | | | | | |
|
| 141 |
+
| Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 3.230 | 3.273 | <ins>3.177</ins> | <ins>0.958</ins> | <ins>0.53</ins> | <ins>0.941</ins> |
|
| 142 |
+
| MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | 3.210 | <ins>3.288</ins> | 3.094 | 0.947 | 0.67 | 0.896 |
|
| 143 |
+
| MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | <ins>3.239</ins> | 3.203 | **3.536** | **0.968** | 0.62 | **0.954** |
|
| 144 |
+
| **RubatoCodec, depth 16** | 12.5 | 2.43 | **3.486** | **3.314** | 3.029 | 0.950 | **0.50** | 0.928 |
|
| 145 |
|
| 146 |
**JVNV (Japanese emotional speech with nonverbal vocalizations)**
|
| 147 |
|
|
|
|
| 151 |
| Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 β | DNSMOS β | PESQ-WB β | STOI β | dCER (%) β | SIM β |
|
| 152 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 153 |
| *~1.2 kbit/s* | | | | | | | | |
|
| 154 |
+
| FlexiCodec, Ο=1, depth 8 | 12.5 | 1.24 | 1.811 | 3.128 | 2.062 | 0.862 | 5.52 | 0.786 |
|
| 155 |
+
| MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | **2.681** | **3.173** | 2.295 | 0.872 | 7.88 | 0.821 |
|
| 156 |
+
| MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 2.396 | 3.089 | **3.032** | **0.917** | <ins>5.44</ins> | **0.924** |
|
| 157 |
| DTM-Codec, 80-Hz target | 46.1 | 0.77 | 2.523 | 3.154 | 2.081 | 0.862 | 14.82 | 0.783 |
|
| 158 |
+
| **RubatoCodec, depth 8** | 12.5 | 1.23 | <ins>2.628</ins> | <ins>3.164</ins> | <ins>2.650</ins> | <ins>0.907</ins> | **1.92** | <ins>0.907</ins> |
|
| 159 |
| *~2.4 kbit/s* | | | | | | | | |
|
| 160 |
+
| Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 2.638 | 3.097 | <ins>3.041</ins> | <ins>0.931</ins> | <ins>3.01</ins> | <ins>0.951</ins> |
|
| 161 |
+
| MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | **2.708** | **3.176** | 2.949 | 0.911 | 4.21 | 0.885 |
|
| 162 |
+
| MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | 2.383 | 3.057 | **3.496** | **0.941** | 3.96 | **0.963** |
|
| 163 |
+
| **RubatoCodec, depth 16** | 12.5 | 2.43 | <ins>2.651</ins> | <ins>3.138</ins> | 3.032 | 0.929 | **1.54** | 0.946 |
|
| 164 |
|
| 165 |
**Quality versus bitrate** (RubatoCodec is marked with a red star)
|
| 166 |
|
assets/english_reconstruction.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/multilingual_reconstruction.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|