Aratako commited on
Commit
1a315bc
Β·
verified Β·
1 Parent(s): 91f9ad6

Align speaker similarity results with the paper's WavLM model

Browse files
README.md CHANGED
@@ -121,7 +121,7 @@ comparators achieve higher PESQ-WB and STOI.
121
 
122
  dWER/dCER compare ASR transcripts of the reconstructed and original audio
123
  (Whisper-large-v3 for English and Japanese, Paraformer-zh for Mandarin). SIM is
124
- ECAPA-TDNN speaker similarity. Bitrates include duration information.
125
 
126
  The tables below show a subset of the compared codecs. **Bold** and
127
  <ins>underline</ins> mark the best and second-best scores within each bitrate
@@ -132,16 +132,16 @@ band.
132
  | Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 ↑ | DNSMOS ↑ | PESQ-WB ↑ | STOI ↑ | dWER (%) ↓ | SIM ↑ |
133
  |---|---:|---:|---:|---:|---:|---:|---:|---:|
134
  | *~1.2 kbit/s* | | | | | | | | |
135
- | FlexiCodec, Ο„=1, depth 8 | 12.5 | 1.24 | 3.194 | 3.284 | 2.813 | 0.940 | <ins>0.83</ins> | 0.805 |
136
- | MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | 3.183 | 3.291 | 2.677 | 0.927 | 0.86 | <ins>0.879</ins> |
137
- | MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 3.230 | 3.217 | **3.052** | **0.949** | 0.89 | **0.885** |
138
- | DTM-Codec, 80-Hz target | 81.5 | 1.28 | <ins>3.421</ins> | **3.334** | <ins>2.971</ins> | <ins>0.947</ins> | 1.41 | 0.845 |
139
- | **RubatoCodec, depth 8** | 12.5 | 1.23 | **3.478** | <ins>3.327</ins> | 2.684 | 0.931 | **0.75** | 0.862 |
140
  | *~2.4 kbit/s* | | | | | | | | |
141
- | Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 3.230 | 3.273 | <ins>3.177</ins> | <ins>0.958</ins> | <ins>0.53</ins> | 0.931 |
142
- | MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | 3.210 | <ins>3.288</ins> | 3.094 | 0.947 | 0.67 | <ins>0.933</ins> |
143
- | MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | <ins>3.239</ins> | 3.203 | **3.536** | **0.968** | 0.62 | **0.943** |
144
- | **RubatoCodec, depth 16** | 12.5 | 2.43 | **3.486** | **3.314** | 3.029 | 0.950 | **0.50** | 0.920 |
145
 
146
  **JVNV (Japanese emotional speech with nonverbal vocalizations)**
147
 
@@ -151,16 +151,16 @@ every other compared codec exceeds 3%.
151
  | Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 ↑ | DNSMOS ↑ | PESQ-WB ↑ | STOI ↑ | dCER (%) ↓ | SIM ↑ |
152
  |---|---:|---:|---:|---:|---:|---:|---:|---:|
153
  | *~1.2 kbit/s* | | | | | | | | |
154
- | FlexiCodec, Ο„=1, depth 8 | 12.5 | 1.24 | 1.811 | 3.128 | 2.062 | 0.862 | 5.52 | 0.771 |
155
- | MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | **2.681** | **3.173** | 2.295 | 0.872 | 7.88 | 0.860 |
156
- | MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 2.396 | 3.089 | **3.032** | **0.917** | <ins>5.44</ins> | **0.909** |
157
  | DTM-Codec, 80-Hz target | 46.1 | 0.77 | 2.523 | 3.154 | 2.081 | 0.862 | 14.82 | 0.783 |
158
- | **RubatoCodec, depth 8** | 12.5 | 1.23 | <ins>2.628</ins> | <ins>3.164</ins> | <ins>2.650</ins> | <ins>0.907</ins> | **1.92** | <ins>0.903</ins> |
159
  | *~2.4 kbit/s* | | | | | | | | |
160
- | Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 2.638 | 3.097 | <ins>3.041</ins> | <ins>0.931</ins> | <ins>3.01</ins> | <ins>0.945</ins> |
161
- | MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | **2.708** | **3.176** | 2.949 | 0.911 | 4.21 | 0.927 |
162
- | MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | 2.383 | 3.057 | **3.496** | **0.941** | 3.96 | **0.953** |
163
- | **RubatoCodec, depth 16** | 12.5 | 2.43 | <ins>2.651</ins> | <ins>3.138</ins> | 3.032 | 0.929 | **1.54** | 0.941 |
164
 
165
  **Quality versus bitrate** (RubatoCodec is marked with a red star)
166
 
 
121
 
122
  dWER/dCER compare ASR transcripts of the reconstructed and original audio
123
  (Whisper-large-v3 for English and Japanese, Paraformer-zh for Mandarin). SIM is
124
+ cosine similarity of speaker embeddings from a fine-tuned WavLM-Large speaker verification model. Bitrates include duration information.
125
 
126
  The tables below show a subset of the compared codecs. **Bold** and
127
  <ins>underline</ins> mark the best and second-best scores within each bitrate
 
132
  | Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 ↑ | DNSMOS ↑ | PESQ-WB ↑ | STOI ↑ | dWER (%) ↓ | SIM ↑ |
133
  |---|---:|---:|---:|---:|---:|---:|---:|---:|
134
  | *~1.2 kbit/s* | | | | | | | | |
135
+ | FlexiCodec, Ο„=1, depth 8 | 12.5 | 1.24 | 3.194 | 3.284 | 2.813 | 0.940 | <ins>0.83</ins> | 0.846 |
136
+ | MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | 3.183 | 3.291 | 2.677 | 0.927 | 0.86 | 0.848 |
137
+ | MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 3.230 | 3.217 | **3.052** | **0.949** | 0.89 | **0.904** |
138
+ | DTM-Codec, 80-Hz target | 81.5 | 1.28 | <ins>3.421</ins> | **3.334** | <ins>2.971</ins> | <ins>0.947</ins> | 1.41 | <ins>0.871</ins> |
139
+ | **RubatoCodec, depth 8** | 12.5 | 1.23 | **3.478** | <ins>3.327</ins> | 2.684 | 0.931 | **0.75** | <ins>0.871</ins> |
140
  | *~2.4 kbit/s* | | | | | | | | |
141
+ | Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 3.230 | 3.273 | <ins>3.177</ins> | <ins>0.958</ins> | <ins>0.53</ins> | <ins>0.941</ins> |
142
+ | MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | 3.210 | <ins>3.288</ins> | 3.094 | 0.947 | 0.67 | 0.896 |
143
+ | MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | <ins>3.239</ins> | 3.203 | **3.536** | **0.968** | 0.62 | **0.954** |
144
+ | **RubatoCodec, depth 16** | 12.5 | 2.43 | **3.486** | **3.314** | 3.029 | 0.950 | **0.50** | 0.928 |
145
 
146
  **JVNV (Japanese emotional speech with nonverbal vocalizations)**
147
 
 
151
  | Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 ↑ | DNSMOS ↑ | PESQ-WB ↑ | STOI ↑ | dCER (%) ↓ | SIM ↑ |
152
  |---|---:|---:|---:|---:|---:|---:|---:|---:|
153
  | *~1.2 kbit/s* | | | | | | | | |
154
+ | FlexiCodec, Ο„=1, depth 8 | 12.5 | 1.24 | 1.811 | 3.128 | 2.062 | 0.862 | 5.52 | 0.786 |
155
+ | MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | **2.681** | **3.173** | 2.295 | 0.872 | 7.88 | 0.821 |
156
+ | MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 2.396 | 3.089 | **3.032** | **0.917** | <ins>5.44</ins> | **0.924** |
157
  | DTM-Codec, 80-Hz target | 46.1 | 0.77 | 2.523 | 3.154 | 2.081 | 0.862 | 14.82 | 0.783 |
158
+ | **RubatoCodec, depth 8** | 12.5 | 1.23 | <ins>2.628</ins> | <ins>3.164</ins> | <ins>2.650</ins> | <ins>0.907</ins> | **1.92** | <ins>0.907</ins> |
159
  | *~2.4 kbit/s* | | | | | | | | |
160
+ | Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 2.638 | 3.097 | <ins>3.041</ins> | <ins>0.931</ins> | <ins>3.01</ins> | <ins>0.951</ins> |
161
+ | MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | **2.708** | **3.176** | 2.949 | 0.911 | 4.21 | 0.885 |
162
+ | MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | 2.383 | 3.057 | **3.496** | **0.941** | 3.96 | **0.963** |
163
+ | **RubatoCodec, depth 16** | 12.5 | 2.43 | <ins>2.651</ins> | <ins>3.138</ins> | 3.032 | 0.929 | **1.54** | 0.946 |
164
 
165
  **Quality versus bitrate** (RubatoCodec is marked with a red star)
166
 
assets/english_reconstruction.png CHANGED

Git LFS Details

  • SHA256: 5a949195c32ee18e4ac27d6730f4cf3d508ece93ab4e878af16c0f7597b12599
  • Pointer size: 131 Bytes
  • Size of remote file: 156 kB

Git LFS Details

  • SHA256: 205c6ef72050604d5755846f8c3569d217217bbde0288bc4c6d73ea7b1a1aff2
  • Pointer size: 131 Bytes
  • Size of remote file: 132 kB
assets/multilingual_reconstruction.png CHANGED

Git LFS Details

  • SHA256: e2df2fc430a26f2027cba67815355c84e45340d83727c333075fc6f055a53af8
  • Pointer size: 131 Bytes
  • Size of remote file: 215 kB

Git LFS Details

  • SHA256: 41d7bca8ae3650e1b2cbe248fd88c93d272aa1d704c7b5a646b46f601fff2b4e
  • Pointer size: 131 Bytes
  • Size of remote file: 183 kB