| --- |
| license: apache-2.0 |
| tags: |
| - speaker-recognition |
| - uncertainty |
| - speaker-verification |
| - speech |
| - voxceleb |
| --- |
| |
| # 🎙️ Uncertainty-Aware Speaker Recognition |
|
|
| This project provides uncertainty-aware speaker recognition models based on |
| ECAPA-TDNN, ResNet34, and ReDimNet-B2. |
|
|
| The core idea is to estimate the uncertainty of speaker embeddings and use it |
| during representation learning, classification, and verification scoring. |
| This allows the model to distinguish between reliable and unreliable speech |
| segments, improving robustness in both in-domain and cross-domain speaker |
| verification. |
|
|
| The implementation is based on the following two works: |
|
|
| - [$\mathcal{U}^3$-xi: Pushing the Boundaries of Speaker Recognition via Incorporating Uncertainty](https://arxiv.org/abs/2601.15719) |
| - [Towards Robust Uncertainty-Aware Speaker Modeling](https://arxiv.org/abs/2607.04937) |
|
|
| The current release focuses on uncertainty-aware speaker embedding learning |
| and speaker verification. The implementation is based on the |
| [WeSpeaker](https://github.com/wenet-e2e/wespeaker) codebase. |
|
|
| ## ✨ Key Highlights |
|
|
| The main components of this project include: |
|
|
| - **Uncertainty-aware speaker embedding learning** |
| - **Uncertainty estimation based on multi-view self-attention** |
| - **Uncertainty-aware pooling and classification** |
| - **Inter- and intra-speaker-aware uncertainty modeling** |
| - **Uncertainty-aware cosine scoring** |
|
|
| > **Note:** The current repository provides the implementation of |
| > inter-/intra-speaker-aware uncertainty softmax. The |
| > Uncertainty-Calibrated Domain Adaptation (UCDA) method is described in the |
| > paper but is not currently included in this release. |
|
|
| ## 📦 Installation |
|
|
| Clone the repository and install the dependencies: |
|
|
| ```bash |
| git clone https://github.com/mrjunjieli/wespeaker_u_cube.git |
| cd wespeaker_u_cube |
| |
| pip install -r requirements.txt |
| pip install -e . |
| ``` |
|
|
| If you plan to use SpeechBrain interfaces or recipes: |
|
|
| ```bash |
| pip install speechbrain |
| ``` |
|
|
| ## 🏋️ Training |
|
|
| The VoxCeleb experiments are located in: |
|
|
| ```text |
| examples/voxceleb/v2 |
| ``` |
|
|
| An example training command is: |
|
|
| ```bash |
| cd examples/voxceleb/v2 |
| |
| bash run.sh \ |
| --stage 1 \ |
| --stop_stage 5 \ |
| --config conf/ecapa_tdnn_u_cube.yaml |
| ``` |
|
|
| Available uncertainty-aware configurations include: |
|
|
| ```text |
| conf/ecapa_tdnn_u_cube.yaml |
| conf/resnet_u_cube.yaml |
| conf/redimnet_u_cube.yaml |
| ``` |
|
|
| For inter-/intra-speaker-aware uncertainty modeling, use the corresponding |
| projection types implemented in |
| `wespeaker/models/projections.py`: |
|
|
| ```text |
| arc_margin_uncertainty_inter_intra |
| add_margin_uncertainty_inter_intra |
| sphereface2_uncertainty_arcguide |
| ``` |
|
|
|
|
|
|
| ## 📊 Results |
|
|
| <table style="border-collapse: collapse; border-top: 3px solid #000; border-bottom: 3px solid #000;"> |
| <thead> |
| <tr> |
| <th rowspan="2">Model</th> |
| <th rowspan="2"># Param.</th> |
| <th rowspan="2">Loss</th> |
| <th rowspan="2">Uncertainty-aware cosine score</th> |
| <th colspan="7">In-domain</th> |
| <th colspan="3">Cross-domain</th> |
| </tr> |
| <tr> |
| <th colspan="2">Vox1-O</th> |
| <th colspan="2">Vox1-E</th> |
| <th colspan="2">Vox1-H</th> |
| <th rowspan="2">RI (%)</th> |
| <th colspan="2">CNCeleb</th> |
| <th rowspan="2">RI (%)</th> |
| </tr> |
| <tr> |
| <th colspan="4"></th> |
| <th>EER</th> |
| <th>minDCF</th> |
| <th>EER</th> |
| <th>minDCF</th> |
| <th>EER</th> |
| <th>minDCF</th> |
| <th>EER</th> |
| <th>minDCF</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr style="border-top: 6px double #000;"><td rowspan="11">ECAPA512</td><td rowspan="11">6.19 M</td><td>AAM-Softmax</td><td>No</td><td>1.069</td><td>0.122</td><td>1.209</td><td>0.136</td><td>2.310</td><td>0.226</td><td>Benchmark</td><td>15.314</td><td>0.633</td><td>Benchmark</td></tr> |
| <tr style="border-top: 3px solid #000;"><td rowspan="2">UAAM-Softmax</td><td>No</td><td>0.856</td><td>0.109</td><td>1.064</td><td>0.121</td><td>1.982</td><td>0.195</td><td>13.57</td><td>13.706</td><td>0.608</td><td>7.23</td></tr> |
| <tr style="border-top: 3px solid #000;"><td>Yes</td><td>0.782</td><td>0.100</td><td>1.016</td><td>0.115</td><td>1.888</td><td>0.187</td><td>18.64</td><td>10.271</td><td>1.000</td><td>-12.52</td></tr> |
| <tr style="border-top: 3px solid #000;"><td rowspan="2"><a href="https://drive.google.com/drive/folders/1PcI8UM5blXePt2ZCnyNycnj0co5a2XLa?usp=sharing">UAAM-Softmax inter-intra</a></td><td>No</td><td>0.936</td><td>0.102</td><td>1.050</td><td>0.122</td><td>1.978</td><td>0.195</td><td>13.40</td><td>13.974</td><td>0.581</td><td>8.48</td></tr> |
| <tr style="border-top: 3px solid #000;"><td>Yes</td><td>0.840</td><td>0.086</td><td><strong>0.965</strong></td><td>0.110</td><td>1.833</td><td>0.189</td><td><strong>21.22</strong></td><td>10.781</td><td>0.835</td><td>-1.16</td></tr> |
| <tr style="border-top: 3px solid #000;"><td>AM-Softmax</td><td>No</td><td>1.005</td><td>0.107</td><td>1.206</td><td>0.133</td><td>2.254</td><td>0.221</td><td>Benchmark</td><td>14.162</td><td>0.611</td><td>Benchmark</td></tr> |
| <tr style="border-top: 3px solid #000;"><td rowspan="2"><a href="https://drive.google.com/drive/folders/1t9SOt2XlNk4RGV8Q5TfEjZb2iUltIWJl?usp=sharing">UAM-Softmax inter-intra</a></td><td>No</td><td>0.888</td><td>0.099</td><td>1.076</td><td>0.119</td><td>1.973</td><td>0.186</td><td>11.46</td><td>12.436</td><td>0.553</td><td><strong>10.84</strong></td></tr> |
| <tr style="border-top: 3px solid #000;"><td>Yes</td><td>0.808</td><td><strong>0.084</strong></td><td>0.991</td><td>0.109</td><td>1.794</td><td><strong>0.178</strong></td><td>19.46</td><td><strong>9.411</strong></td><td>1.000</td><td>-15.03</td></tr> |
| <tr style="border-top: 3px solid #000;"><td>SphereFace2</td><td>No</td><td>0.963</td><td>0.108</td><td>1.121</td><td>0.125</td><td>1.967</td><td>0.199</td><td>Benchmark</td><td>12.582</td><td>0.573</td><td>Benchmark</td></tr> |
| <tr style="border-top: 3px solid #000;"><td rowspan="2"><a href="https://drive.google.com/drive/folders/1Luy6GQ7OBWhdbfTkLcgSN7QV4bz2dl0W?usp=sharing">USphereFace2 inter-intra</a></td><td>No</td><td>0.856</td><td>0.104</td><td>1.035</td><td>0.119</td><td>1.918</td><td>0.196</td><td>5.21</td><td>12.265</td><td><strong>0.550</strong></td><td>3.27</td></tr> |
| <tr style="border-top: 3px solid #000;"><td>Yes</td><td><strong>0.739</strong></td><td>0.102</td><td><strong>0.965</strong></td><td><strong>0.108</strong></td><td><strong>1.771</strong></td><td><strong>0.178</strong></td><td>12.81</td><td>10.560</td><td>0.624</td><td>3.59</td></tr> |
| <tr style="border-top: 6px double #000;"><td rowspan="7">ResNet34</td><td rowspan="7">6.63 M</td><td>AAM-Softmax</td><td>No</td><td>0.867</td><td>0.091</td><td>1.049</td><td>0.121</td><td>1.960</td><td>0.192</td><td>Benchmark</td><td>11.090</td><td><strong>0.488</strong></td><td>Benchmark</td></tr> |
| <tr style="border-top: 3px solid #000;"><td rowspan="2">UAAM-Softmax</td><td>No</td><td>0.888</td><td>0.085</td><td>0.900</td><td>0.099</td><td>1.712</td><td>0.175</td><td>9.68</td><td>11.732</td><td>0.513</td><td>-5.46</td></tr> |
| <tr style="border-top: 3px solid #000;"><td>Yes</td><td>0.867</td><td>0.078</td><td>0.868</td><td>0.095</td><td>1.641</td><td>0.172</td><td>13.29</td><td>10.082</td><td>0.541</td><td>-0.89</td></tr> |
| <tr style="border-top: 3px solid #000;"><td rowspan="2"><a href="https://drive.google.com/drive/folders/1c9ohIlh6re-R4MdqW3x2wGb6nGvyq6Au?usp=sharing">UAAM-Softmax inter-intra</a></td><td>No</td><td>0.904</td><td><strong>0.070</strong></td><td>0.933</td><td>0.098</td><td>1.658</td><td><strong>0.165</strong></td><td>13.06</td><td>12.116</td><td>0.505</td><td>-6.37</td></tr> |
| <tr style="border-top: 3px solid #000;"><td>Yes</td><td><strong>0.813</strong></td><td>0.075</td><td><strong>0.847</strong></td><td><strong>0.091</strong></td><td><strong>1.532</strong></td><td>0.167</td><td><strong>17.12</strong></td><td><strong>9.631</strong></td><td>0.539</td><td><strong>1.35</strong></td></tr> |
| <tr style="border-top: 3px solid #000;"><td rowspan="2">USphereFace2</td><td>No</td><td>1.483</td><td>0.148</td><td>1.451</td><td>0.156</td><td>2.112</td><td>0.206</td><td>-36.00</td><td>11.441</td><td>0.512</td><td>-4.04</td></tr> |
| <tr style="border-top: 3px solid #000;"><td>Yes</td><td>1.340</td><td>0.156</td><td>1.357</td><td>0.150</td><td>1.986</td><td>0.193</td><td>-30.19</td><td>10.949</td><td>0.499</td><td>-0.49</td></tr> |
| <tr style="border-top: 6px double #000;"><td rowspan="7">ReDimNet-B2</td><td rowspan="7">4.89 M</td><td>AAM-Softmax</td><td>No</td><td>0.782</td><td>0.064</td><td>0.907</td><td>0.097</td><td>1.667</td><td>0.162</td><td>Benchmark</td><td>12.385</td><td>0.552</td><td>Benchmark</td></tr> |
| <tr style="border-top: 3px solid #000;"><td rowspan="2">UAAM-Softmax</td><td>No</td><td>0.649</td><td>0.073</td><td>0.801</td><td>0.089</td><td>1.532</td><td>0.153</td><td>6.09</td><td>13.464</td><td>0.552</td><td>-4.36</td></tr> |
| <tr style="border-top: 3px solid #000;"><td>Yes</td><td><strong>0.606</strong></td><td>0.065</td><td>0.779</td><td>0.091</td><td>1.494</td><td>0.157</td><td>9.12</td><td>9.479</td><td>1.000</td><td>-28.85</td></tr> |
| <tr style="border-top: 3px solid #000;"><td rowspan="2"><a href="https://drive.google.com/drive/folders/1vROZR5yv-CG897qEguFXR9xRBvZjQ590?usp=sharing">UAAM-Softmax inter-intra</a></td><td>No</td><td>0.686</td><td>0.070</td><td>0.802</td><td>0.090</td><td>1.536</td><td>0.151</td><td>6.06</td><td>12.132</td><td>0.516</td><td>4.28</td></tr> |
| <tr style="border-top: 3px solid #000;"><td>Yes</td><td>0.627</td><td>0.064</td><td><strong>0.758</strong></td><td>0.088</td><td>1.434</td><td>0.153</td><td>10.84</td><td><strong>8.607</strong></td><td>0.838</td><td>-10.65</td></tr> |
| <tr style="border-top: 3px solid #000;"><td rowspan="2"><a href="https://drive.google.com/drive/folders/17CKqWU2bq4ugsNpFU2DYxCgbh1te5ikT?usp=sharing">USphereFace2 inter-intra</a></td><td>No</td><td>0.622</td><td>0.052</td><td>0.776</td><td>0.085</td><td>1.440</td><td>0.146</td><td>14.92</td><td>12.081</td><td>0.515</td><td>4.58</td></tr> |
| <tr style="border-top: 3px solid #000;"><td>Yes</td><td>0.622</td><td><strong>0.051</strong></td><td>0.774</td><td><strong>0.084</strong></td><td><strong>1.433</strong></td><td><strong>0.145</strong></td><td><strong>15.56</strong></td><td>11.899</td><td><strong>0.506</strong></td><td><strong>6.13</strong></td></tr> |
| </tbody> |
| </table> |
| |
|
|
| ## 📚 Citation |
|
|
| If you use this project, please cite the following papers: |
|
|
| ```bibtex |
| @article{li2026ucubexi, |
| title={{U^3}-xi: Pushing the Boundaries of Speaker Recognition via Incorporating Uncertainty}, |
| author={Li, Junjie and Lee, Kong Aik}, |
| journal={arXiv preprint arXiv:2601.15719}, |
| year={2026} |
| } |
| ``` |
|
|
| ```bibtex |
| @article{li2026robust, |
| author = {Junjie Li and Yang Xiao and Kong Aik Lee}, |
| title = {Towards Robust Uncertainty-Aware Speaker Modeling}, |
| journal = {arXiv preprint arXiv:2607.04937}, |
| year = {2026}, |
| url = {https://arxiv.org/abs/2607.04937} |
| } |
| ``` |
|
|
| ## 📧 Contact |
|
|
| For questions, please contact: |
|
|
| junjie98.li@connect.polyu.hk |
|
|
