README.md

---
license: apache-2.0
tags:
- speaker-recognition
- uncertainty
- speaker-verification
- speech
- voxceleb
---

# 🎙️ Uncertainty-Aware Speaker Recognition

This project provides uncertainty-aware speaker recognition models based on
ECAPA-TDNN, ResNet34, and ReDimNet-B2.

The core idea is to estimate the uncertainty of speaker embeddings and use it
during representation learning, classification, and verification scoring.
This allows the model to distinguish between reliable and unreliable speech
segments, improving robustness in both in-domain and cross-domain speaker
verification.

The implementation is based on the following two works:

- [$\mathcal{U}^3$-xi: Pushing the Boundaries of Speaker Recognition via Incorporating Uncertainty](https://arxiv.org/abs/2601.15719)
- [Towards Robust Uncertainty-Aware Speaker Modeling](https://arxiv.org/abs/2607.04937)

The current release focuses on uncertainty-aware speaker embedding learning
and speaker verification. The implementation is based on the
[WeSpeaker](https://github.com/wenet-e2e/wespeaker) codebase.

## ✨ Key Highlights

The main components of this project include:

- **Uncertainty-aware speaker embedding learning**
- **Uncertainty estimation based on multi-view self-attention**
- **Uncertainty-aware pooling and classification**
- **Inter- and intra-speaker-aware uncertainty modeling**
- **Uncertainty-aware cosine scoring**

> **Note:** The current repository provides the implementation of
> inter-/intra-speaker-aware uncertainty softmax. The
> Uncertainty-Calibrated Domain Adaptation (UCDA) method is described in the
> paper but is not currently included in this release.

## 📦 Installation

Clone the repository and install the dependencies:

```bash
git clone https://github.com/mrjunjieli/wespeaker_u_cube.git
cd wespeaker_u_cube

pip install -r requirements.txt
pip install -e .
```

If you plan to use SpeechBrain interfaces or recipes:

```bash
pip install speechbrain
```

## 🏋️ Training

The VoxCeleb experiments are located in:

```text
examples/voxceleb/v2
```

An example training command is:

```bash
cd examples/voxceleb/v2

bash run.sh \
  --stage 1 \
  --stop_stage 5 \
  --config conf/ecapa_tdnn_u_cube.yaml
```

Available uncertainty-aware configurations include:

```text
conf/ecapa_tdnn_u_cube.yaml
conf/resnet_u_cube.yaml
conf/redimnet_u_cube.yaml
```

For inter-/intra-speaker-aware uncertainty modeling, use the corresponding
projection types implemented in
`wespeaker/models/projections.py`:

```text
arc_margin_uncertainty_inter_intra
add_margin_uncertainty_inter_intra
sphereface2_uncertainty_arcguide
```



## 📊 Results

<table style="border-collapse: collapse; border-top: 3px solid #000; border-bottom: 3px solid #000;">
  <thead>
    <tr>
      <th rowspan="2">Model</th>
      <th rowspan="2"># Param.</th>
      <th rowspan="2">Loss</th>
      <th rowspan="2">Uncertainty-aware cosine score</th>
      <th colspan="7">In-domain</th>
      <th colspan="3">Cross-domain</th>
    </tr>
    <tr>
      <th colspan="2">Vox1-O</th>
      <th colspan="2">Vox1-E</th>
      <th colspan="2">Vox1-H</th>
      <th rowspan="2">RI (%)</th>
      <th colspan="2">CNCeleb</th>
      <th rowspan="2">RI (%)</th>
    </tr>
    <tr>
      <th colspan="4"></th>
      <th>EER</th>
      <th>minDCF</th>
      <th>EER</th>
      <th>minDCF</th>
      <th>EER</th>
      <th>minDCF</th>
      <th>EER</th>
      <th>minDCF</th>
    </tr>
  </thead>
  <tbody>
    <tr style="border-top: 6px double #000;"><td rowspan="11">ECAPA512</td><td rowspan="11">6.19 M</td><td>AAM-Softmax</td><td>No</td><td>1.069</td><td>0.122</td><td>1.209</td><td>0.136</td><td>2.310</td><td>0.226</td><td>Benchmark</td><td>15.314</td><td>0.633</td><td>Benchmark</td></tr>
    <tr style="border-top: 3px solid #000;"><td rowspan="2">UAAM-Softmax</td><td>No</td><td>0.856</td><td>0.109</td><td>1.064</td><td>0.121</td><td>1.982</td><td>0.195</td><td>13.57</td><td>13.706</td><td>0.608</td><td>7.23</td></tr>
    <tr style="border-top: 3px solid #000;"><td>Yes</td><td>0.782</td><td>0.100</td><td>1.016</td><td>0.115</td><td>1.888</td><td>0.187</td><td>18.64</td><td>10.271</td><td>1.000</td><td>-12.52</td></tr>
    <tr style="border-top: 3px solid #000;"><td rowspan="2"><a href="https://drive.google.com/drive/folders/1PcI8UM5blXePt2ZCnyNycnj0co5a2XLa?usp=sharing">UAAM-Softmax inter-intra</a></td><td>No</td><td>0.936</td><td>0.102</td><td>1.050</td><td>0.122</td><td>1.978</td><td>0.195</td><td>13.40</td><td>13.974</td><td>0.581</td><td>8.48</td></tr>
    <tr style="border-top: 3px solid #000;"><td>Yes</td><td>0.840</td><td>0.086</td><td><strong>0.965</strong></td><td>0.110</td><td>1.833</td><td>0.189</td><td><strong>21.22</strong></td><td>10.781</td><td>0.835</td><td>-1.16</td></tr>
    <tr style="border-top: 3px solid #000;"><td>AM-Softmax</td><td>No</td><td>1.005</td><td>0.107</td><td>1.206</td><td>0.133</td><td>2.254</td><td>0.221</td><td>Benchmark</td><td>14.162</td><td>0.611</td><td>Benchmark</td></tr>
    <tr style="border-top: 3px solid #000;"><td rowspan="2"><a href="https://drive.google.com/drive/folders/1t9SOt2XlNk4RGV8Q5TfEjZb2iUltIWJl?usp=sharing">UAM-Softmax inter-intra</a></td><td>No</td><td>0.888</td><td>0.099</td><td>1.076</td><td>0.119</td><td>1.973</td><td>0.186</td><td>11.46</td><td>12.436</td><td>0.553</td><td><strong>10.84</strong></td></tr>
    <tr style="border-top: 3px solid #000;"><td>Yes</td><td>0.808</td><td><strong>0.084</strong></td><td>0.991</td><td>0.109</td><td>1.794</td><td><strong>0.178</strong></td><td>19.46</td><td><strong>9.411</strong></td><td>1.000</td><td>-15.03</td></tr>
    <tr style="border-top: 3px solid #000;"><td>SphereFace2</td><td>No</td><td>0.963</td><td>0.108</td><td>1.121</td><td>0.125</td><td>1.967</td><td>0.199</td><td>Benchmark</td><td>12.582</td><td>0.573</td><td>Benchmark</td></tr>
    <tr style="border-top: 3px solid #000;"><td rowspan="2"><a href="https://drive.google.com/drive/folders/1Luy6GQ7OBWhdbfTkLcgSN7QV4bz2dl0W?usp=sharing">USphereFace2 inter-intra</a></td><td>No</td><td>0.856</td><td>0.104</td><td>1.035</td><td>0.119</td><td>1.918</td><td>0.196</td><td>5.21</td><td>12.265</td><td><strong>0.550</strong></td><td>3.27</td></tr>
    <tr style="border-top: 3px solid #000;"><td>Yes</td><td><strong>0.739</strong></td><td>0.102</td><td><strong>0.965</strong></td><td><strong>0.108</strong></td><td><strong>1.771</strong></td><td><strong>0.178</strong></td><td>12.81</td><td>10.560</td><td>0.624</td><td>3.59</td></tr>
    <tr style="border-top: 6px double #000;"><td rowspan="7">ResNet34</td><td rowspan="7">6.63 M</td><td>AAM-Softmax</td><td>No</td><td>0.867</td><td>0.091</td><td>1.049</td><td>0.121</td><td>1.960</td><td>0.192</td><td>Benchmark</td><td>11.090</td><td><strong>0.488</strong></td><td>Benchmark</td></tr>
    <tr style="border-top: 3px solid #000;"><td rowspan="2">UAAM-Softmax</td><td>No</td><td>0.888</td><td>0.085</td><td>0.900</td><td>0.099</td><td>1.712</td><td>0.175</td><td>9.68</td><td>11.732</td><td>0.513</td><td>-5.46</td></tr>
    <tr style="border-top: 3px solid #000;"><td>Yes</td><td>0.867</td><td>0.078</td><td>0.868</td><td>0.095</td><td>1.641</td><td>0.172</td><td>13.29</td><td>10.082</td><td>0.541</td><td>-0.89</td></tr>
    <tr style="border-top: 3px solid #000;"><td rowspan="2"><a href="https://drive.google.com/drive/folders/1c9ohIlh6re-R4MdqW3x2wGb6nGvyq6Au?usp=sharing">UAAM-Softmax inter-intra</a></td><td>No</td><td>0.904</td><td><strong>0.070</strong></td><td>0.933</td><td>0.098</td><td>1.658</td><td><strong>0.165</strong></td><td>13.06</td><td>12.116</td><td>0.505</td><td>-6.37</td></tr>
    <tr style="border-top: 3px solid #000;"><td>Yes</td><td><strong>0.813</strong></td><td>0.075</td><td><strong>0.847</strong></td><td><strong>0.091</strong></td><td><strong>1.532</strong></td><td>0.167</td><td><strong>17.12</strong></td><td><strong>9.631</strong></td><td>0.539</td><td><strong>1.35</strong></td></tr>
    <tr style="border-top: 3px solid #000;"><td rowspan="2">USphereFace2</td><td>No</td><td>1.483</td><td>0.148</td><td>1.451</td><td>0.156</td><td>2.112</td><td>0.206</td><td>-36.00</td><td>11.441</td><td>0.512</td><td>-4.04</td></tr>
    <tr style="border-top: 3px solid #000;"><td>Yes</td><td>1.340</td><td>0.156</td><td>1.357</td><td>0.150</td><td>1.986</td><td>0.193</td><td>-30.19</td><td>10.949</td><td>0.499</td><td>-0.49</td></tr>
    <tr style="border-top: 6px double #000;"><td rowspan="7">ReDimNet-B2</td><td rowspan="7">4.89 M</td><td>AAM-Softmax</td><td>No</td><td>0.782</td><td>0.064</td><td>0.907</td><td>0.097</td><td>1.667</td><td>0.162</td><td>Benchmark</td><td>12.385</td><td>0.552</td><td>Benchmark</td></tr>
    <tr style="border-top: 3px solid #000;"><td rowspan="2">UAAM-Softmax</td><td>No</td><td>0.649</td><td>0.073</td><td>0.801</td><td>0.089</td><td>1.532</td><td>0.153</td><td>6.09</td><td>13.464</td><td>0.552</td><td>-4.36</td></tr>
    <tr style="border-top: 3px solid #000;"><td>Yes</td><td><strong>0.606</strong></td><td>0.065</td><td>0.779</td><td>0.091</td><td>1.494</td><td>0.157</td><td>9.12</td><td>9.479</td><td>1.000</td><td>-28.85</td></tr>
    <tr style="border-top: 3px solid #000;"><td rowspan="2"><a href="https://drive.google.com/drive/folders/1vROZR5yv-CG897qEguFXR9xRBvZjQ590?usp=sharing">UAAM-Softmax inter-intra</a></td><td>No</td><td>0.686</td><td>0.070</td><td>0.802</td><td>0.090</td><td>1.536</td><td>0.151</td><td>6.06</td><td>12.132</td><td>0.516</td><td>4.28</td></tr>
    <tr style="border-top: 3px solid #000;"><td>Yes</td><td>0.627</td><td>0.064</td><td><strong>0.758</strong></td><td>0.088</td><td>1.434</td><td>0.153</td><td>10.84</td><td><strong>8.607</strong></td><td>0.838</td><td>-10.65</td></tr>
    <tr style="border-top: 3px solid #000;"><td rowspan="2"><a href="https://drive.google.com/drive/folders/17CKqWU2bq4ugsNpFU2DYxCgbh1te5ikT?usp=sharing">USphereFace2 inter-intra</a></td><td>No</td><td>0.622</td><td>0.052</td><td>0.776</td><td>0.085</td><td>1.440</td><td>0.146</td><td>14.92</td><td>12.081</td><td>0.515</td><td>4.58</td></tr>
    <tr style="border-top: 3px solid #000;"><td>Yes</td><td>0.622</td><td><strong>0.051</strong></td><td>0.774</td><td><strong>0.084</strong></td><td><strong>1.433</strong></td><td><strong>0.145</strong></td><td><strong>15.56</strong></td><td>11.899</td><td><strong>0.506</strong></td><td><strong>6.13</strong></td></tr>
  </tbody>
</table>


## 📚 Citation

If you use this project, please cite the following papers:

```bibtex
@article{li2026ucubexi,
  title={{U^3}-xi: Pushing the Boundaries of Speaker Recognition via Incorporating Uncertainty},
  author={Li, Junjie and Lee, Kong Aik},
  journal={arXiv preprint arXiv:2601.15719},
  year={2026}
}
```

```bibtex
@article{li2026robust,
  author  = {Junjie Li and Yang Xiao and Kong Aik Lee},
  title   = {Towards Robust Uncertainty-Aware Speaker Modeling},
  journal = {arXiv preprint arXiv:2607.04937},
  year    = {2026},
  url     = {https://arxiv.org/abs/2607.04937}
}
```

## 📧 Contact

For questions, please contact:

junjie98.li@connect.polyu.hk