JeffeLI/u_cube_xi

🎙️ Uncertainty-Aware Speaker Recognition

This project provides uncertainty-aware speaker recognition models based on ECAPA-TDNN, ResNet34, and ReDimNet-B2.

The core idea is to estimate the uncertainty of speaker embeddings and use it during representation learning, classification, and verification scoring. This allows the model to distinguish between reliable and unreliable speech segments, improving robustness in both in-domain and cross-domain speaker verification.

The implementation is based on the following two works:

• $\mathcal{U}^3$-xi: Pushing the Boundaries of Speaker Recognition via Incorporating Uncertainty
• Towards Robust Uncertainty-Aware Speaker Modeling

The current release focuses on uncertainty-aware speaker embedding learning and speaker verification. The implementation is based on the WeSpeaker codebase.

✨ Key Highlights

The main components of this project include:

• Uncertainty-aware speaker embedding learning
• Uncertainty estimation based on multi-view self-attention
• Uncertainty-aware pooling and classification
• Inter- and intra-speaker-aware uncertainty modeling
• Uncertainty-aware cosine scoring

Note: The current repository provides the implementation of inter-/intra-speaker-aware uncertainty softmax. The Uncertainty-Calibrated Domain Adaptation (UCDA) method is described in the paper but is not currently included in this release.

📦 Installation

Clone the repository and install the dependencies:

git clone https://github.com/mrjunjieli/wespeaker_u_cube.git
cd wespeaker_u_cube

pip install -r requirements.txt
pip install -e .

If you plan to use SpeechBrain interfaces or recipes:

pip install speechbrain

🏋️ Training

The VoxCeleb experiments are located in:

examples/voxceleb/v2

An example training command is:

cd examples/voxceleb/v2

bash run.sh \
  --stage 1 \
  --stop_stage 5 \
  --config conf/ecapa_tdnn_u_cube.yaml

Available uncertainty-aware configurations include:

conf/ecapa_tdnn_u_cube.yaml
conf/resnet_u_cube.yaml
conf/redimnet_u_cube.yaml

For inter-/intra-speaker-aware uncertainty modeling, use the corresponding projection types implemented in wespeaker/models/projections.py:

arc_margin_uncertainty_inter_intra
add_margin_uncertainty_inter_intra
sphereface2_uncertainty_arcguide

📊 Results

Model	# Param.	Loss	Uncertainty-aware cosine score	In-domain							Cross-domain
				Vox1-O		Vox1-E		Vox1-H		RI (%)	CNCeleb		RI (%)
				EER	minDCF	EER	minDCF	EER	minDCF		EER	minDCF
ECAPA512	6.19 M	AAM-Softmax	No	1.069	0.122	1.209	0.136	2.310	0.226	Benchmark	15.314	0.633	Benchmark
		UAAM-Softmax	No	0.856	0.109	1.064	0.121	1.982	0.195	13.57	13.706	0.608	7.23
			Yes	0.782	0.100	1.016	0.115	1.888	0.187	18.64	10.271	1.000	-12.52
		UAAM-Softmax inter-intra	No	0.936	0.102	1.050	0.122	1.978	0.195	13.40	13.974	0.581	8.48
			Yes	0.840	0.086	0.965	0.110	1.833	0.189	21.22	10.781	0.835	-1.16
		AM-Softmax	No	1.005	0.107	1.206	0.133	2.254	0.221	Benchmark	14.162	0.611	Benchmark
		UAM-Softmax inter-intra	No	0.888	0.099	1.076	0.119	1.973	0.186	11.46	12.436	0.553	10.84
			Yes	0.808	0.084	0.991	0.109	1.794	0.178	19.46	9.411	1.000	-15.03
		SphereFace2	No	0.963	0.108	1.121	0.125	1.967	0.199	Benchmark	12.582	0.573	Benchmark
		USphereFace2 inter-intra	No	0.856	0.104	1.035	0.119	1.918	0.196	5.21	12.265	0.550	3.27
			Yes	0.739	0.102	0.965	0.108	1.771	0.178	12.81	10.560	0.624	3.59
ResNet34	6.63 M	AAM-Softmax	No	0.867	0.091	1.049	0.121	1.960	0.192	Benchmark	11.090	0.488	Benchmark
		UAAM-Softmax	No	0.888	0.085	0.900	0.099	1.712	0.175	9.68	11.732	0.513	-5.46
			Yes	0.867	0.078	0.868	0.095	1.641	0.172	13.29	10.082	0.541	-0.89
		UAAM-Softmax inter-intra	No	0.904	0.070	0.933	0.098	1.658	0.165	13.06	12.116	0.505	-6.37
			Yes	0.813	0.075	0.847	0.091	1.532	0.167	17.12	9.631	0.539	1.35
		USphereFace2	No	1.483	0.148	1.451	0.156	2.112	0.206	-36.00	11.441	0.512	-4.04
			Yes	1.340	0.156	1.357	0.150	1.986	0.193	-30.19	10.949	0.499	-0.49
ReDimNet-B2	4.89 M	AAM-Softmax	No	0.782	0.064	0.907	0.097	1.667	0.162	Benchmark	12.385	0.552	Benchmark
		UAAM-Softmax	No	0.649	0.073	0.801	0.089	1.532	0.153	6.09	13.464	0.552	-4.36
			Yes	0.606	0.065	0.779	0.091	1.494	0.157	9.12	9.479	1.000	-28.85
		UAAM-Softmax inter-intra	No	0.686	0.070	0.802	0.090	1.536	0.151	6.06	12.132	0.516	4.28
			Yes	0.627	0.064	0.758	0.088	1.434	0.153	10.84	8.607	0.838	-10.65
		USphereFace2 inter-intra	No	0.622	0.052	0.776	0.085	1.440	0.146	14.92	12.081	0.515	4.58
			Yes	0.622	0.051	0.774	0.084	1.433	0.145	15.56	11.899	0.506	6.13

📚 Citation

If you use this project, please cite the following papers:

@article{li2026ucubexi,
  title={{U^3}-xi: Pushing the Boundaries of Speaker Recognition via Incorporating Uncertainty},
  author={Li, Junjie and Lee, Kong Aik},
  journal={arXiv preprint arXiv:2601.15719},
  year={2026}
}

@article{li2026robust,
  author  = {Junjie Li and Yang Xiao and Kong Aik Lee},
  title   = {Towards Robust Uncertainty-Aware Speaker Modeling},
  journal = {arXiv preprint arXiv:2607.04937},
  year    = {2026},
  url     = {https://arxiv.org/abs/2607.04937}
}

📧 Contact

For questions, please contact:

junjie98.li@connect.polyu.hk