Curry

CURriculum Ranking Loss for Large-Scale Speaker Verification
What and When to Learn

📖 Read our Paper · 💬 Questions? Contact Us

---

Introduction

Curry (CURriculum Ranking) is an adaptive loss for speaker verification. Fixed-margin losses such as ArcFace treat every training sample the same, so mislabeled or heavily degraded utterances inject noisy gradients. Curry estimates each sample's difficulty online from its Sub-center ArcFace confidence, sorts samples into easy / medium / hard tiers using running batch statistics (no extra labels), and learns tier weights that move training from stable identity learning, to manifold refinement, to boundary sharpening.

This repository hosts the Curry speaker-embedding model from the paper: a fully fine-tuned W2V-BERT 2.0 frontend with an MFA backend, trained on 500K+ speaker identities (VoxCeleb1/2, VoxBlink2, CommonVoice), the largest-scale speaker verification system trained to date. It reaches 0.38% EER on VoxCeleb1-O and 1.60% EER on SITW, an 86.8% and 60.0% relative EER reduction over the Sub-center ArcFace baseline.

Quick Start

Installation

pip install torch torchaudio transformers safetensors

Usage

from transformers import AutoModel
import torch
import torchaudio

# Load model (weights + custom code are in this repo)
model = AutoModel.from_pretrained("cmu-mlsp/Curry", trust_remote_code=True)
model.eval()

# Load audio (16 kHz, mono)
waveform, sr = torchaudio.load("audio.wav")
waveform = waveform.mean(dim=0, keepdim=True)
if sr != 16000:
    waveform = torchaudio.functional.resample(waveform, sr, 16000)

# Extract the 192-d speaker embedding
with torch.no_grad():
    outputs = model(waveform)
    speaker_embedding = outputs.embeddings          # [batch, 192]
    frame_features    = outputs.last_hidden_state   # [batch, time, 1024]

Speaker Verification Example

import torch.nn.functional as F

def verify_speaker(model, audio1, audio2, threshold=0.35):
    """Check if two 16 kHz waveforms [1, T] are from the same speaker."""
    with torch.no_grad():
        emb1 = model(audio1).embeddings
        emb2 = model(audio2).embeddings
        similarity = F.cosine_similarity(emb1, emb2)
    return similarity.item() > threshold, similarity.item()

same_speaker, score = verify_speaker(model, waveform1, waveform2)
print(f"Same speaker: {same_speaker}, Score: {score:.4f}")

# or simply, from file paths:
score = model.verify("spk1_a.wav", "spk1_b.wav")

The threshold of 0.35 is the EER operating point on VoxCeleb1-O after eval-set mean subtraction; tune it for your domain.

Evaluate on VoxCeleb1-O

python inference.py --trials veri_test.txt --root /path/to/VoxCeleb1/wav/

---

Performance on Benchmarks

Speaker Verification (EER % ↓ / minDCF ↓)

Loss	VoxCeleb1-O EER	VoxCeleb1-O minDCF	SITW EER	SITW minDCF
Sub-center ArcFace	2.87	0.45	4.00	0.27
Curry (this model)	0.38	0.04	1.60	0.07

The VoxCeleb1-O numbers were re-measured from the released model.safetensors with inference.py (cosine scoring, eval-set mean subtraction, minDCF at p_target = 0.001) and match Table 1 of the paper.

EARS demographic subgroups (EER % ↓)

Subgroup	Sub-center ArcFace	Curry
Female (59 spk)	6.23	1.09
Male (43 spk)	9.95	1.84

See the paper for the full age × gender breakdown.

---

Architecture

Component	Configuration
Frontend	W2V-BERT 2.0, 24 Conformer layers, 1024-d, fully fine-tuned
Input	160-bin log-mel (25 ms window / 10 ms hop, stride 2), computed internally from 16 kHz waveform
Layer aggregation	Learnable softmax-weighted sum over all 25 hidden states
Backend	MFA residual adapter 1024 → 128 → 1024
Pooling	Attentive Statistics Pooling (2048-d)
Head	BatchNorm → Linear(2048, 192) → BatchNorm
Embedding	192-d, cosine scoring
Parameters	581 M

The training classifier (Sub-center ArcFace, K = 3 sub-centers, 385,986 classes) is not shipped; only the embedding network is.

---

Training Details

Setting	Value
Loss	Curry over Sub-center ArcFace + inter-top-k (s = 32, K = 3, top-k = 5, mp = 0.06), code in curry_loss.py
Difficulty tiers	easy: s > μ̂ + σ̂, hard: s < μ̂ − σ̂, medium otherwise; μ̂, σ̂ running stats with momentum 0.01
Curriculum	Phase I easy only → Phase II + medium → Phase III all tiers, then tier logits learnable
Margin schedule	0.2 → 0.3 → 0.35 across phases
Data	VoxCeleb1 + VoxCeleb2 (7K spk), VoxBlink2 (111K spk), CommonVoice (340K+ spk); 3 s crops, Gaussian noise
Optimizer	AdamW, wd 1e-4; lr 5e-6 (frontend), 5e-5 (backend, classifier), 1e-3 (tier logits)
Schedule	Linear warm-up + cosine decay
Batch	128 per GPU, effective 1024
Hardware	8× NVIDIA H100
Precision	fp16 mixed

---

Use Cases

• ✅ Speaker Verification — Verify if two audio samples are from the same speaker
• ✅ Speaker Identification — Enrol speakers and match by cosine similarity
• ✅ Speaker Diarization — Cluster segments by speaker embedding
• ✅ Forensic Audio Analysis — Robust embeddings on degraded, large-scale, multilingual data

---

Files

File	Purpose
model.safetensors	embedding-network weights (fp32)
config.json	model config (includes the embedded W2V-BERT 2.0 config, no extra downloads)
configuration_curry.py, modeling_curry.py	trust_remote_code model definition
curry_loss.py	the Curry loss used for training (reference)
inference.py	pair scoring and VoxCeleb1-O evaluation
veri_test.txt	VoxCeleb1-O trial list

---

Citation

If you find Curry useful, please cite our paper:

@article{baali2026curry,
  title={What and When to Learn: {CURriculum} Ranking Loss for Large-Scale Speaker Verification},
  author={Baali, Massa and Bisht, Sarthak and Singh, Rita and Raj, Bhiksha},
  journal={arXiv preprint arXiv:2603.24432},
  year={2026}
}

---

Authors

Massa Baali, Sarthak Bisht, Rita Singh, Bhiksha Raj
Carnegie Mellon University
📧 mbaali@cs.cmu.edu