Update Model Card for Auden-Voice

README.md

@@ -1,3 +1,154 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+---
+# Auden-Voice
+
+**Auden-Voice** is a general-purpose voice encoder trained to learn robust speaker representations.
+
+The model is trained using **multi-task learning**, where jointly optimizing speaker identification, emotion, gender, and age classification objectives leads to more general and transferable voice representations.
+
+---
+
+## Model Details
+
+- **Model type**: Voice encoder
+- **Architecture**: Zipformer
+- **Embedding dimension**: 768
+- **Number of parameters**: ~156M
+- **Framework**: PyTorch
+- **Output**: Frame-level embeddings `[B, T, D]`
+- **Pooling**: User-defined (e.g., mean pooling for utterance-level embeddings)
+
+---
+
+## Training
+
+### Training Strategy
+
+Multi-task learning was found to work best. The model is jointly trained on the following tasks:
+
+- Speaker identification
+- Emotion classification
+- Gender classification
+- Age classification
+
+This setup encourages the encoder to learn robust and general-purpose voice representations.
+
+### Training Data
+
+The model is trained on publicly available academic speech datasets, totaling approximately **2050 hours** of audio.
+
+| Task | Dataset(s) | #Samples | Hours |
+|-----|-----------|----------|-------|
+| Speaker Identification | VoxCeleb2 | 974k | 2026 |
+| Paralinguistic Tasks | CREMA-D, RAVDESS, IEMOCAP, TESS | 18.3k | 20 |
+
+### Training Code
+
+Full training scripts and configurations are available at:  
+https://github.com/AudenAI/Auden/tree/main/examples/voice
+
+---
+
+## Intended Use
+
+This model is intended to be used as a **general-purpose voice encoder** for:
+
+- Speaker identification and verification
+- Speaker diarization
+- Emotion, gender, and age classification
+- Audio–text and text–audio retrieval
+- Speech-related downstream tasks that benefit from pretrained voice embeddings
+
+---
+
+## How to Use
+
+### Load the Encoder
+
+```python
+from auden.auto.auto_model import AutoModel
+import torch
+
+encoder = AutoModel.from_pretrained("AudenAI/auden-encoder-voice")
+encoder = encoder.to("cuda" if torch.cuda.is_available() else "cpu")
+
+# Extract Voice Embeddings
+import torch.nn.functional as F
+
+audio_files = ["/path/to/audio1.wav", "/path/to/audio2.wav"]
+embeddings_list = []
+
+for audio_file in audio_files:
+    x, x_lens = encoder.extract_feature([audio_file])
+    x, x_lens = x.to(device), x_lens.to(device)
+
+    with torch.no_grad():
+        encoder_output = encoder(x, x_lens)
+        frame_embeddings = encoder_output["encoder_out"]  # [B, T, D]
+
+        # Global average pooling (example for speaker verification)
+        T = frame_embeddings.size(1)
+        mask = (torch.arange(T, device=device).unsqueeze(0) < x_lens.unsqueeze(1)).unsqueeze(-1).float()
+        utterance_embedding = (frame_embeddings * mask).sum(dim=1) / mask.sum(dim=1)
+
+        embeddings_list.append(utterance_embedding)
+
+embeddings = torch.cat(embeddings_list, dim=0)  # [N, D]
+embeddings = F.normalize(embeddings, p=2, dim=-1)
+
+similarity = torch.matmul(embeddings[0], embeddings[1])
+print(f"Cosine similarity: {similarity:.4f}")
+
+
+# Expected Output
+🎵 Audio 1:
+   Frame embeddings shape: torch.Size([1, 97, 768])
+   Utterance embedding shape: torch.Size([1, 768])
+
+🎵 Audio 2: 
+   Frame embeddings shape: torch.Size([1, 138, 768])
+   Utterance embedding shape: torch.Size([1, 768])
+
+Cosine similarity: 0.7234
+Same speaker: YES
+```
+
+## Performance
+
+| Task - Dataset | Metric |
+|----------------|--------|
+| Speaker Identification - VoxCeleb2 | Accuracy 95.25% |
+| Speaker Verification - VoxCeleb1-O | EER 3% |
+| Speaker Diarization - VoxConverse | DER 17% |
+| Age Classification - CREMA-D | Accuracy 93.91% |
+| Gender Classification - CREMA-D | Accuracy 99.72% |
+| Gender Classification - RAVDESS | Accuracy 100% |
+| Emotion Classification - CREMA-D | Accuracy 83.99% |
+| Emotion Classification - RAVDESS | Accuracy 89.71% |
+| Audio → Text Retrieval - ParaspeechCaps | R@1 63.31 |
+| Text → Audio Retrieval - ParaspeechCaps | R@1 61.69 |
+| LLM-QA Emotion - AirBench-MELD | Accuracy 27.23% |
+| LLM-QA Emotion - AirBench-IEMOCAP | Accuracy 84.70% |
+| LLM-QA Gender - AirBench-MELD | Accuracy 81.58% |
+| LLM-QA Gender - AirBench-CommonVoice | Accuracy 93.15% |
+| LLM-QA Age - AirBench-CommonVoice | Accuracy 58.27% |
+
+
+## Limitations
+
+- The model is trained primarily on English speech data and may not generalize well to other languages.
+- The model is not evaluated on generative tasks such as speech synthesis or voice conversion.
+- Utterance-level representations depend on the pooling strategy selected by the user.
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```bibtex
+@article{huo2025auden,
+  title={Auden-Voice: General-Purpose Voice Encoder for Speech and Language Understanding},
+  author={Huo, Mingyue and Tseng, Wei-Cheng and Shao, Yiwen and Zhang, Hao and Yu, Dong},
+  journal={arXiv preprint arXiv:2511.15145},
+  year={2025}
+}