ChristophSchuhmann commited on
Commit
441d220
·
verified ·
1 Parent(s): 875b8af

Add 55 emotion experts from Empathic-Insight-Voice-Small + 4 new quality experts (DNSMOS + AudioBox)

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +378 -0
  2. model_Affection_best.pth +3 -0
  3. model_Age_best.pth +3 -0
  4. model_Amusement_best.pth +3 -0
  5. model_Anger_best.pth +3 -0
  6. model_Arousal_best.pth +3 -0
  7. model_Astonishment_Surprise_best.pth +3 -0
  8. model_Authenticity_best.pth +3 -0
  9. model_Awe_best.pth +3 -0
  10. model_Background_Noise_best.pth +3 -0
  11. model_Bitterness_best.pth +3 -0
  12. model_Concentration_best.pth +3 -0
  13. model_Confident_vs._Hesitant_best.pth +3 -0
  14. model_Confusion_best.pth +3 -0
  15. model_Contemplation_best.pth +3 -0
  16. model_Contempt_best.pth +3 -0
  17. model_Contentment_best.pth +3 -0
  18. model_Disappointment_best.pth +3 -0
  19. model_Disgust_best.pth +3 -0
  20. model_Distress_best.pth +3 -0
  21. model_Doubt_best.pth +3 -0
  22. model_Elation_best.pth +3 -0
  23. model_Embarrassment_best.pth +3 -0
  24. model_Emotional_Numbness_best.pth +3 -0
  25. model_Fatigue_Exhaustion_best.pth +3 -0
  26. model_Fear_best.pth +3 -0
  27. model_Gender_best.pth +3 -0
  28. model_Helplessness_best.pth +3 -0
  29. model_High-Pitched_vs._Low-Pitched_best.pth +3 -0
  30. model_Hope_Enthusiasm_Optimism_best.pth +3 -0
  31. model_Impatience_and_Irritability_best.pth +3 -0
  32. model_Infatuation_best.pth +3 -0
  33. model_Interest_best.pth +3 -0
  34. model_Intoxication_Altered_States_of_Consciousness_best.pth +3 -0
  35. model_Jealousy_&_Envy_best.pth +3 -0
  36. model_Longing_best.pth +3 -0
  37. model_Malevolence_Malice_best.pth +3 -0
  38. model_Monotone_vs._Expressive_best.pth +3 -0
  39. model_Pain_best.pth +3 -0
  40. model_Pleasure_Ecstasy_best.pth +3 -0
  41. model_Pride_best.pth +3 -0
  42. model_Recording_Quality_best.pth +3 -0
  43. model_Relief_best.pth +3 -0
  44. model_Sadness_best.pth +3 -0
  45. model_Serious_vs._Humorous_best.pth +3 -0
  46. model_Sexual_Lust_best.pth +3 -0
  47. model_Shame_best.pth +3 -0
  48. model_Soft_vs._Harsh_best.pth +3 -0
  49. model_Sourness_best.pth +3 -0
  50. model_Submissive_vs._Dominant_best.pth +3 -0
README.md ADDED
@@ -0,0 +1,378 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Empathic-Insight-Voice-Plus
2
+ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1WR-B6j--Y5RdhIyRGF_tJ3YdFF8BkUA2)
3
+
4
+ **Empathic-Insight-Voice-Plus** extends [laion/Empathic-Insight-Voice-Small](https://huggingface.co/laion/Empathic-Insight-Voice-Small) with **4 additional audio quality expert models**. These new experts predict overall audio quality, speech quality, background noise quality, and content enjoyment from the same frozen [laion/BUD-E-Whisper](https://huggingface.co/laion/BUD-E-Whisper) encoder embeddings.
5
+
6
+ This repository is fully compatible with the original Empathic-Insight-Voice-Small suite. It uses the same Whisper encoder (based on OpenAI Whisper Small) and the same MLP hidden architecture (`proj=64, hidden=[64, 32, 16]`). The only difference is that the new quality experts use **pooled features** (mean + min + max + std pooling over the encoder sequence dimension, yielding a 3072-d input) instead of the full flattened sequence, making them significantly more compact (~200K parameters each vs. ~73.7M for the full-sequence emotion experts).
7
+
8
+ This work is based on the research paper:
9
+ **"EMONET-VOICE: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detection"**
10
+
11
+
12
+ ## Example Video Analyses (Top 3 Emotions)
13
+ <div style='display: flex; flex-wrap: wrap; justify-content: flex-start; gap: 15px;'>
14
+ <div style='flex: 0 1 auto; margin-bottom: 15px; text-align: center; width: 480px; max-width: 480px;'>
15
+ <a href='https://www.youtube.com/watch?v=TsTVKCmqHhk' target='_blank' title='Watch video TsTVKCmqHhk'>
16
+ <img src='https://img.youtube.com/vi/TsTVKCmqHhk/hqdefault.jpg' alt='YouTube Thumbnail for TsTVKCmqHhk' style='width: 100%; height: auto; border: 1px solid #ccc; border-radius: 4px; display: block;'>
17
+ </a>
18
+ <p style='font-size: 0.8em; margin-top: 5px; word-break: break-all;'>ID: TsTVKCmqHhk</p>
19
+ </div>
20
+ <div style='flex: 0 1 auto; margin-bottom: 15px; text-align: center; width: 480px; max-width: 480px;'>
21
+ <a href='https://www.youtube.com/watch?v=sErqFgL4vA8' target='_blank' title='Watch video sErqFgL4vA8'>
22
+ <img src='https://img.youtube.com/vi/sErqFgL4vA8/hqdefault.jpg' alt='YouTube Thumbnail for sErqFgL4vA8' style='width: 100%; height: auto; border: 1px solid #ccc; border-radius: 4px; display: block;'>
23
+ </a>
24
+ <p style='font-size: 0.8em; margin-top: 5px; word-break: break-all;'>ID: sErqFgL4vA8</p>
25
+ </div>
26
+ <div style='flex: 0 1 auto; margin-bottom: 15px; text-align: center; width: 480px; max-width: 480px;'>
27
+ <a href='https://www.youtube.com/watch?v=BUnfuiwE_IM' target='_blank' title='Watch video BUnfuiwE_IM'>
28
+ <img src='https://img.youtube.com/vi/BUnfuiwE_IM/hqdefault.jpg' alt='YouTube Thumbnail for BUnfuiwE_IM' style='width: 100%; height: auto; border: 1px solid #ccc; border-radius: 4px; display: block;'>
29
+ </a>
30
+ <p style='font-size: 0.8em; margin-top: 5px; word-break: break-all;'>ID: BUnfuiwE_IM</p>
31
+ </div>
32
+ <div style='flex: 0 1 auto; margin-bottom: 15px; text-align: center; width: 480px; max-width: 480px;'>
33
+ <a href='https://www.youtube.com/watch?v=dDrmjcUq8W4' target='_blank' title='Watch video dDrmjcUq8W4'>
34
+ <img src='https://img.youtube.com/vi/dDrmjcUq8W4/hqdefault.jpg' alt='YouTube Thumbnail for dDrmjcUq8W4' style='width: 100%; height: auto; border: 1px solid #ccc; border-radius: 4px; display: block;'>
35
+ </a>
36
+ <p style='font-size: 0.8em; margin-top: 5px; word-break: break-all;'>ID: dDrmjcUq8W4</p>
37
+ </div>
38
+ </div>
39
+
40
+
41
+ ## Model Description
42
+
43
+ The Empathic-Insight-Voice-Plus suite combines the original 54+ emotion/attribute MLP experts from [Empathic-Insight-Voice-Small](https://huggingface.co/laion/Empathic-Insight-Voice-Small) with 4 new audio quality regression experts. All models use embeddings from the fine-tuned Whisper model [laion/BUD-E-Whisper](https://huggingface.co/laion/BUD-E-Whisper).
44
+
45
+ The original emotion experts were trained on the large-scale, multilingual synthetic voice-acting dataset LAION'S GOT TALENT (~5,000 hours) & an "in the wild" dataset of voice snippets (~5,000 hours). The new quality experts were trained on the [mitermix/balanced-audio-score-datasets](https://huggingface.co/datasets/mitermix/balanced-audio-score-datasets) dataset.
46
+
47
+ The quality scores are distilled from two established audio quality models:
48
+ - **Overall Quality, Speech Quality, Background Quality**: Distilled from Microsoft's [DNSMOS](https://github.com/microsoft/DNS-Challenge) (Deep Noise Suppression Mean Opinion Score), a non-intrusive speech quality estimator.
49
+ - **Content Enjoyment**: Distilled from Meta's [AudioBox](https://ai.meta.com/research/publications/audiobox-unified-audio-generation-with-natural-language-prompts/) content enjoyment scoring model.
50
+
51
+
52
+ ## New Quality Expert Scores
53
+
54
+ | Expert | Description | Source | Expected Range | Typical Mean | Unit |
55
+ |--------|-------------|--------|---------------|--------------|------|
56
+ | **Overall Quality** | Overall perceived audio quality score. Higher is better. | DNSMOS | 1.0 - 3.7 | ~2.4 | MOS-like |
57
+ | **Speech Quality** | Quality of the speech signal itself (clarity, naturalness). Higher is better. | DNSMOS | 1.0 - 2.4 | ~1.9 | MOS-like |
58
+ | **Background Quality** | Quality of the background (absence of noise/artifacts). Higher is better. | DNSMOS | 1.0 - 4.3 | ~3.2 | MOS-like |
59
+ | **Content Enjoyment** | How engaging/enjoyable the spoken content is. Higher is better. | Meta AudioBox | 1.9 - 5.1 | ~4.1 | MOS-like |
60
+
61
+ These scores were trained on the [mitermix/balanced-audio-score-datasets](https://huggingface.co/datasets/mitermix/balanced-audio-score-datasets) dataset.
62
+
63
+ ### Validation Performance
64
+
65
+ | Expert | Val MAE | Pearson r | Val Samples |
66
+ |--------|---------|-----------|-------------|
67
+ | Overall Quality | 0.26 | 0.899 | 200 |
68
+ | Speech Quality | 0.30 | 0.517 | 200 |
69
+ | Background Quality | 0.35 | 0.865 | 200 |
70
+ | Content Enjoyment | 0.34 | 0.691 | 200 |
71
+
72
+
73
+ ## How to Use
74
+
75
+ ### Full Inference: Emotion Scores + Quality Scores
76
+
77
+ The following example loads both the original 54 emotion/attribute experts and the 4 new quality experts, then runs inference on a single audio file.
78
+
79
+ ```python
80
+ import torch
81
+ import torch.nn as nn
82
+ import numpy as np
83
+ import librosa
84
+ from pathlib import Path
85
+ from transformers import WhisperModel, WhisperFeatureExtractor
86
+ from huggingface_hub import snapshot_download
87
+ import gc
88
+
89
+ # --- Configuration ---
90
+ SAMPLING_RATE = 16000
91
+ MAX_AUDIO_SECONDS = 30.0
92
+ WHISPER_MODEL_ID = "laion/BUD-E-Whisper"
93
+
94
+ # Original emotion experts
95
+ HF_EMOTION_REPO_ID = "laion/Empathic-Insight-Voice-Small"
96
+ # New quality experts (this repo)
97
+ HF_QUALITY_REPO_ID = "laion/Empathic-Insight-Voice-Plus"
98
+
99
+ WHISPER_SEQ_LEN = 1500
100
+ WHISPER_EMBED_DIM = 768
101
+ PROJECTION_DIM = 64
102
+ MLP_HIDDEN_DIMS = [64, 32, 16]
103
+ MLP_DROPOUTS = [0.0, 0.1, 0.1, 0.1]
104
+ POOLED_DIM = 3072 # 4 * 768 (mean + min + max + std)
105
+
106
+ DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
107
+
108
+
109
+ # --- Model Definitions ---
110
+
111
+ class FullEmbeddingMLP(nn.Module):
112
+ """Original emotion expert architecture (full sequence input)."""
113
+ def __init__(self, seq_len, embed_dim, projection_dim, mlp_hidden_dims, mlp_dropout_rates):
114
+ super().__init__()
115
+ self.flatten = nn.Flatten()
116
+ self.proj = nn.Linear(seq_len * embed_dim, projection_dim)
117
+ layers = [nn.ReLU(), nn.Dropout(mlp_dropout_rates[0])]
118
+ current_dim = projection_dim
119
+ for i, h_dim in enumerate(mlp_hidden_dims):
120
+ layers.extend([nn.Linear(current_dim, h_dim), nn.ReLU(), nn.Dropout(mlp_dropout_rates[i + 1])])
121
+ current_dim = h_dim
122
+ layers.append(nn.Linear(current_dim, 1))
123
+ self.mlp = nn.Sequential(*layers)
124
+
125
+ def forward(self, x):
126
+ if x.ndim == 4 and x.shape[1] == 1:
127
+ x = x.squeeze(1)
128
+ return self.mlp(self.proj(self.flatten(x)))
129
+
130
+
131
+ class PooledEmbeddingMLP(nn.Module):
132
+ """New quality expert architecture (pooled features input)."""
133
+ def __init__(self, input_dim, projection_dim, mlp_hidden_dims, mlp_dropout_rates):
134
+ super().__init__()
135
+ self.proj = nn.Linear(input_dim, projection_dim)
136
+ layers = [nn.ReLU(), nn.Dropout(mlp_dropout_rates[0])]
137
+ current_dim = projection_dim
138
+ for i, h_dim in enumerate(mlp_hidden_dims):
139
+ layers.extend([nn.Linear(current_dim, h_dim), nn.ReLU(), nn.Dropout(mlp_dropout_rates[i + 1])])
140
+ current_dim = h_dim
141
+ layers.append(nn.Linear(current_dim, 1))
142
+ self.mlp = nn.Sequential(*layers)
143
+
144
+ def forward(self, x):
145
+ return self.mlp(self.proj(x))
146
+
147
+
148
+ def pool_embedding(embedding):
149
+ """Pool encoder output [1, seq_len, 768] -> [1, 3072] using mean+min+max+std."""
150
+ mean_pool = embedding.mean(dim=1)
151
+ min_pool = embedding.min(dim=1).values
152
+ max_pool = embedding.max(dim=1).values
153
+ std_pool = embedding.std(dim=1)
154
+ return torch.cat([mean_pool, min_pool, max_pool, std_pool], dim=1)
155
+
156
+
157
+ # Quality expert file mapping
158
+ QUALITY_EXPERTS = {
159
+ "Overall_Quality": "model_score_overall_quality_best.pth",
160
+ "Speech_Quality": "model_score_speech_quality_best.pth",
161
+ "Background_Quality": "model_score_background_quality_best.pth",
162
+ "Content_Enjoyment": "model_score_content_enjoyment_best.pth",
163
+ }
164
+
165
+
166
+ @torch.no_grad()
167
+ def analyze_audio(audio_path: str):
168
+ """Run full inference: emotions + quality scores."""
169
+
170
+ # Load Whisper encoder
171
+ print("Loading Whisper encoder...")
172
+ feature_extractor = WhisperFeatureExtractor.from_pretrained(WHISPER_MODEL_ID)
173
+ whisper = WhisperModel.from_pretrained(WHISPER_MODEL_ID, low_cpu_mem_usage=True)
174
+ encoder = whisper.get_encoder().to(DEVICE).eval()
175
+ del whisper; gc.collect()
176
+
177
+ # Load audio
178
+ print(f"Loading audio: {audio_path}")
179
+ waveform, sr = librosa.load(audio_path, sr=SAMPLING_RATE, mono=True)
180
+ max_samples = int(MAX_AUDIO_SECONDS * SAMPLING_RATE)
181
+ if len(waveform) > max_samples:
182
+ waveform = waveform[:max_samples]
183
+ print(f" Duration: {len(waveform)/SAMPLING_RATE:.2f}s")
184
+
185
+ # Get encoder embedding
186
+ inputs = feature_extractor(waveform, sampling_rate=SAMPLING_RATE, return_tensors="pt")
187
+ input_features = inputs.input_features.to(DEVICE)
188
+ embedding = encoder(input_features).last_hidden_state # [1, 1500, 768]
189
+
190
+ # Prepare pooled features for quality experts
191
+ pooled = pool_embedding(embedding.float()) # [1, 3072]
192
+
193
+ # Download model repos
194
+ emotion_dir = Path(snapshot_download(HF_EMOTION_REPO_ID, allow_patterns=["*.pth"]))
195
+ quality_dir = Path(snapshot_download(HF_QUALITY_REPO_ID, allow_patterns=["*.pth"]))
196
+
197
+ results = {}
198
+
199
+ # --- Run emotion experts (full sequence) ---
200
+ print("\n--- Emotion Scores ---")
201
+ embedding_cpu = embedding.cpu().float()
202
+ for pth_file in sorted(emotion_dir.glob("model_*_best.pth")):
203
+ name = pth_file.stem.replace("model_", "").replace("_best", "")
204
+ model = FullEmbeddingMLP(WHISPER_SEQ_LEN, WHISPER_EMBED_DIM, PROJECTION_DIM, MLP_HIDDEN_DIMS, MLP_DROPOUTS)
205
+ state_dict = torch.load(pth_file, map_location="cpu", weights_only=True)
206
+ if any(k.startswith("_orig_mod.") for k in state_dict):
207
+ state_dict = {k.replace("_orig_mod.", ""): v for k, v in state_dict.items()}
208
+ model.load_state_dict(state_dict)
209
+ model.eval()
210
+ score = model(embedding_cpu).item()
211
+ results[name] = score
212
+ del model; gc.collect()
213
+
214
+ # Print top emotions
215
+ sorted_emotions = sorted(results.items(), key=lambda x: x[1], reverse=True)
216
+ for name, score in sorted_emotions[:5]:
217
+ print(f" {name}: {score:.4f}")
218
+
219
+ # --- Run quality experts (pooled features) ---
220
+ print("\n--- Quality Scores ---")
221
+ pooled_cpu = pooled.cpu().float()
222
+ for label, filename in QUALITY_EXPERTS.items():
223
+ pth_path = quality_dir / filename
224
+ if not pth_path.exists():
225
+ print(f" {label}: model not found")
226
+ continue
227
+ model = PooledEmbeddingMLP(POOLED_DIM, PROJECTION_DIM, MLP_HIDDEN_DIMS, MLP_DROPOUTS)
228
+ model.load_state_dict(torch.load(pth_path, map_location="cpu", weights_only=True))
229
+ model.eval()
230
+ score = model(pooled_cpu).item()
231
+ results[label] = score
232
+ print(f" {label}: {score:.4f}")
233
+ del model; gc.collect()
234
+
235
+ return results
236
+
237
+
238
+ # --- Example Usage ---
239
+ # results = analyze_audio("your_audio_file.mp3")
240
+ ```
241
+
242
+ ### Quality Scores Only (Lightweight)
243
+
244
+ If you only need the 4 quality scores without the emotion experts:
245
+
246
+ ```python
247
+ import torch
248
+ import torch.nn as nn
249
+ import numpy as np
250
+ import librosa
251
+ from transformers import WhisperModel, WhisperFeatureExtractor
252
+ from huggingface_hub import hf_hub_download
253
+
254
+ DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
255
+
256
+ class PooledEmbeddingMLP(nn.Module):
257
+ def __init__(self, input_dim=3072, projection_dim=64,
258
+ mlp_hidden_dims=[64, 32, 16], mlp_dropout_rates=[0.0, 0.1, 0.1, 0.1]):
259
+ super().__init__()
260
+ self.proj = nn.Linear(input_dim, projection_dim)
261
+ layers = [nn.ReLU(), nn.Dropout(mlp_dropout_rates[0])]
262
+ current_dim = projection_dim
263
+ for i, h_dim in enumerate(mlp_hidden_dims):
264
+ layers.extend([nn.Linear(current_dim, h_dim), nn.ReLU(), nn.Dropout(mlp_dropout_rates[i + 1])])
265
+ current_dim = h_dim
266
+ layers.append(nn.Linear(current_dim, 1))
267
+ self.mlp = nn.Sequential(*layers)
268
+
269
+ def forward(self, x):
270
+ return self.mlp(self.proj(x))
271
+
272
+ # Load encoder
273
+ feature_extractor = WhisperFeatureExtractor.from_pretrained("laion/BUD-E-Whisper")
274
+ whisper = WhisperModel.from_pretrained("laion/BUD-E-Whisper", low_cpu_mem_usage=True)
275
+ encoder = whisper.get_encoder().to(DEVICE).eval()
276
+
277
+ # Load audio
278
+ waveform, sr = librosa.load("your_audio.mp3", sr=16000, mono=True)
279
+ inputs = feature_extractor(waveform, sampling_rate=16000, return_tensors="pt")
280
+ with torch.no_grad():
281
+ embedding = encoder(inputs.input_features.to(DEVICE)).last_hidden_state # [1, 1500, 768]
282
+
283
+ # Pool: mean + min + max + std
284
+ emb = embedding.float()
285
+ pooled = torch.cat([emb.mean(1), emb.min(1).values, emb.max(1).values, emb.std(1)], dim=1) # [1, 3072]
286
+
287
+ # Run quality experts
288
+ EXPERTS = {
289
+ "Overall_Quality": "model_score_overall_quality_best.pth",
290
+ "Speech_Quality": "model_score_speech_quality_best.pth",
291
+ "Background_Quality": "model_score_background_quality_best.pth",
292
+ "Content_Enjoyment": "model_score_content_enjoyment_best.pth",
293
+ }
294
+
295
+ for label, filename in EXPERTS.items():
296
+ path = hf_hub_download("laion/Empathic-Insight-Voice-Plus", filename)
297
+ model = PooledEmbeddingMLP()
298
+ model.load_state_dict(torch.load(path, map_location="cpu", weights_only=True))
299
+ model.eval()
300
+ with torch.no_grad():
301
+ score = model(pooled.cpu()).item()
302
+ print(f"{label}: {score:.4f}")
303
+ ```
304
+
305
+
306
+ ## Architecture
307
+
308
+ The quality experts use a `PooledEmbeddingMLP` architecture:
309
+
310
+ ```
311
+ Input: [batch, 3072] (4 * 768: mean + min + max + std pooling over Whisper encoder sequence)
312
+ -> Linear(3072, 64) -> ReLU -> Dropout(0.0)
313
+ -> Linear(64, 64) -> ReLU -> Dropout(0.1)
314
+ -> Linear(64, 32) -> ReLU -> Dropout(0.1)
315
+ -> Linear(32, 16) -> ReLU -> Dropout(0.1)
316
+ -> Linear(16, 1)
317
+ Output: scalar score
318
+ ```
319
+
320
+ ~203K parameters per expert. Trained with Huber loss (delta=1.0), AdamW optimizer (lr=1e-3, weight_decay=1e-4), cosine annealing LR schedule over 50 epochs.
321
+
322
+
323
+ ## Training Data
324
+
325
+ Trained on [mitermix/balanced-audio-score-datasets](https://huggingface.co/datasets/mitermix/balanced-audio-score-datasets) (32.2 GB), which contains balanced distributions of audio quality scores across 5 dimensions. Each subset contains paired audio files and JSON metadata with ground truth scores.
326
+
327
+ | Subset | Source | Training Samples |
328
+ |--------|--------|-----------------|
329
+ | Overall Quality | DNSMOS | 39,800 |
330
+ | Speech Quality | DNSMOS | 9,800 |
331
+ | Background Quality | DNSMOS | 39,800 |
332
+ | Content Enjoyment | Meta AudioBox | 19,800 |
333
+
334
+
335
+ ## Based On
336
+
337
+ - **Whisper Encoder**: [laion/BUD-E-Whisper](https://huggingface.co/laion/BUD-E-Whisper) (OpenAI Whisper Small, fine-tuned)
338
+ - **MLP Architecture**: Same hidden layer structure as [laion/Empathic-Insight-Voice-Small](https://huggingface.co/laion/Empathic-Insight-Voice-Small)
339
+ - **Emotion Experts**: Fully compatible with the 54 emotion/attribute experts from Empathic-Insight-Voice-Small
340
+
341
+
342
+ ## Files
343
+
344
+ **New quality experts (this repo):**
345
+ - `model_score_overall_quality_best.pth` - Overall Quality expert (DNSMOS)
346
+ - `model_score_speech_quality_best.pth` - Speech Quality expert (DNSMOS)
347
+ - `model_score_background_quality_best.pth` - Background Quality expert (DNSMOS)
348
+ - `model_score_content_enjoyment_best.pth` - Content Enjoyment expert (Meta AudioBox)
349
+
350
+ **Original emotion experts** are loaded from [laion/Empathic-Insight-Voice-Small](https://huggingface.co/laion/Empathic-Insight-Voice-Small) (54 `.pth` files).
351
+
352
+
353
+ ## Emotion Taxonomy
354
+
355
+ The core 40 emotion categories are (from EMONET-VOICE, Appendix A.1):
356
+ Affection, Amusement, Anger, Astonishment/Surprise, Awe, Bitterness, Concentration, Confusion, Contemplation, Contempt, Contentment, Disappointment, Disgust, Distress, Doubt, Elation, Embarrassment, Emotional Numbness, Fatigue/Exhaustion, Fear, Helplessness, Hope/Enthusiasm/Optimism, Impatience and Irritability, Infatuation, Interest, Intoxication/Altered States of Consciousness, Jealousy & Envy, Longing, Malevolence/Malice, Pain, Pleasure/Ecstasy, Pride, Relief, Sadness, Sexual Lust, Shame, Sourness, Teasing, Thankfulness/Gratitude, Triumph.
357
+
358
+ Additional vocal attributes (e.g., Valence, Arousal, Gender, Age, Pitch characteristics) are also predicted by corresponding MLP models in the suite. The full list of predictable dimensions can be inferred from the `FILENAME_PART_TO_TARGET_KEY_MAP` in the [Colab notebook](https://colab.research.google.com/drive/1WR-B6j--Y5RdhIyRGF_tJ3YdFF8BkUA2).
359
+
360
+
361
+ ## Intended Use
362
+
363
+ These models are intended for research purposes in affective computing, speech emotion recognition (SER), human-AI interaction, and voice AI development. They can be used to:
364
+ * Analyze and predict fine-grained emotional states and vocal attributes from speech.
365
+ * Assess audio quality, speech clarity, background noise levels, and content enjoyment.
366
+ * Serve as a baseline for developing more advanced SER and audio quality assessment systems.
367
+
368
+ **Out-of-Scope Use:**
369
+ These models are trained on synthetic speech and their generalization to spontaneous real-world speech needs further evaluation. They should not be used for making critical decisions about individuals, for surveillance, or in any manner that could lead to discriminatory outcomes or infringe on privacy without due diligence and ethical review.
370
+
371
+
372
+ ## Ethical Considerations
373
+
374
+ The EMONET-VOICE suite was developed with ethical considerations as a priority:
375
+
376
+ Privacy Preservation: The use of synthetic voice generation fundamentally circumvents privacy concerns associated with collecting real human emotional expressions, especially for sensitive states.
377
+
378
+ Responsible Use: These models are released for research. Users are urged to consider the ethical implications of their applications and avoid misuse, such as for emotional manipulation, surveillance, or in ways that could lead to unfair, biased, or harmful outcomes. The broader societal implications and mitigation of potential misuse of SER technology remain important ongoing considerations.
model_Affection_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2cb3b80015949ba8ed8bd062563873199fc4eab382547b425af8c1a5037b2cde
3
+ size 294944277
model_Age_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fb66eed7bb1df19a31cebc77557542d9641e0ce5f8d86c223ea795897cb8800d
3
+ size 294944181
model_Amusement_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:20339a29b72b06dadfaa2c91c2540ba9b589d446120da82a09f847638902b006
3
+ size 294944277
model_Anger_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7193c17d0519819daa84b3076a67037d32794eb46a3c5c6666a9cacb8d92d936
3
+ size 294944213
model_Arousal_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b14b31aa0a47b29c3cb5587b0d4966039221f933b1438bdc897b21edcced3737
3
+ size 294944245
model_Astonishment_Surprise_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3ab2fa3d4e3ba8bae4fb719ed58ef5531f7ec657bce5a2d1f077a35cb8f362a3
3
+ size 294944533
model_Authenticity_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f24301cfcd1cb8f3b6f084340f7d650cd97e903694f7774bfac97b77c799ac0a
3
+ size 294944389
model_Awe_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:677e6eadd5bae126cb6f0d1b6f76257f4ad0cf92f9c299328b1f32dcb8f80c0a
3
+ size 294944181
model_Background_Noise_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:12e433405f5424d410d7eea6bbd3eb77496d520e7ff1c3513c0cff8ac165f532
3
+ size 294944453
model_Bitterness_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe423a55e3e9221ea2620a12e07c9f185628f781838e4764ab78bfdb2594db68
3
+ size 294944293
model_Concentration_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ba6b214e6085385e934161384ce37b94118ad24a6afe249c5acf9558e511242
3
+ size 294944405
model_Confident_vs._Hesitant_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b6db73d4c2477c5edca4e62fb76a35ff772e495777eef3acaeef5647c05d6629
3
+ size 294944549
model_Confusion_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3328c129f33743252c96ec7d1a695caac8e53c95033dff98381296de4770c835
3
+ size 294944277
model_Contemplation_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b80e1ea9f458fdee525fac692e0e3195b0dbffdd53d56650488f73a82e57d83
3
+ size 294944405
model_Contempt_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:93c8b276bd3775a23b56205a045daa6864031e418f603c4937d09bdc355e3ddd
3
+ size 294944261
model_Contentment_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9d0f87eb40619e5446491860d010f7c6c995318aedb6c43889ceac5d740a5378
3
+ size 294944373
model_Disappointment_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5284d7e00699667810c4b4ce57c5e14fd720e7ee398dfd21194d09271b311640
3
+ size 294944421
model_Disgust_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1015b54279531e2761fdce7e09e0a53465c3d1b8fc5d3d6a2b1c44aa3b7f0a85
3
+ size 294944245
model_Distress_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9fbffa8cf062ff244a3cf22873a879d34a05a03208d44cc8004c9eedf5134058
3
+ size 294944261
model_Doubt_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0a45d0c2f596c384423ccb30fd5f82c248705172e0f667f67068ee0adecbbc46
3
+ size 294944213
model_Elation_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a28a871c3160acf0a246722e43a4e1aebc9cd42a3b6e74ceccd2cf1642c6ff82
3
+ size 294944245
model_Embarrassment_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4f964cff1fbbf1f36062dc03549e6a516576c234c6859cc10e9e75ba9eb0157b
3
+ size 294944405
model_Emotional_Numbness_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1a4a0404f68f87299c4b336c06006ab9bcf32fa5ff3b980dd41730b356e4632c
3
+ size 294944485
model_Fatigue_Exhaustion_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:991007f02f7f81a5ef3858302df0d91d2f5764ba9907595964a155e3fef27fa6
3
+ size 294944485
model_Fear_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:18f7596fe98b1929fa606104db0e2f9fbc5c62bdba463597881afbb7f911cf66
3
+ size 294944197
model_Gender_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:608ba1f1efdfefafe2f006e5ca8d5dc743f6971c7cc52444ccbd3f6dd0d9bbc8
3
+ size 294944229
model_Helplessness_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4408052e892f6b2eb3a7c6c4e7da8b747f1f3922f1bf8ca094c856f7023b13ae
3
+ size 294944389
model_High-Pitched_vs._Low-Pitched_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7da700e13b9bb6ebf6a361fb5b77b54f8404dd80b9ccb56c1eb314a024f2b88a
3
+ size 294944645
model_Hope_Enthusiasm_Optimism_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3632c99cd5ce9c9bd9198f8a89cf21586bffbbb3d98ed34ca0f618449ae41249
3
+ size 294944581
model_Impatience_and_Irritability_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:56ebef395765ebfd81a8ecd31b56960b56a1dab4836ab09e37f933f15d77fb59
3
+ size 294944629
model_Infatuation_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:947f3e8dff35e16e57a46050ba5f953235804053f43980d22a1338bccefe4129
3
+ size 294944373
model_Interest_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3b955a33a66578f6cd06c5ccd0a1dae49e1afd9f1c88e33d551fcb41f0f6eadd
3
+ size 294944261
model_Intoxication_Altered_States_of_Consciousness_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:57bff66a300a7daee8eef6cbbbe47a1befb98251b68dd851c29ad4c6088c2717
3
+ size 294945029
model_Jealousy_&_Envy_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f9ecb3a0f1d8e3aaee752cf08476c9f8b9666ea3a0a5566852ea06f151e36076
3
+ size 294944437
model_Longing_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:37a837bebe8f32e1fe90e19a66eab44adfd85200be0640a2ef7b621bba90bb8a
3
+ size 294944245
model_Malevolence_Malice_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fc8c25456c87f72931df2f6ef9c7c1d7c34c33d764bba48617d106d87ff6226b
3
+ size 294944485
model_Monotone_vs._Expressive_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:35d0efce6cf03cb5d183d2584e7322c131f91236724424595b3b440e7ccd357a
3
+ size 294944565
model_Pain_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6d7af519e37a84998984ed19504f452522b5965d9345ed3cd3609566d9c159bc
3
+ size 294944197
model_Pleasure_Ecstasy_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:02e09b2e231ed731de0875f4d908578afd4992e3ed23584dae40c0cfaf6d27f3
3
+ size 294944453
model_Pride_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3a46d5798dd17a2464dc1645c1f71dd5c0a8620769b595a3651f4f598c438b3b
3
+ size 294944213
model_Recording_Quality_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:61f74b262bd80ba38ff6e4d3365bf931dc8eee037e911b53326dffbc349642dd
3
+ size 294944469
model_Relief_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:11a997a03bd57c68b2609c64216a7563d67208c742f020e502fa25960d39a7e1
3
+ size 294944229
model_Sadness_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:07409e62bf314596a533264f75a094303cc57af1cf66ea3f2920ec1261a4e245
3
+ size 294944245
model_Serious_vs._Humorous_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:19c6176384b82ab23c177258fe8c303c26b723b7015ba989c138d48e08637f6e
3
+ size 294944517
model_Sexual_Lust_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:856fe33053517e41e1758dc65ccfb407398be92a278eec2e9659855d4905fcec
3
+ size 294944373
model_Shame_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6d3a515d8b606d725f71eda295c2cd903c593643a690a951f6de177e8bcd1fe5
3
+ size 294944213
model_Soft_vs._Harsh_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ebbc72d8a7a9b22609777f0abfe24a81fe3aae1d767fdda6cb9ae93f15aa220d
3
+ size 294944421
model_Sourness_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:83b5c1a7f3ad37b48e6854fb31fb09d936f6201c4536b9db1ea3705491755c5e
3
+ size 294944261
model_Submissive_vs._Dominant_best.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6e347b16f92844781b6306e7a7592c43e995dc6f11940d7a3ac847428552ca22
3
+ size 294944565