# Empathic-Insight-Voice-Plus [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1WR-B6j--Y5RdhIyRGF_tJ3YdFF8BkUA2) **Empathic-Insight-Voice-Plus** extends [laion/Empathic-Insight-Voice-Small](https://huggingface.co/laion/Empathic-Insight-Voice-Small) with **4 additional audio quality expert models**. These new experts predict overall audio quality, speech quality, background noise quality, and content enjoyment from the same frozen [laion/BUD-E-Whisper](https://huggingface.co/laion/BUD-E-Whisper) encoder embeddings. This repository is fully compatible with the original Empathic-Insight-Voice-Small suite. It uses the same Whisper encoder (based on OpenAI Whisper Small) and the same MLP hidden architecture (`proj=64, hidden=[64, 32, 16]`). The only difference is that the new quality experts use **pooled features** (mean + min + max + std pooling over the encoder sequence dimension, yielding a 3072-d input) instead of the full flattened sequence, making them significantly more compact (~200K parameters each vs. ~73.7M for the full-sequence emotion experts). This work is based on the research paper: **"EMONET-VOICE: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detection"** ## Example Video Analyses (Top 3 Emotions)

ID: TsTVKCmqHhk

ID: sErqFgL4vA8

ID: BUnfuiwE_IM

ID: dDrmjcUq8W4

## Model Description The Empathic-Insight-Voice-Plus suite combines the original 54+ emotion/attribute MLP experts from [Empathic-Insight-Voice-Small](https://huggingface.co/laion/Empathic-Insight-Voice-Small) with 4 new audio quality regression experts. All models use embeddings from the fine-tuned Whisper model [laion/BUD-E-Whisper](https://huggingface.co/laion/BUD-E-Whisper). The original emotion experts were trained on the large-scale, multilingual synthetic voice-acting dataset LAION'S GOT TALENT (~5,000 hours) & an "in the wild" dataset of voice snippets (~5,000 hours). The new quality experts were trained on the [mitermix/balanced-audio-score-datasets](https://huggingface.co/datasets/mitermix/balanced-audio-score-datasets) dataset. The quality scores are distilled from two established audio quality models: - **Overall Quality, Speech Quality, Background Quality**: Distilled from Microsoft's [DNSMOS](https://github.com/microsoft/DNS-Challenge) (Deep Noise Suppression Mean Opinion Score), a non-intrusive speech quality estimator. - **Content Enjoyment**: Distilled from Meta's [AudioBox](https://ai.meta.com/research/publications/audiobox-unified-audio-generation-with-natural-language-prompts/) content enjoyment scoring model. ## New Quality Expert Scores | Expert | Description | Source | Expected Range | Typical Mean | Unit | |--------|-------------|--------|---------------|--------------|------| | **Overall Quality** | Overall perceived audio quality score. Higher is better. | DNSMOS | 1.0 - 3.7 | ~2.4 | MOS-like | | **Speech Quality** | Quality of the speech signal itself (clarity, naturalness). Higher is better. | DNSMOS | 1.0 - 2.4 | ~1.9 | MOS-like | | **Background Quality** | Quality of the background (absence of noise/artifacts). Higher is better. | DNSMOS | 1.0 - 4.3 | ~3.2 | MOS-like | | **Content Enjoyment** | How engaging/enjoyable the spoken content is. Higher is better. | Meta AudioBox | 1.9 - 5.1 | ~4.1 | MOS-like | ### Validation Performance | Expert | Val MAE | Pearson r | Val Samples | |--------|---------|-----------|-------------| | Overall Quality | 0.26 | 0.899 | 200 | | Speech Quality | 0.30 | 0.517 | 200 | | Background Quality | 0.35 | 0.865 | 200 | | Content Enjoyment | 0.34 | 0.691 | 200 | ## How to Use ### Full Inference: Emotion Scores + Quality Scores The following example loads both the original 54 emotion/attribute experts and the 4 new quality experts, then runs inference on a single audio file. ```python import torch import torch.nn as nn import numpy as np import librosa from pathlib import Path from transformers import WhisperModel, WhisperFeatureExtractor from huggingface_hub import snapshot_download import gc # --- Configuration --- SAMPLING_RATE = 16000 MAX_AUDIO_SECONDS = 30.0 WHISPER_MODEL_ID = "laion/BUD-E-Whisper" # Original emotion experts HF_EMOTION_REPO_ID = "laion/Empathic-Insight-Voice-Small" # New quality experts (this repo) HF_QUALITY_REPO_ID = "laion/Empathic-Insight-Voice-Plus" WHISPER_SEQ_LEN = 1500 WHISPER_EMBED_DIM = 768 PROJECTION_DIM = 64 MLP_HIDDEN_DIMS = [64, 32, 16] MLP_DROPOUTS = [0.0, 0.1, 0.1, 0.1] POOLED_DIM = 3072 # 4 * 768 (mean + min + max + std) DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu") # --- Model Definitions --- class FullEmbeddingMLP(nn.Module): """Original emotion expert architecture (full sequence input).""" def __init__(self, seq_len, embed_dim, projection_dim, mlp_hidden_dims, mlp_dropout_rates): super().__init__() self.flatten = nn.Flatten() self.proj = nn.Linear(seq_len * embed_dim, projection_dim) layers = [nn.ReLU(), nn.Dropout(mlp_dropout_rates[0])] current_dim = projection_dim for i, h_dim in enumerate(mlp_hidden_dims): layers.extend([nn.Linear(current_dim, h_dim), nn.ReLU(), nn.Dropout(mlp_dropout_rates[i + 1])]) current_dim = h_dim layers.append(nn.Linear(current_dim, 1)) self.mlp = nn.Sequential(*layers) def forward(self, x): if x.ndim == 4 and x.shape[1] == 1: x = x.squeeze(1) return self.mlp(self.proj(self.flatten(x))) class PooledEmbeddingMLP(nn.Module): """New quality expert architecture (pooled features input).""" def __init__(self, input_dim, projection_dim, mlp_hidden_dims, mlp_dropout_rates): super().__init__() self.proj = nn.Linear(input_dim, projection_dim) layers = [nn.ReLU(), nn.Dropout(mlp_dropout_rates[0])] current_dim = projection_dim for i, h_dim in enumerate(mlp_hidden_dims): layers.extend([nn.Linear(current_dim, h_dim), nn.ReLU(), nn.Dropout(mlp_dropout_rates[i + 1])]) current_dim = h_dim layers.append(nn.Linear(current_dim, 1)) self.mlp = nn.Sequential(*layers) def forward(self, x): return self.mlp(self.proj(x)) def pool_embedding(embedding): """Pool encoder output [1, seq_len, 768] -> [1, 3072] using mean+min+max+std.""" mean_pool = embedding.mean(dim=1) min_pool = embedding.min(dim=1).values max_pool = embedding.max(dim=1).values std_pool = embedding.std(dim=1) return torch.cat([mean_pool, min_pool, max_pool, std_pool], dim=1) # Quality expert file mapping QUALITY_EXPERTS = { "Overall_Quality": "model_score_overall_quality_best.pth", "Speech_Quality": "model_score_speech_quality_best.pth", "Background_Quality": "model_score_background_quality_best.pth", "Content_Enjoyment": "model_score_content_enjoyment_best.pth", } @torch.no_grad() def analyze_audio(audio_path: str): """Run full inference: emotions + quality scores.""" # Load Whisper encoder print("Loading Whisper encoder...") feature_extractor = WhisperFeatureExtractor.from_pretrained(WHISPER_MODEL_ID) whisper = WhisperModel.from_pretrained(WHISPER_MODEL_ID, low_cpu_mem_usage=True) encoder = whisper.get_encoder().to(DEVICE).eval() del whisper; gc.collect() # Load audio print(f"Loading audio: {audio_path}") waveform, sr = librosa.load(audio_path, sr=SAMPLING_RATE, mono=True) max_samples = int(MAX_AUDIO_SECONDS * SAMPLING_RATE) if len(waveform) > max_samples: waveform = waveform[:max_samples] print(f" Duration: {len(waveform)/SAMPLING_RATE:.2f}s") # Get encoder embedding inputs = feature_extractor(waveform, sampling_rate=SAMPLING_RATE, return_tensors="pt") input_features = inputs.input_features.to(DEVICE) embedding = encoder(input_features).last_hidden_state # [1, 1500, 768] # Prepare pooled features for quality experts pooled = pool_embedding(embedding.float()) # [1, 3072] # Download model repos emotion_dir = Path(snapshot_download(HF_EMOTION_REPO_ID, allow_patterns=["*.pth"])) quality_dir = Path(snapshot_download(HF_QUALITY_REPO_ID, allow_patterns=["*.pth"])) results = {} # --- Run emotion experts (full sequence) --- print("\n--- Emotion Scores ---") embedding_cpu = embedding.cpu().float() for pth_file in sorted(emotion_dir.glob("model_*_best.pth")): name = pth_file.stem.replace("model_", "").replace("_best", "") model = FullEmbeddingMLP(WHISPER_SEQ_LEN, WHISPER_EMBED_DIM, PROJECTION_DIM, MLP_HIDDEN_DIMS, MLP_DROPOUTS) state_dict = torch.load(pth_file, map_location="cpu", weights_only=True) if any(k.startswith("_orig_mod.") for k in state_dict): state_dict = {k.replace("_orig_mod.", ""): v for k, v in state_dict.items()} model.load_state_dict(state_dict) model.eval() score = model(embedding_cpu).item() results[name] = score del model; gc.collect() # Print top emotions sorted_emotions = sorted(results.items(), key=lambda x: x[1], reverse=True) for name, score in sorted_emotions[:5]: print(f" {name}: {score:.4f}") # --- Run quality experts (pooled features) --- print("\n--- Quality Scores ---") pooled_cpu = pooled.cpu().float() for label, filename in QUALITY_EXPERTS.items(): pth_path = quality_dir / filename if not pth_path.exists(): print(f" {label}: model not found") continue model = PooledEmbeddingMLP(POOLED_DIM, PROJECTION_DIM, MLP_HIDDEN_DIMS, MLP_DROPOUTS) model.load_state_dict(torch.load(pth_path, map_location="cpu", weights_only=True)) model.eval() score = model(pooled_cpu).item() results[label] = score print(f" {label}: {score:.4f}") del model; gc.collect() return results # --- Example Usage --- # results = analyze_audio("your_audio_file.mp3") ``` ### Quality Scores Only (Lightweight) If you only need the 4 quality scores without the emotion experts: ```python import torch import torch.nn as nn import numpy as np import librosa from transformers import WhisperModel, WhisperFeatureExtractor from huggingface_hub import hf_hub_download DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu") class PooledEmbeddingMLP(nn.Module): def __init__(self, input_dim=3072, projection_dim=64, mlp_hidden_dims=[64, 32, 16], mlp_dropout_rates=[0.0, 0.1, 0.1, 0.1]): super().__init__() self.proj = nn.Linear(input_dim, projection_dim) layers = [nn.ReLU(), nn.Dropout(mlp_dropout_rates[0])] current_dim = projection_dim for i, h_dim in enumerate(mlp_hidden_dims): layers.extend([nn.Linear(current_dim, h_dim), nn.ReLU(), nn.Dropout(mlp_dropout_rates[i + 1])]) current_dim = h_dim layers.append(nn.Linear(current_dim, 1)) self.mlp = nn.Sequential(*layers) def forward(self, x): return self.mlp(self.proj(x)) # Load encoder feature_extractor = WhisperFeatureExtractor.from_pretrained("laion/BUD-E-Whisper") whisper = WhisperModel.from_pretrained("laion/BUD-E-Whisper", low_cpu_mem_usage=True) encoder = whisper.get_encoder().to(DEVICE).eval() # Load audio waveform, sr = librosa.load("your_audio.mp3", sr=16000, mono=True) inputs = feature_extractor(waveform, sampling_rate=16000, return_tensors="pt") with torch.no_grad(): embedding = encoder(inputs.input_features.to(DEVICE)).last_hidden_state # [1, 1500, 768] # Pool: mean + min + max + std emb = embedding.float() pooled = torch.cat([emb.mean(1), emb.min(1).values, emb.max(1).values, emb.std(1)], dim=1) # [1, 3072] # Run quality experts EXPERTS = { "Overall_Quality": "model_score_overall_quality_best.pth", "Speech_Quality": "model_score_speech_quality_best.pth", "Background_Quality": "model_score_background_quality_best.pth", "Content_Enjoyment": "model_score_content_enjoyment_best.pth", } for label, filename in EXPERTS.items(): path = hf_hub_download("laion/Empathic-Insight-Voice-Plus", filename) model = PooledEmbeddingMLP() model.load_state_dict(torch.load(path, map_location="cpu", weights_only=True)) model.eval() with torch.no_grad(): score = model(pooled.cpu()).item() print(f"{label}: {score:.4f}") ``` ## Architecture The quality experts use a `PooledEmbeddingMLP` architecture: ``` Input: [batch, 3072] (4 * 768: mean + min + max + std pooling over Whisper encoder sequence) -> Linear(3072, 64) -> ReLU -> Dropout(0.0) -> Linear(64, 64) -> ReLU -> Dropout(0.1) -> Linear(64, 32) -> ReLU -> Dropout(0.1) -> Linear(32, 16) -> ReLU -> Dropout(0.1) -> Linear(16, 1) Output: scalar score ``` ~203K parameters per expert. Trained with Huber loss (delta=1.0), AdamW optimizer (lr=1e-3, weight_decay=1e-4), cosine annealing LR schedule over 50 epochs. ## Training Data Trained on [mitermix/balanced-audio-score-datasets](https://huggingface.co/datasets/mitermix/balanced-audio-score-datasets) (32.2 GB), which contains balanced distributions of audio quality scores across 5 dimensions. Each subset contains paired audio files and JSON metadata with ground truth scores. | Subset | Source | Training Samples | |--------|--------|-----------------| | Overall Quality | DNSMOS | 39,800 | | Speech Quality | DNSMOS | 9,800 | | Background Quality | DNSMOS | 39,800 | | Content Enjoyment | Meta AudioBox | 19,800 | ## Based On - **Whisper Encoder**: [laion/BUD-E-Whisper](https://huggingface.co/laion/BUD-E-Whisper) (OpenAI Whisper Small, fine-tuned) - **MLP Architecture**: Same hidden layer structure as [laion/Empathic-Insight-Voice-Small](https://huggingface.co/laion/Empathic-Insight-Voice-Small) - **Emotion Experts**: Fully compatible with the 54 emotion/attribute experts from Empathic-Insight-Voice-Small ## Files **New quality experts (this repo):** - `model_score_overall_quality_best.pth` - Overall Quality expert (DNSMOS) - `model_score_speech_quality_best.pth` - Speech Quality expert (DNSMOS) - `model_score_background_quality_best.pth` - Background Quality expert (DNSMOS) - `model_score_content_enjoyment_best.pth` - Content Enjoyment expert (Meta AudioBox) **Original emotion experts** are loaded from [laion/Empathic-Insight-Voice-Small](https://huggingface.co/laion/Empathic-Insight-Voice-Small) (54 `.pth` files). ## Emotion Taxonomy The core 40 emotion categories are (from EMONET-VOICE, Appendix A.1): Affection, Amusement, Anger, Astonishment/Surprise, Awe, Bitterness, Concentration, Confusion, Contemplation, Contempt, Contentment, Disappointment, Disgust, Distress, Doubt, Elation, Embarrassment, Emotional Numbness, Fatigue/Exhaustion, Fear, Helplessness, Hope/Enthusiasm/Optimism, Impatience and Irritability, Infatuation, Interest, Intoxication/Altered States of Consciousness, Jealousy & Envy, Longing, Malevolence/Malice, Pain, Pleasure/Ecstasy, Pride, Relief, Sadness, Sexual Lust, Shame, Sourness, Teasing, Thankfulness/Gratitude, Triumph. Additional vocal attributes (e.g., Valence, Arousal, Gender, Age, Pitch characteristics) are also predicted by corresponding MLP models in the suite. The full list of predictable dimensions can be inferred from the `FILENAME_PART_TO_TARGET_KEY_MAP` in the [Colab notebook](https://colab.research.google.com/drive/1WR-B6j--Y5RdhIyRGF_tJ3YdFF8BkUA2). ## Intended Use These models are intended for research purposes in affective computing, speech emotion recognition (SER), human-AI interaction, and voice AI development. They can be used to: * Analyze and predict fine-grained emotional states and vocal attributes from speech. * Assess audio quality, speech clarity, background noise levels, and content enjoyment. * Serve as a baseline for developing more advanced SER and audio quality assessment systems. **Out-of-Scope Use:** These models are trained on synthetic speech and their generalization to spontaneous real-world speech needs further evaluation. They should not be used for making critical decisions about individuals, for surveillance, or in any manner that could lead to discriminatory outcomes or infringe on privacy without due diligence and ethical review. ## Ethical Considerations The EMONET-VOICE suite was developed with ethical considerations as a priority: Privacy Preservation: The use of synthetic voice generation fundamentally circumvents privacy concerns associated with collecting real human emotional expressions, especially for sensitive states. Responsible Use: These models are released for research. Users are urged to consider the ethical implications of their applications and avoid misuse, such as for emotional manipulation, surveillance, or in ways that could lead to unfair, biased, or harmful outcomes. The broader societal implications and mitigation of potential misuse of SER technology remain important ongoing considerations.