Update README.md

README.md

@@ -1,3 +1,323 @@
----
-license: cc-by-4.0
----
+Okay, I will generate a model card for "Empathic Insight Voice" based on the structure of the "Empathic Insight Face" model card and the provided Python code. This will include:
+
+An adapted model card in Markdown format.
+
+Embedded YouTube video previews from your Cell 0.
+
+A "How to Use" section with example Python code for:
+
+Performing inference on a single audio file using all emotion MLPs (experts).
+
+Mention of batch processing to JSON and HTML reports as available in the full Colab.
+
+A separate section for the video emotion subtitling script (from your Cell 5).
+
+Here's the generated model card:
+
+---
+license: cc-by-4.0
+---
+
+# Empathic-Insight-Voice-Small
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1WR-B6j--Y5RdhIyRGF_tJ3YdFF8BkUA2)
+
+## Example Video Analyses (Top 3 Emotions)
+<!-- This section will be populated by the HTML from Cell 0 -->
+{{YOUTUBE_PREVIEWS_HTML}}
+
+**Empathic-Insight-Voice-Small** is a suite of 40+ emotion and attribute regression models trained on the EMONET-VOICE benchmark dataset, which is derived from the large-scale, multilingual synthetic voice-acting dataset LAION'S GOT TALENT. Each model is designed to predict the intensity of a specific fine-grained emotion or attribute from speech audio. These models leverage embeddings from a fine-tuned Whisper model (mkrausio/EmoWhisper-AnS-Small-v0.1) followed by dedicated MLP regression heads for each dimension.
+
+This work is based on the research paper:
+**"EMONET-VOICE: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detection"**
+*Authors: Anonymous Author(s) (as per the provided OCR text)*
+*(Please refer to the full paper when published for complete author list and affiliations).*
+*Paper link: (To be added when EMONET-VOICE paper is available, e.g., ArXiv/Conference link)*
+
+The models and datasets (LAION'S GOT TALENT, EMONET-VOICE benchmark) are intended to be released under a permissive license (e.g., CC-BY-4.0, Apache 2.0, as mentioned in the NeurIPS checklist from OCR).
+
+## Model Description
+
+The Empathic-Insight-Voice-Small suite consists of over 50 individual MLP models (40 for primary emotions, plus others for attributes like valence, arousal, gender, etc.). Each model takes a Whisper audio embedding as input and outputs a continuous score for one of the emotion/attribute categories defined in the EMONET-VOICE taxonomy and extended attribute set.
+
+The models were trained on a large dataset of synthetic speech, with labels generated via a sophisticated process involving LLMs (Gemini Flash 2.0) for initial 40-dimensional emotion scores and subsequent expert verification for the EMONET-VOICE benchmark subset.
+
+**Key Features:**
+*   **Fine-grained Emotions & Attributes:** Covers a 40-category emotion taxonomy plus additional vocal attributes.
+*   **Synthetic Data Foundation:** Trained on LAION'S GOT TALENT, a large-scale (5,000+ hours) synthetic voice-acting dataset across 11 voices, 40 emotions, and 4 languages.
+*   **Expert-Verified Benchmark:** The EMONET-VOICE subset features rigorous validation by human experts with psychology degrees.
+*   **Multilingual Potential:** The foundation dataset includes English, German, Spanish, and French.
+*   **Open:** Publicly released models, datasets, and taxonomy are planned.
+
+## Intended Use
+
+These models are intended for research purposes in affective computing, speech emotion recognition (SER), human-AI interaction, and voice AI development. They can be used to:
+*   Analyze and predict fine-grained emotional states and vocal attributes from speech.
+*   Serve as a baseline for developing more advanced SER systems.
+*   Facilitate research into nuanced emotional understanding in voice AI.
+*   Explore multilingual and cross-cultural aspects of speech emotion (given the foundation dataset).
+
+**Out-of-Scope Use:**
+These models are trained on synthetic speech and their generalization to spontaneous real-world speech needs further evaluation. They should not be used for making critical decisions about individuals, for surveillance, or in any manner that could lead to discriminatory outcomes or infringe on privacy without due diligence and ethical review.
+
+## How to Use
+
+The primary way to use these models is through the provided [Google Colab Notebook](https://colab.research.google.com/drive/1WR-B6j--Y5RdhIyRGF_tJ3YdFF8BkUA2). The notebook handles dependencies, model loading, audio processing, and provides examples for:
+*   Batch processing a folder of audio files.
+*   Generating a comprehensive HTML report with per-file emotion scores, waveforms, and audio players.
+*   Generating individual JSON files with all predicted scores for each audio file.
+
+Below is a conceptual example of how to perform inference for a single audio file, extracting all emotion and attribute scores. For the full, runnable version, please refer to the Colab notebook.
+
+**Conceptual Python Example for Single Audio File Inference:**
+
+```python
+import torch
+import torch.nn as nn
+import librosa
+import numpy as np
+from pathlib import Path
+from transformers import WhisperProcessor, WhisperForConditionalGeneration
+from huggingface_hub import snapshot_download # For downloading MLP models
+import gc # For memory management
+
+# --- Configuration (should match Cell 2 of the Colab) ---
+SAMPLING_RATE = 16000
+MAX_AUDIO_SECONDS = 30.0
+WHISPER_MODEL_ID = "mkrausio/EmoWhisper-AnS-Small-v0.1"
+HF_MLP_REPO_ID = "laion/Empathic-Insight-Voice-Small" # Or -Large if using those
+LOCAL_MLP_MODELS_DOWNLOAD_DIR = Path("./empathic_insight_voice_small_models_downloaded")
+
+WHISPER_SEQ_LEN = 1500
+WHISPER_EMBED_DIM = 768
+PROJECTION_DIM_FOR_FULL_EMBED = 64 # For 'Small' models
+MLP_HIDDEN_DIMS = [64, 32, 16]    # For 'Small' models
+MLP_DROPOUTS = [0.0, 0.1, 0.1, 0.1] # For 'Small' models
+
+# Mapping from .pth file name parts to human-readable dimension keys
+# (Abridged, full map in Colab Cell 2)
+FILENAME_PART_TO_TARGET_KEY_MAP = {
+    "Affection": "Affection", "Amusement": "Amusement", "Anger": "Anger",
+    "Arousal": "Arousal", "Valence": "Valence", # ... and many more
+    # Add all 40 emotions and other attributes as per Colab Cell 2
+}
+TARGET_EMOTION_KEYS_FOR_REPORT = [ # The 40 primary emotions
+    "Amusement", "Elation", # ... (full list from Colab Cell 2)
+]
+
+
+# --- MLP Model Definition (from Colab Cell 2) ---
+class FullEmbeddingMLP(nn.Module):
+    def __init__(self, seq_len, embed_dim, projection_dim, mlp_hidden_dims, mlp_dropout_rates):
+        super().__init__()
+        if len(mlp_dropout_rates) != len(mlp_hidden_dims) + 1:
+            raise ValueError("Dropout rates length error.")
+        self.flatten = nn.Flatten()
+        self.proj = nn.Linear(seq_len * embed_dim, projection_dim)
+        layers = [nn.ReLU(), nn.Dropout(mlp_dropout_rates[0])]
+        current_dim = projection_dim
+        for i, h_dim in enumerate(mlp_hidden_dims):
+            layers.extend([nn.Linear(current_dim, h_dim), nn.ReLU(), nn.Dropout(mlp_dropout_rates[i+1])])
+            current_dim = h_dim
+        layers.append(nn.Linear(current_dim, 1))
+        self.mlp = nn.Sequential(*layers)
+    def forward(self, x):
+        if x.ndim == 4 and x.shape[1] == 1: x = x.squeeze(1)
+        return self.mlp(self.proj(self.flatten(x)))
+
+# --- Global Model Placeholders ---
+whisper_model_global = None
+whisper_processor_global = None
+all_mlp_model_paths_dict = {} # To be populated
+WHISPER_DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+MLP_DEVICE = torch.device("cpu") # As per USE_CPU_OFFLOADING_FOR_MLPS in Colab
+
+def initialize_models():
+    global whisper_model_global, whisper_processor_global, all_mlp_model_paths_dict
+
+    print(f"Whisper will run on: {WHISPER_DEVICE}")
+    print(f"MLPs will run on: {MLP_DEVICE}")
+
+    # Load Whisper
+    if whisper_model_global is None:
+        print(f"Loading Whisper model '{WHISPER_MODEL_ID}'...")
+        whisper_processor_global = WhisperProcessor.from_pretrained(WHISPER_MODEL_ID)
+        whisper_model_global = WhisperForConditionalGeneration.from_pretrained(WHISPER_MODEL_ID).to(WHISPER_DEVICE).eval()
+        print("Whisper model loaded.")
+
+    # Download and map MLPs (paths only, models loaded on-demand)
+    if not all_mlp_model_paths_dict:
+        print(f"Downloading MLP checkpoints from {HF_MLP_REPO_ID} to {LOCAL_MLP_MODELS_DOWNLOAD_DIR}...")
+        LOCAL_MLP_MODELS_DOWNLOAD_DIR.mkdir(parents=True, exist_ok=True)
+        snapshot_download(
+            repo_id=HF_MLP_REPO_ID,
+            local_dir=LOCAL_MLP_MODELS_DOWNLOAD_DIR,
+            local_dir_use_symlinks=False,
+            allow_patterns=["*.pth"],
+            repo_type="model"
+        )
+        print("MLP checkpoints downloaded.")
+
+        # Map .pth files to target keys (simplified from Colab Cell 2)
+        for pth_file in LOCAL_MLP_MODELS_DOWNLOAD_DIR.glob("model_*_best.pth"):
+            try:
+                filename_part = pth_file.name.split("model_")[1].split("_best.pth")[0]
+                if filename_part in FILENAME_PART_TO_TARGET_KEY_MAP:
+                    target_key = FILENAME_PART_TO_TARGET_KEY_MAP[filename_part]
+                    all_mlp_model_paths_dict[target_key] = pth_file
+            except IndexError:
+                print(f"Warning: Could not parse filename part from {pth_file.name}")
+        print(f"Mapped {len(all_mlp_model_paths_dict)} MLP model paths.")
+        if not all_mlp_model_paths_dict:
+             raise RuntimeError("No MLP model paths could be mapped. Check FILENAME_PART_TO_TARGET_KEY_MAP and downloaded files.")
+
+
+@torch.no_grad()
+def get_whisper_embedding(audio_waveform_np):
+    if whisper_model_global is None or whisper_processor_global is None:
+        raise RuntimeError("Whisper model not initialized. Call initialize_models() first.")
+
+    input_features = whisper_processor_global(
+        audio_waveform_np, sampling_rate=SAMPLING_RATE, return_tensors="pt"
+    ).input_features.to(WHISPER_DEVICE).to(whisper_model_global.dtype)
+
+    encoder_outputs = whisper_model_global.get_encoder()(input_features=input_features)
+    embedding = encoder_outputs.last_hidden_state
+
+    current_seq_len = embedding.shape[1]
+    if current_seq_len < WHISPER_SEQ_LEN:
+        padding = torch.zeros((1, WHISPER_SEQ_LEN - current_seq_len, WHISPER_EMBED_DIM),
+                              device=WHISPER_DEVICE, dtype=embedding.dtype)
+        embedding = torch.cat((embedding, padding), dim=1)
+    elif current_seq_len > WHISPER_SEQ_LEN:
+        embedding = embedding[:, :WHISPER_SEQ_LEN, :]
+    return embedding
+
+def load_single_mlp(model_path, target_key):
+    # Simplified loading for example (Colab Cell 2 has more robust loading)
+    # For this example, assumes USE_HALF_PRECISION_FOR_MLPS=False, USE_TORCH_COMPILE_FOR_MLPS=False
+    print(f"  Loading MLP for '{target_key}'...")
+    model_instance = FullEmbeddingMLP(
+        WHISPER_SEQ_LEN, WHISPER_EMBED_DIM, PROJECTION_DIM_FOR_FULL_EMBED,
+        MLP_HIDDEN_DIMS, MLP_DROPOUTS
+    )
+    state_dict = torch.load(model_path, map_location='cpu')
+    # Handle potential '_orig_mod.' prefix if model was torch.compile'd during training
+    if any(k.startswith("_orig_mod.") for k in state_dict.keys()):
+        state_dict = {k.replace("_orig_mod.", ""): v for k, v in state_dict.items()}
+    model_instance.load_state_dict(state_dict)
+    model_instance = model_instance.to(MLP_DEVICE).eval()
+    return model_instance
+
+@torch.no_grad()
+def predict_with_mlp(embedding, mlp_model):
+    embedding_for_mlp = embedding.to(MLP_DEVICE)
+    # Ensure dtype matches (simplified)
+    mlp_dtype = next(mlp_model.parameters()).dtype
+    prediction = mlp_model(embedding_for_mlp.to(mlp_dtype))
+    return prediction.item()
+
+def process_audio_file(audio_file_path_str: str) -> Dict[str, float]:
+    if not all_mlp_model_paths_dict:
+        initialize_models() # Ensure models are ready
+
+    print(f"Processing audio file: {audio_file_path_str}")
+    try:
+        waveform, sr = librosa.load(audio_file_path_str, sr=SAMPLING_RATE, mono=True)
+        max_samples = int(MAX_AUDIO_SECONDS * SAMPLING_RATE)
+        if len(waveform) > max_samples:
+            waveform = waveform[:max_samples]
+        print(f"Audio loaded. Duration: {len(waveform)/SAMPLING_RATE:.2f}s")
+    except Exception as e:
+        print(f"Error loading audio {audio_file_path_str}: {e}")
+        return {}
+
+    embedding = get_whisper_embedding(waveform)
+    del waveform; gc.collect();
+    if WHISPER_DEVICE.type == 'cuda': torch.cuda.empty_cache()
+
+    all_scores: Dict[str, float] = {}
+    for target_key, mlp_model_path in all_mlp_model_paths_dict.items():
+        if target_key not in FILENAME_PART_TO_TARGET_KEY_MAP.values(): # Only process mapped keys
+            continue
+
+        current_mlp_model = load_single_mlp(mlp_model_path, target_key)
+        if current_mlp_model:
+            score = predict_with_mlp(embedding, current_mlp_model)
+            all_scores[target_key] = score
+            print(f"    {target_key}: {score:.4f}")
+            del current_mlp_model # Unload after use
+            gc.collect()
+            if MLP_DEVICE.type == 'cuda': torch.cuda.empty_cache()
+        else:
+            all_scores[target_key] = float('nan')
+
+    del embedding; gc.collect();
+    if WHISPER_DEVICE.type == 'cuda': torch.cuda.empty_cache()
+
+    # Optional: Calculate Softmax for the 40 primary emotions
+    emotion_raw_scores = [all_scores.get(k, -float('inf')) for k in TARGET_EMOTION_KEYS_FOR_REPORT if k in all_scores]
+    if emotion_raw_scores:
+        softmax_probs = torch.softmax(torch.tensor(emotion_raw_scores, dtype=torch.float32), dim=0)
+        print("\nTop 3 Emotions (Softmax Probabilities):")
+        # Create a dictionary of {emotion_key: softmax_prob}
+        emotion_softmax_dict = {
+            key: prob.item()
+            for key, prob in zip(
+                [k for k in TARGET_EMOTION_KEYS_FOR_REPORT if k in all_scores], # only keys that had scores
+                softmax_probs
+            )
+        }
+        sorted_emotions = sorted(emotion_softmax_dict.items(), key=lambda item: item[1], reverse=True)
+        for i, (emotion, prob) in enumerate(sorted_emotions[:3]):
+            print(f"  {i+1}. {emotion}: {prob:.4f} (Raw: {all_scores.get(emotion, float('nan')):.4f})")
+    return all_scores
+
+# --- Example Usage (Run this after defining functions and initializing models) ---
+# Make sure to have an audio file (e.g., "sample.mp3") in your current directory or provide a full path.
+# And ensure FILENAME_PART_TO_TARGET_KEY_MAP and TARGET_EMOTION_KEYS_FOR_REPORT are fully populated.
+#
+# initialize_models() # Call this once
+#
+# # Create a dummy sample.mp3 for testing if it doesn't exist
+# if not Path("sample.mp3").exists():
+#     print("Creating dummy sample.mp3 for testing...")
+#     dummy_sr = 16000
+#     dummy_duration = 5 # seconds
+#     dummy_tone_freq = 440 # A4 note
+#     t = np.linspace(0, dummy_duration, int(dummy_sr * dummy_duration), endpoint=False)
+#     dummy_waveform = 0.5 * np.sin(2 * np.pi * dummy_tone_freq * t)
+#     import soundfile as sf
+#     sf.write("sample.mp3", dummy_waveform, dummy_sr)
+#     print("Dummy sample.mp3 created.")
+#
+# if Path("sample.mp3").exists() and FILENAME_PART_TO_TARGET_KEY_MAP and TARGET_EMOTION_KEYS_FOR_REPORT:
+#    results = process_audio_file("sample.mp3")
+#    # print("\nFull Scores Dictionary:", results)
+# else:
+#    print("Skipping example usage: 'sample.mp3' not found or maps are not fully populated.")
+```
+
+Batch Processing and Reporting:
+The Google Colab Notebook provides a complete pipeline (Cells 3 and 4) for:
+
+Processing all audio files in a specified input folder.
+
+Generating a detailed HTML report summarizing predictions for all files, including waveforms, audio players, and scores for all dimensions.
+
+Saving per-file JSON outputs containing all raw prediction scores.
+
+## Taxonomy
+
+The core 40 emotion categories are (from EMONET-VOICE, Appendix A.1):
+Affection, Amusement, Anger, Astonishment/Surprise, Awe, Bitterness, Concentration, Confusion, Contemplation, Contempt, Contentment, Disappointment, Disgust, Distress, Doubt, Elation, Embarrassment, Emotional Numbness, Fatigue/Exhaustion, Fear, Helplessness, Hope/Enthusiasm/Optimism, Impatience and Irritability, Infatuation, Interest, Intoxication/Altered States of Consciousness, Jealousy & Envy, Longing, Malevolence/Malice, Pain, Pleasure/Ecstasy, Pride, Relief, Sadness, Sexual Lust, Shame, Sourness, Teasing, Thankfulness/Gratitude, Triumph.
+
+Additional vocal attributes (e.g., Valence, Arousal, Gender, Age, Pitch characteristics) are also predicted by corresponding MLP models in the suite. The full list of predictable dimensions can be inferred from the FILENAME_PART_TO_TARGET_KEY_MAP in the Colab notebook (Cell 2).
+
+
+## Ethical Considerations
+
+The EMONET-VOICE suite was developed with ethical considerations as a priority:
+
+Privacy Preservation: The use of synthetic voice generation fundamentally circumvents privacy concerns associated with collecting real human emotional expressions, especially for sensitive states.
+
+Responsible Use: These models are released for research. Users are urged to consider the ethical implications of their applications and avoid misuse, such as for emotional manipulation, surveillance, or in ways that could lead to unfair, biased, or harmful outcomes. The broader societal implications and mitigation of potential misuse of SER technology remain important ongoing considerations.