Update README.md

README.md

@@ -5,6 +5,8 @@ tags:
 - music-classification
 - math-rock
 - pytorch
+- wavlm
+- transformer
 datasets:
 - anggars/neural-mathrock
 language:
@@ -16,50 +18,50 @@ pipeline_tag: audio-classification
 
 # Neural Mathrock: Multimodal Emotion and Personality Analysis
 
-This repository hosts a multimodal deep learning framework specialized in the affective and psychological analysis of Math Rock and Midwest Emo music. By integrating lyrical semantics and acoustic patterns, the system extracts features to classify emotional and personality-based characteristics.
+This repository hosts a multimodal deep learning framework specialized in the affective and psychological analysis of Math Rock and Midwest Emo music. By integrating lyrical semantics and raw acoustic representations, the system extracts features to classify emotional and personality-based characteristics.
 
 ## Project Objectives
 The research is structured to prioritize emotional resonance and genre-specific complexities:
-- **Emotion & Vibe Recognition:** Identifying affective states and general vibes (e.g., Melancholic, Aggressive) through the synergy of lyrical themes and audio arrays.
+- **Emotion & Vibe Recognition:** Identifying affective states and general vibes (e.g., Melancholic, Aggressive) through the synergy of lyrical themes and audio.
 - **Personality (MBTI) Profiling:** Correlating complex musical arrangements and introspective lyrics with personality archetypes.
 - **Acoustic Feature Extraction:** Analyzing technical attributes like syncopation and odd time signatures using robust audio signal processing.
 
 ## Technical Architecture: Late Fusion Multimodal
-The system utilizes a custom Multimodal PyTorch architecture:
+The system utilizes a custom Multimodal PyTorch architecture combining NLP and state-of-the-art audio transformers:
 
 ### 1. Lyrical Stream (NLP)
-- **Encoder:** `xlm-roberta-base`
-- **Logic:** Extracts high-level semantic embeddings from song lyrics (768-dim). The base model weights are frozen to maintain stable pre-trained representations.
+- **Encoder:** `xlm-roberta-base` (or equivalent transformer)
+- **Logic:** Extracts high-level semantic embeddings from song lyrics. The base model weights are frozen to maintain stable pre-trained representations.
 
 ### 2. Acoustic Stream (DSP)
-- **Model:** 2D-Convolutional Neural Network (CNN) followed by a Transformer Encoder.
-- **Input:** 3-channel stacked features (Mel-spectrogram, 13-coefficient MFCC, and Chroma).
-- **Logic:** Captures the complex guitar textures and erratic drum patterns common in the genre, pooling them into a 256-dim feature vector.
+- **Model:** Pre-trained Audio Transformer (`WavLMModel`)
+- **Input:** Raw audio waveform processed via `AutoFeatureExtractor`.
+- **Logic:** Captures the complex guitar textures and erratic drum patterns common in the genre natively from the waveform, replacing the legacy 2D-CNN Mel-spectrogram approach.
 
 ### 3. Fusion Layer
-- **Method:** Feature concatenation (768-dim Text + 256-dim Audio) into a 1024-dim representation, processed through a 512-dim feed-forward layer with dropout regularization.
+- **Method:** Feature concatenation (Text Embeddings + Audio Embeddings) into a unified representation, processed through feed-forward layers with dropout regularization.
 - **Heads:** Multi-task fully connected layers for joint classification of MBTI, Emotion, Vibe, Intensity, and Tempo.
 
-## Performance Summary and Academic Context
-This project is an undergraduate thesis developed at Sekolah Tinggi Teknologi Cipasung (STTC), Informatics Department, Class of 2022. It explores the extreme class imbalances naturally found in niche music genres.
+## Academic Context
+This project is an undergraduate thesis developed at Sekolah Tinggi Teknologi Cipasung (STTC), Informatics Department. 
+
+**Thesis Title:** *RANCANG BANGUN SISTEM ANALISIS MULTIMODAL EMOSI DAN KEPRIBADIAN MBTI PADA LIRIK MUSIK MIDWEST EMO MENGGUNAKAN ARSITEKTUR TRANSFORMER DAN EKSTRAKSI FITUR AUDIO DALAM MUSIK MATH ROCK*
 
 **Key Findings:**
-- **High-Performance Metrics:** The architecture successfully identifies broad acoustic targets, achieving an **F1-Score of 0.77 for High Intensity** and **0.72 for Melancholic Vibe**, proving the viability of the multimodal feature extraction pipeline.
-- **Class Imbalance in Niche Genres:** For 16-class targets like MBTI and Emotion, the model highlights the natural bias of the dataset. Classes with sufficient samples (e.g., ESTJ, Desire, Pride) perform adequately, while minority classes struggle due to lack of representation (Macro F1 ~0.07). This serves as a realistic baseline for future research regarding data sparsity in highly subjective Music Information Retrieval (MIR) tasks.
+- **High-Performance Metrics:** The architecture successfully identifies broad acoustic targets, proving the viability of the multimodal feature extraction pipeline using WavLM.
+- **Class Imbalance in Niche Genres:** For 16-class targets like MBTI and Emotion, the model highlights the natural bias of the dataset. Classes with sufficient samples perform adequately, while minority classes struggle due to lack of representation. This serves as a realistic baseline for future research regarding data sparsity in highly subjective Music Information Retrieval (MIR) tasks.
 
 ## How to Use
-Since this is a custom PyTorch architecture, it cannot be loaded via the standard `pipeline` API. You must define the model classes locally and load the state dictionary.
+Since this is a custom PyTorch architecture incorporating Hugging Face transformers, it must be loaded by defining the model class locally before loading the state dictionary.
 
 ```python
 import torch
+from transformers import WavLMModel, AutoFeatureExtractor
 
-# 1. Define the MultimodalMathRock architecture classes here first
-# (Refer to the training script for the AudioCNNTransformer, TextTransformer, and Fusion module)
-
+# 1. Define your custom multimodal architecture class matching the training script
 # 2. Load the weights
 model = MultimodalMathRock()
-state_dict = torch.load("model.pt", map_location="cpu")
-model.load_state_dict(state_dict['model_state'])
+model.load_state_dict(torch.load("model.pt", map_location="cpu"))
 model.eval()
 
 # Model is ready for inference