wav2vec2-ser-ravdess-optimized

facebook/wav2vec2-base-960h fine-tuned for 8-class speech emotion recognition on RAVDESS.

• Test accuracy: 68.3% (speaker-independent — evaluated on 3 actors never seen during training)
• Macro F1: 0.676
• Labels: angry, calm, disgust, fearful, happy, neutral, sad, surprised

Code and full writeup: https://github.com/sbhardwaj1304/Speech-Emotion-Recognition

Usage

import torch
from transformers import Wav2Vec2ForSequenceClassification, Wav2Vec2FeatureExtractor

MODEL_ID = "sbh013/wav2vec2-ser-ravdess-optimized"
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(MODEL_ID)
model = Wav2Vec2ForSequenceClassification.from_pretrained(MODEL_ID)
model.eval()

import librosa
waveform, _ = librosa.load("speech.wav", sr=16000, mono=True)
inputs = feature_extractor(waveform, sampling_rate=16000, return_tensors="pt", padding=True)

with torch.no_grad():
    probs = torch.softmax(model(**inputs).logits, dim=-1).squeeze().numpy()

id2label = model.config.id2label
print({id2label[i]: float(probs[i]) for i in range(len(probs))})

Training details

Setting	Value
Backbone	facebook/wav2vec2-base-960h
CNN feature encoder	Frozen
Epochs	25
LR (encoder / head)	2e-5 / 5e-5
Batch size	8, grad accumulation 2 (effective 16)
LR schedule	Cosine, 6% warmup
Class weights	Inverse frequency, extra boost on sad/angry
Augmentation	Noise, gain, small time shift (70% of training batches)
Precision	fp32
Split	Speaker-independent — 19 train / 2 val / 3 test actors

Per-class results (test set)

Emotion	Precision	Recall	F1
angry	0.917	0.458	0.611
calm	0.810	0.708	0.756
disgust	0.947	0.750	0.837
fearful	0.679	0.792	0.731
happy	0.500	0.833	0.625
neutral	0.588	0.833	0.690
sad	0.500	0.292	0.368
surprised	0.724	0.875	0.793

Limitations

• Trained and evaluated only on RAVDESS (acted, English, studio-quality speech) — real-world microphone conditions and spontaneous (non-acted) speech may perform differently.
• sad is the weakest class, frequently confused with fearful and neutral.

Citation

• Livingstone, S. R., & Russo, F. A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). PLoS ONE, 13(5).
• Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. NeurIPS.