README.md

metadata

license: apache-2.0
pipeline_tag: audio-classification
language:
  - en
tags:
  - audio-classification
  - speech-emotion-recognition
  - emotion2vec
  - ravdess
  - multimodal
  - emotion-aware-agent

emotion2vec classification head for 6-class speech emotion recognition

A small trained classifier that sits on top of the frozen iic/emotion2vec_base_finetuned speech encoder. Given a speech clip, it predicts one of six emotions: angry, disgust, fearful, happy, sad, surprised.

It is the audio branch of the LLM-Based Multimodal Behaviour-Change Support project, where speech emotion is combined with text and facial-expression signals before a language model writes a reply.

This repository holds only the head (about 0.2M parameters). The backbone is not included. You load the backbone separately and attach this head to it.

At a glance

Task	Utterance-level speech emotion classification
Classes	6: angry, disgust, fearful, happy, sad, surprised
Input	Speech waveform, 16 kHz, mono
Backbone	iic/emotion2vec_base_finetuned (frozen, 768-dimensional frame embeddings)
Trainable parameters	198,918 (the head only)
Training data	RAVDESS, 768 training clips
Test accuracy / macro F1	92.7% / 0.927 on 192 held-out clips from unseen speakers
Weights file	model_state_dict.pt

How it works

waveform (16 kHz, mono)
   -> emotion2vec backbone (frozen)     768 numbers per frame
   -> mean over time frames             768
   -> Linear(768 -> 256)
   -> LayerNorm(256)
   -> GELU
   -> Dropout(0.1)
   -> Linear(256 -> 6)                  6 logits -> softmax

Layer	Parameters
Linear 768 to 256	196,864
LayerNorm 256	512
Linear 256 to 6	1,542
Total	198,918

Label ids: 0 angry, 1 disgust, 2 fearful, 3 happy, 4 sad, 5 surprised.

The saved keys carry a classifier. prefix. Strip it before loading into a bare head, as shown below.

How to use

Install the dependencies:

pip install torch funasr huggingface_hub

import torch
import torch.nn as nn
from funasr import AutoModel
from huggingface_hub import hf_hub_download

LABELS = ["angry", "disgust", "fearful", "happy", "sad", "surprised"]

def build_head(hidden=768, num_labels=6):
    return nn.Sequential(
        nn.Linear(hidden, 256), nn.LayerNorm(256), nn.GELU(),
        nn.Dropout(0.1), nn.Linear(256, num_labels),
    )

# 1. Backbone (downloaded at runtime)
backbone = AutoModel(model="iic/emotion2vec_base_finetuned")

# 2. Head: load the weights and strip the "classifier." prefix from the keys
head = build_head()
path = hf_hub_download(repo_id="<your-username>/<this-repo>", filename="model_state_dict.pt")
state = torch.load(path, map_location="cpu")
state = {k.removeprefix("classifier."): v for k, v in state.items()}
head.load_state_dict(state)
head.eval()

# 3. Predict
out = backbone.generate("clip.wav", granularity="frame", extract_embedding=True)
frames = torch.tensor(out[0]["feats"])          # (time, 768)
probs = head(frames.mean(dim=0)).softmax(-1)    # mean over time, as in training
print({l: round(float(p), 3) for l, p in zip(LABELS, probs)})

The reference implementation is src/pipeline/audio_pipeline.py in the source project. If it differs from the snippet above (for example in how the embedding is extracted), follow the project code.

Training

Setting	Value
Dataset	RAVDESS, 6 classes (neutral and calm excluded)
Samples	1,152 total: 768 train, 192 validation, 192 test
Split	By actor, so no speaker appears in two splits: actors 1 to 16 train, 17 to 20 validation, 21 to 24 test
Epochs	10
Batch size	8
Learning rate	2e-5
Optimizer	AdamW, weight decay 0.01
Gradient clipping	Norm 1.0
Mixed precision	Yes
Seed	42
Loss	Cross-entropy
Checkpoint choice	Lowest validation loss (epoch 2)
Training time	232 seconds
Hardware	Single RTX 3050 6 GB laptop GPU; the backbone runs on the CPU

Only the head is trained. The backbone embeddings are used as fixed features, which is why training is fast and overfits less than full fine-tuning.

Evaluation

Test set: 192 clips from four unseen actors, 32 clips per class.

Metric	Value
Accuracy	92.71%
Macro F1	0.9271
Weighted F1	0.9271

Emotion	Precision	Recall	F1
angry	1.000	1.000	1.000
disgust	0.889	1.000	0.941
fearful	0.929	0.813	0.867
happy	1.000	0.938	0.968
sad	0.794	0.844	0.818
surprised	0.969	0.969	0.969

Most errors: 6 fearful clips predicted as sad, and 4 sad clips predicted as disgust.

Comparison with other speech models

All three were trained with identical settings on the same split.

Model	What was trained	Test accuracy	Macro F1
wav2vec2-base	Whole model	68.2%	0.669
WavLM-base	Whole model	72.4%	0.711
emotion2vec + this head	Head only	92.7%	0.927

Intended use

• Research and prototyping of emotion-aware conversational agents.
• One input signal among several: it is designed to be combined with text and face signals, not to be the only source of truth.

Out-of-scope use

• Diagnosing or inferring any mental-health or clinical condition from speech. The labels describe expressed emotion in short clips, not a person's health.
• Decisions that affect people, such as hiring, insurance, policing or education.
• Recording or analysing someone's voice without their knowledge and consent.

Limitations and bias

• Acted speech. RAVDESS is made of professional actors reading fixed sentences. Natural, spontaneous speech is harder, and the 92.7% figure should not be expected to carry over.
• Small and narrow data. 24 speakers, one language (English), six emotions. Accents, other languages, age groups, noisy rooms and phone-quality audio have not been tested.
• Closed label set. Neutral and calm were removed, so the model must pick one of six emotions even when none fits.
• Backbone overlap not checked. The backbone checkpoint is itself fine-tuned for emotion recognition. We have not verified whether its fine-tuning data overlaps RAVDESS, so the score may overstate how well the model generalises. Testing on a second dataset is planned.
• Backbone access. The backbone is downloaded at runtime and may be gated or rate-limited depending on your account and token.
• Audio quality, speaker identity and language all affect predictions.

Files

File	Description
model_state_dict.pt	Head weights (keys prefixed with classifier.)
README.md	This model card

Source

• Weights: models/audio/runs_emotion2vec/model_state_dict.pt
• Training script: models/audio/train.py
• Inference pipeline: src/pipeline/audio_pipeline.py

Citation

If you use this head, please also cite the backbone and the dataset.

@article{ma2023emotion2vec,
  title   = {emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation},
  author  = {Ma, Ziyang and Zheng, Zhisheng and Ye, Jiaxin and Li, Jinchao and Gao, Zhifu and Zhang, Shiliang and Chen, Xie},
  journal = {arXiv preprint arXiv:2312.15185},
  year    = {2023}
}

@article{livingstone2018ravdess,
  title   = {The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS)},
  author  = {Livingstone, Steven R. and Russo, Frank A.},
  journal = {PLOS ONE},
  volume  = {13},
  number  = {5},
  pages   = {e0196391},
  year    = {2018}
}