---
license: apache-2.0
pipeline_tag: audio-classification
language:
- en
tags:
- audio-classification
- speech-emotion-recognition
- emotion2vec
- ravdess
- multimodal
- emotion-aware-agent
---

# emotion2vec classification head for 6-class speech emotion recognition

A small trained classifier that sits on top of the frozen [`iic/emotion2vec_base_finetuned`](https://modelscope.cn/models/iic/emotion2vec_base_finetuned) speech encoder. Given a speech clip, it predicts one of six emotions: **angry, disgust, fearful, happy, sad, surprised**.

It is the audio branch of the *LLM-Based Multimodal Behaviour-Change Support* project, where speech emotion is combined with text and facial-expression signals before a language model writes a reply.

> **This repository holds only the head (about 0.2M parameters). The backbone is not included.** You load the backbone separately and attach this head to it.

## At a glance

| | |
|---|---|
| Task | Utterance-level speech emotion classification |
| Classes | 6: angry, disgust, fearful, happy, sad, surprised |
| Input | Speech waveform, 16 kHz, mono |
| Backbone | `iic/emotion2vec_base_finetuned` (frozen, 768-dimensional frame embeddings) |
| Trainable parameters | 198,918 (the head only) |
| Training data | RAVDESS, 768 training clips |
| Test accuracy / macro F1 | **92.7% / 0.927** on 192 held-out clips from unseen speakers |
| Weights file | `model_state_dict.pt` |

## How it works

```
waveform (16 kHz, mono)
   -> emotion2vec backbone (frozen)     768 numbers per frame
   -> mean over time frames             768
   -> Linear(768 -> 256)
   -> LayerNorm(256)
   -> GELU
   -> Dropout(0.1)
   -> Linear(256 -> 6)                  6 logits -> softmax
```

| Layer | Parameters |
|---|---|
| Linear 768 to 256 | 196,864 |
| LayerNorm 256 | 512 |
| Linear 256 to 6 | 1,542 |
| **Total** | **198,918** |

Label ids: `0 angry, 1 disgust, 2 fearful, 3 happy, 4 sad, 5 surprised`.

The saved keys carry a `classifier.` prefix. Strip it before loading into a bare head, as shown below.

## How to use

Install the dependencies:

```bash
pip install torch funasr huggingface_hub
```

```python
import torch
import torch.nn as nn
from funasr import AutoModel
from huggingface_hub import hf_hub_download

LABELS = ["angry", "disgust", "fearful", "happy", "sad", "surprised"]

def build_head(hidden=768, num_labels=6):
    return nn.Sequential(
        nn.Linear(hidden, 256), nn.LayerNorm(256), nn.GELU(),
        nn.Dropout(0.1), nn.Linear(256, num_labels),
    )

# 1. Backbone (downloaded at runtime)
backbone = AutoModel(model="iic/emotion2vec_base_finetuned")

# 2. Head: load the weights and strip the "classifier." prefix from the keys
head = build_head()
path = hf_hub_download(repo_id="<your-username>/<this-repo>", filename="model_state_dict.pt")
state = torch.load(path, map_location="cpu")
state = {k.removeprefix("classifier."): v for k, v in state.items()}
head.load_state_dict(state)
head.eval()

# 3. Predict
out = backbone.generate("clip.wav", granularity="frame", extract_embedding=True)
frames = torch.tensor(out[0]["feats"])          # (time, 768)
probs = head(frames.mean(dim=0)).softmax(-1)    # mean over time, as in training
print({l: round(float(p), 3) for l, p in zip(LABELS, probs)})
```

The reference implementation is `src/pipeline/audio_pipeline.py` in the source project. If it differs from the snippet above (for example in how the embedding is extracted), follow the project code.

## Training

| Setting | Value |
|---|---|
| Dataset | RAVDESS, 6 classes (neutral and calm excluded) |
| Samples | 1,152 total: 768 train, 192 validation, 192 test |
| Split | By actor, so no speaker appears in two splits: actors 1 to 16 train, 17 to 20 validation, 21 to 24 test |
| Epochs | 10 |
| Batch size | 8 |
| Learning rate | 2e-5 |
| Optimizer | AdamW, weight decay 0.01 |
| Gradient clipping | Norm 1.0 |
| Mixed precision | Yes |
| Seed | 42 |
| Loss | Cross-entropy |
| Checkpoint choice | Lowest validation loss (epoch 2) |
| Training time | 232 seconds |
| Hardware | Single RTX 3050 6 GB laptop GPU; the backbone runs on the CPU |

Only the head is trained. The backbone embeddings are used as fixed features, which is why training is fast and overfits less than full fine-tuning.

## Evaluation

Test set: 192 clips from four unseen actors, 32 clips per class.

| Metric | Value |
|---|---|
| Accuracy | 92.71% |
| Macro F1 | 0.9271 |
| Weighted F1 | 0.9271 |

| Emotion | Precision | Recall | F1 |
|---|---|---|---|
| angry | 1.000 | 1.000 | 1.000 |
| disgust | 0.889 | 1.000 | 0.941 |
| fearful | 0.929 | 0.813 | 0.867 |
| happy | 1.000 | 0.938 | 0.968 |
| sad | 0.794 | 0.844 | 0.818 |
| surprised | 0.969 | 0.969 | 0.969 |

Most errors: 6 fearful clips predicted as sad, and 4 sad clips predicted as disgust.

### Comparison with other speech models

All three were trained with identical settings on the same split.

| Model | What was trained | Test accuracy | Macro F1 |
|---|---|---|---|
| wav2vec2-base | Whole model | 68.2% | 0.669 |
| WavLM-base | Whole model | 72.4% | 0.711 |
| **emotion2vec + this head** | Head only | **92.7%** | **0.927** |

## Intended use

- Research and prototyping of emotion-aware conversational agents.
- One input signal among several: it is designed to be combined with text and face signals, not to be the only source of truth.

## Out-of-scope use

- Diagnosing or inferring any mental-health or clinical condition from speech. The labels describe expressed emotion in short clips, not a person's health.
- Decisions that affect people, such as hiring, insurance, policing or education.
- Recording or analysing someone's voice without their knowledge and consent.

## Limitations and bias

- **Acted speech.** RAVDESS is made of professional actors reading fixed sentences. Natural, spontaneous speech is harder, and the 92.7% figure should not be expected to carry over.
- **Small and narrow data.** 24 speakers, one language (English), six emotions. Accents, other languages, age groups, noisy rooms and phone-quality audio have not been tested.
- **Closed label set.** Neutral and calm were removed, so the model must pick one of six emotions even when none fits.
- **Backbone overlap not checked.** The backbone checkpoint is itself fine-tuned for emotion recognition. We have not verified whether its fine-tuning data overlaps RAVDESS, so the score may overstate how well the model generalises. Testing on a second dataset is planned.
- **Backbone access.** The backbone is downloaded at runtime and may be gated or rate-limited depending on your account and token.
- Audio quality, speaker identity and language all affect predictions.

## Files

| File | Description |
|---|---|
| `model_state_dict.pt` | Head weights (keys prefixed with `classifier.`) |
| `README.md` | This model card |

## Source

- Weights: `models/audio/runs_emotion2vec/model_state_dict.pt`
- Training script: `models/audio/train.py`
- Inference pipeline: `src/pipeline/audio_pipeline.py`

## Citation

If you use this head, please also cite the backbone and the dataset.

```bibtex
@article{ma2023emotion2vec,
  title   = {emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation},
  author  = {Ma, Ziyang and Zheng, Zhisheng and Ye, Jiaxin and Li, Jinchao and Gao, Zhifu and Zhang, Shiliang and Chen, Xie},
  journal = {arXiv preprint arXiv:2312.15185},
  year    = {2023}
}

@article{livingstone2018ravdess,
  title   = {The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS)},
  author  = {Livingstone, Steven R. and Russo, Frank A.},
  journal = {PLOS ONE},
  volume  = {13},
  number  = {5},
  pages   = {e0196391},
  year    = {2018}
}
```