|
|
--- |
|
|
license: mit |
|
|
datasets: |
|
|
- confit/cremad-parquet |
|
|
language: |
|
|
- en |
|
|
metrics: |
|
|
- accuracy |
|
|
- f1 |
|
|
base_model: |
|
|
- facebook/wav2vec2-base-960h |
|
|
library_name: transformers |
|
|
tags: |
|
|
- speech |
|
|
- speech-emotion-recognition |
|
|
- ser |
|
|
- wav2vec2 |
|
|
--- |
|
|
|
|
|
# SER Wav2Vec2 — Speech Emotion Recognition |
|
|
|
|
|
## Model Summary |
|
|
- Architecture: Wav2Vec2 encoder with a sequence classification head (`Wav2Vec2ForSequenceClassification`). |
|
|
- Sampling rate: 16 kHz mono inputs. |
|
|
- Label set: `angry`, `calm`, `disgust`, `fear`, `happy`, `neutral`, `sad`, `surprise`. |
|
|
- Base: `facebook/wav2vec2-base-960h` fine‑tuned for emotion classification. |
|
|
|
|
|
## Intended Use |
|
|
- Input: short audio clips of speech. |
|
|
- Output: per‑class probabilities for the above emotions and a dominant label. |
|
|
- Use cases: UX feedback, call‑center analytics, demo apps, research prototypes. |
|
|
|
|
|
## Out‑of‑Scope |
|
|
- Clinical or safety‑critical decisions. |
|
|
- Non‑speech audio (music, noise) or multilingual speech without adaptation. |
|
|
|
|
|
## Data and Training |
|
|
- Dataset: CREMA‑D (via `confit/cremad-parquet`). |
|
|
- Preprocessing: resample to 16 kHz, convert to mono, optional padding for very short clips. |
|
|
- Fine‑tuning objective: cross‑entropy classification. |
|
|
|
|
|
## Configuration Snapshot |
|
|
- Hidden size: 768 |
|
|
- Attention heads: 12 |
|
|
- Hidden layers: 12 |
|
|
- Classifier projection: 256 |
|
|
- Final dropout: 0.0 |
|
|
- Feature extractor norm: group |
|
|
|
|
|
## Metrics |
|
|
- Reported metrics: accuracy, F1‑score. |
|
|
- Benchmark numbers depend on split and preprocessing; fill in with your evaluation results. |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Python (Transformers) |
|
|
```python |
|
|
import torch, numpy as np, soundfile as sf, librosa |
|
|
from transformers import Wav2Vec2ForSequenceClassification, AutoFeatureExtractor |
|
|
|
|
|
model_dir = "path/to/best_model" # local directory or Hub repo id |
|
|
model = Wav2Vec2ForSequenceClassification.from_pretrained(model_dir) |
|
|
fe = AutoFeatureExtractor.from_pretrained(model_dir) |
|
|
model.eval() |
|
|
|
|
|
def load_audio(path, sr=fe.sampling_rate): |
|
|
y, s = sf.read(path, always_2d=False) |
|
|
if isinstance(y, np.ndarray): |
|
|
if y.ndim > 1: |
|
|
y = np.mean(y, axis=1) |
|
|
if s != sr: |
|
|
y = librosa.resample(y.astype(np.float32), orig_sr=s, target_sr=sr) |
|
|
y = y.astype(np.float32) |
|
|
else: |
|
|
y = np.array(y, dtype=np.float32) |
|
|
if y.size < sr // 10: |
|
|
y = np.pad(y, (0, max(0, sr - y.size))) |
|
|
return y |
|
|
|
|
|
audio = load_audio("sample.wav") |
|
|
inputs = fe(audio, sampling_rate=fe.sampling_rate, return_tensors="pt") |
|
|
with torch.no_grad(): |
|
|
logits = model(**inputs).logits |
|
|
probs = torch.softmax(logits, dim=-1)[0].cpu().numpy() |
|
|
labels = [model.config.id2label[str(i)] if isinstance(list(model.config.id2label.keys())[0], str) else model.config.id2label[i] for i in range(len(probs))] |
|
|
pairs = sorted(zip(labels, probs), key=lambda x: x[1], reverse=True) |
|
|
print(pairs[:3]) |
|
|
``` |
|
|
|
|
|
### FastAPI Integration |
|
|
This repository includes a FastAPI server that exposes `POST /predict` and returns sorted label probabilities and the dominant emotion. |
|
|
|
|
|
## Input and Output Schema |
|
|
- Input: audio file (`wav`, `mp3`, `m4a`, etc.). Internally converted to 16 kHz mono. |
|
|
- Output JSON: |
|
|
```json |
|
|
{ |
|
|
"results": [{ "label": "happy", "score": 0.81 }, ...], |
|
|
"dominant": { "label": "happy", "score": 0.81 } |
|
|
} |
|
|
``` |
|
|
|
|
|
## Limitations and Bias |
|
|
- Emotion labels are subjective; datasets may reflect staged emotions. |
|
|
- Performance can degrade with strong noise, reverberation, or accents not present in training. |
|
|
- Not suitable for sensitive decision‑making without rigorous validation. |
|
|
|
|
|
## Ethical Considerations |
|
|
- Obtain consent where required. |
|
|
- Be transparent about the model’s limitations and intended use. |
|
|
- Avoid deployment in contexts where misclassification can cause harm. |
|
|
|
|
|
## Citation |
|
|
If you use this model, please cite: |
|
|
- Baevski et al., "wav2vec 2.0: A Framework for Self‑Supervised Learning of Speech Representations", NeurIPS 2020. |
|
|
- CREMA‑D: Cao et al., "CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset", IEEE Transactions on Affective Computing. |
|
|
|
|
|
## License |
|
|
MIT |
|
|
|
|
|
|
