mahdikarimiii/wav2vec2-modified-shemo-augmented

Persian Speech Emotion Recognition (Wav2Vec2 Augmented)

This model is a fine-tuned version of the superb/wav2vec2-base-superb-er architecture, specifically trained to recognize human emotions from Persian (Farsi) speech. It classifies audio into 6 distinct emotional states.

Model Details

Model Description

This model provides a robust solution for Persian Speech Emotion Recognition (SER). It addresses the inherent severe class imbalance in Persian emotional datasets (specifically the lack of "happiness" samples) by utilizing targeted Data Augmentation (Pitch Shifting, Time Stretching, and White Noise Injection) and strategic Class Weighting during training. This ensures the model does not just default to majority classes like "neutral" or "angry", achieving a balanced and highly accurate F1-Macro score.

• Developed by: Mahdi Karimi (Department of Computer Science and Engineering, Shahid Beheshti University)
• Model type: Audio Classification (Wav2Vec2ForSequenceClassification)
• Language(s) (NLP): Persian (Farsi)
• License: Apache 2.0 (Inherited from the base dataset/model)
• Finetuned from model: superb/wav2vec2-base-superb-er

Model Sources

• Dataset: aliyzd95/modified_shemo

Uses

Direct Use

The model is optimized for direct integration into production environments, such as Python-based backends (FastAPI/Flask) for web applications. It accepts 16kHz mono audio files and outputs the predicted emotional state, which can be used to drive UI animations (e.g., dynamic emojis in a React frontend).

Out-of-Scope Use

This model is trained specifically on semi-natural Persian speech. It may not perform optimally on:

• Highly noisy environments without prior noise-cancellation preprocessing.
• Languages other than Persian.
• Extreme outlier emotional states not represented in the ShEMO dataset.

Bias, Risks, and Limitations

While the original ShEMO dataset suffers from a severe imbalance (e.g., "happiness" making up less than 7% of the data), this model explicitly mitigates this bias through synthetic data generation for minority classes.

However, users should be aware that the classes fear and surprise still have inherently lower recall due to the extreme scarcity of source samples. The model performs exceptionally well on neutral, anger, and sadness.

Recommendations

For downstream integration, it is recommended to chunk incoming audio streams into 3-second segments, pad them to max_length=64000 (at 16kHz), and use f1_macro rather than plain accuracy for any further re-evaluation.

How to Get Started with the Model

Use the code below to get started with the model. Note that the preprocessor_config.json is fully integrated.

from transformers import AutoFeatureExtractor, AutoModelForAudioClassification
import torch
import librosa

model_id = "mahdikarimiii/wav2vec2-modified-shemo-augmented"

# Load extractor and model
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = AutoModelForAudioClassification.from_pretrained(model_id)

# Load and preprocess audio
audio_data, sr = librosa.load("path_to_audio.wav", sr=16000)

inputs = feature_extractor(
    audio_data,
    sampling_rate=16000,
    return_tensors="pt",
    padding="max_length",
    max_length=64000,
    truncation=True
)

with torch.no_grad():
    logits = model(**inputs).logits

predicted_id = torch.argmax(logits, dim=-1).item()
predicted_emotion = model.config.id2label[predicted_id]
print(f"Predicted Emotion: {predicted_emotion}")

Training Details

Training Data

The model was trained on the Modified ShEMO (Sharif Emotional Speech Database), which corrects significant transcription and alignment errors present in the original dataset. To combat class imbalance, the happiness class was heavily augmented using librosa prior to feature extraction.

Training Procedure

Preprocessing

• Sampling Rate: All files strictly resampled to 16,000 Hz.
• Chunking/Padding: Fixed sequence length of 64,000 samples (4 seconds).
• Augmentation: White noise injection, pitch shifting (-2 to +2 semitones), and time stretching (0.8x to 1.2x) applied exclusively to minority classes.

Training Hyperparameters

• Training regime: fp32
• Learning Rate: 3e-5
• Epochs: 5
• Train Batch Size: 8 (per device)
• Gradient Accumulation Steps: 2
• Eval Batch Size: 8
• Eval Accumulation Steps: 4
• Loss Function: Weighted Cross-Entropy Loss (calculated via sklearn.utils.class_weight)

Evaluation

Testing Data, Factors & Metrics

• Testing Data: 20% stratified split of the Modified ShEMO dataset (600 instances).
• Metrics: F1-Macro (primary metric to ensure minority class representation) and Accuracy.

Results

The model achieves strong generalization, successfully pulling the recall of the "happiness" class from 0.00 (in non-augmented baseline runs) to 0.51, while maintaining high precision on majority classes.

Summary

Metric	Value
Accuracy	77.0%
F1-Macro	58.0%
Validation Loss	1.093

Environmental Impact

• Hardware Type: Dual NVIDIA T4 (Kaggle Environment)
• Hours used: ~0.6 hours

Citation

If you use this model, please ensure you also cite the creators of the original and modified ShEMO datasets:

BibTeX:

@Article{MohamadNezami2019,
    author="Mohamad Nezami, Omid and Jamshid Lou, Paria and Karami, Mansoureh",
    title="ShEMO: a large-scale validated database for Persian speech emotion detection",
    journal="Language Resources and Evaluation",
    year="2019"
}

@misc{yazdani2022persian,
    title={A Persian ASR-based SER: Modification of Sharif Emotional Speech Database},
    author={Ali Yazdani and Yasser Shekofteh},
    year={2022},
    eprint={2211.09956},
    archivePrefix={arXiv}
}