Vocal Console — Speech Emotion Recognition

A React (Vite + Tailwind) frontend and FastAPI backend around your fine-tuned WavLM-Large SER model (from ser_wavlm_large_multi.py). Record from the mic or upload an audio file, get per-emotion probabilities, and swap in new model checkpoints from the UI as you train better versions.

ser-app/
├── backend/          FastAPI service, model loading, /predict
│   ├── app/
│   │   ├── main.py   API routes
│   │   └── model.py  architecture — must match the training script
│   ├── models/        drop your .pt checkpoints here
│   └── requirements.txt
└── frontend/          React + Vite + Tailwind UI
    └── src/

1. Backend

Requires ffmpeg on PATH (used by pydub to decode webm/ogg audio from the browser's MediaRecorder). Install it first:

# macOS
brew install ffmpeg
# Ubuntu/Debian
sudo apt-get install ffmpeg

Then:

cd backend
python -m venv .venv && source .venv/Scripts/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Copy the checkpoint your Kaggle notebook produces (wavlm_large_multi.pt, from OUT / "wavlm_large_multi.pt" in the training script) into backend/models/, then start the API:

uvicorn app.main:app --reload --port 8000

On startup it auto-loads the first .pt file it finds in backend/models/ (or whichever one was last activated, tracked in models/_active.json). If backend/models/ is empty, the server still starts — use the model-upload panel in the UI, or POST /api/models/upload, to add one.

The first request downloads microsoft/wavlm-large's base weights from Hugging Face (a few hundred MB) before your fine-tuned weights are applied on top — this only happens once per checkpoint activation, cached locally afterward.

API summary

2. Frontend

cd frontend
npm install
npm run dev

Open http://localhost:5173 — Vite proxies /api/* to http://localhost:8000 (see vite.config.js), so both must be running together in dev.

For production, run npm run build and serve frontend/dist/ from whatever you like (nginx, the FastAPI app itself via StaticFiles, etc.), pointing /api at your deployed backend.

3. Model versions

Description

The models uses SSl model WavLM Large fine-tuned on a combined corpus of:

• CREMA-D
• IEMOCAP
• RAVDESS

Version 1 — WavLM Large (8-Class Emotion Recognition)

The model performs classification into 8 emotion categories: angry, calm, disgusted, fearful, happy, neutral, sad, surprised.

Evaluation Results (Emo-Box)

---

Version 2 — WavLM Large with Class Weighted Loss

Description

This version improves handling of class imbalance by applying: Inverse Frequency Class Weighted Loss

Minority classes receive higher weights during training, encouraging the model to learn underrepresented emotions. We also drop classes like clam and surprised as they are underrepresented in the training corpus.

Emotion classes: angry, disgusted, fearful, happy, neutral, sad.

Evaluation Results (Emo-Box)

---

Version 3 — WavLM Large with Sampling-Based Class Balancing

Description

This version addresses class imbalance using a combination of:

• Under-sampling majority classes
• Over-sampling minority classes

The median class distribution was selected as the balancing target.

Evaluation Results (Emo-Box)

---

📊 Performance Comparison

---

🤗 Hugging Face Models

The trained models are uploaded on Hugging Face for easy access.

You can download the pretrained model files from the following Hugging Face Repository: https://huggingface.co/arijitraj01/SER_WavLM_Fine-tuning

4. Adding a new fine-tuned model version

Your training script already saves checkpoints in the right shape:

torch.save({"state": best_state, "classes": classes, "dev_ua": best_ua}, ...)

Any .pt file with that structure works. To roll out a new version:

1. Download the checkpoint from your Kaggle run's Output tab.
2. In the app's Model version panel, click + upload .pt and pick the file.
3. It's activated immediately — no restart needed. Older versions stay listed so you can switch back at any time.

Notes

• Inference caps clips at 10s (matching EVAL_CAP_S in training) and applies the same per-utterance normalization as the eval path.
• /api/predict returns a full softmax over the checkpoint's union label space; it doesn't mask to a specific corpus's classes (that masking only applied during training/evaluation against known-corpus test sets).
• GPU is used automatically if torch.cuda.is_available(); otherwise CPU (fine for single-clip inference, just slower per request).