---
language:
- en
pipeline_tag: audio-classification
tags:
- pytorch
- speech-emotion-recognition
- speech-enhancement
- mixture-of-experts
- multi-task-learning
- wavlm
- MSP-Podcast
- arxiv:2509.08470
base_model: microsoft/wavlm-large
---

# Sparse MERIT

Official pretrained checkpoints for **Sparse Mixture-of-Experts Representation Integration Technique (Sparse MERIT)**, introduced in:

**Joint Learning Using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognition**  
Jing-Tong Tzeng, Carlos Busso, and Chi-Chun Lee  
*IEEE Transactions on Audio, Speech and Language Processing*, vol. 34, pp. 3026–3038, 2026.

[Paper](https://doi.org/10.1109/TASLPRO.2026.3688928) · [arXiv](https://arxiv.org/abs/2509.08470) · [Code and evaluation scripts](https://github.com/RogerTzeng/Sparse-MERIT)

## Model overview

Sparse MERIT jointly learns **speech enhancement (SE)** and **speech emotion recognition (SER)** from noisy speech. A shared WavLM-Large backbone supplies frame-level representations to a mixture of experts, with separate task-specific routers selecting expert features for enhancement and emotion classification. This design supports task specialization within a shared representation framework.

![Sparse MERIT architecture](Framework.png)

| Component | Description |
| --- | --- |
| Speech backbone | WavLM-Large; 24 Transformer layers plus the pre-Transformer representation |
| Representation integration | Concatenation of 25 representations, each with 1,024 dimensions |
| Experts | Three shared feed-forward experts; 25,600 → 4,096 → 1,024 dimensions |
| Routing | Task-specific, frame-wise Top-1 expert selection |
| SER head | Attentive statistics pooling followed by a classification head |
| SE head | BSSE-SE-based decoder combining expert features with the noisy log-magnitude spectrogram |
| Audio input | Mono speech at 16 kHz, using training-set waveform normalisation |
| SER output order | `0: Angry`, `1: Sad`, `2: Happy`, `3: Neutral` |

These are component checkpoints for the custom PyTorch implementation in the linked GitHub repository. Use its model definitions and loading code to assemble the system.

## Released files

The root directory contains `WavLM-Large.pt` and the first-stage head checkpoints `pretrained_pool.pt`, `pretrained_ser.pt`, and `pretrained_se.pth.tar`.

The final checkpoint components are stored under:

```text
bsse_noisy_weight_cat_ser_4_classes_git_ws_1_1_5e-5_SSL_0_5_Original_weight_switch_mmoe_3experts/wavLM_adamW/7/
```

| File | Contents |
| --- | --- |
| `final_ssl.pt` | Fine-tuned speech backbone |
| `final_mmoe.pt` | Experts and task-specific routing modules |
| `final_pool.pt` | SER pooling module |
| `final_ser.pt` | Emotion classification head |
| `final_se.pt` | Speech enhancement head |
| `final_loss.pt` | Saved training loss-module state |
| `train_norm_stat.pkl` | Noisy-training waveform mean and standard deviation, used for evaluation input |
| `clean_train_norm_stat.pkl` | Clean-training waveform mean and standard deviation |

## Download and use

Clone the implementation and install its dependencies. The repository documents Python 3.9 as its development environment.

```bash
git clone https://github.com/RogerTzeng/Sparse-MERIT.git
cd Sparse-MERIT
pip install -r requirements.txt
pip install huggingface_hub
```

Run the following Python code from the cloned repository to download weights and normalisation files into the expected directory layout:

```python
from huggingface_hub import HfApi, hf_hub_download

repo_id = "RogerTzeng/Sparse-MERIT"
extensions = (".pt", ".pth", ".pth.tar", ".safetensors", ".ckpt", ".pkl")

for filename in HfApi().list_repo_files(repo_id):
    if filename.endswith(extensions):
        hf_hub_download(
            repo_id=repo_id,
            filename=filename,
            local_dir="model" if "/" in filename else "pretrained_models",
        )
```

For dataset evaluation, configure the audio and label paths in `config_cat.json` and `eval.sh`, and set `--model_path` to the downloaded checkpoint directory under `model/`. Select the intended dataset split and noise condition. Align the WavLM checkpoint path in `eval.py` with `pretrained_models/WavLM-Large.pt` and use the WavLM implementation supplied by the repository. See [`eval.py`](https://github.com/RogerTzeng/Sparse-MERIT/blob/main/eval.py) for component loading and the SER forward pass, and [`train.py`](https://github.com/RogerTzeng/Sparse-MERIT/blob/main/train.py) for joint SE/SER processing.

### Waveform normalisation

Each `.pkl` contains a `(mean, std)` tuple:

| Statistics file | Mean | Standard deviation |
| --- | ---: | ---: |
| `train_norm_stat.pkl` | -0.00016752422864340985 | 0.09842836134288799 |
| `clean_train_norm_stat.pkl` | -0.00008088798101574731 | 0.08598362556210874 |

The repository computes sample-weighted global statistics across the loaded training waveforms, before duration truncation or batch padding:

```text
N = total number of waveform samples
mean = sum(x) / N
std = sqrt(sum(x²) / N - mean²)
x_normalised = (x - mean) / (std + 1e-8)
```

Use `train_norm_stat.pkl` for the evaluation waveform input, as in `eval.py`. Retain these training statistics when evaluating other conditions; do not recompute them on the test set. The repository's waveform dataset uses a default maximum duration of 12 seconds.

## Training and evaluation in the paper

The experiments use **MSP-Podcast v1.11** with four emotion categories: anger, sadness, happiness, and neutral. The paper describes excluding background music and overlapping speech and retaining source recordings with predicted SNR above 20 dB. The Train partition is used for training, Development for model selection and early stopping, and Test1 for evaluation.

Training and development speech is mixed with CRSS-4ENGLISH-14 babble noise at **5 dB SNR**. Evaluation covers **−5, 0, 5, and 10 dB SNR**, using CRSS noise and unseen Freesound and ICASSP 2023 DNS Challenge noise. Overlapping DNS/Freesound segments are removed, and room impulse responses are excluded.

The paper uses two training phases:

1. Freeze WavLM and train the SE and SER heads independently with AdamW at `5e-5`: 130 epochs with batch size 16 for SE, and 20 epochs with batch size 32 for SER.
2. Jointly fine-tune for 20 epochs with batch size 32, using `5e-5` for experts, routers, and task heads, and `2.5e-5` for WavLM Transformer layers. The convolutional feature extractor remains frozen.

The joint objective combines class-weighted cross-entropy for SER and L1 spectral reconstruction loss for SE.

SER is evaluated with macro- and micro-F1 across four training seeds and five non-overlapping Test1 subsets per condition. SE is evaluated using a fixed training seed with PESQ, CSIG, CBAK, COVL, segmental SNR, and STOI. Full condition-specific results and statistical comparisons appear in Tables II and III of the [paper](https://doi.org/10.1109/TASLPRO.2026.3688928).

The paper reports improved SER robustness under low-SNR, unseen-noise conditions and competitive enhancement performance. Its aggregate SER results cover multiple training runs; this Hub repository currently provides the final checkpoint set in the `7/` directory.

## Scope and data access

The model is intended for research on noisy-speech emotion classification, speech enhancement, and multi-task representation learning. Its evaluation covers English podcast speech and the four listed emotion classes. Performance on other languages, recording domains, emotion taxonomies, or noise distributions requires separate evaluation. Emotion predictions reflect learned annotation categories rather than direct measurements of a speaker's internal state.

MSP-Podcast audio and noise datasets are not distributed here. Obtain MSP-Podcast through its [official access process](https://www.lab-msp.com/MSP/MSP-Podcast.html) and follow the terms of each dataset. Results using a different corpus release or noise construction should be identified with that experimental configuration.

The source code is distributed under the [GitHub repository's MIT license](https://github.com/RogerTzeng/Sparse-MERIT/blob/main/LICENSE). Refer to the original providers for terms applicable to third-party pretrained resources and datasets.

## Citation

```bibtex
@article{tzeng_2026_taslp,
  author  = {Tzeng, Jing-Tong and Busso, Carlos and Lee, Chi-Chun},
  title   = {Joint Learning Using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognition},
  journal = {IEEE Transactions on Audio, Speech and Language Processing},
  year    = {2026},
  volume  = {34},
  pages   = {3026--3038},
  doi     = {10.1109/TASLPRO.2026.3688928}
}
```

## Acknowledgements

The framework builds on WavLM and adapts speech enhancement components from [BSSE-SE](https://github.com/khhungg/BSSE-SE). We thank the authors for sharing their implementations and pretrained resources.