| --- |
| license: apache-2.0 |
| pipeline_tag: audio-to-audio |
| --- |
| # StuPASE: Studio-Quality PASE |
|
|
| StuPASE is a state-of-the-art generative speech enhancement model trained to remove noise and reverberation while preserving linguistic content and speaker identity, and achieving studio-level perceptual quality. It operates on 16 kHz mono audio. |
|
|
| --- |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
|
|
| <img src="framework_stupase.png" alt="High-level system design" width="80%"> |
|
|
| StuPASE contains three main components: |
|
|
| - **DeWavLM-R:** Performs low-hallucination phonetic enhancement, fine‑tuned from DeWavLM using dry targets for improved dereverberation. |
| - **Inputs**: Noisy speech |
| - **Outputs**: Enhanced phonetic representations |
|
|
| - **CFM:** Performs phonetic-guided acoustic enhancement. |
| - **Inputs**: Noisy Mel features + Enhanced phonetic representations |
| - **Outputs**: Enhanced Mel features |
|
|
| - **Mel Vocoder:** Reconstructs enhanced wavforms. |
| - **Inputs**: Enhanced Mel features |
| - **Outputs**: Enhanced speech |
|
|
| **Developed by:** Copyright © 2026 by Cisco Systems, Inc. All rights reserved. |
| **Cisco product group**: Collaboration AI: Xiaobin Rong, Mansur Yesilbursa, Kamil Wojcicki |
| **Model type:** Generative Speech Enhancement |
| **License:** Apache 2.0 |
| **Finetuned from:** [WavLM-Large](https://github.com/microsoft/unilm/tree/master/wavlm), [DeWavLM](https://github.com/cisco-open/pase) |
|
|
|
|
| --- |
|
|
| ### Model Sources |
|
|
| - **Repository:** https://github.com/cisco-open/pase |
| - **Paper:** https://arxiv.org/abs/2603.09234 |
| - **Demo:** https://xiaobin-rong.github.io/stupase_demo/ |
| --- |
| ## Uses |
| ### Direct Use |
| - Enhance noisy or reverberant speech recordings |
| - Improve perceptual quality and intelligibility |
| - Preserve speaker identity and linguistic content |
| - Supports **16 kHz mono audio** |
| ### Out-of-Scope Use |
| - Medical, legal, or safety‑critical decisions |
| - Voice conversion or identity manipulation |
| - Non‑speech audio enhancement |
| --- |
| ## How to Get Started |
| Refer to the repository for quick-start code and examples: |
| https://github.com/cisco-open/pase |
| |
| --- |
| ## Training Details |
| ### Training Data |
| We release a StuPASE checkpoint that has been trained on an updated list of datasets. For this release, training used: |
| |
| - Clean speech: |
| - DNS5 Challenge clean-speech resources derived from the LibriVox public-domain subset |
| - [LibriSpeech](https://www.openslr.org/12/) |
| - [LibriTTS](https://www.openslr.org/60/) |
| - [VCTK](https://datashare.ed.ac.uk/handle/10283/3443) |
| - Noise: |
| - DNS5 Challenge noise resources |
| - Room impulse responses: |
| - [OpenSLR26](https://www.openslr.org/26/) |
| - [OpenSLR28](https://www.openslr.org/28/) |
| |
| These source datasets were used to prepare training mixtures and train the released model. The model card and repository do not redistribute the underlying dataset contents; please refer to the original dataset pages and licenses below. |
| |
| ### Dataset Attribution |
| - DNS5 Challenge clean speech (LibriVox subset): clean-speech material prepared from [LibriVox](https://librivox.org/) through the [DNS Challenge](https://github.com/microsoft/DNS-Challenge). The LibriVox recordings used for this portion are [public domain](https://librivox.org/pages/public-domain/) and were used as clean-speech training data for the released checkpoint. |
| - LibriSpeech: [LibriSpeech](https://www.openslr.org/12/) by Vassil Panayotov et al., licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). It was used as clean-speech training data for the released checkpoint. |
| - LibriTTS: [LibriTTS](https://www.openslr.org/60/) by Heiga Zen et al., licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). It was used as clean-speech training data for the released checkpoint. |
| - VCTK Corpus: the [VCTK dataset](https://datashare.ed.ac.uk/handle/10283/3443) from the Centre for Speech Technology Research, University of Edinburgh, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). It was used as clean-speech training data for the released checkpoint. |
| - DNS5 Challenge noise resources: noise data prepared through the [DNS Challenge](https://github.com/microsoft/DNS-Challenge) and used to synthesize noisy training mixtures for the released checkpoint. For this release, the DNS5 noise resources draw on [AudioSet](https://research.google.com/audioset/index.html) material licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), selected [Freesound](https://freesound.org/) files licensed under [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/), and [DEMAND](https://zenodo.org/record/1227121#.XRKKxYhKiUk) environmental recordings licensed under [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/deed.en_CA). |
| - OpenSLR26 and OpenSLR28: [OpenSLR26](https://www.openslr.org/26/) and [OpenSLR28](https://www.openslr.org/28/) room impulse response resources, both licensed under Apache 2.0, were used to add reverberation during training. |
| |
| All audio was resampled to 16 kHz. |
| |
| ### Training Procedure |
| #### Preprocessing |
| - Mixtures generated dynamically |
| - SNR sampled from –5 to 15 dB |
| - Reverberation applied with 50% probability |
| #### Training Hyperparameters |
| - **DeWavLM-R:** 50k steps, LR 2e-5, batch size 20 |
| - **CFM:** 100k steps, LR 1e‑4, batch size 60 |
| - **Mel Vocoder:** 200k steps, LR 2e-4, batch size 60 |
| - Optimizer: AdamW with warmup + cosine decay |
| - Hardware: 2 × NVIDIA RTX 4090 GPUs |
| #### Speeds, Sizes, Times |
| - Total parameters: ~561M |
| - Inference compute: ~104 GMAC/s |
| --- |
| ## Evaluation |
| ### Testing Data |
| - Simulated [LibriSpeech](https://www.openslr.org/12/) test set (using test split) |
| - [DNS1 test set](https://github.com/microsoft/DNS-Challenge/tree/interspeech2020/master/datasets/test_set/synthetic) with/without reverberation |
| ### Metrics |
| - DNSMOS, UTMOS |
| - LPS, SpeechBERTScore (SBS) |
| - Speaker Similarity, using [WavLM-Large based TCAPA-TDNN (finetuned)](https://github.com/microsoft/UniSpeech/tree/main/downstreams/speaker_verification) |
| - WER, using Whisper-Large-v3 |
| |
| ### Results |
| |
| The performance of the retrained version compared to the original one: |
| | Model | DNSMOS | UTMOS | SBS | LPS | SpkSim | WER (%) | |
| |:-----:|:------:|:-----:|:---:|:---:|:------:|:-------:| |
| | DeWavLM-R (orig.) | 3.35 | 3.94 | 0.84 | 0.88 | 0.49 | 13.22 | |
| | **DeWavLM-R (retrained)** | 3.38 | 3.62 | 0.85 | 0.89 | 0.41 | 12.27 | |
| | StuPASE (orig.) | 3.37 | 4.08 | 0.85 |0.90 |0.68 | 11.57| |
| | **StuPASE (retrained)** | 3.36 | 4.02 | 0.85 | 0.89 | 0.66 | 12.06 | |
| |
| It can be seen that the retrained version achieves performance very close to that of the original version on our simulated test set. |
| |
| Overall, StuPASE achieves: |
| - Lowest WER among evaluated generative and discriminative baselines |
| - Highest speaker similarity (SpkSim) |
| - Highest perceptual quality with low hallucination rates |
| - Consistent performance across noisy and reverberant conditions |
| |
| --- |
| ## Bias, Risks, and Limitations |
| - Model trained primarily on English speech; performance may degrade for other languages. |
| - Very strong noise or mismatched reverberation conditions can introduce artifacts. |
| - Speaker characteristics are preserved but not guaranteed perfectly. |
| |
| --- |
| ### Recommendations |
| Evaluate outputs for your specific use case. Avoid deployments where misunderstanding enhanced speech could have safety or legal consequences. |
| |
| --- |
| ## Citation |
| If you use StuPASE in your research, please cite: |
| ```bibtex |
| @misc{StuPASE, |
| title={{StuPASE: Towards Low-Hallucination Studio-Quality Generative Speech Enhancement}}, |
| author={Xiaobin Rong and Jun Gao and Zheng Wang and Mansur Yesilbursa and Kamil Wojcicki and Jing Lu}, |
| year={2026}, |
| eprint={2603.09234}, |
| archivePrefix={arXiv}, |
| primaryClass={eess.AS}, |
| url={https://arxiv.org/abs/2603.09234}, |
| } |
| ``` |
| Copyright © 2026 by Cisco Systems, Inc. All rights reserved. |
| ## Model Card Authorship & Contact |
| - Mansur Yesilbursa: myesilbu@cisco.com |
| |
