MossFormer2 ONNX

ONNX builds of MossFormer2, made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each precision is one folder; what the graphs take and return, and the signal processing that goes around them, is under How to run.

Files

The last column is the benchmark on the VoiceBank-DEMAND test set (824 noisy utterances; details under How to run): each precision's speech-quality scores, and how far its output is from the original PyTorch model's.

Precision File Size sha256 Benchmark
fp16 fp16/model.onnx 117.6 MB 6efde5265274 fp16 storage, fp32 math: PESQ 3.102, STOI 0.9538, SI-SDR 19.42 dB; vs original 72.9 / 57.3 dB; worst file -0.03 dB (CUDA)
int8 int8/model.onnx 64.2 MB 127b799c4db8 experimental: PESQ 3.091, STOI 0.9539, SI-SDR 19.31 dB; vs original 35.9 / 27.6 dB; worst file -0.90 dB, 10 files below -0.5 (CPU)
int4 int4/model.onnx 47.4 MB d33cae85992c experimental: PESQ 3.085, STOI 0.9538, SI-SDR 19.31 dB; vs original 35.4 / 29.8 dB; worst file -0.71 dB, 8 files below -0.5 (CPU)

How to run

MossFormer2 SE 48K (ClearerVoice-Studio's speech enhancement: removes noise from speech at 48 kHz) as one ONNX graph: the network that turns Kaldi filterbank features into a spectral mask. Input fbanks [1, T, 180] (60 log-mel bands plus their first and second deltas; frames of 40 ms every 8 ms at 48 kHz), output mask [1, T, 961]; T is free. The features, the STFT, applying the mask and the ISTFT stay outside the graph, exactly as in ClearerVoice's own code, so the simplest correct pipeline is ClearerVoice's decode function with its network swapped for this graph:

import numpy as np, onnxruntime as ort, soundfile as sf, torch
import clearvoice.utils.decode as dec                      # pip install clearvoice==0.1.2
from clearvoice.network_wrapper import network_wrapper

w = network_wrapper(); w.model_name = "MossFormer2_SE_48K"; w.load_args_se()
args = w.args; args.network = "MossFormer2_SE_48K"
sess = ort.InferenceSession("int8/model.onnx", providers=["CPUExecutionProvider"])   # fp16: CUDAExecutionProvider

class Net:                                                  # what decode() calls: net(fbanks)[-1] is the mask
    def __call__(self, fbanks):
        return [torch.from_numpy(sess.run(None, {"fbanks": fbanks.numpy().astype(np.float32)})[0])]

noisy, sr = sf.read("noisy_48k.wav", dtype="float32")      # 48 kHz mono
clean = dec.decode_one_audio(Net(), torch.device("cpu"), noisy[None].astype(np.float64), args)
sf.write("enhanced.wav", np.asarray(clean, np.float32).reshape(-1)[: len(noisy)] / 32768, 48000)

ClearerVoice computes its filterbank with dither=1.0, so its output differs slightly from run to run; set dec.compute_fbank to call torchaudio.compliance.kaldi.fbank(..., dither=0.0) for repeatable output (the benchmark below does). Files longer than 20 s are processed in 4 s windows by decode_one_audio itself.

Precisions. fp16/: the weights stored in fp16 and cast to fp32 where they are read, so the math is fp32's: the network's gated FSMN layers reach values around 1e10, far past fp16's range, and a true fp16 graph outputs NaN. int8/: weight-only int8 (every MatMul and Conv weight int8 per output channel, activations fp32), for the CPU provider. int4/: the MatMul weights 4-bit (ONNX Runtime MatMulNBits, block 32) and the convolutions int8; MatMuls hold 82 % of the weights. int8/ and int4/ are experimental: on about 1 % of files their SI-SDR is more than 0.5 dB below the original's (worst 0.9 dB), though the averages barely move. No bf16 (no bf16 convolution in ONNX Runtime).

Benchmark. VoiceBank-DEMAND test set (Valentini et al.: 824 utterances by two unseen speakers, 48 kHz), enhanced with ClearerVoice's own decode code, filterbank dither 0. PESQ is wide-band (ITU-T P.862.2, at 16 kHz), STOI at 48 kHz, SI-SDR against the clean speech; "vs original" is the SI-SDR of each output against the PyTorch model's (clearvoice 0.1.2, fp32, CPU), mean / worst file; "worst file" is the largest SI-SDR drop on any one file against the original's. Pass (the speech-enhancement threshold used across these repositories): no file more than 0.5 dB below. Real-time factor with 4 CPU threads (Intel laptop CPU) or on an RTX 3080 Laptop GPU, whole pipeline.

PESQ STOI SI-SDR (dB) vs original (dB) Worst file vs original Real-time factor
noisy input 1.97 0.921 8.39 - - -
original (PyTorch fp32, CPU) 3.102 0.9538 19.42 - - 0.26
ONNX fp32 (CPU, not published) 3.102 0.9538 19.42 125 / 107 -0.00 dB 0.18
fp16/ (CUDA) 3.102 0.9538 19.42 72.9 / 57.3 -0.03 dB 0.13
int8/ (CPU), experimental 3.091 0.9539 19.31 35.9 / 27.6 -0.90 dB (10 files below -0.5) 0.30
int4/ (CPU), experimental 3.085 0.9538 19.31 35.4 / 29.8 -0.71 dB (8 files below -0.5) 0.21

How these were made

From MossFormer2 at revision eff8c97925, starting from an fp32 ONNX graph (where it came from is said above). fp16: onnxconverter-common with inputs and outputs kept fp32, or, where the network's activations leave fp16's range, fp16 weights with fp32 math (said above when so). int8: weight-only, every Conv / MatMul / GRU weight int8 with a per-output-channel scale behind DequantizeLinear and activations in fp32, so it runs on the CPU provider without a calibration set. int4: ONNX Runtime's MatMulNBits (weight-only, block 32) on the MatMul weights, the rest int8 as above, where the graph has enough MatMul weights to matter. No bf16: ONNX Runtime has no bf16 convolution or GRU kernels. Every graph was benchmarked before upload; the sha256 column above is what the files must hash to.

License

Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.


Original model card

Introduction

The MossFormer2_SE_48K model weights for 48 kHz speech enhancement in ClearerVoice-Studio repo.

This model is trained on large scale datasets inclduing open-sourced and private data.

It enhances speech audios by removing background noise.

Install

Clone the Repository

git clone https://github.com/modelscope/ClearerVoice-Studio.git

Create Conda Environment

cd ClearerVoice-Studio
conda create -n clearvoice python=3.8
conda activate clearvoice
pip install -r requirements.txt

Run Script

Go to clearvoice/ and use the following examples. The MossFormer2_SE_48K model will be downloaded from huggingface automatically.

Sample example 1: use speech enhancement model MossFormer2_SE_48K to process one wave file of samples/input.wav and save the output wave file to samples/output_MossFormer2_SE_48K.wav

from clearvoice import ClearVoice

myClearVoice = ClearVoice(task='speech_enhancement', model_names=['MossFormer2_SE_48K'])

output_wav = myClearVoice(input_path='samples/input.wav', online_write=False)

myClearVoice.write(output_wav, output_path='samples/output_MossFormer2_SE_48K.wav')

Sample example 2: use speech enhancement model MossFormer2_SE_48K to process all input wave files in samples/path_to_input_wavs/ and save all output files to samples/path_to_output_wavs

from clearvoice import ClearVoice

myClearVoice = ClearVoice(task='speech_enhancement', model_names=['MossFormer2_SE_48K'])

myClearVoice(input_path='samples/path_to_input_wavs', online_write=True, output_path='samples/path_to_output_wavs')

Sample example 3: use speech enhancement model MossFormer2_SE_48K to process wave files listed in `samples/audio_samples.scp' file, and save all output files to 'samples/path_to_output_wavs_scp/'

from clearvoice import ClearVoice

myClearVoice = ClearVoice(task='speech_enhancement', model_names=['MossFormer2_SE_48K'])

myClearVoice(input_path='samples/scp/audio_samples.scp', online_write=True, output_path='samples/path_to_output_wavs_scp')
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for moonbox-dev/MossFormer2_SE_48K-ONNX

Quantized
(6)
this model