MossFormer2 ONNX
ONNX builds of MossFormer2, made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each precision is one folder; what the graphs take and return, and the signal processing that goes around them, is under How to run.
Files
The last column is the benchmark on the VoiceBank-DEMAND test set (824 noisy utterances; details under How to run): each precision's speech-quality scores, and how far its output is from the original PyTorch model's.
| Precision | File | Size | sha256 | Benchmark |
|---|---|---|---|---|
| fp16 | fp16/model.onnx |
117.6 MB | 6efde5265274 |
fp16 storage, fp32 math: PESQ 3.102, STOI 0.9538, SI-SDR 19.42 dB; vs original 72.9 / 57.3 dB; worst file -0.03 dB (CUDA) |
| int8 | int8/model.onnx |
64.2 MB | 127b799c4db8 |
experimental: PESQ 3.091, STOI 0.9539, SI-SDR 19.31 dB; vs original 35.9 / 27.6 dB; worst file -0.90 dB, 10 files below -0.5 (CPU) |
| int4 | int4/model.onnx |
47.4 MB | d33cae85992c |
experimental: PESQ 3.085, STOI 0.9538, SI-SDR 19.31 dB; vs original 35.4 / 29.8 dB; worst file -0.71 dB, 8 files below -0.5 (CPU) |
How to run
MossFormer2 SE 48K (ClearerVoice-Studio's speech enhancement: removes noise from speech at 48 kHz) as one ONNX graph:
the network that turns Kaldi filterbank features into a spectral mask. Input fbanks [1, T, 180] (60 log-mel bands
plus their first and second deltas; frames of 40 ms every 8 ms at 48 kHz), output mask [1, T, 961]; T is
free. The features, the STFT, applying the mask and the ISTFT stay outside the graph, exactly as in ClearerVoice's
own code, so the simplest correct pipeline is ClearerVoice's decode function with its network swapped for this graph:
import numpy as np, onnxruntime as ort, soundfile as sf, torch
import clearvoice.utils.decode as dec # pip install clearvoice==0.1.2
from clearvoice.network_wrapper import network_wrapper
w = network_wrapper(); w.model_name = "MossFormer2_SE_48K"; w.load_args_se()
args = w.args; args.network = "MossFormer2_SE_48K"
sess = ort.InferenceSession("int8/model.onnx", providers=["CPUExecutionProvider"]) # fp16: CUDAExecutionProvider
class Net: # what decode() calls: net(fbanks)[-1] is the mask
def __call__(self, fbanks):
return [torch.from_numpy(sess.run(None, {"fbanks": fbanks.numpy().astype(np.float32)})[0])]
noisy, sr = sf.read("noisy_48k.wav", dtype="float32") # 48 kHz mono
clean = dec.decode_one_audio(Net(), torch.device("cpu"), noisy[None].astype(np.float64), args)
sf.write("enhanced.wav", np.asarray(clean, np.float32).reshape(-1)[: len(noisy)] / 32768, 48000)
ClearerVoice computes its filterbank with dither=1.0, so its output differs slightly from run to run; set
dec.compute_fbank to call torchaudio.compliance.kaldi.fbank(..., dither=0.0) for repeatable output (the
benchmark below does). Files longer than 20 s are processed in 4 s windows by decode_one_audio itself.
Precisions. fp16/: the weights stored in fp16 and cast to fp32 where they are read, so the math is fp32's: the
network's gated FSMN layers reach values around 1e10, far past fp16's range, and a true fp16 graph outputs NaN.
int8/: weight-only int8 (every MatMul and Conv weight int8 per output channel, activations fp32), for the CPU
provider. int4/: the MatMul weights 4-bit (ONNX Runtime MatMulNBits, block 32) and the convolutions int8; MatMuls
hold 82 % of the weights. int8/ and int4/ are experimental: on about 1 % of files their SI-SDR is more than
0.5 dB below the original's (worst 0.9 dB), though the averages barely move. No bf16 (no bf16 convolution in
ONNX Runtime).
Benchmark. VoiceBank-DEMAND test set (Valentini et al.: 824 utterances by two unseen speakers, 48 kHz),
enhanced with ClearerVoice's own decode code, filterbank dither 0. PESQ is wide-band (ITU-T P.862.2, at 16 kHz), STOI
at 48 kHz, SI-SDR against the clean speech; "vs original" is the SI-SDR of each output against the PyTorch model's
(clearvoice 0.1.2, fp32, CPU), mean / worst file; "worst file" is the largest SI-SDR drop on any one file against
the original's. Pass (the speech-enhancement threshold used across these repositories): no file more than 0.5 dB
below. Real-time factor with 4 CPU threads (Intel laptop CPU) or on an RTX 3080 Laptop GPU, whole pipeline.
| PESQ | STOI | SI-SDR (dB) | vs original (dB) | Worst file vs original | Real-time factor | |
|---|---|---|---|---|---|---|
| noisy input | 1.97 | 0.921 | 8.39 | - | - | - |
| original (PyTorch fp32, CPU) | 3.102 | 0.9538 | 19.42 | - | - | 0.26 |
| ONNX fp32 (CPU, not published) | 3.102 | 0.9538 | 19.42 | 125 / 107 | -0.00 dB | 0.18 |
fp16/ (CUDA) |
3.102 | 0.9538 | 19.42 | 72.9 / 57.3 | -0.03 dB | 0.13 |
int8/ (CPU), experimental |
3.091 | 0.9539 | 19.31 | 35.9 / 27.6 | -0.90 dB (10 files below -0.5) | 0.30 |
int4/ (CPU), experimental |
3.085 | 0.9538 | 19.31 | 35.4 / 29.8 | -0.71 dB (8 files below -0.5) | 0.21 |
How these were made
From MossFormer2 at revision eff8c97925, starting from an fp32 ONNX graph (where it came from is said above).
fp16: onnxconverter-common with inputs and outputs kept fp32, or, where the network's activations leave fp16's
range, fp16 weights with fp32 math (said above when so). int8: weight-only, every Conv / MatMul / GRU weight int8
with a per-output-channel scale behind DequantizeLinear and activations in fp32, so it runs on the CPU provider
without a calibration set. int4: ONNX Runtime's MatMulNBits (weight-only, block 32) on the MatMul weights, the rest
int8 as above, where the graph has enough MatMul weights to matter. No bf16: ONNX Runtime has no bf16 convolution or
GRU kernels. Every graph was benchmarked before upload; the sha256 column above is what the files must hash to.
License
Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.
Original model card
Introduction
The MossFormer2_SE_48K model weights for 48 kHz speech enhancement in ClearerVoice-Studio repo.
This model is trained on large scale datasets inclduing open-sourced and private data.
It enhances speech audios by removing background noise.
Install
Clone the Repository
git clone https://github.com/modelscope/ClearerVoice-Studio.git
Create Conda Environment
cd ClearerVoice-Studio
conda create -n clearvoice python=3.8
conda activate clearvoice
pip install -r requirements.txt
Run Script
Go to clearvoice/ and use the following examples. The MossFormer2_SE_48K model will be downloaded from huggingface automatically.
Sample example 1: use speech enhancement model MossFormer2_SE_48K to process one wave file of samples/input.wav and save the output wave file to samples/output_MossFormer2_SE_48K.wav
from clearvoice import ClearVoice
myClearVoice = ClearVoice(task='speech_enhancement', model_names=['MossFormer2_SE_48K'])
output_wav = myClearVoice(input_path='samples/input.wav', online_write=False)
myClearVoice.write(output_wav, output_path='samples/output_MossFormer2_SE_48K.wav')
Sample example 2: use speech enhancement model MossFormer2_SE_48K to process all input wave files in samples/path_to_input_wavs/ and save all output files to samples/path_to_output_wavs
from clearvoice import ClearVoice
myClearVoice = ClearVoice(task='speech_enhancement', model_names=['MossFormer2_SE_48K'])
myClearVoice(input_path='samples/path_to_input_wavs', online_write=True, output_path='samples/path_to_output_wavs')
Sample example 3: use speech enhancement model MossFormer2_SE_48K to process wave files listed in `samples/audio_samples.scp' file, and save all output files to 'samples/path_to_output_wavs_scp/'
from clearvoice import ClearVoice
myClearVoice = ClearVoice(task='speech_enhancement', model_names=['MossFormer2_SE_48K'])
myClearVoice(input_path='samples/scp/audio_samples.scp', online_write=True, output_path='samples/path_to_output_wavs_scp')
Model tree for moonbox-dev/MossFormer2_SE_48K-ONNX
Base model
alibabasglab/MossFormer2_SE_48K