moonbox-dev/FRCRN_SE_16K-ONNX

FRCRN ONNX

ONNX builds of FRCRN, made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each precision is one folder; what the graphs take and return, and the signal processing that goes around them, is under How to run.

Files

The last column is the benchmark on the VoiceBank-DEMAND test set (824 noisy utterances; details under How to run): each precision's speech-quality scores, and how far its output is from the original PyTorch model's.

Precision	File	Size	sha256	Benchmark
fp16	fp16/model.onnx	29.4 MB	b726166d44ce	fp16 storage, fp32 math: PESQ 3.033, STOI 0.9530, SI-SDR 18.68 dB; vs fp32 66.6 / 41.0 dB; worst file -0.09 dB (CUDA)

How to run

FRCRN SE 16K (ClearerVoice-Studio's speech enhancement: removes noise from speech at 16 kHz; a complex-valued convolutional recurrent network on the STFT) as one ONNX graph: input wave [1, N] (16 kHz mono, float), output enhanced [1, N]. The STFT, the complex mask and the inverse STFT are inside the graph (FRCRN computes its STFT with convolutions), so the graph is the whole denoiser; N is free. Around it, ClearerVoice brings the input to a fixed loudness (audio_norm, -25 dBFS) and scales the output back, and its decode function pads clips shorter than 1 s and runs anything up to 120 s in one call. Both are kept here, as in the benchmark:

import numpy as np, onnxruntime as ort, soundfile as sf, torch
import clearvoice.utils.decode as dec                      # pip install clearvoice==0.1.2 (only for its helpers)
from clearvoice.dataloader.dataloader import audio_norm
from clearvoice.network_wrapper import network_wrapper

w = network_wrapper(); w.model_name = "FRCRN_SE_16K"; w.load_args_se()
args = w.args; args.network = "FRCRN_SE_16K"
sess = ort.InferenceSession("fp16/model.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"])

class Net:                                                  # what decode() calls
    def inference(self, x):
        return torch.from_numpy(sess.run(None, {"wave": x.numpy().astype(np.float32)})[0])

noisy, sr = sf.read("noisy_16k.wav", dtype="float32")      # 16 kHz mono
x, scale = audio_norm(noisy.astype(np.float64))
clean = dec.decode_one_audio(Net(), torch.device("cpu"), x[None].astype(np.float32), args)
sf.write("enhanced.wav", np.asarray(clean, np.float32).reshape(-1)[: len(noisy)] * scale, 16000)

Without ClearerVoice: scale the waveform to about -25 dBFS RMS, pad it with zeros to at least 1 s, run the graph once, cut the output back to the input's length and undo the scaling.

Precision. Only fp16/: the weights stored in fp16 and cast to fp32 where they are read, so the math is fp32's (a true fp16 graph loses a lot: PESQ 2.24 instead of 3.29 on our first test clips). Use a GPU provider: on ONNX Runtime's CPU provider the graph runs about 8 times slower than real time on a laptop CPU (as does the PyTorch original). No int8 or int4: both fail. Weight-only int8 puts 583 of the 824 test files more than 0.5 dB below the original (mean -4.6 dB, worst 29 dB). Int4 (MatMulNBits) all but undoes the enhancement (PESQ 2.02, against 1.97 for the noisy input). The model is sensitive to weight rounding in every layer type we tried. No bf16 (no bf16 convolution in ONNX Runtime).

Benchmark. VoiceBank-DEMAND test set (Valentini et al.: 824 utterances by two unseen speakers), resampled to 16 kHz, enhanced with ClearerVoice's own loudness normalization and decode code. PESQ is wide-band (ITU-T P.862.2), STOI and SI-SDR against the clean speech. The PyTorch original (clearvoice 0.1.2, fp32) runs 11 times slower than real time on this laptop's CPU, so it was run on the first 50 files. There, the ONNX fp32 graph on CUDA matches it at 74.3 dB mean (56.2 dB worst file), and no file is more than 0.08 dB SI-SDR below it. The full set then compares against that fp32 graph. "vs fp32" is the SI-SDR of each output against the fp32 graph's, mean / worst file; "worst file" is the largest SI-SDR drop on any one file. Pass (the speech-enhancement threshold used across these repositories): no file more than 0.5 dB below. Real-time factor on an RTX 3080 Laptop GPU, whole pipeline.

	Files	PESQ	STOI	SI-SDR (dB)	vs fp32 (dB)	Worst file	Real-time factor
noisy input	824	1.97	0.921	8.39	-	-	-
original (PyTorch fp32, CPU)	first 50	3.184	0.9457	19.32	-	-	10.8
ONNX fp32 (CUDA, not published)	first 50	3.184	0.9457	19.32	-	-0.08 dB vs original	-
ONNX fp32 (CUDA, not published)	824	3.035	0.9530	18.67	-	-	0.087
fp16/ (CUDA)	824	3.033	0.9530	18.68	66.6 / 41.0	-0.09 dB	0.087
int8 (not published)	824	2.929	0.9490	14.10	20.8 / -7.0	-29.2 dB (583 files below -0.5)	0.090
int4 (not published)	824	2.023	0.9170	8.82	9.7 / -11.2	-30.1 dB	0.090

How these were made

From FRCRN at revision 3766e6a64b, starting from an fp32 ONNX graph (where it came from is said above). fp16: onnxconverter-common with inputs and outputs kept fp32, or, where the network's activations leave fp16's range, fp16 weights with fp32 math (said above when so). int8: weight-only, every Conv / MatMul / GRU weight int8 with a per-output-channel scale behind DequantizeLinear and activations in fp32, so it runs on the CPU provider without a calibration set. int4: ONNX Runtime's MatMulNBits (weight-only, block 32) on the MatMul weights, the rest int8 as above, where the graph has enough MatMul weights to matter. No bf16: ONNX Runtime has no bf16 convolution or GRU kernels. Every graph was benchmarked before upload; the sha256 column above is what the files must hash to.

License

Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.

---

Original model card

The FRCRN_SE_16K model weights for 16 kHz speech enhancement in ClearerVoice-Studio repo.

This model is trained on large scale datasets inclduing open-sourced and private data.

It enhances speech audios by removing background noise.