Resemble Enhance ONNX

ONNX builds of Resemble Enhance, made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each precision is one folder; what the graphs take and return, and the signal processing that goes around them, is under How to run.

Files

The last column is the benchmark on the VoiceBank-DEMAND test set (824 noisy utterances; details under How to run): each precision's speech-quality scores, and how far its output is from the original PyTorch model's.

Precision File Size sha256 Benchmark
fp16 fp16/cfm.onnx 138.2 MB 3f6583040d20 fp16 storage (the checkpoint's own weights), fp32 math: denoise PESQ 2.823, SI-SDR 12.42 dB, vs original 87.1 / 66.0 dB, worst file -0.01 dB; enhance PESQ 2.333, STOI 0.9128, vs original 53.8 / 24.4 dB, worst file PESQ -0.095 (CUDA)
fp16 fp16/config.json 0.1 kB 8ad7fb74a0ea fp16 storage (the checkpoint's own weights), fp32 math: denoise PESQ 2.823, SI-SDR 12.42 dB, vs original 87.1 / 66.0 dB, worst file -0.01 dB; enhance PESQ 2.333, STOI 0.9128, vs original 53.8 / 24.4 dB, worst file PESQ -0.095 (CUDA)
fp16 fp16/decoder.onnx 448.9 MB ad65ec440d6e fp16 storage (the checkpoint's own weights), fp32 math: denoise PESQ 2.823, SI-SDR 12.42 dB, vs original 87.1 / 66.0 dB, worst file -0.01 dB; enhance PESQ 2.333, STOI 0.9128, vs original 53.8 / 24.4 dB, worst file PESQ -0.095 (CUDA)
fp16 fp16/denoiser.onnx 21.5 MB a4f497e3b15d fp16 storage (the checkpoint's own weights), fp32 math: denoise PESQ 2.823, SI-SDR 12.42 dB, vs original 87.1 / 66.0 dB, worst file -0.01 dB; enhance PESQ 2.333, STOI 0.9128, vs original 53.8 / 24.4 dB, worst file PESQ -0.095 (CUDA)
fp16 fp16/encoder.onnx 101.8 MB 5e425e50c11f fp16 storage (the checkpoint's own weights), fp32 math: denoise PESQ 2.823, SI-SDR 12.42 dB, vs original 87.1 / 66.0 dB, worst file -0.01 dB; enhance PESQ 2.333, STOI 0.9128, vs original 53.8 / 24.4 dB, worst file PESQ -0.095 (CUDA)
fp16 fp16/pipeline.py 7.9 kB 311e89e6bd3a fp16 storage (the checkpoint's own weights), fp32 math: denoise PESQ 2.823, SI-SDR 12.42 dB, vs original 87.1 / 66.0 dB, worst file -0.01 dB; enhance PESQ 2.333, STOI 0.9128, vs original 53.8 / 24.4 dB, worst file PESQ -0.095 (CUDA)

How to run

Resemble Enhance (Resemble AI's speech enhancer: removes noise and restores the quality of speech recordings, output at 44.1 kHz) as four ONNX graphs plus the Python pipeline that runs them:

  • denoiser.onnx: the UNet denoiser on the 1680-point STFT. Inputs mag, cos, sin [1, 841, T] (magnitude and phase of the spectrum, T a multiple of 16), outputs the denoised sep_mag, sep_cos, sep_sin.
  • encoder.onnx: the IRMAE encoder. Input mel [1, 128, T] (normalized 128-band mel), output the latent [1, 64, T].
  • cfm.onnx: the flow-matching velocity network. Inputs psi [1, 64, T], cond [1, 128, T], t [1]; output velocity; called 32 times per clip.
  • decoder.onnx: the IRMAE decoder and the UnivNet vocoder. Inputs latent [1, 64, T] and noise [1, 128, T + 10], output wav, the 44.1 kHz waveform [1, 420 T].

T is free (frames of 420 samples at 44.1 kHz). The STFT, the mel spectrogram, the resampling and the ODE solver loop stay outside the graphs, in pipeline.py (a copy in the folder). It reproduces resemble-enhance 0.0.1's denoise() and enhance(nfe=32, solver="midpoint", lambd=0.5, tau=0.5). It needs numpy, scipy, soundfile, torch and torchaudio (CPU builds are enough: they do only the STFT, mel and resampling), and onnxruntime:

python fp16/pipeline.py noisy.wav enhanced.wav --cuda                  # denoise + enhance (44.1 kHz output)
python fp16/pipeline.py noisy.wav denoised.wav --cuda --mode denoise   # the denoiser alone

Or from Python:

import soundfile as sf, sys
sys.path.insert(0, "fp16")
import pipeline

parts = pipeline.OrtParts("fp16", "cuda")                  # or "cpu"
wav, sr = sf.read("noisy.wav", dtype="float32", always_2d=True)
out, out_sr = pipeline.run(parts, parts.cfg, wav.mean(1), sr, mode="enhance", seed=0)
sf.write("enhanced.wav", out, out_sr)

Differences from the original. The enhancer draws random noise twice (the flow's starting point and the vocoder's noise). The original takes both from torch's global generator, so its output changes from run to run; here both are inputs to the graphs, drawn from seed, so a run is repeatable. Files longer than 30 s are cut into 30 s chunks overlapping by 1 s as in the original, but joined with plain cross-fades instead of the original's merge, so long files can differ slightly at the joins. On CUDA, the pipeline lets ONNX Runtime shrink its memory arena after each call.

Precisions. fp16/: the original checkpoint is stored in fp16, so this folder keeps exactly those weights (fp16 storage, fp32 math). No int8: weight-only int8 passes in denoise mode but in enhance mode puts 13 of the 824 test files more than 0.1 PESQ below the original (worst -1.14), so it isn't published. No int4 (MatMul weights are under 10 % of every graph, so a 4-bit MatMul build is the int8 one). No bf16. True fp16 math works but was no faster and further from the original than fp16 storage.

Benchmark. VoiceBank-DEMAND test set (Valentini et al.: 824 utterances by two unseen speakers, 48 kHz), through the pipeline above and through the PyTorch original (resemble-enhance 0.0.1, CUDA) given the same noise draws. PESQ is wide-band (ITU-T P.862.2) and STOI, both against the clean speech at 16 kHz. Thresholds:

  • Denoise mode: no file's SI-SDR (against the clean speech) more than 0.5 dB below the original's (the speech-enhancement threshold used across these repositories).
  • Enhance mode: the enhancer resynthesizes the speech through a vocoder and extends its bandwidth, so its waveform doesn't line up with the clean recording: SI-SDR against the clean speech is about -19 dB for the original too. Instead, no file's PESQ may be more than 0.1 below the original's.

"vs original" is the SI-SDR of each output against the original's output, mean / worst file. Real-time factor on an RTX 3080 Laptop GPU, whole pipeline (ONNX Runtime 1.30, CUDA provider). The ONNX pipeline is slower than the PyTorch original on the GPU: the solver loop runs 32 separate graph calls from Python. CPU speed was measured only for the denoiser (int8 build, 4 threads: real-time factor 0.23 on 20 files); a full enhance on the CPU is much slower.

Mode Files PESQ STOI vs original (dB) Worst file Real-time factor
noisy input 824 1.97 0.921 - - -
denoise original (PyTorch, CUDA) 824 2.823 0.9472 - - 0.03
denoise fp16/ (CUDA) 824 2.823 0.9472 87.1 / 66.0 SI-SDR -0.01 dB 0.05
denoise int8 (not published) 824 2.822 0.9472 50.1 / 30.3 SI-SDR -0.41 dB (PESQ -1.26 on one file) 0.05
enhance original (PyTorch, CUDA) 824 2.333 0.9128 - - 0.35
enhance fp16/ (CUDA) 824 2.333 0.9128 53.8 / 24.4 PESQ -0.095 0.49
enhance int8 (not published) 824 2.328 0.9126 22.9 / 2.7 PESQ -1.14 (13 files below -0.1) 0.61

Denoise-mode SI-SDR against the clean speech: original 12.42 dB, fp16/ 12.42 dB. The fp16 graphs hold the checkpoint's own weights, so the small differences in enhance mode come from GPU arithmetic order, amplified through the 32 solver steps.

How these were made

From Resemble Enhance at revision 4e3510ce4a, starting from an fp32 ONNX graph (where it came from is said above). fp16: onnxconverter-common with inputs and outputs kept fp32, or, where the network's activations leave fp16's range, fp16 weights with fp32 math (said above when so). int8: weight-only, every Conv / MatMul / GRU weight int8 with a per-output-channel scale behind DequantizeLinear and activations in fp32, so it runs on the CPU provider without a calibration set. int4: ONNX Runtime's MatMulNBits (weight-only, block 32) on the MatMul weights, the rest int8 as above, where the graph has enough MatMul weights to matter. No bf16: ONNX Runtime has no bf16 convolution or GRU kernels. Every graph was benchmarked before upload; the sha256 column above is what the files must hash to.

License

MIT. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.


Original model card

Resemble Enhance is an AI-powered tool that aims to improve the overall quality of speech by performing denoising and enhancement. It consists of two modules: a denoiser, which separates speech from a noisy audio, and an enhancer, which further boosts the perceptual audio quality by restoring audio distortions and extending the audio bandwidth. The two models are trained on high-quality 44.1kHz speech data that guarantees the enhancement of your speech with high quality.

Below are three videos with separate examples of Resemble Enhance's denoiser module and then enhancer module improving speech quality.

Background Music - https://youtu.be/gl--IMtQ0XQ

Street Noise - https://youtu.be/zC87BjtsZVA

Archived Speech - https://youtu.be/6dALaLMJhSQ

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for moonbox-dev/resemble-enhance-ONNX

Quantized
(3)
this model