Resemble Enhance ONNX
ONNX builds of Resemble Enhance, made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each precision is one folder; what the graphs take and return, and the signal processing that goes around them, is under How to run.
Files
The last column is the benchmark on the VoiceBank-DEMAND test set (824 noisy utterances; details under How to run): each precision's speech-quality scores, and how far its output is from the original PyTorch model's.
| Precision | File | Size | sha256 | Benchmark |
|---|---|---|---|---|
| fp16 | fp16/cfm.onnx |
138.2 MB | 3f6583040d20 |
fp16 storage (the checkpoint's own weights), fp32 math: denoise PESQ 2.823, SI-SDR 12.42 dB, vs original 87.1 / 66.0 dB, worst file -0.01 dB; enhance PESQ 2.333, STOI 0.9128, vs original 53.8 / 24.4 dB, worst file PESQ -0.095 (CUDA) |
| fp16 | fp16/config.json |
0.1 kB | 8ad7fb74a0ea |
fp16 storage (the checkpoint's own weights), fp32 math: denoise PESQ 2.823, SI-SDR 12.42 dB, vs original 87.1 / 66.0 dB, worst file -0.01 dB; enhance PESQ 2.333, STOI 0.9128, vs original 53.8 / 24.4 dB, worst file PESQ -0.095 (CUDA) |
| fp16 | fp16/decoder.onnx |
448.9 MB | ad65ec440d6e |
fp16 storage (the checkpoint's own weights), fp32 math: denoise PESQ 2.823, SI-SDR 12.42 dB, vs original 87.1 / 66.0 dB, worst file -0.01 dB; enhance PESQ 2.333, STOI 0.9128, vs original 53.8 / 24.4 dB, worst file PESQ -0.095 (CUDA) |
| fp16 | fp16/denoiser.onnx |
21.5 MB | a4f497e3b15d |
fp16 storage (the checkpoint's own weights), fp32 math: denoise PESQ 2.823, SI-SDR 12.42 dB, vs original 87.1 / 66.0 dB, worst file -0.01 dB; enhance PESQ 2.333, STOI 0.9128, vs original 53.8 / 24.4 dB, worst file PESQ -0.095 (CUDA) |
| fp16 | fp16/encoder.onnx |
101.8 MB | 5e425e50c11f |
fp16 storage (the checkpoint's own weights), fp32 math: denoise PESQ 2.823, SI-SDR 12.42 dB, vs original 87.1 / 66.0 dB, worst file -0.01 dB; enhance PESQ 2.333, STOI 0.9128, vs original 53.8 / 24.4 dB, worst file PESQ -0.095 (CUDA) |
| fp16 | fp16/pipeline.py |
7.9 kB | 311e89e6bd3a |
fp16 storage (the checkpoint's own weights), fp32 math: denoise PESQ 2.823, SI-SDR 12.42 dB, vs original 87.1 / 66.0 dB, worst file -0.01 dB; enhance PESQ 2.333, STOI 0.9128, vs original 53.8 / 24.4 dB, worst file PESQ -0.095 (CUDA) |
How to run
Resemble Enhance (Resemble AI's speech enhancer: removes noise and restores the quality of speech recordings, output at 44.1 kHz) as four ONNX graphs plus the Python pipeline that runs them:
denoiser.onnx: the UNet denoiser on the 1680-point STFT. Inputsmag,cos,sin[1, 841, T] (magnitude and phase of the spectrum, T a multiple of 16), outputs the denoisedsep_mag,sep_cos,sep_sin.encoder.onnx: the IRMAE encoder. Inputmel[1, 128, T] (normalized 128-band mel), output the latent [1, 64, T].cfm.onnx: the flow-matching velocity network. Inputspsi[1, 64, T],cond[1, 128, T],t[1]; outputvelocity; called 32 times per clip.decoder.onnx: the IRMAE decoder and the UnivNet vocoder. Inputslatent[1, 64, T] andnoise[1, 128, T + 10], outputwav, the 44.1 kHz waveform [1, 420 T].
T is free (frames of 420 samples at 44.1 kHz). The STFT, the mel spectrogram, the resampling and the ODE solver loop
stay outside the graphs, in pipeline.py (a copy in the folder). It reproduces resemble-enhance 0.0.1's
denoise() and enhance(nfe=32, solver="midpoint", lambd=0.5, tau=0.5). It needs numpy, scipy, soundfile, torch
and torchaudio (CPU builds are enough: they do only the STFT, mel and resampling), and onnxruntime:
python fp16/pipeline.py noisy.wav enhanced.wav --cuda # denoise + enhance (44.1 kHz output)
python fp16/pipeline.py noisy.wav denoised.wav --cuda --mode denoise # the denoiser alone
Or from Python:
import soundfile as sf, sys
sys.path.insert(0, "fp16")
import pipeline
parts = pipeline.OrtParts("fp16", "cuda") # or "cpu"
wav, sr = sf.read("noisy.wav", dtype="float32", always_2d=True)
out, out_sr = pipeline.run(parts, parts.cfg, wav.mean(1), sr, mode="enhance", seed=0)
sf.write("enhanced.wav", out, out_sr)
Differences from the original. The enhancer draws random noise twice (the flow's starting point and the vocoder's
noise). The original takes both from torch's global generator, so its output changes from run to run; here both are
inputs to the graphs, drawn from seed, so a run is repeatable. Files longer than 30 s are cut into 30 s chunks
overlapping by 1 s as in the original, but joined with plain cross-fades instead of the original's merge, so long files
can differ slightly at the joins. On CUDA, the pipeline lets ONNX Runtime shrink its memory arena after each call.
Precisions. fp16/: the original checkpoint is stored in fp16, so this folder keeps exactly those weights (fp16
storage, fp32 math). No int8: weight-only int8 passes in denoise mode but in enhance mode puts 13 of the 824 test
files more than 0.1 PESQ below the original (worst -1.14), so it isn't published. No int4 (MatMul weights are
under 10 % of every graph, so a 4-bit MatMul build is the int8 one). No bf16. True fp16 math works but was no faster
and further from the original than fp16 storage.
Benchmark. VoiceBank-DEMAND test set (Valentini et al.: 824 utterances by two unseen speakers, 48 kHz), through the pipeline above and through the PyTorch original (resemble-enhance 0.0.1, CUDA) given the same noise draws. PESQ is wide-band (ITU-T P.862.2) and STOI, both against the clean speech at 16 kHz. Thresholds:
- Denoise mode: no file's SI-SDR (against the clean speech) more than 0.5 dB below the original's (the speech-enhancement threshold used across these repositories).
- Enhance mode: the enhancer resynthesizes the speech through a vocoder and extends its bandwidth, so its waveform doesn't line up with the clean recording: SI-SDR against the clean speech is about -19 dB for the original too. Instead, no file's PESQ may be more than 0.1 below the original's.
"vs original" is the SI-SDR of each output against the original's output, mean / worst file. Real-time factor on an RTX 3080 Laptop GPU, whole pipeline (ONNX Runtime 1.30, CUDA provider). The ONNX pipeline is slower than the PyTorch original on the GPU: the solver loop runs 32 separate graph calls from Python. CPU speed was measured only for the denoiser (int8 build, 4 threads: real-time factor 0.23 on 20 files); a full enhance on the CPU is much slower.
| Mode | Files | PESQ | STOI | vs original (dB) | Worst file | Real-time factor | |
|---|---|---|---|---|---|---|---|
| noisy input | 824 | 1.97 | 0.921 | - | - | - | |
| denoise | original (PyTorch, CUDA) | 824 | 2.823 | 0.9472 | - | - | 0.03 |
| denoise | fp16/ (CUDA) |
824 | 2.823 | 0.9472 | 87.1 / 66.0 | SI-SDR -0.01 dB | 0.05 |
| denoise | int8 (not published) | 824 | 2.822 | 0.9472 | 50.1 / 30.3 | SI-SDR -0.41 dB (PESQ -1.26 on one file) | 0.05 |
| enhance | original (PyTorch, CUDA) | 824 | 2.333 | 0.9128 | - | - | 0.35 |
| enhance | fp16/ (CUDA) |
824 | 2.333 | 0.9128 | 53.8 / 24.4 | PESQ -0.095 | 0.49 |
| enhance | int8 (not published) | 824 | 2.328 | 0.9126 | 22.9 / 2.7 | PESQ -1.14 (13 files below -0.1) | 0.61 |
Denoise-mode SI-SDR against the clean speech: original 12.42 dB, fp16/ 12.42 dB. The fp16 graphs hold the
checkpoint's own weights, so the small differences in enhance mode come from GPU arithmetic order, amplified through
the 32 solver steps.
How these were made
From Resemble Enhance at revision 4e3510ce4a, starting from an fp32 ONNX graph (where it came from is said above).
fp16: onnxconverter-common with inputs and outputs kept fp32, or, where the network's activations leave fp16's
range, fp16 weights with fp32 math (said above when so). int8: weight-only, every Conv / MatMul / GRU weight int8
with a per-output-channel scale behind DequantizeLinear and activations in fp32, so it runs on the CPU provider
without a calibration set. int4: ONNX Runtime's MatMulNBits (weight-only, block 32) on the MatMul weights, the rest
int8 as above, where the graph has enough MatMul weights to matter. No bf16: ONNX Runtime has no bf16 convolution or
GRU kernels. Every graph was benchmarked before upload; the sha256 column above is what the files must hash to.
License
MIT. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.
Original model card
Resemble Enhance is an AI-powered tool that aims to improve the overall quality of speech by performing denoising and enhancement. It consists of two modules: a denoiser, which separates speech from a noisy audio, and an enhancer, which further boosts the perceptual audio quality by restoring audio distortions and extending the audio bandwidth. The two models are trained on high-quality 44.1kHz speech data that guarantees the enhancement of your speech with high quality.
Below are three videos with separate examples of Resemble Enhance's denoiser module and then enhancer module improving speech quality.
Background Music - https://youtu.be/gl--IMtQ0XQ
Street Noise - https://youtu.be/zC87BjtsZVA
Archived Speech - https://youtu.be/6dALaLMJhSQ
Model tree for moonbox-dev/resemble-enhance-ONNX
Base model
ResembleAI/resemble-enhance