--- license: mit pipeline_tag: audio-to-audio library_name: onnx tags: - onnx - moonbox - onnxruntime - deepfilternet - speech-enhancement - denoising - noise-suppression - 48khz - real-time --- # DeepFilterNet3 ONNX ONNX builds of [DeepFilterNet3](https://github.com/Rikorose/DeepFilterNet), made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each precision is one folder; what the graphs take and return, and the signal processing that goes around them, is under *How to run*. ## Files The last column is the benchmark on the VoiceBank-DEMAND test set (824 noisy utterances; details under *How to run*): each precision's speech-quality scores, and how far its output is from the original PyTorch model's. | Precision | File | Size | sha256 | Benchmark | |---|---|---|---|---| | fp16 | [`fp16/config.ini`](https://huggingface.co/moonbox-dev/DeepFilterNet3-ONNX/blob/main/fp16/config.ini) | 2.1 kB | `415eb925d449` | PESQ 3.163, STOI 0.9452, SI-SDR 18.91 dB; vs original 65.2 / 55.2 dB (CUDA) | | fp16 | [`fp16/df_dec.onnx`](https://huggingface.co/moonbox-dev/DeepFilterNet3-ONNX/blob/main/fp16/df_dec.onnx) | 1.7 MB | `a3380e4164a9` | PESQ 3.163, STOI 0.9452, SI-SDR 18.91 dB; vs original 65.2 / 55.2 dB (CUDA) | | fp16 | [`fp16/enc.onnx`](https://huggingface.co/moonbox-dev/DeepFilterNet3-ONNX/blob/main/fp16/enc.onnx) | 1.0 MB | `694385ef6060` | PESQ 3.163, STOI 0.9452, SI-SDR 18.91 dB; vs original 65.2 / 55.2 dB (CUDA) | | fp16 | [`fp16/erb_dec.onnx`](https://huggingface.co/moonbox-dev/DeepFilterNet3-ONNX/blob/main/fp16/erb_dec.onnx) | 1.7 MB | `72126330232d` | PESQ 3.163, STOI 0.9452, SI-SDR 18.91 dB; vs original 65.2 / 55.2 dB (CUDA) | | int8 | [`int8/config.ini`](https://huggingface.co/moonbox-dev/DeepFilterNet3-ONNX/blob/main/int8/config.ini) | 2.1 kB | `415eb925d449` | PESQ 3.159, STOI 0.9448, SI-SDR 18.89 dB; vs original 44.4 / 32.2 dB (CPU) | | int8 | [`int8/df_dec.onnx`](https://huggingface.co/moonbox-dev/DeepFilterNet3-ONNX/blob/main/int8/df_dec.onnx) | 0.9 MB | `7b729627da56` | PESQ 3.159, STOI 0.9448, SI-SDR 18.89 dB; vs original 44.4 / 32.2 dB (CPU) | | int8 | [`int8/enc.onnx`](https://huggingface.co/moonbox-dev/DeepFilterNet3-ONNX/blob/main/int8/enc.onnx) | 0.5 MB | `41ac70016f7f` | PESQ 3.159, STOI 0.9448, SI-SDR 18.89 dB; vs original 44.4 / 32.2 dB (CPU) | | int8 | [`int8/erb_dec.onnx`](https://huggingface.co/moonbox-dev/DeepFilterNet3-ONNX/blob/main/int8/erb_dec.onnx) | 0.9 MB | `394ba8bd8f85` | PESQ 3.159, STOI 0.9448, SI-SDR 18.89 dB; vs original 44.4 / 32.2 dB (CPU) | ## How to run DeepFilterNet3 (real-time speech enhancement at 48 kHz: ERB-band gains plus deep filtering of the low band) as the three ONNX graphs of the official export (`models/DeepFilterNet3_onnx.tar.gz` in the GitHub repository): `enc.onnx` (encoder), `erb_dec.onnx` (ERB-gain decoder) and `df_dec.onnx` (deep-filter decoder), with the model's `config.ini`. The time axis is free, so a whole file runs in one call. Our `fp16/` and `int8/` are made from those graphs; the fp32 originals are in the release itself. The graphs are the neural network only: the STFT, the ERB and complex features, applying the gains and the deep filter, and the ISTFT go around them. The simplest way to get all of that exactly right is the official `deepfilternet` package with its three networks swapped for these graphs (how this repository was benchmarked): ```python import numpy as np, onnxruntime as ort, soundfile as sf, torch from df.enhance import enhance, init_df # pip install deepfilternet==0.5.6 (torch / torchaudio 2.1) model, df_state, _ = init_df() # the official DeepFilterNet3 checkpoint prov = ["CPUExecutionProvider"] # int8: CPU; fp16: ["CUDAExecutionProvider"] class Graph(torch.nn.Module): def __init__(self, path, first_only): super().__init__() self.s, self.first_only = ort.InferenceSession(path, providers=prov), first_only self.names = [i.name for i in self.s.get_inputs()] def forward(self, *xs): out = [torch.from_numpy(o) for o in self.s.run(None, {n: x.numpy() for n, x in zip(self.names, xs)})] return out[0] if self.first_only else tuple(out) model.enc, model.erb_dec, model.df_dec = (Graph("int8/enc.onnx", False), Graph("int8/erb_dec.onnx", True), Graph("int8/df_dec.onnx", True)) noisy, sr = sf.read("noisy_48k.wav", dtype="float32") # 48 kHz mono with torch.no_grad(): clean = enhance(model, df_state, torch.from_numpy(noisy)[None]) sf.write("enhanced.wav", clean.squeeze(0).numpy(), 48000) ``` Without Python, the DeepFilterNet project's Rust crate (`libDF`, with its `tract` runtime) implements the same feature pipeline around these three graphs. **Precisions.** `int8/`: weights int8 per output channel behind DequantizeLinear (every Conv, grouped linear and GRU weight), activations fp32; 2.2 MB, runs on the CPU provider, the one to use for a real-time CPU denoiser. `fp16/`: weights and math in fp16, inputs and outputs fp32; for GPU providers (ONNX Runtime's CPU provider has no fp16 GRU). No int4 and no bf16 (no 4-bit or bf16 GRU kernels in ONNX Runtime). **Benchmark.** VoiceBank-DEMAND test set (Valentini et al.: 824 utterances by two unseen speakers, 48 kHz, the set the DeepFilterNet papers report), enhanced in one call per file with the post-filter off (the package default). PESQ is wide-band (ITU-T P.862.2, at 16 kHz), STOI at 48 kHz, SI-SDR against the clean speech; "vs original" is the SI-SDR of each output against the PyTorch model's output (mean / worst file). Real-time factor with 4 CPU threads on an Intel laptop CPU (int8) or the RTX 3080 Laptop GPU (fp16), whole pipeline. | | PESQ | STOI | SI-SDR (dB) | vs original (dB) | Real-time factor | |---|---|---|---|---|---| | noisy input | 1.97 | 0.921 | 8.39 | - | - | | original (PyTorch fp32, CPU) | 3.163 | 0.9452 | 18.91 | - | 0.043 | | official ONNX fp32 (CPU) | 3.163 | 0.9452 | 18.91 | 135 / 130 | 0.017 | | `fp16/` (CUDA) | 3.163 | 0.9452 | 18.91 | 65.2 / 55.2 | 0.047 | | `int8/` (CPU) | 3.159 | 0.9448 | 18.89 | 44.4 / 32.2 | 0.018 | ## How these were made From [DeepFilterNet3](https://github.com/Rikorose/DeepFilterNet) at revision `7b14b895fe`, starting from an fp32 ONNX graph (where it came from is said above). fp16: onnxconverter-common with inputs and outputs kept fp32, or, where the network's activations leave fp16's range, fp16 weights with fp32 math (said above when so). int8: weight-only, every Conv / MatMul / GRU weight int8 with a per-output-channel scale behind DequantizeLinear and activations in fp32, so it runs on the CPU provider without a calibration set. int4: ONNX Runtime's MatMulNBits (weight-only, block 32) on the MatMul weights, the rest int8 as above, where the graph has enough MatMul weights to matter. No bf16: ONNX Runtime has no bf16 convolution or GRU kernels. Every graph was benchmarked before upload; the `sha256` column above is what the files must hash to. ## License MIT/Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged. --- # Original model card # DeepFilterNet A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) using on Deep Filtering. ![deepfilternet3](https://user-images.githubusercontent.com/16517898/225623209-a54fea75-ca00-404c-a394-c91d2d1146d2.svg) For PipeWire integration as a virtual noise suppression microphone look [here](https://github.com/Rikorose/DeepFilterNet/blob/main/ladspa/README.md). ### Demo https://github.com/Rikorose/DeepFilterNet/assets/16517898/79679fd7-de73-4c22-948c-891927c7d2ca To run the demo (linux only) use: ```bash cargo +nightly run -p df-demo --features ui --bin df-demo --release ``` ### News - New DeepFilterNet Demo: *DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement* - Paper: https://arxiv.org/abs/2305.08227 - Video: https://youtu.be/EO7n96YwnyE - New Multi-Frame Filtering Paper: *Deep Multi-Frame Filtering for Hearing Aids* - Paper: https://arxiv.org/abs/2305.08225 - Real-time version and a LADSPA plugin - [Pre-compiled binary](#deep-filter), no python dependencies. Usage: `deep-filter audio-file.wav` - [LADSPA plugin](https://raw.githubusercontent.com/Rikorose/DeepFilterNet/7b14b895fe7fc42cab72b4a4c1e49c982bd6404b/ladspa/) with pipewire filter-chain integration for real-time noise reduction on your mic. - DeepFilterNet2 Paper: *DeepFilterNet2: Towards Real-Time Speech Enhancement on Embedded Devices for Full-Band Audio* - Paper: https://arxiv.org/abs/2205.05474 - Samples: https://rikorose.github.io/DeepFilterNet2-Samples/ - Demo: https://huggingface.co/spaces/hshr/DeepFilterNet2 - Original DeepFilterNet Paper: *DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep Filtering* - Paper: https://arxiv.org/abs/2110.05588 - Samples: https://rikorose.github.io/DeepFilterNet-Samples/ - Demo: https://huggingface.co/spaces/hshr/DeepFilterNet - Video Lecture: https://youtu.be/it90gBqkY6k ## Usage ### deep-filter Download a pre-compiled deep-filter binary from the [release page](https://github.com/Rikorose/DeepFilterNet/releases/). You can use `deep-filter` to suppress noise in noisy .wav audio files. Currently, only wav files with a sampling rate of 48kHz are supported. ```bash USAGE: deep-filter [OPTIONS] [FILES]... ARGS: ... OPTIONS: -D, --compensate-delay Compensate delay of STFT and model lookahead -h, --help Print help information -m, --model Path to model tar.gz. Defaults to DeepFilterNet2. -o, --out-dir [default: out] --pf Enable postfilter -v, --verbose Logging verbosity -V, --version Print version information ``` If you want to use the pytorch backend e.g. for GPU processing, see further below for the Python usage. ### DeepFilterNet Framework This framework supports Linux, MacOS and Windows. Training is only tested under Linux. The framework is structured as follows: * `libDF` contains Rust code used for data loading and augmentation. * `DeepFilterNet` contains DeepFilterNet code training, evaluation and visualization as well as pretrained model weights. * `pyDF` contains a Python wrapper of libDF STFT/ISTFT processing loop. * `pyDF-data` contains a Python wrapper of libDF dataset functionality and provides a pytorch data loader. * `ladspa` contains a LADSPA plugin for real-time noise suppression. * `models` contains pretrained for usage in DeepFilterNet (Python) or libDF/deep-filter (Rust) ### DeepFilterNet Python: PyPI Install the DeepFilterNet Python wheel via pip: ```bash # Install cpu/cuda pytorch (>=1.9) dependency from pytorch.org, e.g.: pip install torch torchaudio -f https://download.pytorch.org/whl/cpu/torch_stable.html # Install DeepFilterNet pip install deepfilternet # Or install DeepFilterNet including data loading functionality for training (Linux only) pip install deepfilternet[train] ``` To enhance noisy audio files using DeepFilterNet run ```bash # Specify an output directory with --output-dir [OUTPUT_DIR] deepFilter path/to/noisy_audio.wav ``` ### Manual Installation Install cargo via [rustup](https://rustup.rs/). Usage of a `conda` or `virtualenv` recommended. Please read the comments and only execute the commands that you need. Installation of python dependencies and libDF: ```bash cd path/to/DeepFilterNet/ # cd into repository # Recommended: Install or activate a python env # Mandatory: Install cpu/cuda pytorch (>=1.8) dependency from pytorch.org, e.g.: pip install torch torchaudio -f https://download.pytorch.org/whl/cpu/torch_stable.html # Install build dependencies used to compile libdf and DeepFilterNet python wheels pip install maturin poetry # Install remaining DeepFilterNet python dependencies # *Option A:* Install DeepFilterNet python wheel globally within your environment. Do this if you want use # this repos as is, and don't want to develop within this repository. poetry -C DeepFilterNet install -E train -E eval # *Option B:* If you want to develop within this repo, install only dependencies and work with the repository version poetry -C DeepFilterNet install -E train -E eval --no-root export PYTHONPATH=$PWD/DeepFilterNet # And set the python path correctly # Build and install libdf python package required for enhance.py maturin develop --release -m pyDF/Cargo.toml # *Optional*: Install libdfdata python package with dataset and dataloading functionality for training # Required build dependency: HDF5 headers (e.g. ubuntu: libhdf5-dev) maturin develop --release -m pyDF-data/Cargo.toml # If you have troubles with hdf5 you may try to build and link hdf5 statically: maturin develop --release --features hdf5-static -m pyDF-data/Cargo.toml ``` ### Use DeepFilterNet from command line To enhance noisy audio files using DeepFilterNet run ```bash $ python DeepFilterNet/df/enhance.py --help usage: enhance.py [-h] [--model-base-dir MODEL_BASE_DIR] [--pf] [--output-dir OUTPUT_DIR] [--log-level LOG_LEVEL] [--compensate-delay] noisy_audio_files [noisy_audio_files ...] positional arguments: noisy_audio_files List of noise files to mix with the clean speech file. optional arguments: -h, --help show this help message and exit --model-base-dir MODEL_BASE_DIR, -m MODEL_BASE_DIR Model directory containing checkpoints and config. To load a pretrained model, you may just provide the model name, e.g. `DeepFilterNet`. By default, the pretrained DeepFilterNet2 model is loaded. --pf Post-filter that slightly over-attenuates very noisy sections. --output-dir OUTPUT_DIR, -o OUTPUT_DIR Directory in which the enhanced audio files will be stored. --log-level LOG_LEVEL Logger verbosity. Can be one of (debug, info, error, none) --compensate-delay, -D Add some paddig to compensate the delay introduced by the real-time STFT/ISTFT implementation. # Enhance audio with original DeepFilterNet python DeepFilterNet/df/enhance.py -m DeepFilterNet path/to/noisy_audio.wav # Enhance audio with DeepFilterNet2 python DeepFilterNet/df/enhance.py -m DeepFilterNet2 path/to/noisy_audio.wav ``` ### Use DeepFilterNet within your Python script ```py from df import enhance, init_df model, df_state, _ = init_df() # Load default model enhanced_audio = enhance(model, df_state, noisy_audio) ``` See [here](https://github.com/Rikorose/DeepFilterNet/blob/main/scripts/external_usage.py) for a full example. ### Training The entry point is `DeepFilterNet/df/train.py`. It expects a data directory containing HDF5 dataset as well as a dataset configuration json file. So, you first need to create your datasets in HDF5 format. Each dataset typically only holds training, validation, or test set of noise, speech or RIRs. ```py # Install additional dependencies for dataset creation pip install h5py librosa soundfile # Go to DeepFilterNet python package cd path/to/DeepFilterNet/DeepFilterNet # Prepare text file (e.g. called training_set.txt) containing paths to .wav files # # usage: prepare_data.py [-h] [--num_workers NUM_WORKERS] [--max_freq MAX_FREQ] [--sr SR] [--dtype DTYPE] # [--codec CODEC] [--mono] [--compression COMPRESSION] # type audio_files hdf5_db # # where: # type: One of `speech`, `noise`, `rir` # audio_files: Text file containing paths to audio files to include in the dataset # hdf5_db: Output HDF5 dataset. python df/scripts/prepare_data.py --sr 48000 speech training_set.txt TRAIN_SET_SPEECH.hdf5 ``` All datasets should be made available in one dataset folder for the train script. The dataset configuration file should contain 3 entries: "train", "valid", "test". Each of those contains a list of datasets (e.g. a speech, noise and a RIR dataset). You can use multiple speech or noise dataset. Optionally, a sampling factor may be specified that can be used to over/under-sample the dataset. Say, you have a specific dataset with transient noises and want to increase the amount of non-stationary noises by oversampling. In most cases you want to set this factor to 1.
Dataset config example:

`dataset.cfg` ```json { "train": [ [ "TRAIN_SET_SPEECH.hdf5", 1.0 ], [ "TRAIN_SET_NOISE.hdf5", 1.0 ], [ "TRAIN_SET_RIR.hdf5", 1.0 ] ], "valid": [ [ "VALID_SET_SPEECH.hdf5", 1.0 ], [ "VALID_SET_NOISE.hdf5", 1.0 ], [ "VALID_SET_RIR.hdf5", 1.0 ] ], "test": [ [ "TEST_SET_SPEECH.hdf5", 1.0 ], [ "TEST_SET_NOISE.hdf5", 1.0 ], [ "TEST_SET_RIR.hdf5", 1.0 ] ] } ```

Finally, start the training script. The training script may create a model `base_dir` if not existing used for logging, some audio samples, model checkpoints, and config. If no config file is found, it will create a default config. See [DeepFilterNet/pretrained_models/DeepFilterNet](https://github.com/Rikorose/DeepFilterNet/blob/main/DeepFilterNet/pretrained_models/DeepFilterNet/config.ini) for a config file. ```py # usage: train.py [-h] [--debug] data_config_file data_dir base_dir python df/train.py path/to/dataset.cfg path/to/data_dir/ path/to/base_dir/ ``` ## Citation Guide To reproduce any metrics, we recomend to use the python implementation via `pip install deepfilternet`. If you use this framework, please cite: *DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep Filtering* ```bibtex @inproceedings{schroeter2022deepfilternet, title={{DeepFilterNet}: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep Filtering}, author = {Schröter, Hendrik and Escalante-B., Alberto N. and Rosenkranz, Tobias and Maier, Andreas}, booktitle={ICASSP 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, year={2022}, organization={IEEE} } ``` If you use the DeepFilterNet2 model, please cite: *DeepFilterNet2: Towards Real-Time Speech Enhancement on Embedded Devices for Full-Band Audio* ```bibtex @inproceedings{schroeter2022deepfilternet2, title = {{DeepFilterNet2}: Towards Real-Time Speech Enhancement on Embedded Devices for Full-Band Audio}, author = {Schröter, Hendrik and Escalante-B., Alberto N. and Rosenkranz, Tobias and Maier, Andreas}, booktitle={17th International Workshop on Acoustic Signal Enhancement (IWAENC 2022)}, year = {2022}, } ``` If you use the DeepFilterNet3 model, please cite: *DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement* ```bibtex @inproceedings{schroeter2023deepfilternet3, title = {{DeepFilterNet}: Perceptually Motivated Real-Time Speech Enhancement}, author = {Schröter, Hendrik and Rosenkranz, Tobias and Escalante-B., Alberto N. and Maier, Andreas}, booktitle={INTERSPEECH}, year = {2023}, } ``` If you use the multi-frame beamforming algorithms. please cite *Deep Multi-Frame Filtering for Hearing Aids* ```bibtex @inproceedings{schroeter2023deep_mf, title = {Deep Multi-Frame Filtering for Hearing Aids}, author = {Schröter, Hendrik and Rosenkranz, Tobias and Escalante-B., Alberto N. and Maier, Andreas}, booktitle={INTERSPEECH}, year = {2023}, } ``` ## License DeepFilterNet is free and open source! All code and model weights in this repository are dual-licensed under either: * MIT License ([LICENSE-MIT](https://raw.githubusercontent.com/Rikorose/DeepFilterNet/7b14b895fe7fc42cab72b4a4c1e49c982bd6404b/LICENSE-MIT) or [http://opensource.org/licenses/MIT](http://opensource.org/licenses/MIT)) * Apache License, Version 2.0 ([LICENSE-APACHE](https://raw.githubusercontent.com/Rikorose/DeepFilterNet/7b14b895fe7fc42cab72b4a4c1e49c982bd6404b/LICENSE-APACHE) or [http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)) at your option. This means you can select the license you prefer! Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.