jennyxue commited on
Commit
e1513f7
·
verified ·
1 Parent(s): c1a9a4c

Model card and the original license

Browse files
Files changed (4) hide show
  1. LICENSE +12 -0
  2. LICENSE-APACHE +201 -0
  3. LICENSE-MIT +19 -0
  4. README.md +443 -0
LICENSE ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## License
2
+
3
+ Licensed under either of
4
+ * Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
5
+ * MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
6
+ at your option.
7
+
8
+ ### Contribution
9
+
10
+ Unless you explicitly state otherwise, any contribution intentionally submitted
11
+ for inclusion in the work by you, as defined in the Apache-2.0 license, shall
12
+ be dual licensed as above, without any additional terms or conditions.
LICENSE-APACHE ADDED
@@ -0,0 +1,201 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "[]"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright 2021 Hendrik Schröter
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
LICENSE-MIT ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ The MIT License (MIT)
2
+ Copyright (c) 2021 Hendrik Schröter
3
+
4
+ Permission is hereby granted, free of charge, to any person obtaining a copy of
5
+ this software and associated documentation files (the "Software"), to deal in
6
+ the Software without restriction, including without limitation the rights to
7
+ use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
8
+ the Software, and to permit persons to whom the Software is furnished to do so,
9
+ subject to the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be included in all
12
+ copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
16
+ FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
17
+ COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
18
+ IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
19
+ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
README.md ADDED
@@ -0,0 +1,443 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ pipeline_tag: audio-to-audio
4
+ library_name: onnx
5
+ tags:
6
+ - onnx
7
+ - moonbox
8
+ - onnxruntime
9
+ - deepfilternet
10
+ - speech-enhancement
11
+ - denoising
12
+ - noise-suppression
13
+ - 48khz
14
+ - real-time
15
+ ---
16
+
17
+ # DeepFilterNet3 ONNX
18
+
19
+ ONNX builds of [DeepFilterNet3](https://github.com/Rikorose/DeepFilterNet), made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each
20
+ precision is one folder; what the graphs take and return, and the signal processing that goes around them, is
21
+ under *How to run*.
22
+
23
+ ## Files
24
+
25
+ The last column is the benchmark on the VoiceBank-DEMAND test set (824 noisy utterances; details under *How to
26
+ run*): each precision's speech-quality scores, and how far its output is from the original PyTorch model's.
27
+
28
+ | Precision | File | Size | sha256 | Benchmark |
29
+ |---|---|---|---|---|
30
+ | fp16 | [`fp16/config.ini`](https://huggingface.co/moonbox-dev/DeepFilterNet3-ONNX/blob/main/fp16/config.ini) | 2.1 kB | `415eb925d449` | PESQ 3.163, STOI 0.9452, SI-SDR 18.91 dB; vs original 65.2 / 55.2 dB (CUDA) |
31
+ | fp16 | [`fp16/df_dec.onnx`](https://huggingface.co/moonbox-dev/DeepFilterNet3-ONNX/blob/main/fp16/df_dec.onnx) | 1.7 MB | `a3380e4164a9` | PESQ 3.163, STOI 0.9452, SI-SDR 18.91 dB; vs original 65.2 / 55.2 dB (CUDA) |
32
+ | fp16 | [`fp16/enc.onnx`](https://huggingface.co/moonbox-dev/DeepFilterNet3-ONNX/blob/main/fp16/enc.onnx) | 1.0 MB | `694385ef6060` | PESQ 3.163, STOI 0.9452, SI-SDR 18.91 dB; vs original 65.2 / 55.2 dB (CUDA) |
33
+ | fp16 | [`fp16/erb_dec.onnx`](https://huggingface.co/moonbox-dev/DeepFilterNet3-ONNX/blob/main/fp16/erb_dec.onnx) | 1.7 MB | `72126330232d` | PESQ 3.163, STOI 0.9452, SI-SDR 18.91 dB; vs original 65.2 / 55.2 dB (CUDA) |
34
+
35
+ ## How to run
36
+
37
+ DeepFilterNet3 (real-time speech enhancement at 48 kHz: ERB-band gains plus deep filtering of the low band) as the
38
+ three ONNX graphs of the official export (`models/DeepFilterNet3_onnx.tar.gz` in the GitHub repository): `enc.onnx`
39
+ (encoder), `erb_dec.onnx` (ERB-gain decoder) and `df_dec.onnx` (deep-filter decoder), with the model's
40
+ `config.ini`. The time axis is free, so a whole file runs in one call. Our `fp16/` and `int8/` are made from those
41
+ graphs; the fp32 originals are in the release itself.
42
+
43
+ The graphs are the neural network only: the STFT, the ERB and complex features, applying the gains and the deep
44
+ filter, and the ISTFT go around them. The simplest way to get all of that exactly right is the official
45
+ `deepfilternet` package with its three networks swapped for these graphs (how this repository was benchmarked):
46
+
47
+ ```python
48
+ import numpy as np, onnxruntime as ort, soundfile as sf, torch
49
+ from df.enhance import enhance, init_df # pip install deepfilternet==0.5.6 (torch / torchaudio 2.1)
50
+
51
+ model, df_state, _ = init_df() # the official DeepFilterNet3 checkpoint
52
+ prov = ["CPUExecutionProvider"] # int8: CPU; fp16: ["CUDAExecutionProvider"]
53
+
54
+ class Graph(torch.nn.Module):
55
+ def __init__(self, path, first_only):
56
+ super().__init__()
57
+ self.s, self.first_only = ort.InferenceSession(path, providers=prov), first_only
58
+ self.names = [i.name for i in self.s.get_inputs()]
59
+ def forward(self, *xs):
60
+ out = [torch.from_numpy(o) for o in self.s.run(None, {n: x.numpy() for n, x in zip(self.names, xs)})]
61
+ return out[0] if self.first_only else tuple(out)
62
+
63
+ model.enc, model.erb_dec, model.df_dec = (Graph("int8/enc.onnx", False), Graph("int8/erb_dec.onnx", True),
64
+ Graph("int8/df_dec.onnx", True))
65
+ noisy, sr = sf.read("noisy_48k.wav", dtype="float32") # 48 kHz mono
66
+ with torch.no_grad():
67
+ clean = enhance(model, df_state, torch.from_numpy(noisy)[None])
68
+ sf.write("enhanced.wav", clean.squeeze(0).numpy(), 48000)
69
+ ```
70
+
71
+ Without Python, the DeepFilterNet project's Rust crate (`libDF`, with its `tract` runtime) implements the same
72
+ feature pipeline around these three graphs.
73
+
74
+ **Precisions.** `int8/`: weights int8 per output channel behind DequantizeLinear (every Conv, grouped linear and GRU
75
+ weight), activations fp32; 2.2 MB, runs on the CPU provider, the one to use for a real-time CPU denoiser. `fp16/`:
76
+ weights and math in fp16, inputs and outputs fp32; for GPU providers (ONNX Runtime's CPU provider has no fp16 GRU).
77
+ No int4 and no bf16 (no 4-bit or bf16 GRU kernels in ONNX Runtime).
78
+
79
+ **Benchmark.** VoiceBank-DEMAND test set (Valentini et al.: 824 utterances by two unseen speakers, 48 kHz, the set
80
+ the DeepFilterNet papers report), enhanced in one call per file with the post-filter off (the package default).
81
+ PESQ is wide-band (ITU-T P.862.2, at 16 kHz), STOI at 48 kHz, SI-SDR against the clean speech; "vs original" is
82
+ the SI-SDR of each output against the PyTorch model's output (mean / worst file). Real-time factor with 4 CPU threads
83
+ on an Intel laptop CPU (int8) or the RTX 3080 Laptop GPU (fp16), whole pipeline.
84
+
85
+ | | PESQ | STOI | SI-SDR (dB) | vs original (dB) | Real-time factor |
86
+ |---|---|---|---|---|---|
87
+ | noisy input | 1.97 | 0.921 | 8.39 | - | - |
88
+ | original (PyTorch fp32, CPU) | 3.163 | 0.9452 | 18.91 | - | 0.043 |
89
+ | official ONNX fp32 (CPU) | 3.163 | 0.9452 | 18.91 | 135 / 130 | 0.017 |
90
+ | `fp16/` (CUDA) | 3.163 | 0.9452 | 18.91 | 65.2 / 55.2 | 0.047 |
91
+ | `int8/` (CPU) | 3.159 | 0.9448 | 18.89 | 44.4 / 32.2 | 0.018 |
92
+
93
+ ## How these were made
94
+
95
+ From [DeepFilterNet3](https://github.com/Rikorose/DeepFilterNet) at revision `7b14b895fe`, starting from an fp32 ONNX graph (where it came from is said above).
96
+ fp16: onnxconverter-common with inputs and outputs kept fp32, or, where the network's activations leave fp16's
97
+ range, fp16 weights with fp32 math (said above when so). int8: weight-only, every Conv / MatMul / GRU weight int8
98
+ with a per-output-channel scale behind DequantizeLinear and activations in fp32, so it runs on the CPU provider
99
+ without a calibration set. int4: ONNX Runtime's MatMulNBits (weight-only, block 32) on the MatMul weights, the rest
100
+ int8 as above, where the graph has enough MatMul weights to matter. No bf16: ONNX Runtime has no bf16 convolution or
101
+ GRU kernels. Every graph was benchmarked before upload; the `sha256` column above is what the files must hash to.
102
+
103
+ ## License
104
+
105
+ MIT/Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this
106
+ repository, and it governs your use of these files. The original model card follows unchanged.
107
+
108
+ ---
109
+
110
+ # Original model card
111
+
112
+ # DeepFilterNet
113
+ A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) using on Deep Filtering.
114
+
115
+ ![deepfilternet3](https://user-images.githubusercontent.com/16517898/225623209-a54fea75-ca00-404c-a394-c91d2d1146d2.svg)
116
+
117
+ For PipeWire integration as a virtual noise suppression microphone look [here](https://github.com/Rikorose/DeepFilterNet/blob/main/ladspa/README.md).
118
+
119
+ ### Demo
120
+
121
+ https://github.com/Rikorose/DeepFilterNet/assets/16517898/79679fd7-de73-4c22-948c-891927c7d2ca
122
+
123
+ To run the demo (linux only) use:
124
+ ```bash
125
+ cargo +nightly run -p df-demo --features ui --bin df-demo --release
126
+ ```
127
+
128
+ ### News
129
+
130
+ - New DeepFilterNet Demo: *DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement*
131
+ - Paper: https://arxiv.org/abs/2305.08227
132
+ - Video: https://youtu.be/EO7n96YwnyE
133
+
134
+ - New Multi-Frame Filtering Paper: *Deep Multi-Frame Filtering for Hearing Aids*
135
+ - Paper: https://arxiv.org/abs/2305.08225
136
+
137
+ - Real-time version and a LADSPA plugin
138
+ - [Pre-compiled binary](#deep-filter), no python dependencies. Usage: `deep-filter audio-file.wav`
139
+ - [LADSPA plugin](https://raw.githubusercontent.com/Rikorose/DeepFilterNet/7b14b895fe7fc42cab72b4a4c1e49c982bd6404b/ladspa/) with pipewire filter-chain integration for real-time noise reduction on your mic.
140
+
141
+ - DeepFilterNet2 Paper: *DeepFilterNet2: Towards Real-Time Speech Enhancement on Embedded Devices for Full-Band Audio*
142
+ - Paper: https://arxiv.org/abs/2205.05474
143
+ - Samples: https://rikorose.github.io/DeepFilterNet2-Samples/
144
+ - Demo: https://huggingface.co/spaces/hshr/DeepFilterNet2
145
+
146
+ - Original DeepFilterNet Paper: *DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep Filtering*
147
+ - Paper: https://arxiv.org/abs/2110.05588
148
+ - Samples: https://rikorose.github.io/DeepFilterNet-Samples/
149
+ - Demo: https://huggingface.co/spaces/hshr/DeepFilterNet
150
+ - Video Lecture: https://youtu.be/it90gBqkY6k
151
+
152
+ ## Usage
153
+
154
+ ### deep-filter
155
+
156
+ Download a pre-compiled deep-filter binary from the [release page](https://github.com/Rikorose/DeepFilterNet/releases/).
157
+ You can use `deep-filter` to suppress noise in noisy .wav audio files. Currently, only wav files with a sampling rate of 48kHz are supported.
158
+
159
+ ```bash
160
+ USAGE:
161
+ deep-filter [OPTIONS] [FILES]...
162
+
163
+ ARGS:
164
+ <FILES>...
165
+
166
+ OPTIONS:
167
+ -D, --compensate-delay
168
+ Compensate delay of STFT and model lookahead
169
+ -h, --help
170
+ Print help information
171
+ -m, --model <MODEL>
172
+ Path to model tar.gz. Defaults to DeepFilterNet2.
173
+ -o, --out-dir <OUT_DIR>
174
+ [default: out]
175
+ --pf
176
+ Enable postfilter
177
+ -v, --verbose
178
+ Logging verbosity
179
+ -V, --version
180
+ Print version information
181
+ ```
182
+
183
+ If you want to use the pytorch backend e.g. for GPU processing, see further below for the Python usage.
184
+
185
+ ### DeepFilterNet Framework
186
+
187
+ This framework supports Linux, MacOS and Windows. Training is only tested under Linux. The framework is structured as follows:
188
+
189
+ * `libDF` contains Rust code used for data loading and augmentation.
190
+ * `DeepFilterNet` contains DeepFilterNet code training, evaluation and visualization as well as pretrained model weights.
191
+ * `pyDF` contains a Python wrapper of libDF STFT/ISTFT processing loop.
192
+ * `pyDF-data` contains a Python wrapper of libDF dataset functionality and provides a pytorch data loader.
193
+ * `ladspa` contains a LADSPA plugin for real-time noise suppression.
194
+ * `models` contains pretrained for usage in DeepFilterNet (Python) or libDF/deep-filter (Rust)
195
+
196
+ ### DeepFilterNet Python: PyPI
197
+
198
+ Install the DeepFilterNet Python wheel via pip:
199
+ ```bash
200
+ # Install cpu/cuda pytorch (>=1.9) dependency from pytorch.org, e.g.:
201
+ pip install torch torchaudio -f https://download.pytorch.org/whl/cpu/torch_stable.html
202
+ # Install DeepFilterNet
203
+ pip install deepfilternet
204
+ # Or install DeepFilterNet including data loading functionality for training (Linux only)
205
+ pip install deepfilternet[train]
206
+ ```
207
+
208
+ To enhance noisy audio files using DeepFilterNet run
209
+ ```bash
210
+ # Specify an output directory with --output-dir [OUTPUT_DIR]
211
+ deepFilter path/to/noisy_audio.wav
212
+ ```
213
+
214
+ ### Manual Installation
215
+
216
+ Install cargo via [rustup](https://rustup.rs/). Usage of a `conda` or `virtualenv` recommended.
217
+ Please read the comments and only execute the commands that you need.
218
+
219
+ Installation of python dependencies and libDF:
220
+ ```bash
221
+ cd path/to/DeepFilterNet/ # cd into repository
222
+ # Recommended: Install or activate a python env
223
+ # Mandatory: Install cpu/cuda pytorch (>=1.8) dependency from pytorch.org, e.g.:
224
+ pip install torch torchaudio -f https://download.pytorch.org/whl/cpu/torch_stable.html
225
+ # Install build dependencies used to compile libdf and DeepFilterNet python wheels
226
+ pip install maturin poetry
227
+
228
+ # Install remaining DeepFilterNet python dependencies
229
+ # *Option A:* Install DeepFilterNet python wheel globally within your environment. Do this if you want use
230
+ # this repos as is, and don't want to develop within this repository.
231
+ poetry -C DeepFilterNet install -E train -E eval
232
+ # *Option B:* If you want to develop within this repo, install only dependencies and work with the repository version
233
+ poetry -C DeepFilterNet install -E train -E eval --no-root
234
+ export PYTHONPATH=$PWD/DeepFilterNet # And set the python path correctly
235
+
236
+ # Build and install libdf python package required for enhance.py
237
+ maturin develop --release -m pyDF/Cargo.toml
238
+ # *Optional*: Install libdfdata python package with dataset and dataloading functionality for training
239
+ # Required build dependency: HDF5 headers (e.g. ubuntu: libhdf5-dev)
240
+ maturin develop --release -m pyDF-data/Cargo.toml
241
+ # If you have troubles with hdf5 you may try to build and link hdf5 statically:
242
+ maturin develop --release --features hdf5-static -m pyDF-data/Cargo.toml
243
+ ```
244
+
245
+ ### Use DeepFilterNet from command line
246
+
247
+ To enhance noisy audio files using DeepFilterNet run
248
+ ```bash
249
+ $ python DeepFilterNet/df/enhance.py --help
250
+ usage: enhance.py [-h] [--model-base-dir MODEL_BASE_DIR] [--pf] [--output-dir OUTPUT_DIR] [--log-level LOG_LEVEL] [--compensate-delay]
251
+ noisy_audio_files [noisy_audio_files ...]
252
+
253
+ positional arguments:
254
+ noisy_audio_files List of noise files to mix with the clean speech file.
255
+
256
+ optional arguments:
257
+ -h, --help show this help message and exit
258
+ --model-base-dir MODEL_BASE_DIR, -m MODEL_BASE_DIR
259
+ Model directory containing checkpoints and config.
260
+ To load a pretrained model, you may just provide the model name, e.g. `DeepFilterNet`.
261
+ By default, the pretrained DeepFilterNet2 model is loaded.
262
+ --pf Post-filter that slightly over-attenuates very noisy sections.
263
+ --output-dir OUTPUT_DIR, -o OUTPUT_DIR
264
+ Directory in which the enhanced audio files will be stored.
265
+ --log-level LOG_LEVEL
266
+ Logger verbosity. Can be one of (debug, info, error, none)
267
+ --compensate-delay, -D
268
+ Add some paddig to compensate the delay introduced by the real-time STFT/ISTFT implementation.
269
+
270
+ # Enhance audio with original DeepFilterNet
271
+ python DeepFilterNet/df/enhance.py -m DeepFilterNet path/to/noisy_audio.wav
272
+
273
+ # Enhance audio with DeepFilterNet2
274
+ python DeepFilterNet/df/enhance.py -m DeepFilterNet2 path/to/noisy_audio.wav
275
+ ```
276
+
277
+ ### Use DeepFilterNet within your Python script
278
+
279
+ ```py
280
+ from df import enhance, init_df
281
+
282
+ model, df_state, _ = init_df() # Load default model
283
+ enhanced_audio = enhance(model, df_state, noisy_audio)
284
+ ```
285
+
286
+ See [here](https://github.com/Rikorose/DeepFilterNet/blob/main/scripts/external_usage.py) for a full example.
287
+
288
+ ### Training
289
+
290
+ The entry point is `DeepFilterNet/df/train.py`. It expects a data directory containing HDF5 dataset
291
+ as well as a dataset configuration json file.
292
+
293
+ So, you first need to create your datasets in HDF5 format. Each dataset typically only
294
+ holds training, validation, or test set of noise, speech or RIRs.
295
+ ```py
296
+ # Install additional dependencies for dataset creation
297
+ pip install h5py librosa soundfile
298
+ # Go to DeepFilterNet python package
299
+ cd path/to/DeepFilterNet/DeepFilterNet
300
+ # Prepare text file (e.g. called training_set.txt) containing paths to .wav files
301
+ #
302
+ # usage: prepare_data.py [-h] [--num_workers NUM_WORKERS] [--max_freq MAX_FREQ] [--sr SR] [--dtype DTYPE]
303
+ # [--codec CODEC] [--mono] [--compression COMPRESSION]
304
+ # type audio_files hdf5_db
305
+ #
306
+ # where:
307
+ # type: One of `speech`, `noise`, `rir`
308
+ # audio_files: Text file containing paths to audio files to include in the dataset
309
+ # hdf5_db: Output HDF5 dataset.
310
+ python df/scripts/prepare_data.py --sr 48000 speech training_set.txt TRAIN_SET_SPEECH.hdf5
311
+ ```
312
+ All datasets should be made available in one dataset folder for the train script.
313
+
314
+ The dataset configuration file should contain 3 entries: "train", "valid", "test". Each of those
315
+ contains a list of datasets (e.g. a speech, noise and a RIR dataset). You can use multiple speech
316
+ or noise dataset. Optionally, a sampling factor may be specified that can be used to over/under-sample
317
+ the dataset. Say, you have a specific dataset with transient noises and want to increase the amount
318
+ of non-stationary noises by oversampling. In most cases you want to set this factor to 1.
319
+
320
+ <details>
321
+ <summary>Dataset config example:</summary>
322
+ <p>
323
+
324
+ `dataset.cfg`
325
+
326
+ ```json
327
+ {
328
+ "train": [
329
+ [
330
+ "TRAIN_SET_SPEECH.hdf5",
331
+ 1.0
332
+ ],
333
+ [
334
+ "TRAIN_SET_NOISE.hdf5",
335
+ 1.0
336
+ ],
337
+ [
338
+ "TRAIN_SET_RIR.hdf5",
339
+ 1.0
340
+ ]
341
+ ],
342
+ "valid": [
343
+ [
344
+ "VALID_SET_SPEECH.hdf5",
345
+ 1.0
346
+ ],
347
+ [
348
+ "VALID_SET_NOISE.hdf5",
349
+ 1.0
350
+ ],
351
+ [
352
+ "VALID_SET_RIR.hdf5",
353
+ 1.0
354
+ ]
355
+ ],
356
+ "test": [
357
+ [
358
+ "TEST_SET_SPEECH.hdf5",
359
+ 1.0
360
+ ],
361
+ [
362
+ "TEST_SET_NOISE.hdf5",
363
+ 1.0
364
+ ],
365
+ [
366
+ "TEST_SET_RIR.hdf5",
367
+ 1.0
368
+ ]
369
+ ]
370
+ }
371
+ ```
372
+
373
+ </p>
374
+ </details>
375
+
376
+ Finally, start the training script. The training script may create a model `base_dir` if not
377
+ existing used for logging, some audio samples, model checkpoints, and config. If no config file is
378
+ found, it will create a default config. See
379
+ [DeepFilterNet/pretrained_models/DeepFilterNet](https://github.com/Rikorose/DeepFilterNet/blob/main/DeepFilterNet/pretrained_models/DeepFilterNet/config.ini)
380
+ for a config file.
381
+ ```py
382
+ # usage: train.py [-h] [--debug] data_config_file data_dir base_dir
383
+ python df/train.py path/to/dataset.cfg path/to/data_dir/ path/to/base_dir/
384
+ ```
385
+
386
+ ## Citation Guide
387
+
388
+ To reproduce any metrics, we recomend to use the python implementation via `pip install deepfilternet`.
389
+
390
+ If you use this framework, please cite: *DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep Filtering*
391
+ ```bibtex
392
+ @inproceedings{schroeter2022deepfilternet,
393
+ title={{DeepFilterNet}: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep Filtering},
394
+ author = {Schröter, Hendrik and Escalante-B., Alberto N. and Rosenkranz, Tobias and Maier, Andreas},
395
+ booktitle={ICASSP 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
396
+ year={2022},
397
+ organization={IEEE}
398
+ }
399
+ ```
400
+
401
+ If you use the DeepFilterNet2 model, please cite: *DeepFilterNet2: Towards Real-Time Speech Enhancement on Embedded Devices for Full-Band Audio*
402
+
403
+ ```bibtex
404
+ @inproceedings{schroeter2022deepfilternet2,
405
+ title = {{DeepFilterNet2}: Towards Real-Time Speech Enhancement on Embedded Devices for Full-Band Audio},
406
+ author = {Schröter, Hendrik and Escalante-B., Alberto N. and Rosenkranz, Tobias and Maier, Andreas},
407
+ booktitle={17th International Workshop on Acoustic Signal Enhancement (IWAENC 2022)},
408
+ year = {2022},
409
+ }
410
+ ```
411
+
412
+ If you use the DeepFilterNet3 model, please cite: *DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement*
413
+
414
+ ```bibtex
415
+ @inproceedings{schroeter2023deepfilternet3,
416
+ title = {{DeepFilterNet}: Perceptually Motivated Real-Time Speech Enhancement},
417
+ author = {Schröter, Hendrik and Rosenkranz, Tobias and Escalante-B., Alberto N. and Maier, Andreas},
418
+ booktitle={INTERSPEECH},
419
+ year = {2023},
420
+ }
421
+ ```
422
+
423
+ If you use the multi-frame beamforming algorithms. please cite *Deep Multi-Frame Filtering for Hearing Aids*
424
+
425
+ ```bibtex
426
+ @inproceedings{schroeter2023deep_mf,
427
+ title = {Deep Multi-Frame Filtering for Hearing Aids},
428
+ author = {Schröter, Hendrik and Rosenkranz, Tobias and Escalante-B., Alberto N. and Maier, Andreas},
429
+ booktitle={INTERSPEECH},
430
+ year = {2023},
431
+ }
432
+ ```
433
+
434
+ ## License
435
+
436
+ DeepFilterNet is free and open source! All code and model weights in this repository are dual-licensed under either:
437
+
438
+ * MIT License ([LICENSE-MIT](https://raw.githubusercontent.com/Rikorose/DeepFilterNet/7b14b895fe7fc42cab72b4a4c1e49c982bd6404b/LICENSE-MIT) or [http://opensource.org/licenses/MIT](http://opensource.org/licenses/MIT))
439
+ * Apache License, Version 2.0 ([LICENSE-APACHE](https://raw.githubusercontent.com/Rikorose/DeepFilterNet/7b14b895fe7fc42cab72b4a4c1e49c982bd6404b/LICENSE-APACHE) or [http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0))
440
+
441
+ at your option. This means you can select the license you prefer!
442
+
443
+ Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.