aufklarer commited on
Commit
8e4ece5
Β·
verified Β·
1 Parent(s): 85f4c1b

Upload folder using huggingface_hub

Browse files
README.md ADDED
@@ -0,0 +1,153 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - speech-enhancement
5
+ - denoising
6
+ - mlx
7
+ - apple-silicon
8
+ - deepfilternet
9
+ - safetensors
10
+ base_model: Rikorose/DeepFilterNet3
11
+ library_name: mlx
12
+ pipeline_tag: audio-to-audio
13
+ ---
14
+
15
+ # DeepFilterNet3 β€” MLX
16
+
17
+ Real-time speech enhancement for Apple Silicon. Removes background noise from
18
+ speech audio. FP32 MLX weights converted from the official DeepFilterNet3
19
+ checkpoint, with BatchNorm fused into the convolutions.
20
+
21
+ - **2.1M params**, fp32 safetensors, **8.1 MB**
22
+ - 48 kHz native, 10 ms frames, 2-frame (20 ms) lookahead
23
+ - Network-only export: STFT, ERB feature extraction, and deep-filter
24
+ application live in the runtime (`auxiliary.npz` ships the exact DSP
25
+ constants)
26
+
27
+ ## Model
28
+
29
+ | Detail | Value |
30
+ |--------|-------|
31
+ | Architecture | DeepFilterNet3 (encoder + ERB decoder + deep-filter decoder) |
32
+ | Parameters | 2,131,824 (BatchNorm fused, GRU biases folded) |
33
+ | Precision | float32 |
34
+ | Sample rate | 48 kHz (FFT 960, hop 480) |
35
+ | ERB bands / DF bins | 32 / 96 |
36
+ | Deep-filter order | 5, lookahead 2 frames |
37
+
38
+ ## Files
39
+
40
+ | File | Size | Description |
41
+ |------|------|-------------|
42
+ | `model.safetensors` | 8.1 MB | fp32 network weights in MLX layouts |
43
+ | `auxiliary.npz` | 126 KB | ERB filterbank + inverse, Vorbis window, normalization init states |
44
+ | `config.json` | 1 KB | Model and DSP hyperparameters |
45
+ | `dfn3_mlx.py` | 9 KB | Pure-MLX reference implementation of the network |
46
+
47
+ ## Quality
48
+
49
+ 30 VoiceBank-DEMAND test clips (16 kHz mirror, resampled to 48 kHz for the
50
+ model); the neural network forward runs on MLX while STFT / ERB / deep-filter
51
+ post-processing stay in the upstream PyTorch pipeline β€” the same methodology
52
+ used for the [CoreML variant](https://huggingface.co/aufklarer/DeepFilterNet3-CoreML).
53
+
54
+ | Backend | PESQ | STOI | SI-SDR |
55
+ |---------|------|------|--------|
56
+ | Noisy input | 2.205 | 0.932 | 9.28 |
57
+ | PyTorch FP32 (reference) | 2.900 | 0.947 | 18.19 |
58
+ | **MLX, CPU stream** | **2.900** | **0.947** | **18.19** |
59
+ | **MLX, GPU stream** | **2.902** | **0.947** | **18.18** |
60
+
61
+ Network outputs match PyTorch within 9e-7 on the MLX CPU stream β€”
62
+ numerically exact for fp32. The GPU stream uses Metal fast-math
63
+ transcendentals (max output delta ~4e-3) with no measurable metric impact.
64
+
65
+ ## Latency (Apple M5 Pro, network forward only, Python MLX)
66
+
67
+ | Duration | GPU | RTF | CPU | RTF |
68
+ |----------|-----|-----|-----|-----|
69
+ | 5 s | 0.14 s | 0.028 | 0.18 s | 0.036 |
70
+ | 10 s | 0.29 s | 0.029 | 0.40 s | 0.039 |
71
+ | 20 s | 0.54 s | 0.027 | 0.79 s | 0.040 |
72
+
73
+ ## Usage
74
+
75
+ ```python
76
+ import sys
77
+
78
+ import mlx.core as mx
79
+ from huggingface_hub import snapshot_download
80
+
81
+ model_dir = snapshot_download("aufklarer/DeepFilterNet3-MLX")
82
+ sys.path.append(model_dir)
83
+ from dfn3_mlx import DFN3MLX
84
+
85
+ model = DFN3MLX(model_dir)
86
+
87
+ # Normalized features from your DSP front-end (constants in auxiliary.npz):
88
+ feat_erb = mx.zeros((1, 100, 32, 1)) # ERB features, dB-scaled + mean-normalized
89
+ feat_spec = mx.zeros((1, 100, 96, 2)) # complex spectrum (real, imag), unit-normalized
90
+
91
+ erb_mask, df_coefs, lsnr = model(feat_erb, feat_spec)
92
+ ```
93
+
94
+ Full audio-in/audio-out enhancement additionally needs the DeepFilterNet DSP
95
+ path (STFT β†’ features β†’ ERB mask + deep filtering β†’ iSTFT), documented in the
96
+ [upstream repository](https://github.com/Rikorose/DeepFilterNet).
97
+
98
+ On Apple devices, [speech-swift](https://github.com/soniqo/speech-swift)
99
+ provides ready-made speech enhancement via the
100
+ [CoreML variant](https://huggingface.co/aufklarer/DeepFilterNet3-CoreML):
101
+
102
+ ```swift
103
+ import SpeechEnhancement
104
+
105
+ let enhancer = try await SpeechEnhancer.fromPretrained()
106
+ let clean = try enhancer.enhance(audio: noisyAudio, sampleRate: 48000)
107
+ ```
108
+
109
+ ## Tensor contract
110
+
111
+ Channels-last `[B, T, F, C]` β€” time is the conv H axis, frequency the W axis.
112
+ The 2-frame lookahead shift is applied to the inputs inside the model.
113
+
114
+ | Direction | Name | Shape | Notes |
115
+ |-----------|------|-------|-------|
116
+ | input | `feat_erb` | `[B, T, 32, 1]` | ERB features, dB-scaled, exp-mean-normalized |
117
+ | input | `feat_spec` | `[B, T, 96, 2]` | complex spectrum features, unit-normalized |
118
+ | output | `erb_mask` | `[B, T, 32, 1]` | sigmoid ERB gain mask |
119
+ | output | `df_coefs` | `[B, 5, T, 96, 2]` | deep-filter coefficients (order, real/imag) |
120
+ | output | `lsnr` | `[B, T, 1]` | local SNR estimate, dB in [-15, 35] |
121
+
122
+ ## Weight conventions
123
+
124
+ - **Conv2d**: `[O, kH, kW, I/groups]` (PyTorch `[O, I/g, kH, kW]` transposed);
125
+ causal time padding, symmetric frequency padding
126
+ - **Depthwise ConvTranspose2d** (`erb_dec.convt{1,2}.dwt`): `[C, kH, kW, 1]`,
127
+ true transposed-conv kernels (not pre-flipped); k=(1,3), freq stride 2,
128
+ padding 1, output padding 1
129
+ - **GRU** (`mlx.nn.GRU` convention, gate order r/z/n): `Wx = weight_ih`,
130
+ `Wh = weight_hh`, `b = bias_ih + [bias_hh_r; bias_hh_z; 0]`,
131
+ `bhn = bias_hh_n`; run with an explicit zero initial hidden state
132
+ - **Grouped linear**: `[groups, in/groups, out/groups]`, applied as
133
+ `einsum("btgi,gih->btgh")` then flattened
134
+ - BatchNorm is fused into the preceding conv; the unused `df_fc_a` head is
135
+ dropped
136
+
137
+ ## Source
138
+
139
+ - Base model: [Rikorose/DeepFilterNet](https://github.com/Rikorose/DeepFilterNet)
140
+ v0.5.6 checkpoint (Apache-2.0 / MIT)
141
+ - Paper: [DeepFilterNet3](https://arxiv.org/abs/2305.08227)
142
+
143
+ ## License
144
+
145
+ - Model weights: Apache-2.0 / MIT dual license
146
+ - MLX conversion: Apache-2.0
147
+
148
+ ## Links
149
+
150
+ - [speech-swift](https://github.com/soniqo/speech-swift) β€” Apple SDK
151
+ - [Docs](https://soniqo.audio/getting-started) β€” install and CLI docs
152
+ - [soniqo.audio](https://soniqo.audio) β€” website
153
+ - [blog](https://soniqo.audio/blog) β€” blog
__pycache__/dfn3_mlx.cpython-311.pyc ADDED
Binary file (14.2 kB). View file
 
auxiliary.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3cbba3bebdc165e53b13760d814de48f863e4b47caca2a5ab68504acce5ab3b3
3
+ size 128774
config.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "deepfilternet3",
3
+ "sample_rate": 48000,
4
+ "fft_size": 960,
5
+ "hop_size": 480,
6
+ "nb_erb": 32,
7
+ "nb_df": 96,
8
+ "df_order": 5,
9
+ "df_lookahead": 2,
10
+ "conv_lookahead": 2,
11
+ "conv_ch": 64,
12
+ "emb_hidden_dim": 256,
13
+ "df_hidden_dim": 256,
14
+ "lin_groups": 16,
15
+ "enc_lin_groups": 32,
16
+ "df_gru_linear_groups": 8,
17
+ "lsnr_min": -15,
18
+ "lsnr_max": 35,
19
+ "norm_tau": 1.0,
20
+ "gru_initial_hidden": "zeros"
21
+ }
dfn3_mlx.py ADDED
@@ -0,0 +1,190 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Pure-MLX reference implementation of the DeepFilterNet3 network.
2
+
3
+ Loads the model.safetensors + config.json shipped in this repository and
4
+ mirrors the DeepFilterNet3 inference graph: lookahead shift on the input
5
+ features, encoder, ERB decoder, DF decoder, DF-output reshape. The
6
+ surrounding DSP (STFT, ERB feature extraction, normalization, deep-filter
7
+ application, iSTFT) is the caller's job β€” auxiliary.npz carries the exact
8
+ filterbank / window / normalization constants.
9
+
10
+ Layout: channels-last [B, T, F, C] throughout (time = conv H axis,
11
+ frequency = conv W axis). All GRUs run with a zero initial hidden state.
12
+ """
13
+
14
+ import json
15
+ from pathlib import Path
16
+
17
+ import mlx.core as mx
18
+ import numpy as np
19
+
20
+
21
+ def _relu(x):
22
+ return mx.maximum(x, 0)
23
+
24
+
25
+ def grouped_linear(x, w):
26
+ """x: [B, T, I], w: [groups, I/groups, H/groups] -> [B, T, H]."""
27
+ b, t, _ = x.shape
28
+ g, gi, gh = w.shape
29
+ x = x.reshape(b, t, g, gi)
30
+ y = mx.einsum("btgi,gih->btgh", x, w)
31
+ return y.reshape(b, t, g * gh)
32
+
33
+
34
+ def gru_layer(x, Wx, Wh, b, bhn):
35
+ """mlx.nn.GRU recurrence with an explicit zero initial hidden state.
36
+
37
+ x: [B, T, I] -> [B, T, H]. Gate order r, z, n (same as PyTorch):
38
+ r = sigmoid(x@Wxr.T + h@Whr.T + br)
39
+ z = sigmoid(x@Wxz.T + h@Whz.T + bz)
40
+ n = tanh(x@Wxn.T + bn + r * (h@Whn.T + bhn))
41
+ h' = (1 - z) * n + z * h
42
+ """
43
+ hidden = Wh.shape[1]
44
+ xp = x @ Wx.T + b # [B, T, 3H]
45
+ x_rz = xp[..., : 2 * hidden]
46
+ x_n = xp[..., 2 * hidden :]
47
+ h = mx.zeros((x.shape[0], hidden), dtype=x.dtype)
48
+ outs = []
49
+ for t in range(x.shape[1]):
50
+ hp = h @ Wh.T
51
+ rz = mx.sigmoid(x_rz[:, t] + hp[..., : 2 * hidden])
52
+ r = rz[..., :hidden]
53
+ z = rz[..., hidden:]
54
+ n = mx.tanh(x_n[:, t] + r * (hp[..., 2 * hidden :] + bhn))
55
+ h = (1 - z) * n + z * h
56
+ outs.append(h)
57
+ return mx.stack(outs, axis=1)
58
+
59
+
60
+ def conv_transpose_dw_fstride2(x, w_flipped):
61
+ """Depthwise ConvTranspose2d over frequency, k=(1,3), stride=(1,2),
62
+ padding=(0,1), output_padding=(0,1) β€” via zero-stuffing + correlation.
63
+
64
+ x: [B, T, F, C] -> [B, T, 2F, C]. w_flipped: [C, 1, 3, 1], kernel
65
+ pre-flipped along the frequency axis.
66
+ """
67
+ bsz, t, f, c = x.shape
68
+ z = mx.stack([x, mx.zeros_like(x)], axis=3).reshape(bsz, t, 2 * f, c)
69
+ z = mx.pad(z, [(0, 0), (0, 0), (1, 1), (0, 0)])
70
+ return mx.conv2d(z, w_flipped, stride=(1, 1), padding=(0, 0), groups=c)
71
+
72
+
73
+ class DFN3MLX:
74
+ """DeepFilterNet3 network. Inputs are normalized features:
75
+
76
+ feat_erb [B, T, 32, 1] β€” ERB features (dB-scaled, mean-normalized)
77
+ feat_spec [B, T, 96, 2] β€” complex spectrogram features (unit-normalized)
78
+
79
+ Returns (erb_mask [B, T, 32, 1], df_coefs [B, 5, T, 96, 2], lsnr [B, T, 1]).
80
+ """
81
+
82
+ def __init__(self, model_dir: str | Path):
83
+ model_dir = Path(model_dir)
84
+ self.w = {k: mx.array(v) for k, v in mx.load(str(model_dir / "model.safetensors")).items()}
85
+ with open(model_dir / "config.json") as f:
86
+ self.cfg = json.load(f)
87
+ # Pre-flip transposed-conv kernels for the zero-stuff formulation
88
+ for name in ("erb_dec.convt2.dwt.weight", "erb_dec.convt1.dwt.weight"):
89
+ flipped = np.array(self.w[name])[:, :, ::-1, :]
90
+ self.w[name] = mx.array(np.ascontiguousarray(flipped))
91
+ self.lsnr_scale = self.cfg["lsnr_max"] - self.cfg["lsnr_min"]
92
+ self.lsnr_offset = self.cfg["lsnr_min"]
93
+ self.conv_lookahead = self.cfg["conv_lookahead"]
94
+
95
+ # ── building blocks ──
96
+
97
+ def _inp_conv(self, x, prefix, groups):
98
+ """pad(t=2) + Conv2d k=(3,3) (grouped) [+ pointwise] + ReLU."""
99
+ w = self.w
100
+ x = mx.pad(x, [(0, 0), (2, 0), (0, 0), (0, 0)])
101
+ if f"{prefix}.conv.bias" in w: # erb_conv0: single conv, BN fused into it
102
+ y = mx.conv2d(x, w[f"{prefix}.conv.weight"], stride=(1, 1), padding=(0, 1), groups=groups)
103
+ y = y + w[f"{prefix}.conv.bias"]
104
+ else: # df_conv0: grouped conv + BN-fused pointwise
105
+ y = mx.conv2d(x, w[f"{prefix}.conv.weight"], stride=(1, 1), padding=(0, 1), groups=groups)
106
+ y = mx.conv2d(y, w[f"{prefix}.pw.weight"], stride=(1, 1), padding=(0, 0))
107
+ y = y + w[f"{prefix}.pw.bias"]
108
+ return _relu(y)
109
+
110
+ def _sep_conv(self, x, prefix, fstride):
111
+ """Depthwise k=(1,3) conv + BN-fused pointwise + ReLU."""
112
+ w = self.w
113
+ c = x.shape[-1]
114
+ y = mx.conv2d(x, w[f"{prefix}.dw.weight"], stride=(1, fstride), padding=(0, 1), groups=c)
115
+ y = mx.conv2d(y, w[f"{prefix}.pw.weight"], stride=(1, 1), padding=(0, 0))
116
+ return _relu(y + w[f"{prefix}.pw.bias"])
117
+
118
+ def _pathway(self, x, prefix):
119
+ """Depthwise 1x1 conv (BN fused) + ReLU β€” a per-channel affine."""
120
+ w = self.w
121
+ y = mx.conv2d(x, w[f"{prefix}.dw.weight"], stride=(1, 1), padding=(0, 0), groups=x.shape[-1])
122
+ return _relu(y + w[f"{prefix}.dw.bias"])
123
+
124
+ def _sep_convt(self, x, prefix):
125
+ """Depthwise transposed conv (fstride=2) + BN-fused pointwise + ReLU."""
126
+ w = self.w
127
+ y = conv_transpose_dw_fstride2(x, w[f"{prefix}.dwt.weight"])
128
+ y = mx.conv2d(y, w[f"{prefix}.pw.weight"], stride=(1, 1), padding=(0, 0))
129
+ return _relu(y + w[f"{prefix}.pw.bias"])
130
+
131
+ def _squeezed_gru(self, x, prefix, num_layers, has_linear_out):
132
+ w = self.w
133
+ y = _relu(grouped_linear(x, w[f"{prefix}.linear_in.weight"]))
134
+ if num_layers == 1:
135
+ y = gru_layer(y, w[f"{prefix}.gru.Wx"], w[f"{prefix}.gru.Wh"],
136
+ w[f"{prefix}.gru.b"], w[f"{prefix}.gru.bhn"])
137
+ else:
138
+ for l in range(num_layers):
139
+ p = f"{prefix}.gru.layers.{l}"
140
+ y = gru_layer(y, w[f"{p}.Wx"], w[f"{p}.Wh"], w[f"{p}.b"], w[f"{p}.bhn"])
141
+ if has_linear_out:
142
+ y = _relu(grouped_linear(y, w[f"{prefix}.linear_out.weight"]))
143
+ return y
144
+
145
+ # ── model ──
146
+
147
+ def __call__(self, feat_erb, feat_spec, apply_lookahead_shift: bool = True):
148
+ w = self.w
149
+ la = self.conv_lookahead
150
+ if apply_lookahead_shift and la > 0:
151
+ feat_erb = mx.pad(feat_erb[:, la:], [(0, 0), (0, la), (0, 0), (0, 0)])
152
+ feat_spec = mx.pad(feat_spec[:, la:], [(0, 0), (0, la), (0, 0), (0, 0)])
153
+
154
+ # Encoder
155
+ e0 = self._inp_conv(feat_erb, "enc.erb_conv0", groups=1) # [B,T,32,64]
156
+ e1 = self._sep_conv(e0, "enc.erb_conv1", 2) # [B,T,16,64]
157
+ e2 = self._sep_conv(e1, "enc.erb_conv2", 2) # [B,T,8,64]
158
+ e3 = self._sep_conv(e2, "enc.erb_conv3", 1) # [B,T,8,64]
159
+ c0 = self._inp_conv(feat_spec, "enc.df_conv0", groups=2) # [B,T,96,64]
160
+ c1 = self._sep_conv(c0, "enc.df_conv1", 2) # [B,T,48,64]
161
+
162
+ bsz, t = e3.shape[0], e3.shape[1]
163
+ cemb = _relu(grouped_linear(c1.reshape(bsz, t, -1), w["enc.df_fc_emb.weight"]))
164
+ emb = e3.reshape(bsz, t, -1) + cemb # [B,T,512]
165
+ emb = self._squeezed_gru(emb, "enc.emb_gru", num_layers=1, has_linear_out=True)
166
+ lsnr = mx.sigmoid(emb @ w["enc.lsnr_fc.weight"].T + w["enc.lsnr_fc.bias"])
167
+ lsnr = lsnr * self.lsnr_scale + self.lsnr_offset # [B,T,1]
168
+
169
+ # ERB decoder
170
+ d = self._squeezed_gru(emb, "erb_dec.emb_gru", num_layers=2, has_linear_out=True)
171
+ d = d.reshape(bsz, t, 8, -1) # [B,T,8,64]
172
+ d3 = self._sep_conv(self._pathway(e3, "erb_dec.conv3p") + d, "erb_dec.convt3", 1)
173
+ d2 = self._sep_convt(self._pathway(e2, "erb_dec.conv2p") + d3, "erb_dec.convt2")
174
+ d1 = self._sep_convt(self._pathway(e1, "erb_dec.conv1p") + d2, "erb_dec.convt1")
175
+ m = self._pathway(e0, "erb_dec.conv0p") + d1
176
+ m = mx.conv2d(m, w["erb_dec.conv0_out.conv.weight"], stride=(1, 1), padding=(0, 1))
177
+ erb_mask = mx.sigmoid(m + w["erb_dec.conv0_out.conv.bias"]) # [B,T,32,1]
178
+
179
+ # DF decoder
180
+ c = self._squeezed_gru(emb, "df_dec.df_gru", num_layers=2, has_linear_out=False)
181
+ c = c + grouped_linear(emb, w["df_dec.df_skip.weight"]) # [B,T,256]
182
+ cp = mx.pad(c0, [(0, 0), (4, 0), (0, 0), (0, 0)])
183
+ cp = mx.conv2d(cp, w["df_dec.df_convp.conv.weight"], stride=(1, 1), padding=(0, 0), groups=2)
184
+ cp = mx.conv2d(cp, w["df_dec.df_convp.pw.weight"], stride=(1, 1), padding=(0, 0))
185
+ cp = _relu(cp + w["df_dec.df_convp.pw.bias"]) # [B,T,96,10]
186
+ cf = mx.tanh(grouped_linear(c, w["df_dec.df_out.weight"])) # [B,T,960]
187
+ cf = cf.reshape(bsz, t, 96, 10) + cp
188
+ df_coefs = cf.reshape(bsz, t, 96, 5, 2).transpose(0, 3, 1, 2, 4) # [B,5,T,96,2]
189
+
190
+ return erb_mask, df_coefs, lsnr
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:515656d3689d176fc9e2c6013699baab698ea70cc259aa1ae81ee3ae6f5cdc2e
3
+ size 8533736