Audio-to-Audio
MLX
Safetensors
deepfilternet3
speech-enhancement
denoising
apple-silicon
deepfilternet
Instructions to use aufklarer/DeepFilterNet3-MLX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use aufklarer/DeepFilterNet3-MLX with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download aufklarer/DeepFilterNet3-MLX --local-dir DeepFilterNet3-MLX
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Upload folder using huggingface_hub
Browse files- README.md +153 -0
- __pycache__/dfn3_mlx.cpython-311.pyc +0 -0
- auxiliary.npz +3 -0
- config.json +21 -0
- dfn3_mlx.py +190 -0
- model.safetensors +3 -0
README.md
ADDED
|
@@ -0,0 +1,153 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
tags:
|
| 4 |
+
- speech-enhancement
|
| 5 |
+
- denoising
|
| 6 |
+
- mlx
|
| 7 |
+
- apple-silicon
|
| 8 |
+
- deepfilternet
|
| 9 |
+
- safetensors
|
| 10 |
+
base_model: Rikorose/DeepFilterNet3
|
| 11 |
+
library_name: mlx
|
| 12 |
+
pipeline_tag: audio-to-audio
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# DeepFilterNet3 β MLX
|
| 16 |
+
|
| 17 |
+
Real-time speech enhancement for Apple Silicon. Removes background noise from
|
| 18 |
+
speech audio. FP32 MLX weights converted from the official DeepFilterNet3
|
| 19 |
+
checkpoint, with BatchNorm fused into the convolutions.
|
| 20 |
+
|
| 21 |
+
- **2.1M params**, fp32 safetensors, **8.1 MB**
|
| 22 |
+
- 48 kHz native, 10 ms frames, 2-frame (20 ms) lookahead
|
| 23 |
+
- Network-only export: STFT, ERB feature extraction, and deep-filter
|
| 24 |
+
application live in the runtime (`auxiliary.npz` ships the exact DSP
|
| 25 |
+
constants)
|
| 26 |
+
|
| 27 |
+
## Model
|
| 28 |
+
|
| 29 |
+
| Detail | Value |
|
| 30 |
+
|--------|-------|
|
| 31 |
+
| Architecture | DeepFilterNet3 (encoder + ERB decoder + deep-filter decoder) |
|
| 32 |
+
| Parameters | 2,131,824 (BatchNorm fused, GRU biases folded) |
|
| 33 |
+
| Precision | float32 |
|
| 34 |
+
| Sample rate | 48 kHz (FFT 960, hop 480) |
|
| 35 |
+
| ERB bands / DF bins | 32 / 96 |
|
| 36 |
+
| Deep-filter order | 5, lookahead 2 frames |
|
| 37 |
+
|
| 38 |
+
## Files
|
| 39 |
+
|
| 40 |
+
| File | Size | Description |
|
| 41 |
+
|------|------|-------------|
|
| 42 |
+
| `model.safetensors` | 8.1 MB | fp32 network weights in MLX layouts |
|
| 43 |
+
| `auxiliary.npz` | 126 KB | ERB filterbank + inverse, Vorbis window, normalization init states |
|
| 44 |
+
| `config.json` | 1 KB | Model and DSP hyperparameters |
|
| 45 |
+
| `dfn3_mlx.py` | 9 KB | Pure-MLX reference implementation of the network |
|
| 46 |
+
|
| 47 |
+
## Quality
|
| 48 |
+
|
| 49 |
+
30 VoiceBank-DEMAND test clips (16 kHz mirror, resampled to 48 kHz for the
|
| 50 |
+
model); the neural network forward runs on MLX while STFT / ERB / deep-filter
|
| 51 |
+
post-processing stay in the upstream PyTorch pipeline β the same methodology
|
| 52 |
+
used for the [CoreML variant](https://huggingface.co/aufklarer/DeepFilterNet3-CoreML).
|
| 53 |
+
|
| 54 |
+
| Backend | PESQ | STOI | SI-SDR |
|
| 55 |
+
|---------|------|------|--------|
|
| 56 |
+
| Noisy input | 2.205 | 0.932 | 9.28 |
|
| 57 |
+
| PyTorch FP32 (reference) | 2.900 | 0.947 | 18.19 |
|
| 58 |
+
| **MLX, CPU stream** | **2.900** | **0.947** | **18.19** |
|
| 59 |
+
| **MLX, GPU stream** | **2.902** | **0.947** | **18.18** |
|
| 60 |
+
|
| 61 |
+
Network outputs match PyTorch within 9e-7 on the MLX CPU stream β
|
| 62 |
+
numerically exact for fp32. The GPU stream uses Metal fast-math
|
| 63 |
+
transcendentals (max output delta ~4e-3) with no measurable metric impact.
|
| 64 |
+
|
| 65 |
+
## Latency (Apple M5 Pro, network forward only, Python MLX)
|
| 66 |
+
|
| 67 |
+
| Duration | GPU | RTF | CPU | RTF |
|
| 68 |
+
|----------|-----|-----|-----|-----|
|
| 69 |
+
| 5 s | 0.14 s | 0.028 | 0.18 s | 0.036 |
|
| 70 |
+
| 10 s | 0.29 s | 0.029 | 0.40 s | 0.039 |
|
| 71 |
+
| 20 s | 0.54 s | 0.027 | 0.79 s | 0.040 |
|
| 72 |
+
|
| 73 |
+
## Usage
|
| 74 |
+
|
| 75 |
+
```python
|
| 76 |
+
import sys
|
| 77 |
+
|
| 78 |
+
import mlx.core as mx
|
| 79 |
+
from huggingface_hub import snapshot_download
|
| 80 |
+
|
| 81 |
+
model_dir = snapshot_download("aufklarer/DeepFilterNet3-MLX")
|
| 82 |
+
sys.path.append(model_dir)
|
| 83 |
+
from dfn3_mlx import DFN3MLX
|
| 84 |
+
|
| 85 |
+
model = DFN3MLX(model_dir)
|
| 86 |
+
|
| 87 |
+
# Normalized features from your DSP front-end (constants in auxiliary.npz):
|
| 88 |
+
feat_erb = mx.zeros((1, 100, 32, 1)) # ERB features, dB-scaled + mean-normalized
|
| 89 |
+
feat_spec = mx.zeros((1, 100, 96, 2)) # complex spectrum (real, imag), unit-normalized
|
| 90 |
+
|
| 91 |
+
erb_mask, df_coefs, lsnr = model(feat_erb, feat_spec)
|
| 92 |
+
```
|
| 93 |
+
|
| 94 |
+
Full audio-in/audio-out enhancement additionally needs the DeepFilterNet DSP
|
| 95 |
+
path (STFT β features β ERB mask + deep filtering β iSTFT), documented in the
|
| 96 |
+
[upstream repository](https://github.com/Rikorose/DeepFilterNet).
|
| 97 |
+
|
| 98 |
+
On Apple devices, [speech-swift](https://github.com/soniqo/speech-swift)
|
| 99 |
+
provides ready-made speech enhancement via the
|
| 100 |
+
[CoreML variant](https://huggingface.co/aufklarer/DeepFilterNet3-CoreML):
|
| 101 |
+
|
| 102 |
+
```swift
|
| 103 |
+
import SpeechEnhancement
|
| 104 |
+
|
| 105 |
+
let enhancer = try await SpeechEnhancer.fromPretrained()
|
| 106 |
+
let clean = try enhancer.enhance(audio: noisyAudio, sampleRate: 48000)
|
| 107 |
+
```
|
| 108 |
+
|
| 109 |
+
## Tensor contract
|
| 110 |
+
|
| 111 |
+
Channels-last `[B, T, F, C]` β time is the conv H axis, frequency the W axis.
|
| 112 |
+
The 2-frame lookahead shift is applied to the inputs inside the model.
|
| 113 |
+
|
| 114 |
+
| Direction | Name | Shape | Notes |
|
| 115 |
+
|-----------|------|-------|-------|
|
| 116 |
+
| input | `feat_erb` | `[B, T, 32, 1]` | ERB features, dB-scaled, exp-mean-normalized |
|
| 117 |
+
| input | `feat_spec` | `[B, T, 96, 2]` | complex spectrum features, unit-normalized |
|
| 118 |
+
| output | `erb_mask` | `[B, T, 32, 1]` | sigmoid ERB gain mask |
|
| 119 |
+
| output | `df_coefs` | `[B, 5, T, 96, 2]` | deep-filter coefficients (order, real/imag) |
|
| 120 |
+
| output | `lsnr` | `[B, T, 1]` | local SNR estimate, dB in [-15, 35] |
|
| 121 |
+
|
| 122 |
+
## Weight conventions
|
| 123 |
+
|
| 124 |
+
- **Conv2d**: `[O, kH, kW, I/groups]` (PyTorch `[O, I/g, kH, kW]` transposed);
|
| 125 |
+
causal time padding, symmetric frequency padding
|
| 126 |
+
- **Depthwise ConvTranspose2d** (`erb_dec.convt{1,2}.dwt`): `[C, kH, kW, 1]`,
|
| 127 |
+
true transposed-conv kernels (not pre-flipped); k=(1,3), freq stride 2,
|
| 128 |
+
padding 1, output padding 1
|
| 129 |
+
- **GRU** (`mlx.nn.GRU` convention, gate order r/z/n): `Wx = weight_ih`,
|
| 130 |
+
`Wh = weight_hh`, `b = bias_ih + [bias_hh_r; bias_hh_z; 0]`,
|
| 131 |
+
`bhn = bias_hh_n`; run with an explicit zero initial hidden state
|
| 132 |
+
- **Grouped linear**: `[groups, in/groups, out/groups]`, applied as
|
| 133 |
+
`einsum("btgi,gih->btgh")` then flattened
|
| 134 |
+
- BatchNorm is fused into the preceding conv; the unused `df_fc_a` head is
|
| 135 |
+
dropped
|
| 136 |
+
|
| 137 |
+
## Source
|
| 138 |
+
|
| 139 |
+
- Base model: [Rikorose/DeepFilterNet](https://github.com/Rikorose/DeepFilterNet)
|
| 140 |
+
v0.5.6 checkpoint (Apache-2.0 / MIT)
|
| 141 |
+
- Paper: [DeepFilterNet3](https://arxiv.org/abs/2305.08227)
|
| 142 |
+
|
| 143 |
+
## License
|
| 144 |
+
|
| 145 |
+
- Model weights: Apache-2.0 / MIT dual license
|
| 146 |
+
- MLX conversion: Apache-2.0
|
| 147 |
+
|
| 148 |
+
## Links
|
| 149 |
+
|
| 150 |
+
- [speech-swift](https://github.com/soniqo/speech-swift) β Apple SDK
|
| 151 |
+
- [Docs](https://soniqo.audio/getting-started) β install and CLI docs
|
| 152 |
+
- [soniqo.audio](https://soniqo.audio) β website
|
| 153 |
+
- [blog](https://soniqo.audio/blog) β blog
|
__pycache__/dfn3_mlx.cpython-311.pyc
ADDED
|
Binary file (14.2 kB). View file
|
|
|
auxiliary.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3cbba3bebdc165e53b13760d814de48f863e4b47caca2a5ab68504acce5ab3b3
|
| 3 |
+
size 128774
|
config.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_type": "deepfilternet3",
|
| 3 |
+
"sample_rate": 48000,
|
| 4 |
+
"fft_size": 960,
|
| 5 |
+
"hop_size": 480,
|
| 6 |
+
"nb_erb": 32,
|
| 7 |
+
"nb_df": 96,
|
| 8 |
+
"df_order": 5,
|
| 9 |
+
"df_lookahead": 2,
|
| 10 |
+
"conv_lookahead": 2,
|
| 11 |
+
"conv_ch": 64,
|
| 12 |
+
"emb_hidden_dim": 256,
|
| 13 |
+
"df_hidden_dim": 256,
|
| 14 |
+
"lin_groups": 16,
|
| 15 |
+
"enc_lin_groups": 32,
|
| 16 |
+
"df_gru_linear_groups": 8,
|
| 17 |
+
"lsnr_min": -15,
|
| 18 |
+
"lsnr_max": 35,
|
| 19 |
+
"norm_tau": 1.0,
|
| 20 |
+
"gru_initial_hidden": "zeros"
|
| 21 |
+
}
|
dfn3_mlx.py
ADDED
|
@@ -0,0 +1,190 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Pure-MLX reference implementation of the DeepFilterNet3 network.
|
| 2 |
+
|
| 3 |
+
Loads the model.safetensors + config.json shipped in this repository and
|
| 4 |
+
mirrors the DeepFilterNet3 inference graph: lookahead shift on the input
|
| 5 |
+
features, encoder, ERB decoder, DF decoder, DF-output reshape. The
|
| 6 |
+
surrounding DSP (STFT, ERB feature extraction, normalization, deep-filter
|
| 7 |
+
application, iSTFT) is the caller's job β auxiliary.npz carries the exact
|
| 8 |
+
filterbank / window / normalization constants.
|
| 9 |
+
|
| 10 |
+
Layout: channels-last [B, T, F, C] throughout (time = conv H axis,
|
| 11 |
+
frequency = conv W axis). All GRUs run with a zero initial hidden state.
|
| 12 |
+
"""
|
| 13 |
+
|
| 14 |
+
import json
|
| 15 |
+
from pathlib import Path
|
| 16 |
+
|
| 17 |
+
import mlx.core as mx
|
| 18 |
+
import numpy as np
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def _relu(x):
|
| 22 |
+
return mx.maximum(x, 0)
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def grouped_linear(x, w):
|
| 26 |
+
"""x: [B, T, I], w: [groups, I/groups, H/groups] -> [B, T, H]."""
|
| 27 |
+
b, t, _ = x.shape
|
| 28 |
+
g, gi, gh = w.shape
|
| 29 |
+
x = x.reshape(b, t, g, gi)
|
| 30 |
+
y = mx.einsum("btgi,gih->btgh", x, w)
|
| 31 |
+
return y.reshape(b, t, g * gh)
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def gru_layer(x, Wx, Wh, b, bhn):
|
| 35 |
+
"""mlx.nn.GRU recurrence with an explicit zero initial hidden state.
|
| 36 |
+
|
| 37 |
+
x: [B, T, I] -> [B, T, H]. Gate order r, z, n (same as PyTorch):
|
| 38 |
+
r = sigmoid(x@Wxr.T + h@Whr.T + br)
|
| 39 |
+
z = sigmoid(x@Wxz.T + h@Whz.T + bz)
|
| 40 |
+
n = tanh(x@Wxn.T + bn + r * (h@Whn.T + bhn))
|
| 41 |
+
h' = (1 - z) * n + z * h
|
| 42 |
+
"""
|
| 43 |
+
hidden = Wh.shape[1]
|
| 44 |
+
xp = x @ Wx.T + b # [B, T, 3H]
|
| 45 |
+
x_rz = xp[..., : 2 * hidden]
|
| 46 |
+
x_n = xp[..., 2 * hidden :]
|
| 47 |
+
h = mx.zeros((x.shape[0], hidden), dtype=x.dtype)
|
| 48 |
+
outs = []
|
| 49 |
+
for t in range(x.shape[1]):
|
| 50 |
+
hp = h @ Wh.T
|
| 51 |
+
rz = mx.sigmoid(x_rz[:, t] + hp[..., : 2 * hidden])
|
| 52 |
+
r = rz[..., :hidden]
|
| 53 |
+
z = rz[..., hidden:]
|
| 54 |
+
n = mx.tanh(x_n[:, t] + r * (hp[..., 2 * hidden :] + bhn))
|
| 55 |
+
h = (1 - z) * n + z * h
|
| 56 |
+
outs.append(h)
|
| 57 |
+
return mx.stack(outs, axis=1)
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def conv_transpose_dw_fstride2(x, w_flipped):
|
| 61 |
+
"""Depthwise ConvTranspose2d over frequency, k=(1,3), stride=(1,2),
|
| 62 |
+
padding=(0,1), output_padding=(0,1) β via zero-stuffing + correlation.
|
| 63 |
+
|
| 64 |
+
x: [B, T, F, C] -> [B, T, 2F, C]. w_flipped: [C, 1, 3, 1], kernel
|
| 65 |
+
pre-flipped along the frequency axis.
|
| 66 |
+
"""
|
| 67 |
+
bsz, t, f, c = x.shape
|
| 68 |
+
z = mx.stack([x, mx.zeros_like(x)], axis=3).reshape(bsz, t, 2 * f, c)
|
| 69 |
+
z = mx.pad(z, [(0, 0), (0, 0), (1, 1), (0, 0)])
|
| 70 |
+
return mx.conv2d(z, w_flipped, stride=(1, 1), padding=(0, 0), groups=c)
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
class DFN3MLX:
|
| 74 |
+
"""DeepFilterNet3 network. Inputs are normalized features:
|
| 75 |
+
|
| 76 |
+
feat_erb [B, T, 32, 1] β ERB features (dB-scaled, mean-normalized)
|
| 77 |
+
feat_spec [B, T, 96, 2] β complex spectrogram features (unit-normalized)
|
| 78 |
+
|
| 79 |
+
Returns (erb_mask [B, T, 32, 1], df_coefs [B, 5, T, 96, 2], lsnr [B, T, 1]).
|
| 80 |
+
"""
|
| 81 |
+
|
| 82 |
+
def __init__(self, model_dir: str | Path):
|
| 83 |
+
model_dir = Path(model_dir)
|
| 84 |
+
self.w = {k: mx.array(v) for k, v in mx.load(str(model_dir / "model.safetensors")).items()}
|
| 85 |
+
with open(model_dir / "config.json") as f:
|
| 86 |
+
self.cfg = json.load(f)
|
| 87 |
+
# Pre-flip transposed-conv kernels for the zero-stuff formulation
|
| 88 |
+
for name in ("erb_dec.convt2.dwt.weight", "erb_dec.convt1.dwt.weight"):
|
| 89 |
+
flipped = np.array(self.w[name])[:, :, ::-1, :]
|
| 90 |
+
self.w[name] = mx.array(np.ascontiguousarray(flipped))
|
| 91 |
+
self.lsnr_scale = self.cfg["lsnr_max"] - self.cfg["lsnr_min"]
|
| 92 |
+
self.lsnr_offset = self.cfg["lsnr_min"]
|
| 93 |
+
self.conv_lookahead = self.cfg["conv_lookahead"]
|
| 94 |
+
|
| 95 |
+
# ββ building blocks ββ
|
| 96 |
+
|
| 97 |
+
def _inp_conv(self, x, prefix, groups):
|
| 98 |
+
"""pad(t=2) + Conv2d k=(3,3) (grouped) [+ pointwise] + ReLU."""
|
| 99 |
+
w = self.w
|
| 100 |
+
x = mx.pad(x, [(0, 0), (2, 0), (0, 0), (0, 0)])
|
| 101 |
+
if f"{prefix}.conv.bias" in w: # erb_conv0: single conv, BN fused into it
|
| 102 |
+
y = mx.conv2d(x, w[f"{prefix}.conv.weight"], stride=(1, 1), padding=(0, 1), groups=groups)
|
| 103 |
+
y = y + w[f"{prefix}.conv.bias"]
|
| 104 |
+
else: # df_conv0: grouped conv + BN-fused pointwise
|
| 105 |
+
y = mx.conv2d(x, w[f"{prefix}.conv.weight"], stride=(1, 1), padding=(0, 1), groups=groups)
|
| 106 |
+
y = mx.conv2d(y, w[f"{prefix}.pw.weight"], stride=(1, 1), padding=(0, 0))
|
| 107 |
+
y = y + w[f"{prefix}.pw.bias"]
|
| 108 |
+
return _relu(y)
|
| 109 |
+
|
| 110 |
+
def _sep_conv(self, x, prefix, fstride):
|
| 111 |
+
"""Depthwise k=(1,3) conv + BN-fused pointwise + ReLU."""
|
| 112 |
+
w = self.w
|
| 113 |
+
c = x.shape[-1]
|
| 114 |
+
y = mx.conv2d(x, w[f"{prefix}.dw.weight"], stride=(1, fstride), padding=(0, 1), groups=c)
|
| 115 |
+
y = mx.conv2d(y, w[f"{prefix}.pw.weight"], stride=(1, 1), padding=(0, 0))
|
| 116 |
+
return _relu(y + w[f"{prefix}.pw.bias"])
|
| 117 |
+
|
| 118 |
+
def _pathway(self, x, prefix):
|
| 119 |
+
"""Depthwise 1x1 conv (BN fused) + ReLU β a per-channel affine."""
|
| 120 |
+
w = self.w
|
| 121 |
+
y = mx.conv2d(x, w[f"{prefix}.dw.weight"], stride=(1, 1), padding=(0, 0), groups=x.shape[-1])
|
| 122 |
+
return _relu(y + w[f"{prefix}.dw.bias"])
|
| 123 |
+
|
| 124 |
+
def _sep_convt(self, x, prefix):
|
| 125 |
+
"""Depthwise transposed conv (fstride=2) + BN-fused pointwise + ReLU."""
|
| 126 |
+
w = self.w
|
| 127 |
+
y = conv_transpose_dw_fstride2(x, w[f"{prefix}.dwt.weight"])
|
| 128 |
+
y = mx.conv2d(y, w[f"{prefix}.pw.weight"], stride=(1, 1), padding=(0, 0))
|
| 129 |
+
return _relu(y + w[f"{prefix}.pw.bias"])
|
| 130 |
+
|
| 131 |
+
def _squeezed_gru(self, x, prefix, num_layers, has_linear_out):
|
| 132 |
+
w = self.w
|
| 133 |
+
y = _relu(grouped_linear(x, w[f"{prefix}.linear_in.weight"]))
|
| 134 |
+
if num_layers == 1:
|
| 135 |
+
y = gru_layer(y, w[f"{prefix}.gru.Wx"], w[f"{prefix}.gru.Wh"],
|
| 136 |
+
w[f"{prefix}.gru.b"], w[f"{prefix}.gru.bhn"])
|
| 137 |
+
else:
|
| 138 |
+
for l in range(num_layers):
|
| 139 |
+
p = f"{prefix}.gru.layers.{l}"
|
| 140 |
+
y = gru_layer(y, w[f"{p}.Wx"], w[f"{p}.Wh"], w[f"{p}.b"], w[f"{p}.bhn"])
|
| 141 |
+
if has_linear_out:
|
| 142 |
+
y = _relu(grouped_linear(y, w[f"{prefix}.linear_out.weight"]))
|
| 143 |
+
return y
|
| 144 |
+
|
| 145 |
+
# ββ model ββ
|
| 146 |
+
|
| 147 |
+
def __call__(self, feat_erb, feat_spec, apply_lookahead_shift: bool = True):
|
| 148 |
+
w = self.w
|
| 149 |
+
la = self.conv_lookahead
|
| 150 |
+
if apply_lookahead_shift and la > 0:
|
| 151 |
+
feat_erb = mx.pad(feat_erb[:, la:], [(0, 0), (0, la), (0, 0), (0, 0)])
|
| 152 |
+
feat_spec = mx.pad(feat_spec[:, la:], [(0, 0), (0, la), (0, 0), (0, 0)])
|
| 153 |
+
|
| 154 |
+
# Encoder
|
| 155 |
+
e0 = self._inp_conv(feat_erb, "enc.erb_conv0", groups=1) # [B,T,32,64]
|
| 156 |
+
e1 = self._sep_conv(e0, "enc.erb_conv1", 2) # [B,T,16,64]
|
| 157 |
+
e2 = self._sep_conv(e1, "enc.erb_conv2", 2) # [B,T,8,64]
|
| 158 |
+
e3 = self._sep_conv(e2, "enc.erb_conv3", 1) # [B,T,8,64]
|
| 159 |
+
c0 = self._inp_conv(feat_spec, "enc.df_conv0", groups=2) # [B,T,96,64]
|
| 160 |
+
c1 = self._sep_conv(c0, "enc.df_conv1", 2) # [B,T,48,64]
|
| 161 |
+
|
| 162 |
+
bsz, t = e3.shape[0], e3.shape[1]
|
| 163 |
+
cemb = _relu(grouped_linear(c1.reshape(bsz, t, -1), w["enc.df_fc_emb.weight"]))
|
| 164 |
+
emb = e3.reshape(bsz, t, -1) + cemb # [B,T,512]
|
| 165 |
+
emb = self._squeezed_gru(emb, "enc.emb_gru", num_layers=1, has_linear_out=True)
|
| 166 |
+
lsnr = mx.sigmoid(emb @ w["enc.lsnr_fc.weight"].T + w["enc.lsnr_fc.bias"])
|
| 167 |
+
lsnr = lsnr * self.lsnr_scale + self.lsnr_offset # [B,T,1]
|
| 168 |
+
|
| 169 |
+
# ERB decoder
|
| 170 |
+
d = self._squeezed_gru(emb, "erb_dec.emb_gru", num_layers=2, has_linear_out=True)
|
| 171 |
+
d = d.reshape(bsz, t, 8, -1) # [B,T,8,64]
|
| 172 |
+
d3 = self._sep_conv(self._pathway(e3, "erb_dec.conv3p") + d, "erb_dec.convt3", 1)
|
| 173 |
+
d2 = self._sep_convt(self._pathway(e2, "erb_dec.conv2p") + d3, "erb_dec.convt2")
|
| 174 |
+
d1 = self._sep_convt(self._pathway(e1, "erb_dec.conv1p") + d2, "erb_dec.convt1")
|
| 175 |
+
m = self._pathway(e0, "erb_dec.conv0p") + d1
|
| 176 |
+
m = mx.conv2d(m, w["erb_dec.conv0_out.conv.weight"], stride=(1, 1), padding=(0, 1))
|
| 177 |
+
erb_mask = mx.sigmoid(m + w["erb_dec.conv0_out.conv.bias"]) # [B,T,32,1]
|
| 178 |
+
|
| 179 |
+
# DF decoder
|
| 180 |
+
c = self._squeezed_gru(emb, "df_dec.df_gru", num_layers=2, has_linear_out=False)
|
| 181 |
+
c = c + grouped_linear(emb, w["df_dec.df_skip.weight"]) # [B,T,256]
|
| 182 |
+
cp = mx.pad(c0, [(0, 0), (4, 0), (0, 0), (0, 0)])
|
| 183 |
+
cp = mx.conv2d(cp, w["df_dec.df_convp.conv.weight"], stride=(1, 1), padding=(0, 0), groups=2)
|
| 184 |
+
cp = mx.conv2d(cp, w["df_dec.df_convp.pw.weight"], stride=(1, 1), padding=(0, 0))
|
| 185 |
+
cp = _relu(cp + w["df_dec.df_convp.pw.bias"]) # [B,T,96,10]
|
| 186 |
+
cf = mx.tanh(grouped_linear(c, w["df_dec.df_out.weight"])) # [B,T,960]
|
| 187 |
+
cf = cf.reshape(bsz, t, 96, 10) + cp
|
| 188 |
+
df_coefs = cf.reshape(bsz, t, 96, 5, 2).transpose(0, 3, 1, 2, 4) # [B,5,T,96,2]
|
| 189 |
+
|
| 190 |
+
return erb_mask, df_coefs, lsnr
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:515656d3689d176fc9e2c6013699baab698ea70cc259aa1ae81ee3ae6f5cdc2e
|
| 3 |
+
size 8533736
|