FRCRN_SE_16K-MLX / README.md
jennyxue's picture
Model card and the original license
221cc45 verified
|
Raw History Blame Contribute Delete
8.77 kB
---
base_model: alibabasglab/FRCRN_SE_16K
base_model_relation: quantized
license: apache-2.0
pipeline_tag: audio-to-audio
library_name: mlx
tags:
- mlx
- moonbox
- mlx
- frcrn
- clearervoice
- speech-enhancement
- denoising
- 16khz
---
# FRCRN MLX
MLX builds of [FRCRN](https://huggingface.co/alibabasglab/FRCRN_SE_16K), made by Moonbox for Apple Silicon Macs. Each precision is a folder; keep a folder's
files together. Only the parts listed below are converted: the rest of the pipeline comes from the original
repository unless the section below says otherwise.
## Files
The last column is how far each build is from the original PyTorch model on the same inputs (see the section below
for what was measured).
| Precision | File | Size | sha256 | Checked against the original |
|---|---|---|---|---|
| bf16 | [`bf16/config.json`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/bf16/config.json) | 0.1 kB | `86f0962100d6` | below threshold (experimental): SI-SDR 9.66 dB, DNSMOS 3.06; output vs the original's 16.9 / 2.9 dB |
| bf16 | [`bf16/mlx_model.py`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/bf16/mlx_model.py) | 12.3 kB | `307afe87cb76` | below threshold (experimental): SI-SDR 9.66 dB, DNSMOS 3.06; output vs the original's 16.9 / 2.9 dB |
| bf16 | [`bf16/model.safetensors`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/bf16/model.safetensors) | 29.7 MB | `4f4e44804287` | below threshold (experimental): SI-SDR 9.66 dB, DNSMOS 3.06; output vs the original's 16.9 / 2.9 dB |
| fp16 | [`fp16/config.json`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/fp16/config.json) | 0.1 kB | `86f0962100d6` | below threshold (experimental): SI-SDR 10.31 dB mean but 0.6-1.2 dB lower on two clips, DNSMOS 3.16; output vs the original's 25.2 / 8.2 dB |
| fp16 | [`fp16/mlx_model.py`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/fp16/mlx_model.py) | 12.3 kB | `307afe87cb76` | below threshold (experimental): SI-SDR 10.31 dB mean but 0.6-1.2 dB lower on two clips, DNSMOS 3.16; output vs the original's 25.2 / 8.2 dB |
| fp16 | [`fp16/model.safetensors`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/fp16/model.safetensors) | 29.7 MB | `8b4402088aa8` | below threshold (experimental): SI-SDR 10.31 dB mean but 0.6-1.2 dB lower on two clips, DNSMOS 3.16; output vs the original's 25.2 / 8.2 dB |
| fp32 | [`fp32/config.json`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/fp32/config.json) | 0.1 kB | `86f0962100d6` | SI-SDR 10.30 dB (original 10.30), DNSMOS 3.19 (3.19); output vs the original's 89.3 dB mean / 79.2 dB worst clip |
| fp32 | [`fp32/mlx_model.py`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/fp32/mlx_model.py) | 12.3 kB | `307afe87cb76` | SI-SDR 10.30 dB (original 10.30), DNSMOS 3.19 (3.19); output vs the original's 89.3 dB mean / 79.2 dB worst clip |
| fp32 | [`fp32/model.safetensors`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/fp32/model.safetensors) | 56.0 MB | `a8338a1874a5` | SI-SDR 10.30 dB (original 10.30), DNSMOS 3.19 (3.19); output vs the original's 89.3 dB mean / 79.2 dB worst clip |
| 8bit | [`8bit/config.json`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/8bit/config.json) | 0.2 kB | `18aa7e9b7dd2` | below threshold (experimental): SI-SDR 9.88 dB, DNSMOS 3.07; output vs the original's 18.0 / 4.1 dB |
| 8bit | [`8bit/mlx_model.py`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/8bit/mlx_model.py) | 12.3 kB | `307afe87cb76` | below threshold (experimental): SI-SDR 9.88 dB, DNSMOS 3.07; output vs the original's 18.0 / 4.1 dB |
| 8bit | [`8bit/model.safetensors`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/8bit/model.safetensors) | 27.7 MB | `88af0c96d104` | below threshold (experimental): SI-SDR 9.88 dB, DNSMOS 3.07; output vs the original's 18.0 / 4.1 dB |
| 6bit | [`6bit/config.json`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/6bit/config.json) | 0.2 kB | `d3a700755964` | below threshold (experimental): SI-SDR 10.18 dB, DNSMOS 3.04; output vs the original's 13.8 / 3.0 dB |
| 6bit | [`6bit/mlx_model.py`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/6bit/mlx_model.py) | 12.3 kB | `307afe87cb76` | below threshold (experimental): SI-SDR 10.18 dB, DNSMOS 3.04; output vs the original's 13.8 / 3.0 dB |
| 6bit | [`6bit/model.safetensors`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/6bit/model.safetensors) | 27.2 MB | `7bccd20b2f7a` | below threshold (experimental): SI-SDR 10.18 dB, DNSMOS 3.04; output vs the original's 13.8 / 3.0 dB |
| 4bit | [`4bit/config.json`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/4bit/config.json) | 0.2 kB | `c322f3218326` | below threshold (experimental): SI-SDR 4.92 dB, DNSMOS 2.25; output vs the original's 4.7 / -1.3 dB |
| 4bit | [`4bit/mlx_model.py`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/4bit/mlx_model.py) | 12.3 kB | `307afe87cb76` | below threshold (experimental): SI-SDR 4.92 dB, DNSMOS 2.25; output vs the original's 4.7 / -1.3 dB |
| 4bit | [`4bit/model.safetensors`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/4bit/model.safetensors) | 26.6 MB | `b343da5b02df` | below threshold (experimental): SI-SDR 4.92 dB, DNSMOS 2.25; output vs the original's 4.7 / -1.3 dB |
## How to run
FRCRN SE 16K (ClearerVoice-Studio's speech enhancement for 16 kHz speech: two complex-valued UNets with FSMN memory
and squeeze-excitation layers on a 640-point STFT) ported to MLX. Each folder holds `model.safetensors`, `config.json`
and `mlx_model.py` (the port). Needs `mlx`, `numpy`, `soundfile` (and `librosa` for input that isn't 16 kHz):
```python
import os; os.environ["MLX_ENABLE_TF32"] = "0" # float32 matmuls (MLX uses TF32 otherwise)
import sys; sys.path.insert(0, "fp32") # a downloaded folder
import mlx_model, soundfile as sf
model = mlx_model.load("fp32")
clean = mlx_model.enhance(model, "noisy.wav") # float32, 16 kHz, the input's length
sf.write("clean.wav", clean, 16000)
```
Like ClearerVoice, `enhance` normalizes the input's loudness first and scales the result back, and decodes inputs
over 120 s in 1 s windows. **Use `fp32/`.** The network is unusually sensitive to its weights (its BatchNorm statistics
reach 3e5): every folder computes in float32, but storing the weights in 16 bits (`bf16/`, `fp16/`) or quantizing the
FSMN layers (`8bit/`, `6bit/`, `4bit/`, on bf16 storage) already moves the output far enough to miss the threshold, so
those are marked experimental. BatchNorm parameters and the STFT kernels are float32 in every folder.
**How they were checked.** 8 noisy 16 kHz clips (LibriSpeech speech with babble or pink noise, 0-10 dB SNR), enhanced
by the original (ClearerVoice's `clearvoice` 0.1.2, PyTorch float32 on the CPU) and by every MLX build: SI-SDR against
the clean speech, DNSMOS P.835, and how close each output is to the original's. Pass: SI-SDR on every clip at most
0.5 dB below the original's, mean DNSMOS at most 0.1 below.
| Precision | Size | SI-SDR (dB) | DNSMOS | Output vs the original's, mean / worst clip (dB) |
|---|---|---|---|---|
| noisy input | - | 4.98 | 2.04 | - |
| original (PyTorch fp32, CPU) | - | 10.30 | 3.19 | - |
| `fp32/` | 56 MB | 10.30 | 3.19 | 89.3 / 79.2 |
| `fp16/` (experimental) | 30 MB | 10.31 (1.2 lower on one clip) | 3.16 | 25.2 / 8.2 |
| `bf16/` (experimental) | 30 MB | 9.66 | 3.06 | 16.9 / 2.9 |
| `8bit/` (experimental) | 28 MB | 9.88 | 3.07 | 18.0 / 4.1 |
| `6bit/` (experimental) | 27 MB | 10.18 | 3.04 | 13.8 / 3.0 |
| `4bit/` (experimental) | 27 MB | 4.92 | 2.25 | 4.7 / -1.3 |
About 10x faster than real time on an Apple M5 (the original on the CPU: 0.6x).
## How these were made
From [FRCRN](https://huggingface.co/alibabasglab/FRCRN_SE_16K) at revision `3766e6a64b`: the original weights loaded in MLX and stored as bf16 or fp16, or
quantized with MLX's affine quantization (`8bit`, `6bit`, `4bit`, group size 64; layers MLX can't quantize, such as
convolutions and norms, stay bf16). Every build was checked against the original before upload; the `sha256` column
above is what the files must hash to.
## License
Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this
repository, and it governs your use of these files. The original model card follows unchanged.
---
# Original model card
The FRCRN_SE_16K model weights for 16 kHz speech enhancement in [ClearerVoice-Studio](https://github.com/modelscope/ClearerVoice-Studio/tree/main) repo.
This model is trained on large scale datasets inclduing open-sourced and private data.
It enhances speech audios by removing background noise.