Instructions to use moonbox-dev/FRCRN_SE_16K-MLX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use moonbox-dev/FRCRN_SE_16K-MLX with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download moonbox-dev/FRCRN_SE_16K-MLX --local-dir FRCRN_SE_16K-MLX
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Model card and the original license
Browse files
README.md
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
base_model: alibabasglab/FRCRN_SE_16K
|
| 3 |
+
base_model_relation: quantized
|
| 4 |
+
license: apache-2.0
|
| 5 |
+
pipeline_tag: audio-to-audio
|
| 6 |
+
library_name: mlx
|
| 7 |
+
tags:
|
| 8 |
+
- mlx
|
| 9 |
+
- moonbox
|
| 10 |
+
- mlx
|
| 11 |
+
- frcrn
|
| 12 |
+
- clearervoice
|
| 13 |
+
- speech-enhancement
|
| 14 |
+
- denoising
|
| 15 |
+
- 16khz
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
# FRCRN MLX
|
| 19 |
+
|
| 20 |
+
MLX builds of [FRCRN](https://huggingface.co/alibabasglab/FRCRN_SE_16K), made by Moonbox for Apple Silicon Macs. Each precision is a folder; keep a folder's
|
| 21 |
+
files together. Only the parts listed below are converted: the rest of the pipeline comes from the original
|
| 22 |
+
repository unless the section below says otherwise.
|
| 23 |
+
|
| 24 |
+
## Files
|
| 25 |
+
|
| 26 |
+
The last column is how far each build is from the original PyTorch model on the same inputs (see the section below
|
| 27 |
+
for what was measured).
|
| 28 |
+
|
| 29 |
+
| Precision | File | Size | sha256 | Checked against the original |
|
| 30 |
+
|---|---|---|---|---|
|
| 31 |
+
| bf16 | [`bf16/config.json`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/bf16/config.json) | 0.1 kB | `86f0962100d6` | below threshold (experimental): SI-SDR 9.66 dB, DNSMOS 3.06; output vs the original's 16.9 / 2.9 dB |
|
| 32 |
+
| bf16 | [`bf16/mlx_model.py`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/bf16/mlx_model.py) | 12.3 kB | `307afe87cb76` | below threshold (experimental): SI-SDR 9.66 dB, DNSMOS 3.06; output vs the original's 16.9 / 2.9 dB |
|
| 33 |
+
| bf16 | [`bf16/model.safetensors`](https://huggingface.co/moonbox-dev/FRCRN_SE_16K-MLX/blob/main/bf16/model.safetensors) | 29.7 MB | `4f4e44804287` | below threshold (experimental): SI-SDR 9.66 dB, DNSMOS 3.06; output vs the original's 16.9 / 2.9 dB |
|
| 34 |
+
|
| 35 |
+
## How to run
|
| 36 |
+
|
| 37 |
+
FRCRN SE 16K (ClearerVoice-Studio's speech enhancement for 16 kHz speech: two complex-valued UNets with FSMN memory
|
| 38 |
+
and squeeze-excitation layers on a 640-point STFT) ported to MLX. Each folder holds `model.safetensors`, `config.json`
|
| 39 |
+
and `mlx_model.py` (the port). Needs `mlx`, `numpy`, `soundfile` (and `librosa` for input that isn't 16 kHz):
|
| 40 |
+
|
| 41 |
+
```python
|
| 42 |
+
import os; os.environ["MLX_ENABLE_TF32"] = "0" # float32 matmuls (MLX uses TF32 otherwise)
|
| 43 |
+
import sys; sys.path.insert(0, "fp32") # a downloaded folder
|
| 44 |
+
import mlx_model, soundfile as sf
|
| 45 |
+
|
| 46 |
+
model = mlx_model.load("fp32")
|
| 47 |
+
clean = mlx_model.enhance(model, "noisy.wav") # float32, 16 kHz, the input's length
|
| 48 |
+
sf.write("clean.wav", clean, 16000)
|
| 49 |
+
```
|
| 50 |
+
|
| 51 |
+
Like ClearerVoice, `enhance` normalizes the input's loudness first and scales the result back, and decodes inputs
|
| 52 |
+
over 120 s in 1 s windows. **Use `fp32/`.** The network is unusually sensitive to its weights (its BatchNorm statistics
|
| 53 |
+
reach 3e5): every folder computes in float32, but storing the weights in 16 bits (`bf16/`, `fp16/`) or quantizing the
|
| 54 |
+
FSMN layers (`8bit/`, `6bit/`, `4bit/`, on bf16 storage) already moves the output far enough to miss the threshold, so
|
| 55 |
+
those are marked experimental. BatchNorm parameters and the STFT kernels are float32 in every folder.
|
| 56 |
+
|
| 57 |
+
**How they were checked.** 8 noisy 16 kHz clips (LibriSpeech speech with babble or pink noise, 0-10 dB SNR), enhanced
|
| 58 |
+
by the original (ClearerVoice's `clearvoice` 0.1.2, PyTorch float32 on the CPU) and by every MLX build: SI-SDR against
|
| 59 |
+
the clean speech, DNSMOS P.835, and how close each output is to the original's. Pass: SI-SDR on every clip at most
|
| 60 |
+
0.5 dB below the original's, mean DNSMOS at most 0.1 below.
|
| 61 |
+
|
| 62 |
+
| Precision | Size | SI-SDR (dB) | DNSMOS | Output vs the original's, mean / worst clip (dB) |
|
| 63 |
+
|---|---|---|---|---|
|
| 64 |
+
| noisy input | - | 4.98 | 2.04 | - |
|
| 65 |
+
| original (PyTorch fp32, CPU) | - | 10.30 | 3.19 | - |
|
| 66 |
+
| `fp32/` | 56 MB | 10.30 | 3.19 | 89.3 / 79.2 |
|
| 67 |
+
| `fp16/` (experimental) | 30 MB | 10.31 (1.2 lower on one clip) | 3.16 | 25.2 / 8.2 |
|
| 68 |
+
| `bf16/` (experimental) | 30 MB | 9.66 | 3.06 | 16.9 / 2.9 |
|
| 69 |
+
| `8bit/` (experimental) | 28 MB | 9.88 | 3.07 | 18.0 / 4.1 |
|
| 70 |
+
| `6bit/` (experimental) | 27 MB | 10.18 | 3.04 | 13.8 / 3.0 |
|
| 71 |
+
| `4bit/` (experimental) | 27 MB | 4.92 | 2.25 | 4.7 / -1.3 |
|
| 72 |
+
|
| 73 |
+
About 10x faster than real time on an Apple M5 (the original on the CPU: 0.6x).
|
| 74 |
+
|
| 75 |
+
## How these were made
|
| 76 |
+
|
| 77 |
+
From [FRCRN](https://huggingface.co/alibabasglab/FRCRN_SE_16K) at revision `3766e6a64b`: the original weights loaded in MLX and stored as bf16 or fp16, or
|
| 78 |
+
quantized with MLX's affine quantization (`8bit`, `6bit`, `4bit`, group size 64; layers MLX can't quantize, such as
|
| 79 |
+
convolutions and norms, stay bf16). Every build was checked against the original before upload; the `sha256` column
|
| 80 |
+
above is what the files must hash to.
|
| 81 |
+
|
| 82 |
+
## License
|
| 83 |
+
|
| 84 |
+
Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this
|
| 85 |
+
repository, and it governs your use of these files. The original model card follows unchanged.
|
| 86 |
+
|
| 87 |
+
---
|
| 88 |
+
|
| 89 |
+
# Original model card
|
| 90 |
+
|
| 91 |
+
The FRCRN_SE_16K model weights for 16 kHz speech enhancement in [ClearerVoice-Studio](https://github.com/modelscope/ClearerVoice-Studio/tree/main) repo.
|
| 92 |
+
|
| 93 |
+
This model is trained on large scale datasets inclduing open-sourced and private data.
|
| 94 |
+
|
| 95 |
+
It enhances speech audios by removing background noise.
|