File size: 2,374 Bytes
59b0ccc |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 |
---
license: mit
tags:
- mlx
- audio
- pitch-estimation
- f0
- voice-conversion
- rvc
- apple-silicon
library_name: mlx
pipeline_tag: audio-to-audio
---
# MLX-RMVPE
MLX implementation of [RMVPE](https://arxiv.org/abs/2306.15412) (Robust Model for Vocal Pitch Estimation) for Apple Silicon.
## Model Description
RMVPE extracts **fundamental frequency (F0)** from audio, essential for preserving pitch/melody in voice conversion. Unlike simpler methods (CREPE, pYIN), RMVPE is specifically designed for **polyphonic music**, making it ideal for singing voice conversion where background music may be present.
- **Architecture**: Deep U-Net with BiGRU layers
- **Parameters**: ~15.4M
- **Input**: 16kHz audio
- **Output**: F0 in Hz at 100fps (hop_length=160)
- **Pitch range**: ~32 Hz to ~1975 Hz (360 bins)
## Usage
```bash
pip install mlx-rmvpe
```
```python
import librosa
from mlx_rmvpe import RMVPE
# Load model (auto-downloads weights)
model = RMVPE.from_pretrained()
# Load audio at 16kHz
audio, sr = librosa.load("singing.wav", sr=16000, mono=True)
# Extract F0
f0 = model.infer_from_audio(audio)
print(f"F0 shape: {f0.shape} at 100fps")
print(f"Pitch range: {f0[f0 > 0].min():.1f} - {f0[f0 > 0].max():.1f} Hz")
```
## Manual Loading
```python
from huggingface_hub import hf_hub_download
from mlx_rmvpe import RMVPE
weights_path = hf_hub_download(
repo_id="lexandstuff/mlx-rmvpe",
filename="rmvpe.safetensors"
)
model = RMVPE()
model.load_weights(weights_path)
model.eval()
```
## Technical Details
This implementation is converted from the PyTorch weights and produces numerically similar outputs:
| Metric | Value |
|--------|-------|
| Mean F0 difference | 1.29 Hz |
| Correlation | >0.99 |
See the [GitHub repository](https://github.com/lexandstuff/mlx-rmvpe) for implementation details and the full API reference.
## Citation
```bibtex
@inproceedings{wei2023rmvpe,
title={RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music},
author={Wei, Yongmao and others},
booktitle={ISMIR},
year={2023}
}
```
## License
MIT
## Acknowledgments
- [RMVPE](https://github.com/Dream-High/RMVPE) - Original implementation
- [RVC](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI) - Voice conversion pipeline
- [MLX](https://github.com/ml-explore/mlx) - Apple's machine learning framework
|