Hugging Face's logo

lexandstuff
/
mlx-contentvec

Feature Extraction
MLX
English
audio
speech
contentvec
hubert
voice-conversion
rvc
Model card Files
xet
Community
mlx-contentvec
File size: 2,845 Bytes 
6c6cd43
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
 
---
license: mit
library_name: mlx
tags:
  - mlx
  - audio
  - speech
  - feature-extraction
  - contentvec
  - hubert
  - voice-conversion
  - rvc
datasets:
  - librispeech_asr
language:
  - en
pipeline_tag: feature-extraction
---

# MLX ContentVec / HuBERT Base

MLX-converted weights for ContentVec/HuBERT base model, optimized for Apple Silicon.

This model extracts speaker-agnostic semantic features from audio, primarily used as the feature extraction backbone for [RVC (Retrieval-based Voice Conversion)](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI).

## Model Details

- **Architecture**: HuBERT Base (12 transformer layers)
- **Parameters**: ~90M
- **Input**: 16kHz mono audio
- **Output**: 768-dimensional features (~50 frames/second)
- **Framework**: [MLX](https://github.com/ml-explore/mlx)
- **Format**: SafeTensors (float32)

## Usage

```python
import mlx.core as mx
import librosa
from mlx_contentvec import ContentvecModel

# Load model
model = ContentvecModel(encoder_layers_1=0)
model.load_weights("contentvec_base.safetensors")
model.eval()

# Load audio at 16kHz
audio, sr = librosa.load("input.wav", sr=16000, mono=True)
source = mx.array(audio).reshape(1, -1)

# Extract features
result = model(source)
features = result["x"]  # Shape: (1, num_frames, 768)
```

## Installation

```bash
pip install git+https://github.com/example/mlx-contentvec.git
```

## Download Weights

```python
from huggingface_hub import hf_hub_download

weights_path = hf_hub_download(
    repo_id="lexandstuff/mlx-contentvec",
    filename="contentvec_base.safetensors"
)
```

## Validation

These weights produce **numerically identical** outputs to the original PyTorch implementation:

| Metric | Value |
|--------|-------|
| Max absolute difference | 7.3e-6 |
| Cosine similarity | 1.000000 |

## Source Weights

Converted from [hubert_base.pt](https://huggingface.co/lj1995/VoiceConversionWebUI/resolve/main/hubert_base.pt) (MD5: `b76f784c1958d4e535cd0f6151ca35e4`).

## Use Cases

- **Voice Conversion**: Feature extraction for RVC pipeline
- **Speaker Verification**: Content-based audio embeddings
- **Speech Analysis**: Semantic feature extraction

## Citation

```bibtex
@inproceedings{qian2022contentvec,
  title={ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers},
  author={Qian, Kaizhi and Zhang, Yang and Gao, Heting and Ni, Junrui and Lai, Cheng-I and Cox, David and Hasegawa-Johnson, Mark and Chang, Shiyu},
  booktitle={International Conference on Machine Learning},
  year={2022}
}

@article{hsu2021hubert,
  title={HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units},
  author={Hsu, Wei-Ning and others},
  journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},
  year={2021}
}
```

## License

MIT