| | --- |
| | license: mit |
| | tags: |
| | - mlx |
| | - voice-activity-detection |
| | - speaker-segmentation |
| | - speaker-diarization |
| | - pyannote |
| | - apple-silicon |
| | base_model: pyannote/segmentation-3.0 |
| | library_name: mlx |
| | pipeline_tag: voice-activity-detection |
| | --- |
| | |
| | # Pyannote Segmentation 3.0 — MLX |
| |
|
| | MLX-compatible weights for [pyannote/segmentation-3.0](https://huggingface.co/pyannote/segmentation-3.0) (PyanNet), converted from the official PyTorch Lightning checkpoint with pre-computed SincNet filters. |
| |
|
| | ## Model |
| |
|
| | PyanNet is a speaker segmentation model (~1.5M params) that processes 10-second audio windows and outputs 7-class powerset probabilities for up to 3 simultaneous speakers. Used for both voice activity detection (binary) and speaker diarization (per-speaker). |
| |
|
| | **Architecture:** SincNet → BiLSTM(4 layers) → Linear(2 layers) → 7-class softmax |
| |
|
| | **Output classes:** non-speech, spk1, spk2, spk3, spk1+2, spk1+3, spk2+3 |
| |
|
| | ## Usage (Swift / MLX) |
| |
|
| | ```swift |
| | import SpeechVAD |
| | |
| | // Voice Activity Detection |
| | let vad = try await PyannoteVADModel.fromPretrained() |
| | let segments = vad.detectSpeech(audio: samples, sampleRate: 16000) |
| | for seg in segments { |
| | print("Speech: \(seg.startTime)s - \(seg.endTime)s") |
| | } |
| | |
| | // Speaker Diarization (with WeSpeaker embeddings) |
| | let pipeline = try await DiarizationPipeline.fromPretrained() |
| | let result = pipeline.diarize(audio: samples, sampleRate: 16000) |
| | for seg in result.segments { |
| | print("Speaker \(seg.speakerId): \(seg.startTime)s - \(seg.endTime)s") |
| | } |
| | ``` |
| |
|
| | Part of [qwen3-asr-swift](https://github.com/ivan-digital/qwen3-asr-swift). |
| |
|
| | ## Conversion |
| |
|
| | ```bash |
| | python3 scripts/convert_pyannote.py --token YOUR_HF_TOKEN --upload |
| | ``` |
| |
|
| | Converts the gated pyannote/segmentation-3.0 checkpoint using a custom unpickler (no pyannote.audio dependency required). Key transformations: |
| |
|
| | - **SincNet**: pre-compute 80 sinc bandpass filters (40 cos + 40 sin) from 40 learned `(low_hz, band_hz)` parameter pairs |
| | - **Conv1d**: transpose weights `[O, I, K]` → `[O, K, I]` for MLX channels-last |
| | - **BiLSTM**: split into forward/backward stacks, sum `bias_ih + bias_hh` |
| | - **Linear/classifier**: kept as-is |
| |
|
| | ## Weight Mapping |
| |
|
| | | PyTorch Key | MLX Key | Shape | |
| | |-------------|---------|-------| |
| | | `sincnet.conv1d.0.filterbank.*` (computed) | `sincnet.conv.0.weight` | [80, 251, 1] | |
| | | `sincnet.conv1d.{1,2}.weight` | `sincnet.conv.{1,2}.weight` | [O, K, I] | |
| | | `sincnet.norm1d.{0-2}.*` | `sincnet.norm.{0-2}.*` | varies | |
| | | `lstm.weight_ih_l{i}` | `lstm_fwd.layers.{i}.Wx` | [512, I] | |
| | | `lstm.weight_hh_l{i}` | `lstm_fwd.layers.{i}.Wh` | [512, 128] | |
| | | `lstm.bias_ih_l{i} + bias_hh_l{i}` | `lstm_fwd.layers.{i}.bias` | [512] | |
| | | `lstm.*_reverse` | `lstm_bwd.layers.{i}.*` | same | |
| | | `linear.{0,1}.*` | `linear.{0,1}.*` | varies | |
| | | `classifier.*` | `classifier.*` | [7, 128] | |
| |
|
| | ## License |
| |
|
| | The original pyannote segmentation model is released under the [MIT License](https://github.com/pyannote/pyannote-audio/blob/develop/LICENSE). |
| |
|
