| --- |
| license: other |
| license_name: mit-and-apache-2.0 |
| license_link: https://opensource.org/licenses/MIT |
| base_model: |
| - pyannote/segmentation-3.0 |
| base_model_relation: quantized |
| pipeline_tag: voice-activity-detection |
| library_name: onnx |
| language: |
| - zh |
| - en |
| tags: |
| - speaker-diarization |
| - pyannote |
| - wespeaker |
| - onnx |
| - 42model |
| --- |
| |
| <!-- |
| README for the Hugging Face repo `42ailab/Speaker-Diarization-Models`, uploaded with the diarization model package. |
| Editing this file changes the repo landing page. Multi-source models: pyannote segmentation (MIT) + WeSpeaker embeddings (Apache-2.0). |
| --> |
| # Speaker Diarization Models · Who Spoke When (Fully Local) |
|
|
| [](https://opensource.org/licenses/MIT) |
| [](https://42model.com) |
| [](https://42model.com) |
| [](https://modelscope.cn/models/42ailab/Speaker-Diarization-Models) |
|
|
| **English** | [简体中文](./README_zh.md) · Also on [ModelScope](https://modelscope.cn/models/42ailab/Speaker-Diarization-Models) · [42ailab.com](https://42ailab.com) |
|
|
| Give it a multi-speaker conversation recording and this model marks **who said each segment** (Speaker 1, Speaker 2, …) — entirely on your own computer. **No cloud, free, private.** |
|
|
| > **The models were built by two open-source teams**: [pyannote](https://github.com/pyannote/pyannote-audio) (speech segmentation, MIT) and [WeSpeaker](https://github.com/wenet-e2e/wespeaker) (speaker embeddings, Apache-2.0); the ONNX deployment packaging follows [avencera/speakrs](https://github.com/avencera/speakrs) (Apache-2.0). **This repo is not a new model** — we converted these open models into deployment builds that **run offline on your computer** and verified their quality. It is an **enhancement model** and does not transcribe: transcription is still done by [Qwen3-ASR](https://huggingface.co/Qwen/Qwen3-ASR-0.6B), and this model adds the "who is speaking" information on top. |
|
|
| ## 1. The problem it solves |
|
|
| Speech-to-text tells you "what was said", but in a multi-speaker conversation it often cannot tell you **who said this line**. Many scenarios depend precisely on that distinction: |
|
|
| - **Meeting minutes**: organize notes by speaker so each party's position is visible at a glance; |
| - **Interviews and podcasts**: separate host from guest, making transcripts and quotations easier to prepare; |
| - **Customer service and QA**: distinguish agent from customer and analyze each separately. |
|
|
| Doing speaker diarization used to mean uploading the recording to a cloud service — with both privacy concerns and possible costs. This model keeps that step **entirely local**. |
|
|
| ## 2. How it works |
|
|
| It chains three steps into a "who spoke when" pipeline: |
|
|
| - **Segmentation** (pyannote): first find which stretches of audio contain speech, and where the speaker changes; |
| - **Speaker embeddings** (WeSpeaker): extract a "voiceprint" vector representing the voice characteristics of each short segment; |
| - **Clustering**: group segments with similar voiceprints as the same person, and finally attach a speaker label to each segment. |
|
|
| **What we did**: converted the upstream open weights into locally runnable deployment builds, verified that quality did not degrade in conversion, and made them **run offline** across macOS / Windows / Linux; we also selected the right acceleration path for Apple silicon (CoreML), NVIDIA GPUs (CUDA) and **plain CPU**, so machines without a discrete GPU can use it too. |
|
|
| ## 3. How well it performs |
|
|
| For the complete diarization-quality benchmarks (DER, lower is better), refer to upstream public evaluations — per [speakrs benchmarks](https://github.com/avencera/speakrs/tree/master/benchmarks), this pipeline (CoreML) achieves about **7.1% DER** on the VoxConverse **Dev** subset, essentially on par with pyannote community-1 (about 7.2%). Here we list only **our own verification**: |
|
|
| | What we verified | Result | |
| |---|---| |
| | Correctness on a Chinese two-person conversation (5-minute real recording) | **Correctly separated 2 speakers**, with sensible speaker-change points | |
| | Speed · Apple silicon (CoreML) | **~380× real time** (5 minutes of audio in about 0.8 s) | |
| | Speed · plain CPU | **~1.9× real time** (5 minutes of audio in about 2.6 minutes), usable on machines with no GPU | |
| | Memory use (plain CPU) | About 1.2 GB | |
|
|
| > Note: "correctness", "speed" and "memory" above are our own measurements on an Apple M3 Max; "7.1% DER (VoxConverse Dev)" is a public figure from upstream **speakrs** (a third-party CoreML port's evaluation — neither pyannote official nor measured by us). |
|
|
| ## 4. Limitations and what's next |
|
|
| - **Requires a transcription model**: this model only answers "who is speaking" and does not transcribe; use it together with Qwen3-ASR. |
| - **Plain-CPU speed**: extremely fast with Apple / NVIDIA acceleration; about 1.9× real time on plain CPU — usable, but clearly slower than the accelerated tiers, so process long recordings on an accelerated machine where possible. |
| - **Overlapping speech**: where two people talk at once, speaker attribution may be imprecise (a shared difficulty of speaker diarization). |
| - Next: keep improving plain-CPU speed and handling of overlapping segments. |
|
|
| ## 5. How to download and use it |
|
|
| This model is packaged for [42model](https://42model.com), which is the recommended way to get it: |
|
|
| **Desktop app** |
| 1. Open **Model Library → Transcription** and download **Qwen3-ASR Enhanced** (speaker diarization is built in); |
| 2. When transcribing, turn on **Speaker diarization** in the parameter settings, and results will be segmented by speaker. |
|
|
| ## Files and license |
|
|
| | File | Role | Source · License | |
| |---|---|---| |
| | `segmentation-3.0.onnx` | Speech segmentation / speaker-change detection | pyannote/segmentation-3.0 · **MIT** | |
| | `wespeaker-voxceleb-resnet34.onnx` | Speaker embedding extraction | WeSpeaker voxceleb ResNet34 · **Apache-2.0** | |
| | `plda_lda.npy`, `plda_tr.npy`, `plda_mu.npy`, `plda_psi.npy`, `plda_mean1.npy`, `plda_mean2.npy` | Voiceprint clustering parameters (PLDA / VBx, 6 files) | speakrs pipeline · Apache-2.0 | |
| | `manifest.json` | sha256 / size listing for each file (generated at release, for integrity self-verification) | Generated by this repo | |
|
|
| Per-file sha256 values are in the bundled `manifest.json` and on the Files page, and can be verified independently. CoreML acceleration on Apple silicon is compiled automatically on first local run — no extra files needed. |
|
|
| **License**: this repo contains local deployment builds of open models from several sources — speech segmentation is [pyannote/segmentation-3.0](https://huggingface.co/pyannote/segmentation-3.0) (© pyannote.audio, **MIT**; the upstream HF page requires accepting terms for access, but the license is MIT and redistribution is unrestricted — the ONNX here comes from the same weights under the same license); speaker embeddings are [WeSpeaker](https://github.com/wenet-e2e/wespeaker) voxceleb ResNet34 (© the WeNet community, **Apache-2.0**); the ONNX packaging and pipeline follow [avencera/speakrs](https://github.com/avencera/speakrs) (Apache-2.0). Each file is governed by its original license; by using them you agree to the respective upstream license terms. |
|
|
| ## Citation |
|
|
| For **the models themselves**, please cite upstream pyannote and WeSpeaker: |
|
|
| ```bibtex |
| @inproceedings{Plaquet23, |
| author = {Alexis Plaquet and Herv{\'e} Bredin}, |
| title = {{Powerset multi-class cross entropy loss for neural speaker diarization}}, |
| booktitle = {Proc. INTERSPEECH 2023}, |
| year = {2023}, |
| } |
| @inproceedings{Bredin23, |
| author = {Herv{\'e} Bredin}, |
| title = {{pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe}}, |
| booktitle = {Proc. INTERSPEECH 2023}, |
| year = {2023}, |
| } |
| @inproceedings{wang2023wespeaker, |
| author = {Wang, Hongji and Liang, Chengdong and Wang, Shuai and Chen, Zhengyang and Zhang, Binbin and Xiang, Xu and Deng, Yanlei and Qian, Yanmin}, |
| title = {{Wespeaker: A research and production oriented speaker embedding learning toolkit}}, |
| booktitle = {ICASSP 2023}, |
| year = {2023}, |
| } |
| ``` |
|
|
| If **this repo's local deployment packaging** was useful to you, you may additionally cite: |
|
|
| ```bibtex |
| @misc{yang2026diarizationlocal, |
| title = {Speaker Diarization Models: A Local Build of pyannote + WeSpeaker for On-Device Diarization}, |
| author = {Yang, Zhiping}, |
| year = {2026}, |
| howpublished = {\url{https://huggingface.co/42ailab/Speaker-Diarization-Models}}, |
| organization = {42ailab}, |
| note = {Local deployment packaging (ONNX + cross-platform acceleration); the models themselves are pyannote/segmentation-3.0 (MIT) and WeSpeaker voxceleb ResNet34 (Apache-2.0). Contact: contact@42ailab.com} |
| } |
| ``` |
|
|
| Contact us: **contact@42ailab.com** |
|
|
| ## About us |
|
|
| **[42ailab](https://42ailab.com)** — an AI research lab exploring the boundaries of intelligence. Grounded in cognitive science, we work toward a deep integration of AI and human intelligence — to truly understand and augment intelligence, carbon-based and silicon-based alike. |
|
|
| **[42model](https://42model.com)** — a high-performance local inference engine from 42ailab that runs translation, transcription, recognition, chat and coding on your own machine, free and private; with optional cloud compute for fine-tuning your own models and bringing them back to run locally. |
|
|
