vboguckis's picture
TextSniper's copy of FluidInference/speaker-diarization-coreml at 7eab74f
114d3ec verified
|
Raw History Blame Contribute Delete
3.52 kB
metadata
license: cc-by-4.0
pipeline_tag: voice-activity-detection
library_name: coreml
tags:
  - speaker-diarization
  - coreml
  - pyannote
base_model: pyannote/speaker-diarization-community-1

Speaker diarization (pyannote Community-1), Core ML

Fluid Inference's Core ML build of pyannote's speaker-diarization-community-1: the powerset segmentation model, the WeSpeaker ResNet34 speaker embedding model with its FBank frontend, and BUT Speech@FIT's PLDA. MacEpic hosts this copy so TextSniper can download it at a fixed revision; nothing here is MacEpic's own work, and no file of the model was changed.

Source

FluidInference/speaker-diarization-coreml at commit 7eab74f3463c736418d0eb10296261ae581146c9. These 21 files are byte for byte the same as there (21,602,787 bytes):

  • Segmentation.mlmodelc, FBank.mlmodelc, Embedding.mlmodelc: the compiled Core ML models.
  • plda-parameters.json, xvector-transform.json: BUT Speech@FIT's PLDA as Float32 tensors.
  • LICENSE, NOTICE.md, PROVENANCE.md, provenance.json: the license, its scope and attribution, and where each file comes from.

Changes from the source, as CC BY 4.0 asks them to be noted: PLDA.mlmodelc, PldaRho.mlmodelc, the legacy models, mlpackages/, the plots, config.json and Fluid Inference's README were left out, and this README and bundle.json (every file with its size and SHA-256) were added.

License and attribution

CC BY 4.0, as the source; the full text is in LICENSE, and NOTICE.md sets out which files it covers.

The files are modified Core ML conversions of:

  • pyannote's speaker-diarization-community-1 pipeline (CNRS, pyannoteAI), CC BY 4.0;
  • WeSpeaker's VoxCeleb ResNet34 speaker embedding model, through pyannote, CC BY 4.0;
  • Brno University of Technology / BUT Speech@FIT's PLDA (plda.npz, xvec_transform.npz), CC BY 4.0;

converted to Core ML by Fluid Inference, which introduced fixed and enumerated input shapes, mixed-precision storage, a separate FBank frontend, and compiled the models for Apple platforms.

Citations

Speaker segmentation

@inproceedings{Plaquet23,
  author={Alexis Plaquet and Hervé Bredin},
  title={{Powerset multi-class cross entropy loss for neural speaker diarization}},
  year=2023,
  booktitle={Proc. INTERSPEECH 2023},
}

Speaker embedding

@inproceedings{Wang2023,
  title={Wespeaker: A research and production oriented speaker embedding learning toolkit},
  author={Wang, Hongji and Liang, Chengdong and Wang, Shuai and Chen, Zhengyang and Zhang, Binbin and Xiang, Xu and Deng, Yanlei and Qian, Yanmin},
  booktitle={ICASSP 2023, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={1--5},
  year={2023},
  organization={IEEE}
}

Speaker clustering

@article{Landini2022,
  author={Landini, Federico and Profant, J{\'a}n and Diez, Mireia and Burget, Luk{\'a}{\v{s}}},
  title={{Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks}},
  year={2022},
  journal={Computer Speech & Language},
}