| --- |
| license: other |
| license_name: mixed-upstream |
| license_link: https://github.com/pyannote/pyannote-audio |
| tags: |
| - coreml |
| - speaker-diarization |
| - pyannote |
| - wespeaker |
| - apple-neural-engine |
| library_name: coremlit |
| --- |
| |
| # speakerkit-coreml — fp16-safe CoreML conversions for speaker diarization |
|
|
| CoreML conversions of the pyannote community-1 diarization pipeline's neural components, |
| re-converted with **fp16-survivable numerical guards** so they are correct on the Apple |
| Neural Engine and GPU — not just CPU. |
|
|
| Consumed by the `speakerkit` crate of the [coremlit](https://github.com/findit-studio/coremlit) |
| workspace (Rust, sync, sans-I/O), feeding the `dia` diarization pipeline's clustering. |
|
|
| ## Why these conversions exist |
|
|
| Stock conversions of these models carry numerical guard epsilons **below fp16's smallest |
| subnormal (`2^-24`)**. When a graph runs on the ANE (which computes in fp16 regardless of |
| the declared dtype), those guards round to zero: |
|
|
| - the segmentation model's `softmax → log(eps≈0)` tail saturates `log(0)` to **≈ −45440**, |
| which in the downstream pipeline collapsed an 8-speaker recording to 5 detected speakers |
| (16.6% DER, 100% confusion) while the ONNX reference was frame-perfect; |
| - the embedder's pooling `real_div` guards (`1e-8`) vanish the same way. |
|
|
| Even a *naive re-export* reproduces the defect: `torch.log_softmax` still lowers to the |
| decomposed `softmax → log(eps=0)` under coremltools 9.0. These conversions use an |
| explicitly **fused `x − logsumexp(x)`** tail and raise every guard constant to `0x1p-24`. |
|
|
| ## Contents |
|
|
| | artifact | form | what it is | |
| |---|---|---| |
| | `pyannote_segmentation.{mlpackage,mlmodelc}` | fp16 | pyannote **segmentation-3.0** (PyanNet), fused log-probability tail | |
| | `wespeaker.{mlpackage,mlmodelc}` | fp32 weights | **WeSpeaker ResNet34-LM** speaker embedder, pooling guards raised to `0x1p-24` (otherwise bit-identical numerics to the stock conversion) | |
| | `wespeaker_int8.{mlpackage,mlmodelc}` | 8-bit palettized | same fixed embedder, k-means per-tensor palettization (~3.7× smaller, the production default) | |
|
|
| `.mlpackage` is the canonical distributable; `.mlmodelc` is the compiled form the test |
| suite consumes directly. `CHECKSUMS.sha256` covers every file. |
|
|
| ## Verification (summary) |
|
|
| - Every graph passes a **MIL-level fp16-guard audit** (every guard's *effective floor* |
| ≥ `2^-24`, constants followed through `cast`, unresolvable guards fail the audit). |
| - Segmentation: worst per-window delta vs the ONNX oracle dropped from **45,422 → 0.07** |
| (ANE), argmax agreement 100%; the 8-speaker collapse is fixed at model level. |
| - Embedder: All-vs-CPU cosine 1.0 on real speech; int8 clusters identically to fp32 on the |
| multi-speaker parity corpus (speaker counts never differ). |
| - Conversion toolchain pinned in the recipe: coremltools 9.0, torch 2.5.1, onnx 1.22.0, |
| python 3.11.15. |
|
|
| ## Upstream provenance & licensing |
|
|
| | component | upstream | license | |
| |---|---|---| |
| | segmentation | [pyannote/segmentation-3.0](https://huggingface.co/pyannote/segmentation-3.0) (weights via its ONNX export) | MIT (upstream repo is access-gated; the MIT license permits redistribution with attribution — please also respect pyannote's usage conditions and cite their papers) | |
| | embedder | WeSpeaker ResNet34-LM (voxceleb) | Apache-2.0 (WeSpeaker project) | |
|
|
| These are *derivative conversions*: same weights, corrected graph numerics. If you are the |
| upstream author of either model and want changes to this redistribution, open a discussion. |
|
|
