| --- |
| license: cc-by-4.0 |
| library_name: coreai |
| pipeline_tag: voice-activity-detection |
| base_model: nvidia/diar_streaming_sortformer_4spk-v2 |
| tags: [core-ai, coreaikit, sortformer, speaker-diarization, diarization, streaming, on-device, apple] |
| --- |
| |
| # Streaming Sortformer 4-spk v2 — Core AI |
|
|
| [`nvidia/diar_streaming_sortformer_4spk-v2`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2) |
| (cc-by-4.0, 117M) converted to **Apple Core AI** — streaming **speaker diarization** ("who spoke |
| when", up to 4 speakers) running fully on-device via the |
| [zoo](https://github.com/john-rocky/coreai-model-zoo). Only the neural core is a graph; the NeMo |
| 128-mel frontend, the streaming chunk loop, and the AOSC speaker-cache compression run in the Swift |
| host — a 1:1 port of NeMo `sortformer_modules.py` (inference path). |
|
|
| ⚠️ Use the **streaming v2** checkpoint (cc-by-4.0). The offline `diar_sortformer_4spk-v1` is CC-BY-**NC**. |
|
|
| ## Files |
|
|
| - `sortformer_float16.aimodel` — the static `forward_for_export` core, fp16 (~237 MB, macOS GPU). |
| - `sortformer_float16.h18p.aimodelc` — the same graph AOT-compiled for iPhone (h18p, ~450 MB). |
| - `sortformer_mel_filters_128x257.f32` — librosa-slaney mel filterbank (host log-mel frontend). |
| - `metadata.json` — streaming params + the fixed-buffer graph contract. |
|
|
| ## Fixed-buffer graph contract |
|
|
| ``` |
| inputs: chunk_mel [1,1520,128] host zero-pads each mel chunk |
| spkcache [1,188,512] host-maintained speaker cache |
| valid [1,378] 1 = real frame / 0 = pad (spkcache block [0:len], chunk block [188:188+pe_len]) |
| outputs: preds [1,378,4] sigmoid speaker activity |
| chunk_pe [1,190,512] pre-encode embeddings (host appends them to the speaker cache) |
| ``` |
| Host: NeMo 128-mel (preemph 0.97 → STFT n_fft=512/win=400/hop=160 → slaney mel → log, normalize=NA) |
| → chunk the mel (188·8 frames, ±1 subsample ctx) → run the graph → slice chunk preds → |
| `streaming_update` + `compress_spkcache` (AOSC) → threshold 0.5/frame/speaker (frame = 80 ms) → turns. |
| |
| ## Verification |
| |
| Byte-gated vs NeMo `forward_streaming` at **100.00 % speaker-activity agreement** (@0.5) on a 21.5 s |
| and a 64.5 s clip (the latter exercises the AOSC cache compression ~4×), in Python, in Swift on |
| **Mac GPU**, and on **iPhone 17 Pro** (A19 Pro, AOT h18p) — all driving this exported fp16 graph. |
|
|
| ## Use |
|
|
| Ships in the **coreai-audio** app (Transcribe tab, "Diarize — who said what"): the diarizer segments |
| each speaker turn, then the on-device ASR (Whisper / Qwen3-ASR / Parakeet / Nemotron) transcribes it |
| into a **diarized transcript** — `Speaker 1 [0.3–4.1s]: …`. Speaker diarization already ships |
| on-device elsewhere (e.g. CoreML/ANE); this is speed parity, offered as a diarized transcript wired to |
| the zoo's own ASR. Conversion + Swift host loop: see |
| [`conversion/sortformer_diar`](https://github.com/john-rocky/coreai-model-zoo/tree/main/conversion/sortformer_diar). |
|
|
| Derived from NVIDIA's `diar_streaming_sortformer_4spk-v2` (CC-BY-4.0); this Core AI conversion is |
| released under the same license. |
|
|
| <!-- funnel:v1 --> |
|
|
| --- |
|
|
| **More models in this format:** [Core AI Model Zoo](https://huggingface.co/collections/mlboydaisuke/core-ai-model-zoo-6a7ff330f753e8dcae04671a) — 75 models, each with the recipe that produced it. |
|
|
| **Want a different model on-device?** [Open a request](https://github.com/john-rocky/on-device-requests) — free, open weights only; the export and its measured numbers get published publicly. |
|
|
| <!-- /funnel:v1 --> |
|
|
