| --- |
| language: en |
| license: cc-by-nc-4.0 |
| library_name: pytorch |
| tags: |
| - speaker-diarization |
| - ax650 |
| - axera |
| - onnx |
| - axmodel |
| - wavlm |
| - conformer |
| pipeline_tag: voice-activity-detection |
| --- |
| |
| # DiariZen Speaker Segmentation - AX650 Deployment |
|
|
| CPU+NPU hybrid speaker diarization segmentation for AX650 NPU. |
|
|
| ## Model |
|
|
| - Source: [BUT-FIT/diarizen-wavlm-large-s80-md](https://huggingface.co/BUT-FIT/diarizen-wavlm-large-s80-md) |
| - Architecture: WavLM-Large (pruned) + Conformer |
| - Task: Frame-level speaker activity segmentation (11 classes, 4s @ 16kHz) |
|
|
| ## Pipeline |
|
|
| ``` |
| Audio (16kHz mono, any length) |
| → CPU: resample → 4s sliding window → LayerNorm |
| → AX650 NPU: CNN feature extractor (7 conv, U16, 17.7ms) |
| → CPU: WavLM Transformer (24L) + Conformer (4L) + Classifier (251ms) |
| → Log-probabilities (1, 199, 11) per window |
| ``` |
|
|
| ## Performance |
|
|
| | Stage | Time | Hardware | |
| |-------|------|----------| |
| | CNN | 17.7 ms | AX650 NPU @1GHz | |
| | Backend | 251 ms | CPU ONNX Runtime | |
| | Total | 269 ms | 14.9x real-time | |
|
|
| ## Accuracy |
|
|
| End-to-end cosine: 0.9997 vs full FP32 reference. |
|
|
| ## Directory |
|
|
| ``` |
| models/ cnn_features.axmodel + backend.onnx |
| python/ Python SDK (diarizen_sdk) |
| cpp/ C++ SDK (diarizen_segmenter) |
| model_convert/ ONNX export + Pulsar2 compile config |
| reports/ SDK and simulation reports |
| ``` |
|
|
| ## Quick Start |
|
|
| ```bash |
| pip install -r python/requirements.txt |
| python python/diarizen_sdk/example.py audio.wav \ |
| --cnn-model models/cnn_features.axmodel \ |
| --backend-model models/backend.onnx |
| ``` |
|
|
| ## Known Limitations |
|
|
| - 4s fixed window; longer audio requires sliding window stitching. |
| - CNN requires U16 quantization (U8 insufficient accuracy). |
| - Transformer + Conformer run on CPU (NPU backend limitation with WavLM attention ops). |
| - pos_conv_embed Conv (kernel=128, NLC layout) incompatible with NPU; included in CPU backend. |
|
|
