REAL-TSE BSRNN Target Speaker Extraction — Core ML FP16
Native Core ML conversion of the REAL-TSE Challenge spk_emb_100 baseline.
It extracts an enrolled target speaker from overlapping 16 kHz speech while
preserving the original audio timeline.
The conversion is split into two models so enrollment embeddings can be cached and the signal-processing front end can use Apple's Accelerate framework:
| File | Inputs | Output |
|---|---|---|
BSRNNSpeakerEmbedding.mlpackage / .mlmodelc |
Kaldi-compatible fbank [1, T, 80], T=50...1600 |
ECAPA embedding [1, 192] |
BSRNNSeparator8s.mlpackage / .mlmodelc |
STFT [1, 2, 257, 1001] and embedding [1, 192] |
Estimated target STFT [1, 2, 257, 1001] |
Audio contract
- Mono Float32 PCM at 16,000 Hz.
- STFT: 512-point periodic Hann window, 128-sample hop, centered reflect padding.
- Separator chunk: 128,000 samples (8 seconds).
- Long recordings: overlap-add 8-second chunks.
- Enrollment: at least 0.5 seconds; several seconds of clean solo speech is recommended.
STFT, inverse STFT, Kaldi fbank, resampling, and overlap-add are intentionally outside the Core ML graphs.
Apple runtime recommendation
Use MLComputeUnits.cpuOnly for the separator. MLComputePlan assigns
effectively all separator cost to twelve LSTMs, which are not supported by the
Apple Neural Engine. On an Apple M3 Max, CPU-only execution was faster and used
substantially less memory than CPU+GPU.
Core ML may return padded MLMultiArray output strides on CPU. Consumers must
read MLMultiArray.strides rather than assuming contiguous logical dimensions.
Validation
Measured on an Apple M3 Max with macOS 26.4:
- Swift/Core ML chunk parity versus the Python/Core ML reference: 50.3 dB SI-SDR.
- 33.16-second English target with equal-level German interference:
- mixture SI-SDR: -0.42 dB
- extracted SI-SDR: 10.26 dB
- NVIDIA Parakeet TDT v3 WER: 57.8% → 2.0%
- target-word recall: 43.1% → 98.0%
- extraction real-time factor: 0.197
- peak BSRNN + Parakeet test-process RSS: approximately 594 MB
Provenance
- Architecture and checkpoint: REAL-TSE Challenge WeSep
spk_emb_100 - Separator: non-causal BSRNN, six repeated dual-axis recurrent blocks.
- Speaker encoder: ECAPA-TDNN, 192-dimensional embedding.
- Training configuration: clean Libri2Mix train-100, 16 kHz.
- Conversion: PyTorch trace → Core ML ML Program, FP16 compute precision.
No pruning, palettization, or weight quantization is applied to the published FP16 models.
Licensing and attribution
The upstream checkpoint download does not include a standalone checkpoint
license, so this repository uses the Hugging Face other license marker rather
than asserting broader rights than the upstream authors granted.
- WeSep implementation files used by this conversion identify Apache-2.0.
- The checkpoint configuration identifies clean Libri2Mix train-100 and a VoxCeleb ECAPA speaker encoder. Their published model/data documentation uses CC BY 4.0 attribution terms.
- Users are responsible for confirming that their intended checkpoint use and redistribution comply with upstream REAL-TSE, LibriMix/LibriSpeech, VoxCeleb, and WeSpeaker terms.
Please cite the REAL-TSE Challenge and WeSep authors when using this conversion.
Limitations
- Fixed eight-second separator graph; long audio requires chunking.
- Optimized for Apple silicon and macOS 15 or newer.
- The speaker embedding is for extraction conditioning, not identity verification or biometric authentication.
- Quality may degrade for enrollment audio with strong noise, reverberation, or a voice that differs substantially from the target recording.
- Downloads last month
- -