| --- |
| license: cc-by-4.0 |
| language: [kk, en] |
| library_name: pytorch |
| pipeline_tag: automatic-speech-recognition |
| tags: [target-speaker-asr, speech-recognition, kazakh, english, overlapping-speech, wavlm, ctc] |
| --- |
| |
| # Persona-ASR |
|
|
| Bilingual (Kazakh–English) **target-speaker ASR** for overlapping speech, from the [Persona-ASR](https://github.com/IS2AI/Persona_ASR) project. Given a three-speaker mixture and a short enrollment utterance, the model transcribes only the enrolled target speaker, and rejects the utterance (emitting `<no_target>`) when the target is absent. |
|
|
| ## Checkpoints |
| - `asr_backbone.pt` — recognizer + activity-detection backbone: a frozen ECAPA-TDNN speaker embedding modulates a WavLM-Base+ encoder via FiLM, feeding two language-specific CTC heads (English/Latin, Kazakh/Cyrillic) and a frame-level VAD head. |
| - `presence_gate.pt` — utterance-level target-presence gate (enrollment–mixture matching + speaker-conditioned attention + attentive-statistics pooling), applied on top of the frozen backbone. |
| - `config.json` — backbone configuration (vocab sizes, hyperparameters). |
|
|
| ## Results (three-speaker test sets) |
| | Test set | Raw WER | Gated WER | Detection BAcc | |
| |---|---|---|---| |
| | English (Libri3Mix-100h) | 29.35 | 36.92 | 80.04 | |
| | Kazakh (Kazakh3Mix-100h) | 43.47 | 50.71 | 86.78 | |
|
|
| Presence-gate thresholds (calibrated on validation): τ_EN = 0.502, τ_KK = 0.586. |
|
|
| ## Training |
| Backbone loss L = 0.7·CTC + 0.3·VAD; gate trained with binary cross-entropy on the frozen backbone. Trained on LibriMix (English) and [KazMix-3](https://huggingface.co/datasets/issai/KazMix-3) (Kazakh); also evaluated on [PersonaMix](https://huggingface.co/datasets/issai/PersonaMix). |
|
|
| ## Usage |
| Inference and evaluation code: [github.com/IS2AI/Persona_ASR](https://github.com/IS2AI/Persona_ASR). |
|
|
| ## License and citation |
| Released under CC BY 4.0. Please cite Persona-ASR. |
|
|
| ```bibtex |
| @article{persona_asr, |
| title = {Persona-ASR: Bilingual Target-Speaker Speech Recognition for Kazakh--English Overlapping Speech}, |
| year = {2026} |
| } |
| ``` |
|
|
