---
license: cc-by-4.0
language:
- en
library_name: nemo
pipeline_tag: automatic-speech-recognition
base_model: nvidia/parakeet-tdt-0.6b-v2
tags:
- automatic-speech-recognition
- target-speaker-asr
- speaker-diarization
- multi-talker
- parakeet
- nemo
- rnnt
- tdt
---

# DiCoP — Diarization-Conditioned Parakeet

Target-speaker ASR built on [nvidia/parakeet-tdt-0.6b-v2](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2). Given audio and
a diarization, it transcribes **one speaker at a time**.

The conditioning lives inside the encoder. Every frame is labelled silence / target /
non-target / overlap (STNO), and each Conformer layer applies a small learned per-class
transform — an FDDT block — before the layer runs. A whole meeting is decoded per speaker in one
pass: no segmentation, no speaker embeddings, no separation front-end.

| | |
|---|---|
| Base model | [nvidia/parakeet-tdt-0.6b-v2](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2) |
| Parameters | 618M |
| Encoder | 24 × FastConformer, d_model 1024 |
| Encoder frame rate | 12.5 Hz (80 ms) |
| Vocabulary | 1024 BPE tokens |
| Decoder | TDT (token-and-duration transducer) |
| Sample rate | 16000 Hz |

## Usage

This checkpoint **cannot be loaded by `nemo_toolkit` alone**. Its encoder `_target_` points at a
class that lives in the [DiCoP repository](https://github.com/BUTSpeechFIT/DiCoP), and NeMo only resolves `_target_`s
inside the `nemo` package unless that check is relaxed.

```bash
git clone https://github.com/BUTSpeechFIT/DiCoP && cd DiCoP
pip install -r requirements.txt

python infer.py \
    --checkpoint BUT-FIT/DiCoP_v0.1 \
    --rttm /path/to/rttms/ --audio-dir /path/to/audio/ \
    --output hyp.stm
```

To drive the model directly:

```python
import sys
sys.path.insert(0, "/path/to/DiCoP")

from utils.nemo import allow_external_nemo_targets, register_legacy_nemo_aliases

allow_external_nemo_targets()
register_legacy_nemo_aliases()

from src.model.asr_bpe_model import EncDecRNNTBPEModelSTNO

model = EncDecRNNTBPEModelSTNO.from_pretrained("BUT-FIT/DiCoP_v0.1")
```

`transcribe()` is deliberately disabled on this model. NeMo's transcription path cannot supply a
mask, and an unconditioned encoder returns a fluent transcript of whoever is loudest — which
looks correct but is not target-speaker output. Use `infer.py`, or `transcribe_stno()` with an
STNO mask you build yourself (see `src/data/stno.py`).

## Results
Oracle diarization, cpWER and tcpWER (collar 5s) in percent, `whisper_nsf` normalization applied. AMI's half-hour sessions used
windowed local attention
(`-O model.encoder.self_attention_model=rel_pos_local_attn -O model.encoder.att_context_size=[256,256]`)
to bound memory; every other set is full-context, full-session.

| Set | Sessions | cpWER | tcpWER |
|---|---|---|---|
| AMI-SDM dev / test | 18 / 16 | 13.98 / 15.97 | 14.26 / 16.51 |
| AMI-IHM-mix dev / test | 18 / 16 | 11.20 / 11.75 | 11.41 / 12.15 |
| NOTSOFAR-SDM dev1 / eval | 177 / 160 | 17.44 / 17.56 | 17.93 / 17.94 |
| LibriSpeechMix 2mix dev / test | 2703 / 2620 | 2.62 / 2.54 | 2.62 / 2.54 |
| LibriSpeechMix 3mix dev / test | 2703 / 2620 | 6.79 / 6.34 | 6.80 / 6.35 |
| Libri2Mix dev / test clean | 3000 | 4.13 / 4.40 | 4.16 / 4.41 |
| Libri3Mix dev / test clean | 3000 | 30.93 / 33.12 | 31.00 / 33.19 |

## Contact

If you have any questions, reach out to: [iklement@fit.vut.cz](mailto:iklement@fit.vut.cz)