|
|
--- |
|
|
library_name: transformers |
|
|
tags: |
|
|
- speech |
|
|
- automatic-speech-recognition |
|
|
- whisper |
|
|
- multilingual |
|
|
- speaker-diarization |
|
|
- meeting-transcription |
|
|
- DiCoW |
|
|
- BUT-FIT |
|
|
pipeline_tag: automatic-speech-recognition |
|
|
license: cc-by-4.0 |
|
|
datasets: |
|
|
- microsoft/NOTSOFAR |
|
|
- edinburghcstr/ami |
|
|
--- |
|
|
|
|
|
# 🧠 DiCoW\_v3.2 — BUT-FIT Model for MT-ASR |
|
|
|
|
|
This repository hosts the **DiCoW\_v3.2** model developed by [BUT Speech@FIT](https://github.com/BUTSpeechFIT), tailored for **multi-talker automatic speech recognition (MT-ASR)**. |
|
|
|
|
|
This model is available under the terms of CC BY 4.0. It incorporates an MIT-licensed base model and CC BY 4.0 licensed training data. |
|
|
|
|
|
## 🔧 Key Improvements over DiCoW v1 |
|
|
|
|
|
* **FDDT (Frame-Level Diarization Dependent Transformation)** before positional embeddings |
|
|
* **Less strict suppressive initialization** to ease early training dynamics |
|
|
* **Enhanced sequential decoding** with fallback seeking |
|
|
* **Frozen decoder** during fine-tuning to retain language modeling capabilities |
|
|
|
|
|
### 🧪 Augmentations |
|
|
|
|
|
* Random **STNO** noise injection |
|
|
* Segment-wise random class flipping of **STNO tokens** |
|
|
* **SpecAugment** |
|
|
* **MUSAN** noise mixing |
|
|
|
|
|
### ⚙️ Optimization & Inference Enhancements |
|
|
|
|
|
* Updated **learning schedule** |
|
|
* Improved **hallucination detection & mitigation** during inference |
|
|
|
|
|
--- |
|
|
|
|
|
|
|
|
## 🛠️ Model Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForSpeechSeq2Seq |
|
|
|
|
|
MODEL_NAME = "BUT-FIT/DiCoW_v3_2" |
|
|
dicow = AutoModelForSpeechSeq2Seq.from_pretrained(MODEL_NAME, trust_remote_code=True) |
|
|
``` |
|
|
|
|
|
➡️ For detailed inference pipelines, see: [**DiCoW GitHub (Inference)**](https://github.com/BUTSpeechFIT/DiCoW) |
|
|
|
|
|
|
|
|
--- |
|
|
|
|
|
## 🏆 Performance |
|
|
|
|
|
See how **DiCoW_v3.2** performs on our multi-talker ASR benchmark: |
|
|
|
|
|
- 🔗 [**EMMA-MT ASR Leaderboard**](https://huggingface.co/spaces/BUT-FIT/EMMA_leaderboard) |
|
|
|
|
|
--- |
|
|
|
|
|
|
|
|
## 📦 Model Details |
|
|
|
|
|
* **Base Model:** Whisper large-v3-turbo |
|
|
* **Training Datasets:** |
|
|
|
|
|
* [NOTSOFAR-1](https://github.com/microsoft/NOTSOFAR1-Challenge) |
|
|
* [AMI Meeting Corpus](http://groups.inf.ed.ac.uk/ami/corpus/) |
|
|
* [Libri2Mix](https://github.com/JorisCos/LibriMix) |
|
|
|
|
|
--- |
|
|
|
|
|
## 🧬 Source Repositories |
|
|
|
|
|
* 🔧 [Training Code: TS-ASR-Whisper](https://github.com/BUTSpeechFIT/TS-ASR-Whisper) |
|
|
* 🚀 [Inference](https://github.com/BUTSpeechFIT/DiCoW) |
|
|
|
|
|
--- |
|
|
|
|
|
|
|
|
## 📚 Related Publications |
|
|
|
|
|
* 📰 **Journal Paper:** |
|
|
*DiCoW: Diarization-Conditioned Whisper for Target Speaker Automatic Speech Recognition* |
|
|
[Computer Speech & Language, 2026](https://www.sciencedirect.com/science/article/pii/S088523082500066X) |
|
|
|
|
|
* 📰 **ICASSP 2025:** |
|
|
*Target Speaker ASR with Whisper* |
|
|
[IEEE ICASSP 2025](https://doi.org/10.1109/ICASSP49660.2025.10887683) |
|
|
|
|
|
* 📰 **CHiME-8 System Description:** |
|
|
*BUT/JHU System Description for CHiME-8 NOTSOFAR-1 Challenge* |
|
|
[CHiME 2024 Proceedings](https://doi.org/10.21437/CHiME.2024-4) |
|
|
|
|
|
* 📰 **MLC-SLM Challenge Submission:** |
|
|
*BUT System for the MLC-SLM Challenge* |
|
|
[arXiv:2506.13414](https://arxiv.org/abs/2506.13414) |
|
|
|
|
|
--- |
|
|
|
|
|
## 📝 Citation |
|
|
|
|
|
If you use this model, please cite the following works: |
|
|
|
|
|
```bibtex |
|
|
@article{POLOK2026101841, |
|
|
title = {DiCoW: Diarization-conditioned Whisper for target speaker automatic speech recognition}, |
|
|
journal = {Computer Speech & Language}, |
|
|
volume = {95}, |
|
|
pages = {101841}, |
|
|
year = {2026}, |
|
|
issn = {0885-2308}, |
|
|
doi = {https://doi.org/10.1016/j.csl.2025.101841}, |
|
|
url = {https://www.sciencedirect.com/science/article/pii/S088523082500066X}, |
|
|
author = {Alexander Polok and Dominik Klement and Martin Kocour and Jiangyu Han and Federico Landini and Bolaji Yusuf and Matthew Wiesner and Sanjeev Khudanpur and Jan Černocký and Lukáš Burget}, |
|
|
keywords = {Diarization-conditioned Whisper, Target-speaker ASR, Speaker diarization, Long-form ASR, Whisper adaptation}, |
|
|
} |
|
|
|
|
|
@INPROCEEDINGS{10887683, |
|
|
author={Polok, Alexander and Klement, Dominik and Wiesner, Matthew and Khudanpur, Sanjeev and Černocký, Jan and Burget, Lukáš}, |
|
|
booktitle={ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, |
|
|
title={Target Speaker ASR with Whisper}, |
|
|
year={2025}, |
|
|
volume={}, |
|
|
number={}, |
|
|
pages={1-5}, |
|
|
keywords={Transforms;Signal processing;Transformers;Acoustics;Speech processing;target-speaker ASR;diarization conditioning;multi-speaker ASR;Whisper}, |
|
|
doi={10.1109/ICASSP49660.2025.10887683} |
|
|
} |
|
|
|
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## 📬 Contact |
|
|
|
|
|
For questions or collaboration inquiries: |
|
|
|
|
|
📧 **Email:** [ipoloka@fit.vut.cz](mailto:ipoloka@fit.vut.cz) |
|
|
|
|
|
🏢 **Affiliation:** [BUT Speech@FIT](https://github.com/BUTSpeechFIT), Brno University of Technology |
|
|
|
|
|
🔗 **GitHub:** [BUTSpeechFIT](https://github.com/BUTSpeechFIT) |
