BUT-FIT
/

DiCoW_v3_3

@@ -1,199 +1,151 @@
 ---
 library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
 library_name: transformers
+tags:
+- speech
+- automatic-speech-recognition
+- whisper
+- multilingual
+- speaker-diarization
+- meeting-transcription
+- DiCoW
+- BUT-FIT
+pipeline_tag: automatic-speech-recognition
+license: cc-by-4.0
+datasets:
+- microsoft/NOTSOFAR
+- edinburghcstr/ami
 ---
+# 🧠 DiCoW\_v3.3 — BUT-FIT Model for MT-ASR
+This repository hosts the **DiCoW\_v3.3** model developed by [BUT Speech@FIT](https://github.com/BUTSpeechFIT), tailored for **multi-talker automatic speech recognition (MT-ASR)**.
+This model is available under the terms of CC BY 4.0. It incorporates an MIT-licensed base model and CC BY 4.0 licensed training data.
+## 🔧 Key Improvements over DiCoW v1
+* **FDDT (Frame-Level Diarization Dependent Transformation)** before positional embeddings
+* **Less strict suppressive initialization** to ease early training dynamics
+* **Enhanced sequential decoding** with fallback seeking
+* **Frozen decoder** during fine-tuning to retain language modeling capabilities
+### 🧪 Augmentations
+* Random **STNO** noise injection
+* Segment-wise random class flipping of **STNO tokens**
+* **SpecAugment**
+* **MUSAN** noise mixing
+### ⚙️ Optimization & Inference Enhancements
+* Updated **learning schedule**
+* Improved **hallucination detection & mitigation** during inference
+---
+## 🛠️ Model Usage
+```python
+from transformers import AutoModelForSpeechSeq2Seq
+MODEL_NAME = "BUT-FIT/DiCoW_v3_3"
+dicow = AutoModelForSpeechSeq2Seq.from_pretrained(MODEL_NAME, trust_remote_code=True)
+```
+➡️ For detailed inference pipelines, see: [**DiCoW GitHub (Inference)**](https://github.com/BUTSpeechFIT/DiCoW)
+---
+## 🏆 Performance
+See how **DiCoW_v3.3** performs on our multi-talker ASR benchmark:
+- 🔗 [**EMMA-MT ASR Leaderboard**](https://huggingface.co/spaces/BUT-FIT/EMMA_leaderboard)
+---
+## 📦 Model Details
+* **Base Model:** Whisper large-v3-turbo
+* **Training Datasets:**
+  * [NOTSOFAR-1](https://github.com/microsoft/NOTSOFAR1-Challenge)
+  * [AMI Meeting Corpus](http://groups.inf.ed.ac.uk/ami/corpus/)
+  * [LibriMix](https://github.com/JorisCos/LibriMix)
+---
+## 🧬 Source Repositories
+* 🔧 [Training Code: TS-ASR-Whisper](https://github.com/BUTSpeechFIT/TS-ASR-Whisper)
+* 🚀 [Inference](https://github.com/BUTSpeechFIT/DiCoW)
+---
+## 📚 Related Publications
+* 📰 **Journal Paper:**
+  *DiCoW: Diarization-Conditioned Whisper for Target Speaker Automatic Speech Recognition*
+  [Computer Speech & Language, 2026](https://www.sciencedirect.com/science/article/pii/S088523082500066X)
+* 📰 **ICASSP 2025:**
+  *Target Speaker ASR with Whisper*
+  [IEEE ICASSP 2025](https://doi.org/10.1109/ICASSP49660.2025.10887683)
+* 📰 **CHiME-8 System Description:**
+  *BUT/JHU System Description for CHiME-8 NOTSOFAR-1 Challenge*
+  [CHiME 2024 Proceedings](https://doi.org/10.21437/CHiME.2024-4)
+* 📰 **MLC-SLM Challenge Submission:**
+  *BUT System for the MLC-SLM Challenge*
+  [arXiv:2506.13414](https://arxiv.org/abs/2506.13414)
+---
+## 📝 Citation
+If you use this model, please cite the following works:
+```bibtex
+@article{POLOK2026101841,
+    title = {DiCoW: Diarization-conditioned Whisper for target speaker automatic speech recognition},
+    journal = {Computer Speech & Language},
+    volume = {95},
+    pages = {101841},
+    year = {2026},
+    issn = {0885-2308},
+    doi = {https://doi.org/10.1016/j.csl.2025.101841},
+    url = {https://www.sciencedirect.com/science/article/pii/S088523082500066X},
+    author = {Alexander Polok and Dominik Klement and Martin Kocour and Jiangyu Han and Federico Landini and Bolaji Yusuf and Matthew Wiesner and Sanjeev Khudanpur and Jan Černocký and Lukáš Burget},
+    keywords = {Diarization-conditioned Whisper, Target-speaker ASR, Speaker diarization, Long-form ASR, Whisper adaptation},
+}
+@INPROCEEDINGS{10887683,
+    author={Polok, Alexander and Klement, Dominik and Wiesner, Matthew and Khudanpur, Sanjeev and Černocký, Jan and Burget, Lukáš},
+    booktitle={ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
+    title={Target Speaker ASR with Whisper},
+    year={2025},
+    volume={},
+    number={},
+    pages={1-5},
+    keywords={Transforms;Signal processing;Transformers;Acoustics;Speech processing;target-speaker ASR;diarization conditioning;multi-speaker ASR;Whisper},
+    doi={10.1109/ICASSP49660.2025.10887683}
+}
+```
+---
+## 📬 Contact
+For questions or collaboration inquiries:
+📧 **Email:** [ipoloka@fit.vut.cz](mailto:ipoloka@fit.vut.cz)
+🏢 **Affiliation:** [BUT Speech@FIT](https://github.com/BUTSpeechFIT), Brno University of Technology
+🔗 **GitHub:** [BUTSpeechFIT](https://github.com/BUTSpeechFIT)