---
language:
- ko
- en
metrics:
- wer
- cer
pipeline_tag: automatic-speech-recognition
library_name: transformers
tags:
- audio
- asr
- automatic-speech-recognition
datasets:
- yongchanskii/youtube-data-for-developers
license: cc-by-nc-3.0
---
# Whisper for developers

<!-- Provide a quick summary of what the model is/does. -->

This model is a **fine-tuned version of Whisper-large-v2 model specifically tuned for software developers.** It transcribes words like 'ChatGPT' or 'Webhook' correctly, which previous Whisper models could not do.

## Model Details

This model outperforms previous Whisper models in the transcription accuracy of software related words. I developed new metric to assess transcription accuracy of software related words, which is called DSWES (Domain-Specific Word Embedding Similarity).
Further information about this metric will be provided in upcoming paper.

Please refer to the OpenAI Whisper model card for more details about the backbone model.

### Model Description

<!-- Provide a longer summary of what this model is. -->

- **Developed by:** [yongchanskii](https://huggingface.co/yongchanskii)
- **Shared by:** [yongchanskii](https://huggingface.co/yongchanskii)
- **Model type:** Whisper
- **Language(s):** Korean, English
- **License:** [Attribution-NonCommercial 3.0 Unported](https://creativecommons.org/licenses/by-nc/3.0/)
- **Finetuned from model:** [openai/whisper-large-v2](https://huggingface.co/openai/whisper-large-v2)

### Model Sources

<!-- Provide the basic links for the model. -->

- **Repository:** [cyc9805](https://github.com/cyc9805/domain-specific-whisper)
- **Paper:** _Coming soon_

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

<!-- This should link to a Data Card if possible. -->
Testing data consists of 1 hour of audio data manually recorded from [AIWeek 2023](https://rsvp.withgoogle.com/events/aiweek2023), and 2 hours of audio data from developers conference video uploaded on YouTube. 
Note that testing data can not be provided publicly due to the privacy issue.

#### Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->
Two of the most popular metrics to assess automatic speech recognition model, WER and CER, were used. <br>
Additionally, DSWES was used to specifically check the transcription accuracy of softwared-related words. Note that higher the DSWES, the better.

For accessment, WhisperX was used as a backbone of a fine-tuned model due to its fast inference speed and reduced size. 
Since backbone of WhisperX is Whisper, I can safely assume that the performace of Whisper would very much similar to that of WhisperX.  

### Results

| Models | WER | CER | DSWES |
|------------|------------|--------------|--------------|
| WhisperX-large-v2       | 6.89    | 3.66 | 87
| **WhisperX-for-developers**       | **6.56**        | **2.84** | **91** |