|
|
--- |
|
|
language: |
|
|
- ca |
|
|
license: apache-2.0 |
|
|
base_model: openai/whisper-large-v3 |
|
|
tags: |
|
|
- whisper-event |
|
|
- generated_from_trainer |
|
|
datasets: |
|
|
- mozilla-foundation/common_voice_13_0 |
|
|
metrics: |
|
|
- wer |
|
|
model-index: |
|
|
- name: Whisper Large-V3 Catalan |
|
|
results: |
|
|
- task: |
|
|
name: Automatic Speech Recognition |
|
|
type: automatic-speech-recognition |
|
|
dataset: |
|
|
name: mozilla-foundation/common_voice_13_0 ca |
|
|
type: mozilla-foundation/common_voice_13_0 |
|
|
config: ca |
|
|
split: test |
|
|
args: ca |
|
|
metrics: |
|
|
- name: Wer |
|
|
type: wer |
|
|
value: 5.971420405830237 |
|
|
--- |
|
|
|
|
|
# Whisper Large-V3 Catalan |
|
|
|
|
|
## Model summary |
|
|
|
|
|
**Whisper Large-V3 Catalan** is an automatic speech recognition (ASR) model for **Catalan** speech. It is fine-tuned from [openai/whisper-large-v3] on the Catalan portion of **Mozilla Common Voice 13.0**, achieving a **Word Error Rate (WER) of 5.97%** on the Common Voice test split. |
|
|
|
|
|
The model is intended for high-quality transcription of Catalan speech in a variety of accents and recording conditions, including read and semi-spontaneous speech. |
|
|
|
|
|
--- |
|
|
|
|
|
## Model description |
|
|
|
|
|
* **Architecture:** Transformer-based encoder–decoder (Whisper) |
|
|
* **Base model:** openai/whisper-large-v3 |
|
|
* **Language:** Catalan (ca) |
|
|
* **Task:** Automatic Speech Recognition (ASR) |
|
|
* **Output:** Text transcription in Catalan |
|
|
* **Decoding:** Autoregressive sequence-to-sequence decoding |
|
|
|
|
|
This model leverages Whisper's multilingual pretraining and large-scale speech-text alignment, followed by supervised fine-tuning on Catalan speech data to improve language-specific accuracy. |
|
|
|
|
|
--- |
|
|
|
|
|
## Intended use |
|
|
|
|
|
### Primary use cases |
|
|
|
|
|
* Transcription of Catalan audio recordings |
|
|
* Speech-to-text pipelines for media, education, and research |
|
|
* Accessibility tools (e.g., subtitles, captions) |
|
|
* Offline or batch ASR for Catalan datasets |
|
|
|
|
|
### Intended users |
|
|
|
|
|
* Researchers working on Catalan or low-resource ASR |
|
|
* Developers building Catalan speech applications |
|
|
* Institutions and companies requiring Catalan transcription |
|
|
|
|
|
### Out-of-scope use |
|
|
|
|
|
* Real-time or low-latency ASR without optimization |
|
|
* Speech translation (this model performs transcription only) |
|
|
* Safety-critical applications without additional validation |
|
|
|
|
|
--- |
|
|
|
|
|
## Limitations and known issues |
|
|
|
|
|
* Performance may degrade on: |
|
|
|
|
|
* Highly noisy audio |
|
|
* Strong regional accents underrepresented in Common Voice |
|
|
* Conversational or overlapping speech |
|
|
* The model may produce hallucinated text when audio quality is very poor or silent. |
|
|
* Biases present in the Common Voice dataset (e.g., demographic or accent imbalance) may be reflected in model outputs. |
|
|
|
|
|
Users are encouraged to evaluate the model on their own data before deployment. |
|
|
|
|
|
--- |
|
|
|
|
|
## Training and evaluation data |
|
|
|
|
|
### Training data |
|
|
|
|
|
* **Dataset:** Mozilla Common Voice 13.0 (Catalan subset) |
|
|
* **Data type:** Crowd-sourced, read speech |
|
|
* **Preprocessing:** |
|
|
|
|
|
* Audio resampled to 16 kHz |
|
|
* Text normalized using Whisper tokenizer |
|
|
* Invalid or excessively long samples filtered |
|
|
|
|
|
### Evaluation data |
|
|
|
|
|
* **Dataset:** Common Voice 13.0 (Catalan test split) |
|
|
* **Metric:** Word Error Rate (WER) |
|
|
|
|
|
--- |
|
|
|
|
|
## Evaluation results |
|
|
|
|
|
| Metric | Value | |
|
|
| ---------- | --------- | |
|
|
| WER (test) | **5.97%** | |
|
|
|
|
|
These results indicate strong performance compared to the base Whisper multilingual model on Catalan speech. |
|
|
|
|
|
--- |
|
|
|
|
|
## Training procedure |
|
|
|
|
|
### Training hyperparameters |
|
|
|
|
|
* Learning rate: 1e-5 |
|
|
* Optimizer: Adam (β1=0.9, β2=0.999, ε=1e-8) |
|
|
* LR scheduler: Linear |
|
|
* Warmup steps: 500 |
|
|
* Training steps: 20,000 |
|
|
* Train batch size: 32 |
|
|
* Gradient accumulation steps: 2 |
|
|
* Effective batch size: 64 |
|
|
* Evaluation batch size: 16 |
|
|
* Mixed precision: FP16 (Native AMP) |
|
|
* Seed: 42 |
|
|
|
|
|
### Training results (summary) |
|
|
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Wer | |
|
|
|:-------------:|:-----:|:-----:|:---------------:|:------:| |
|
|
| 0.0988 | 1.95 | 1000 | 0.1487 | 6.5619 | |
|
|
| 0.025 | 3.91 | 2000 | 0.1676 | 6.3155 | |
|
|
| 0.0105 | 5.86 | 3000 | 0.1871 | 6.4035 | |
|
|
| 0.0047 | 7.81 | 4000 | 0.1973 | 6.4870 | |
|
|
| 0.0061 | 9.77 | 5000 | 0.2086 | 6.4836 | |
|
|
| 0.0034 | 11.72 | 6000 | 0.2172 | 6.6442 | |
|
|
| 0.0036 | 13.67 | 7000 | 0.2205 | 6.4041 | |
|
|
| 0.002 | 15.62 | 8000 | 0.2214 | 6.4350 | |
|
|
| 0.0011 | 17.58 | 9000 | 0.2339 | 6.1943 | |
|
|
| 0.0009 | 19.53 | 10000 | 0.2388 | 6.2921 | |
|
|
| 0.0011 | 21.48 | 11000 | 0.2327 | 6.2515 | |
|
|
| 0.0003 | 23.44 | 12000 | 0.2472 | 6.2052 | |
|
|
| 0.0012 | 25.39 | 13000 | 0.2382 | 6.2892 | |
|
|
| 0.0001 | 27.34 | 14000 | 0.2550 | 5.9949 | |
|
|
| 0.0006 | 29.3 | 15000 | 0.2574 | 6.3607 | |
|
|
| 0.0001 | 31.25 | 16000 | 0.2584 | 6.0143 | |
|
|
| 0.0001 | 33.2 | 17000 | 0.2686 | 5.9486 | |
|
|
| 0.0 | 35.16 | 18000 | 0.2736 | 5.9194 | |
|
|
| 0.0 | 37.11 | 19000 | 0.2768 | 5.9646 | |
|
|
| 0.0 | 39.06 | 20000 | 0.2783 | 5.9714 | |
|
|
|
|
|
|
|
|
### Framework versions |
|
|
|
|
|
- Transformers 4.37.2 |
|
|
- Pytorch 2.2.0+cu121 |
|
|
- Datasets 2.16.1 |
|
|
- Tokenizers 0.15.1 |
|
|
|
|
|
--- |
|
|
|
|
|
## How to use |
|
|
|
|
|
```python |
|
|
from transformers import pipeline |
|
|
|
|
|
hf_model = "HiTZ/whisper-large-v3-ca" |
|
|
device = 0 # set to -1 for CPU |
|
|
|
|
|
pipe = pipeline( |
|
|
task="automatic-speech-recognition", |
|
|
model=hf_model, |
|
|
device=device |
|
|
) |
|
|
|
|
|
result = pipe("audio.wav") |
|
|
print(result["text"]) |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Ethical considerations and risks |
|
|
|
|
|
* This model transcribes speech and may process personal data. |
|
|
* Users should ensure compliance with applicable data protection laws (e.g., GDPR). |
|
|
* The model should not be used for surveillance or non-consensual audio processing. |
|
|
|
|
|
--- |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use this model in your research, please cite: |
|
|
|
|
|
```bibtex |
|
|
@misc{dezuazo2025whisperlmimprovingasrmodels, |
|
|
title={Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languages}, |
|
|
author={Xabier de Zuazo and Eva Navas and Ibon Saratxaga and Inma Hernáez Rioja}, |
|
|
year={2025}, |
|
|
eprint={2503.23542}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.CL} |
|
|
} |
|
|
``` |
|
|
|
|
|
Please, check the related paper preprint in |
|
|
[arXiv:2503.23542](https://arxiv.org/abs/2503.23542) |
|
|
for more details. |
|
|
|
|
|
--- |
|
|
|
|
|
## License |
|
|
|
|
|
This model is available under the |
|
|
[Apache-2.0 License](https://www.apache.org/licenses/LICENSE-2.0). |
|
|
You are free to use, modify, and distribute this model as long as you credit |
|
|
the original creators. |
|
|
|
|
|
--- |
|
|
|
|
|
## Contact and attribution |
|
|
|
|
|
* Fine-tuning and evaluation: HiTZ/Aholab - Basque Center for Language Technology |
|
|
* Base model: OpenAI Whisper |
|
|
* Dataset: Mozilla Common Voice |
|
|
|
|
|
For questions or issues, please open an issue in the model repository. |
|
|
|
