| | --- |
| | license: apache-2.0 |
| | datasets: |
| | - KBLab/rixvox |
| | language: |
| | - sv |
| | --- |
| | # Whisper Tiny RixVox Swedish |
| |
|
| | This is a [Whisper tiny](https://huggingface.co/openai/whisper-tiny) finetuned for Swedish using |
| | the [RixVox](https://huggingface.co/datasets/KBLab/rixvox) dataset. |
| |
|
| | Please note that this model, as every other encoder-decoder speech-to-text model, is prone to |
| | hallucinating on unexpected inputs and treats the task as translation rather than transcription. |
| | I.e your mileage may vary depending on filtering and type of data. |
| |
|
| | In this release the entire encoder was frozen. Subsequent releases will not do this **if** the |
| | generalization to other types of data (i.e not parliamentary speeches) is kept when not freezing |
| | the encoder. |
| |
|
| | ## Evaluation |
| |
|
| | * Fleurs WER: 51.68 |
| | * Fleurs WER (normalized*): 48.09 |
| | |
| | *) Normalization is done by applying the following to source and generated texts: |
| |
|
| | ``` |
| | def normalize(s): |
| | return ' '.join([ x for x in sub('[^0-9a-zåäöA-ZÅÄÖ ]', ' ', s.lower()).split() ]) |
| | ``` |
| |
|
| | ## Training |
| |
|
| | Training was done using Huggingface and Deepspeed with ZeRO stage 2. |
| |
|
| | * learning rate: 1e-5 |
| | * optimizer: CPUAdamW (Deepspeed) |
| | * lr scheduler: linear |
| | * warmup steps: 500 |
| | * per device batch size: 32 |
| | * GPUs: 8 x NVIDIA A100 40GB |
| | * total batch size: 160 |
| | * steps: 10000 |
| | * lowercase: no |
| | * fp16 |
| | * entire encoder was frozen |
