Update README.md

README.md

@@ -218,40 +218,77 @@ Stage 1 employed low threshold values (0 to 0.30 BLEU depending on dataset), whe
 | NST         | 250                      | 250          |
 | **Total**   | **56,514**               | **8,533**    |
 
-The default when loading our models through Hugging Face is **Stage 2**. We have however also uploaded continued pretraining checkpoints and tagged them. You can load these other checkpoints by specifying the `revision` in `.from_pretrained()`. The pretrained checkpoints tag can for example be found here: [`pretrained-checkpoint`](https://huggingface.co/KBLab/kb-whisper-large/tree/pretrained-checkpoint). The Stage 2 default model tag is named `standard`. We supply a different stage 2 checkpoint -- with a more condensed style of transcribing -- under the name `subtitle`. 
+
+The default when loading our models through Hugging Face is **Stage 2**. We have however also uploaded continued pretraining checkpoints and tagged them. You can load these other checkpoints by specifying the `revision` in `.from_pretrained()`. The pretrained checkpoints tag can for example be found here: [`pretrained-checkpoint`](https://huggingface.co/KBLab/kb-whisper-large/tree/pretrained-checkpoint). The Stage 2 default model tag is named `standard`. We supply two different stage 2 checkpoints -- one with a more condensed style of transcribing -- under the name `subtitle`, and one more verbose called `strict`. 
 
 ### Evaluation
 
 
-#### WER
+#### WER compared to OpenAI
 | Model size  |  | FLEURS | CommonVoice | NST  |
 |------------|---------|--------|-------------|------|
 | [tiny](https://huggingface.co/KBLab/kb-whisper-tiny)       | **KBLab**   | **13.2**  | **12.9**  | **11.2**  |
 |            | OpenAI  | 59.2   | 67.8   | 85.2   |
 | [base](https://huggingface.co/KBLab/kb-whisper-base)       | **KBLab**   | **9.1**   | **8.7**   | **7.8**   |
 |            | OpenAI  | 39.6   | 52.1   | 53.4   |
-| [small](https://huggingface.co/KBLab/kb-whisper-small)      | **KBLab**   | **7.3**   | **6.4**   | **6.6**   |
+| [small](https://huggingface.co/KBLab/kb-whisper-small)     | **KBLab**   | **7.3**   | **6.4**   | **6.6**   |
 |            | OpenAI  | 20.6   | 26.4   | 26.4   |
-| [medium](https://huggingface.co/KBLab/kb-whisper-medium)     | **KBLab**   | **6.6**   | **5.4**   | **5.8**   |
+| [medium](https://huggingface.co/KBLab/kb-whisper-medium)   | **KBLab**   | **6.6**   | **5.4**   | **5.8**   |
 |            | OpenAI  | 12.1   | 15.8   | 17.1   |
-| [large-v3](https://huggingface.co/KBLab/kb-whisper-large)   | **KBLab**   | **5.4**   | **4.1**   | **5.2**   |
+| [large-v3](https://huggingface.co/KBLab/kb-whisper-large)  | **KBLab**   | **5.4**   | **4.1**   | **5.2**   |
 |            | OpenAI  | 7.8    | 9.5    | 11.3    |
 
+#### WER for different KBLab stage2 versions
 
-#### BLEU Score
+| Model size  |  | FLEURS | CommonVoice | NST  |
+|------------|---------|--------|-------------|------|
+| [tiny](https://huggingface.co/KBLab/kb-whisper-tiny)       | **standard**   | **13.2**  | **12.9**  | **11.2**  |
+|            | strict    | 14.1   | 13.4   | 11.0   |
+|            | subtitle  | 13.3   | 12.9   | 11.4   |
+| [base](https://huggingface.co/KBLab/kb-whisper-base)       | **standard**   | **9.1**   | **8.7**   | **7.8**   |
+|            | strict    | 10.4   | 9.6    | 8.4    |
+|            | subtitle  | 9.1    | 8.7    | 7.9    |
+| [small](https://huggingface.co/KBLab/kb-whisper-small)     | **standard**   | **7.3**   | **6.4**   | **6.6**   |
+|            | strict    | 8.2    | 7.0    | 6.7    |
+|            | subtitle  | 7.3    | 6.4    | 6.6    |
+| [medium](https://huggingface.co/KBLab/kb-whisper-medium)   | **standard**   | **6.6**   | **5.4**   | **5.8**   |
+|            | strict    | 6.8    | 5.4    | 6.0    |
+| [large-v3](https://huggingface.co/KBLab/kb-whisper-large)  | **standard**   | **5.4**   | **4.1**   | **5.2**   |
+|            | strict    | 5.3    | 4.0    | 5.1    |
+
+#### BLEU Score compared to OpenAI
 | Model size  |   | FLEURS | CommonVoice | NST  |
 |------------|---------|--------|-------------|------|
-| tiny       | KBLab   | **76.6**  | **73.7**  | **74.3**  |
+| [tiny](https://huggingface.co/KBLab/kb-whisper-tiny)       | **KBLab**   | **76.6**  | **73.7**  | **74.3**  |
 |            | OpenAI  | 26.9   | 21.1   | 24.0   |
-| base       | KBLab   | **83.2**   | **79.9**   | **78.3**   |
+| [base](https://huggingface.co/KBLab/kb-whisper-base)       | **KBLab**   | **83.2**   | **79.9**   | **78.3**   |
 |            | OpenAI  | 41.1   | 32.5   | 36.9   |
-| small      | KBLab   | **86.6**   | **83.5**   | **79.6**   |
+| [small](https://huggingface.co/KBLab/kb-whisper-small)     | **KBLab**   | **86.6**   | **83.5**   | **79.6**   |
 |            | OpenAI  | 64.0   | 56.5   | 58.2   |
-| medium     | KBLab   | **87.6**   | **85.0**   | **80.2**   |
+| [medium](https://huggingface.co/KBLab/kb-whisper-medium)   | **KBLab**   | **87.6**   | **85.0**   | **80.2**   |
 |            | OpenAI  | 77.1   | 70.1   | 68.9   |
-| large-v3   | KBLab   | **89.8**   | **87.2**   | **81.1**   |
+| [large-v3](https://huggingface.co/KBLab/kb-whisper-large)  | **KBLab**   | **89.8**   | **87.2**   | **81.1**   |
 |            | OpenAI  | 84.9    | 79.1    | 75.1    |
 
+#### BLEU Score for different KBLab stage2 versions
+| Model size  |   | FLEURS | CommonVoice | NST  |
+|------------|---------|--------|-------------|------|
+| [tiny](https://huggingface.co/KBLab/kb-whisper-tiny)       | **standard**   | **76.6**  | **73.7**  | **74.3**  |
+|            | strict      | 75.3    | 72.9    | 74.6    |
+|            | subtitle    | 76.6    | 73.7    | 74.1    |
+| [base](https://huggingface.co/KBLab/kb-whisper-base)       | **standard**   | **83.2**   | **79.9**   | **78.3**   |
+|            | strict      | 81.0    | 78.4    | 77.5    |
+|            | subtitle    | 83.2    | 79.8    | 78.2    |
+| [small](https://huggingface.co/KBLab/kb-whisper-small)     | **standard**   | **86.6**   | **83.5**   | **79.6**   |
+|            | strict      | 84.9    | 82.4    | 79.3    |
+|            | subtitle    | 86.6    | 83.5    | 79.6    |
+| [medium](https://huggingface.co/KBLab/kb-whisper-medium)   | **standard**   | **87.6**   | **85.0**   | **80.2**   |
+|            | strict      | 87.3    | 84.9    | 80.1    |
+| [large-v3](https://huggingface.co/KBLab/kb-whisper-large)  | **standard**   | **89.8**   | **87.2**   | **81.1**   |
+|            | strict      | 90.0    | 87.4    | 81.2    |
+
+
+
 ### Acknowledgements
 
 We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium through an EuroHPC AI and Data-Intensive Applications Access call.