Training in progress, step 160

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ee5cacde1623b15122ac3014d88b82ce77e23f256c7a7413eb98924fec427b1b
+oid sha256:8f9d8969563a430c539e7832a0f581b9fc1ecc43821015672854439cc52fdd59
 size 151098921

runs/Dec13_11-33-05_d7f040c448a8/1670931197.5413718/events.out.tfevents.1670931197.d7f040c448a8.8569.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1fa90eee1b0e722b3cdda4a41b9b95bc5313544b8a27d1aa8ffe808ccd3a22a2
+size 5880

runs/Dec13_11-33-05_d7f040c448a8/events.out.tfevents.1670931197.d7f040c448a8.8569.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc936480041cf78da4f01d76172c6714c24a08c7f7ca674c2d931b264097e697
+size 4743

src/readme.md

@@ -0,0 +1,106 @@
+## Description
+
+Fine-tuning [OpenAI Whisper](https://github.com/openai/whisper) model for Belarusian language during 
+[Whisper fine-tuning Event](https://github.com/huggingface/community-events/tree/main/whisper-fine-tuning-event)
+hosted by HuggingFace x Lambda.
+
+The code in this repository is a modified version of code from 
+[Whisper fine-tuning Event](https://github.com/huggingface/community-events/tree/main/whisper-fine-tuning-event) repo.
+
+## Fine-tuning todos:
+* perform evaluation of fine-tuned model on CommonVoice test set
+* check exact sizes of train, eval, test sets of CommonVoice 11
+
+## Resuming training from exising checkpoint
+When resuming training from existing checkpoint:
+* learning rate gets reset if passing same parameter value to training script as in previour run.<br>
+  need to provide learning rate from the last step of previous run to continue
+  training in a correct way.<br>
+  however even if passing learning rate from the last step, in the new run it has different value than expected
+  (probably because of warmup).
+* it's unclear whether decision on saving current model
+  is made by comparing current metrics with metrics of the best checkpoint. I guess model with worse performance
+  will not overwrite best model checkpoint already exising in the output dir, but need to double check.
+* we can set `ignore_data_skip=True` Training argument not to 
+  skip data items already passed to a model - that will save time on data loads.
+    * it's unclear whether order of input items in the train set (that is shuffled) will be the same 
+      across multiple reruns - i.e. it's unclear whether sampling is the same across reruns.
+    * if the sampling is the same across reruns, `ignore_data_skip=True` will lead to same items been passed to a model
+      in current run. it's OK if previous run ended with large step value on the last epoch.
+      if not, the same elements from the same epoch will be passed to a model again.
+
+## Questions:
+* What checkpoint (best, I guess) is saved in the `output_dir`? 
+  How is it overwritten when resuming training from existing checkpoint?
+
+### Prepended tokens
+* Why are there following lines in Data Collator?
+  ```python
+    # if bos token is appended in previous tokenization step,
+    # cut bos token here as it's append later anyways
+    if (labels[:, 0] == self.decoder_start_token_id).all().cpu().item():
+        labels = labels[:, 1:]
+    ```
+* `tokenizer.bos_token_id` vs `model.config.decoder_start_token_id`.<br>
+  which one to pass to Data Collator as `decoder_start_token_id` parameter?
+* Answer:
+  * In this case, the two are equivalent. You can verify this:
+    ```python
+    print(tokenizer.bos_token_id)
+    print(model.config.decoder_start_token_id)
+    ```
+
+  * Print Output:
+    ```
+    <|startoftranscript|>
+    <|startoftranscript|>
+    ```
+
+  * Technically speaking, the decoder_start_token_id is the correct convention here. Before starting generating any tokens, we initialise the generate method with a starting token, which is the decoder_start_token_id. 
+  See: https://huggingface.co/blog/how-to-generate. The decoder_start_token_id corresponds to the initial context word sequence, and is the zero'th token generated.
+
+  * We remove this token from the encoded labels in the data collator because we always set the zero'th generated token to the decoder_start_token_id. If we leave the decoder_start_token_id as part of the label sequence, then we'll predict the decoder_start_token_id as the zero'th token, and again as the first token! Because we're always forcing it as the zero'th token, we don't need to predict it as the first token, and so we remove it from the target lables
+
+  * These tokens are not forced in the generation process, and so we don't cut them in the data collator. We need to provide them to the model as target labels so that the model can learn the correct tasks from our data
+
+  * The tokens correspond to the audio language, task (translate or transcribe) and whether to predict timestamps
+
+  * We need to tell the model what language the audio corresponds to and what task it's performing during fine-tuning. This way, it learns what audio corresponds to what language, and the difference between transcribing audio vs translating it 
+
+## Notes:
+* using CommonVoice 11 dataset in a streaming way.<br>
+  use `streaming=True` for train & validation & test.<br>
+  as an alternative, we can use `streaming=False` for validation & test sets to save time on data processing.
+  but the size of validation and test sets are unknown (need to check).
+  it's likely they are going to be large - thus pre-download of these sets might not reduce 
+  overall fine-tuning time compared to streaming mode.
+* size of train set is ~370'000 audiofiles. if using `batch_size=64`, then
+  1 epoch will have ~5782 steps. <br>
+  Because of `--eval_steps="1000"` will use `--max_steps="6000"` instead of `--max_steps="5800"`
+  to have evaluation metrics computed in the end of training.
+* if using Google Colab, need to execute  `sudo chmod -R 777 .git` inside hf repo to 
+  to set right permissions to be able to push trained models to HuggingFace Hub
+* Whispers BasicTextNormalizer splits words containing apostrophe:
+  ```python
+  > from transformers.models.whisper.english_normalizer import BasicTextNormalizer
+  > normalizer = BasicTextNormalizer()
+  > normalizer("раз'яднаць")
+  'раз яднаць'
+  ```
+* That's why `BelarusianTextNormalizer` (edited version of `BasicTextNormalizer`) was added to training script:
+  ```python
+  > from run_speech_recognition_seq2seq_streaming import BelarusianTextNormalizer
+  > normalizer_be = BelarusianTextNormalizer()
+  > normalizer_be("раз'яднаць")
+  "раз'яднаць"
+  ```
+* Need to set `use_cache` to False since we're using gradient checkpointing, and the two are incompatible
+* Default Linear scheduler is used 
+* Default Adam optimizer is used
+* To save memory (and increase either model or batch_size) can experiment with:
+    * using Adafactor instead of Adam.
+      Adam requires two optimiser params per one model param, but Adafactor uses only one.
+      > A word of caution: Adafactor is untested for fine-tuning Whisper, 
+        so we are unsure sure how Adafactor performance compares to Adam!
+    * using Adam 8bit from `bitsandbytes` module. 
+      need to provide `optim="adamw_bnb_8bit"` param to `Seq2SeqTrainingArguments`

src/requirements.txt

@@ -0,0 +1,9 @@
+torch>=1.7
+torchaudio
+git+https://github.com/huggingface/transformers
+git+https://github.com/huggingface/datasets
+librosa
+jiwer
+evaluate>=0.3.0
+more-itertools
+tensorboard

src/run.sh

@@ -0,0 +1,42 @@
+python src/run_speech_recognition_seq2seq_streaming.py \
+	--model_name_or_path="openai/whisper-small" \
+	--dataset_name="mozilla-foundation/common_voice_11_0" \
+	--dataset_config_name="be" \
+	--language="be" \
+	--train_split_name="train" \
+	--eval_split_name="validation" \
+	--model_index_name="Whisper Small Belarusian" \
+    \
+	--max_steps="6000" \
+	--output_dir="./" \
+	--per_device_train_batch_size="64" \
+	--per_device_eval_batch_size="32" \
+	--logging_steps="50" \
+	--learning_rate="1e-4" \
+	--warmup_steps="500" \
+	--evaluation_strategy="steps" \
+	--eval_steps="1000" \
+	--save_strategy="steps" \
+	--save_steps="1000" \
+	--gradient_checkpointing \
+	--fp16 \
+    \
+	--shuffle_buffer_size="500" \
+	--generation_max_length="225" \
+	--max_duration_in_seconds="30" \
+	--text_column_name="sentence" \
+	--freeze_feature_encoder="False" \
+	--report_to="tensorboard" \
+	--metric_for_best_model="wer" \
+	--greater_is_better="False" \
+	--load_best_model_at_end \
+    \
+	--do_train \
+	--do_eval \
+	--ignore_data_skip \
+	--predict_with_generate \
+	--do_normalize_eval \
+	--streaming \
+	--use_auth_token \
+	--push_to_hub \
+	--hub_model_id="ales/whisper-small-belarusian"

src/run_debug.sh

@@ -0,0 +1,44 @@
+python src/run_speech_recognition_seq2seq_streaming.py \
+	--model_name_or_path="openai/whisper-tiny" \
+	--dataset_name="mozilla-foundation/common_voice_11_0" \
+	--dataset_config_name="be" \
+	--language="be" \
+	--train_split_name="train" \
+	--eval_split_name="validation" \
+	--model_index_name="Whisper Tiny Belarusian" \
+	\
+	--max_steps="200" \
+	--max_eval_samples="64" \
+	--output_dir="./" \
+	--per_device_train_batch_size="32" \
+	--per_device_eval_batch_size="32" \
+	--logging_steps="10" \
+	--learning_rate="1e-5" \
+	--warmup_steps="0" \
+	--evaluation_strategy="steps" \
+	--eval_steps="10" \
+	--save_strategy="steps" \
+	--save_steps="10" \
+	--gradient_checkpointing \
+	--fp16 \
+	\
+	--shuffle_buffer_size="20" \
+	--generation_max_length="225" \
+	--max_duration_in_seconds="30" \
+	--text_column_name="sentence" \
+	--freeze_feature_encoder="False" \
+	--report_to="tensorboard" \
+	--metric_for_best_model="wer" \
+	--greater_is_better="False" \
+	--load_best_model_at_end \
+	\
+	--do_train \
+	--do_eval \
+	--resume_from_checkpoint="." \
+	--ignore_data_skip \
+	--predict_with_generate \
+	--do_normalize_eval \
+	--streaming \
+	--use_auth_token \
+	--push_to_hub \
+	--hub_model_id="ales/whisper-tiny-be-test"

run_speech_recognition_seq2seq_streaming.py → src/run_speech_recognition_seq2seq_streaming.py

@@ -365,12 +365,26 @@ def main():
     # 3. Detecting last checkpoint and eventually continue from last checkpoint
     last_checkpoint = None
     if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
+        logger.info(f'output_dir already exists. will try to load last checkpoint.')
+        
         last_checkpoint = get_last_checkpoint(training_args.output_dir)
-        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
-            raise ValueError(
-                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
-                "Use --overwrite_output_dir to overcome."
-            )
+        if last_checkpoint is None:
+            logger.info('last_checkpoint is None. will try to read from the model saved in the root of output_dir.')
+
+            dir_content = os.listdir(training_args.output_dir)
+            if len(dir_content) == 0:
+                logger.info('output_dir is empty. can not resume training. will start training from scratch.')
+            else:
+                model_fn = 'pytorch_model.bin'
+                if model_fn in dir_content:
+                    logger.info(f'found {model_fn} inside output_dir. '
+                                f'will continue training treating output_dir as a last checkpoint.')
+                    last_checkpoint = training_args.output_dir
+                else:
+                    raise ValueError(
+                        f"Output directory ({training_args.output_dir}) already exists and is not empty. "
+                        "Use --overwrite_output_dir to overcome."
+                    )
         elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:
             logger.info(
                 f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "

src/setup_env.sh

@@ -0,0 +1,17 @@
+sudo add-apt-repository -y ppa:jonathonf/ffmpeg-4
+sudo apt update
+sudo apt install -y ffmpeg
+
+sudo apt-get install git-lfs
+
+sudo apt-get install tmux
+
+python3 -m venv hf_env
+source hf_env/bin/activate
+echo "source ~/hf_env/bin/activate" >> ~/.bashrc
+
+git clone https://github.com/huggingface/community-events.git
+pip install -r community-events/whisper-fine-tuning-event/requirements.txt
+
+git config --global credential.helper store
+huggingface-cli login

train.log

@@ -72,3 +72,4 @@
   eval_samples_per_second =      3.366
   eval_steps_per_second   =      0.105
   eval_wer                =    55.1282
+{'loss': 0.2716, 'learning_rate': 9.5e-06, 'epoch': 0.05}

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:730a384a3b0a24e4869a732fa2d77f38cdbc3e5018ad72798f612c6759865a2f
+oid sha256:acefe54daf8bd4fb84bd849519c1fad1c02c35d7d3b1b1b321e84097615195a8
 size 3643