Training in progress, step 210

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8f9d8969563a430c539e7832a0f581b9fc1ecc43821015672854439cc52fdd59
+oid sha256:4f56ce3f19074d87c3d8e9a7b783e7309350cd8a1fc62c73f9385a13a9c9157c
 size 151098921

runs/Dec13_12-14-07_d7f040c448a8/1670933661.007405/events.out.tfevents.1670933661.d7f040c448a8.15037.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4cfcfaad843999a1b5c7b7f78449a77aceb6f30d65d780ac91c310870f3a1f4e
+size 5883

runs/Dec13_12-14-07_d7f040c448a8/events.out.tfevents.1670933660.d7f040c448a8.15037.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ff16a5d6ea1bba1706d86a4c1dd60b17f64b22f309363f616133efa89438979
+size 4744

src/readme.md

@@ -1,106 +0,0 @@
-## Description
-
-Fine-tuning [OpenAI Whisper](https://github.com/openai/whisper) model for Belarusian language during 
-[Whisper fine-tuning Event](https://github.com/huggingface/community-events/tree/main/whisper-fine-tuning-event)
-hosted by HuggingFace x Lambda.
-
-The code in this repository is a modified version of code from 
-[Whisper fine-tuning Event](https://github.com/huggingface/community-events/tree/main/whisper-fine-tuning-event) repo.
-
-## Fine-tuning todos:
-* perform evaluation of fine-tuned model on CommonVoice test set
-* check exact sizes of train, eval, test sets of CommonVoice 11
-
-## Resuming training from exising checkpoint
-When resuming training from existing checkpoint:
-* learning rate gets reset if passing same parameter value to training script as in previour run.<br>
-  need to provide learning rate from the last step of previous run to continue
-  training in a correct way.<br>
-  however even if passing learning rate from the last step, in the new run it has different value than expected
-  (probably because of warmup).
-* it's unclear whether decision on saving current model
-  is made by comparing current metrics with metrics of the best checkpoint. I guess model with worse performance
-  will not overwrite best model checkpoint already exising in the output dir, but need to double check.
-* we can set `ignore_data_skip=True` Training argument not to 
-  skip data items already passed to a model - that will save time on data loads.
-    * it's unclear whether order of input items in the train set (that is shuffled) will be the same 
-      across multiple reruns - i.e. it's unclear whether sampling is the same across reruns.
-    * if the sampling is the same across reruns, `ignore_data_skip=True` will lead to same items been passed to a model
-      in current run. it's OK if previous run ended with large step value on the last epoch.
-      if not, the same elements from the same epoch will be passed to a model again.
-
-## Questions:
-* What checkpoint (best, I guess) is saved in the `output_dir`? 
-  How is it overwritten when resuming training from existing checkpoint?
-
-### Prepended tokens
-* Why are there following lines in Data Collator?
-  ```python
-    # if bos token is appended in previous tokenization step,
-    # cut bos token here as it's append later anyways
-    if (labels[:, 0] == self.decoder_start_token_id).all().cpu().item():
-        labels = labels[:, 1:]
-    ```
-* `tokenizer.bos_token_id` vs `model.config.decoder_start_token_id`.<br>
-  which one to pass to Data Collator as `decoder_start_token_id` parameter?
-* Answer:
-  * In this case, the two are equivalent. You can verify this:
-    ```python
-    print(tokenizer.bos_token_id)
-    print(model.config.decoder_start_token_id)
-    ```
-
-  * Print Output:
-    ```
-    <|startoftranscript|>
-    <|startoftranscript|>
-    ```
-
-  * Technically speaking, the decoder_start_token_id is the correct convention here. Before starting generating any tokens, we initialise the generate method with a starting token, which is the decoder_start_token_id. 
-  See: https://huggingface.co/blog/how-to-generate. The decoder_start_token_id corresponds to the initial context word sequence, and is the zero'th token generated.
-
-  * We remove this token from the encoded labels in the data collator because we always set the zero'th generated token to the decoder_start_token_id. If we leave the decoder_start_token_id as part of the label sequence, then we'll predict the decoder_start_token_id as the zero'th token, and again as the first token! Because we're always forcing it as the zero'th token, we don't need to predict it as the first token, and so we remove it from the target lables
-
-  * These tokens are not forced in the generation process, and so we don't cut them in the data collator. We need to provide them to the model as target labels so that the model can learn the correct tasks from our data
-
-  * The tokens correspond to the audio language, task (translate or transcribe) and whether to predict timestamps
-
-  * We need to tell the model what language the audio corresponds to and what task it's performing during fine-tuning. This way, it learns what audio corresponds to what language, and the difference between transcribing audio vs translating it 
-
-## Notes:
-* using CommonVoice 11 dataset in a streaming way.<br>
-  use `streaming=True` for train & validation & test.<br>
-  as an alternative, we can use `streaming=False` for validation & test sets to save time on data processing.
-  but the size of validation and test sets are unknown (need to check).
-  it's likely they are going to be large - thus pre-download of these sets might not reduce 
-  overall fine-tuning time compared to streaming mode.
-* size of train set is ~370'000 audiofiles. if using `batch_size=64`, then
-  1 epoch will have ~5782 steps. <br>
-  Because of `--eval_steps="1000"` will use `--max_steps="6000"` instead of `--max_steps="5800"`
-  to have evaluation metrics computed in the end of training.
-* if using Google Colab, need to execute  `sudo chmod -R 777 .git` inside hf repo to 
-  to set right permissions to be able to push trained models to HuggingFace Hub
-* Whispers BasicTextNormalizer splits words containing apostrophe:
-  ```python
-  > from transformers.models.whisper.english_normalizer import BasicTextNormalizer
-  > normalizer = BasicTextNormalizer()
-  > normalizer("раз'яднаць")
-  'раз яднаць'
-  ```
-* That's why `BelarusianTextNormalizer` (edited version of `BasicTextNormalizer`) was added to training script:
-  ```python
-  > from run_speech_recognition_seq2seq_streaming import BelarusianTextNormalizer
-  > normalizer_be = BelarusianTextNormalizer()
-  > normalizer_be("раз'яднаць")
-  "раз'яднаць"
-  ```
-* Need to set `use_cache` to False since we're using gradient checkpointing, and the two are incompatible
-* Default Linear scheduler is used 
-* Default Adam optimizer is used
-* To save memory (and increase either model or batch_size) can experiment with:
-    * using Adafactor instead of Adam.
-      Adam requires two optimiser params per one model param, but Adafactor uses only one.
-      > A word of caution: Adafactor is untested for fine-tuning Whisper, 
-        so we are unsure sure how Adafactor performance compares to Adam!
-    * using Adam 8bit from `bitsandbytes` module. 
-      need to provide `optim="adamw_bnb_8bit"` param to `Seq2SeqTrainingArguments`

src/requirements.txt

@@ -1,9 +0,0 @@
-torch>=1.7
-torchaudio
-git+https://github.com/huggingface/transformers
-git+https://github.com/huggingface/datasets
-librosa
-jiwer
-evaluate>=0.3.0
-more-itertools
-tensorboard

src/run_debug.sh

@@ -7,7 +7,7 @@ python src/run_speech_recognition_seq2seq_streaming.py \
 	--eval_split_name="validation" \
 	--model_index_name="Whisper Tiny Belarusian" \
 	\
-	--max_steps="200" \
+	--max_steps="300" \
 	--max_eval_samples="64" \
 	--output_dir="./" \
 	--per_device_train_batch_size="32" \
@@ -34,7 +34,6 @@ python src/run_speech_recognition_seq2seq_streaming.py \
 	\
 	--do_train \
 	--do_eval \
-	--resume_from_checkpoint="." \
 	--ignore_data_skip \
 	--predict_with_generate \
 	--do_normalize_eval \

src/run_speech_recognition_seq2seq_streaming.py

@@ -368,28 +368,42 @@ def main():
         logger.info(f'output_dir already exists. will try to load last checkpoint.')
         
         last_checkpoint = get_last_checkpoint(training_args.output_dir)
-        if last_checkpoint is None:
-            logger.info('last_checkpoint is None. will try to read from the model saved in the root of output_dir.')
-
-            dir_content = os.listdir(training_args.output_dir)
-            if len(dir_content) == 0:
-                logger.info('output_dir is empty. can not resume training. will start training from scratch.')
+        if last_checkpoint is not None:
+            if training_args.resume_from_checkpoint is None:
+                logger.info(
+                    f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
+                    "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
+                )
+            else:
+                logger.info(f'Last checkpoint found at: {last_checkpoint}. Will ignore it and resume training '
+                            f'from passed resume_from_checkpoint param: {training_args.resume_from_checkpoint}')
+                assert os.path.isdir(training_args.resume_from_checkpoint)
+        else:    
+            logger.info('last_checkpoint is None. will try to read from training_args.resume_from_checkpoint')
+            
+            if training_args.resume_from_checkpoint is not None and os.path.isdir(training_args.resume_from_checkpoint):
+                logger.info(f'Will resume training from  passed resume_from_checkpoint param: '
+                            f'{training_args.resume_from_checkpoint}')
             else:
-                model_fn = 'pytorch_model.bin'
-                if model_fn in dir_content:
-                    logger.info(f'found {model_fn} inside output_dir. '
-                                f'will continue training treating output_dir as a last checkpoint.')
-                    last_checkpoint = training_args.output_dir
+                logger.info('last_checkpoint is None. resume_from_checkpoint is either None or not existing dir. '
+                            'will try to read from the model saved in the root of output_dir.')
+
+                dir_content = os.listdir(training_args.output_dir)
+                if len(dir_content) == 0:
+                    logger.info('output_dir is empty. will start training from scratch.')                    
                 else:
-                    raise ValueError(
-                        f"Output directory ({training_args.output_dir}) already exists and is not empty. "
-                        "Use --overwrite_output_dir to overcome."
-                    )
-        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:
-            logger.info(
-                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
-                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
-            )
+                    model_fn = 'pytorch_model.bin'
+                    if model_fn in dir_content:
+                        logger.info(f'found {model_fn} inside output_dir. '
+                                    f'will continue training treating output_dir as a last checkpoint.')
+                        last_checkpoint = training_args.output_dir
+                    else:
+                        raise ValueError(
+                            f'Could not find last_checkpoint, resume_from_checkpoint is either None '
+                            'or not existing dir, output_dir is non-empty but does not contain a model.'
+                            'Use --overwrite_output_dir to overcome.'
+                        )
+                        
 
     # Set seed before initializing model.
     set_seed(training_args.seed)

train.log

@@ -97,3 +97,4 @@
   eval_samples_per_second =      3.853
   eval_steps_per_second   =       0.12
   eval_wer                =    54.5788
+{'loss': 0.1922, 'learning_rate': 8.033333333333335e-06, 'epoch': 0.03}

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:acefe54daf8bd4fb84bd849519c1fad1c02c35d7d3b1b1b321e84097615195a8
+oid sha256:ba3e5752f0bcb0a1a12ea56a06813c45fc9e560ac738b4458d08b4141a6bb434
 size 3643