add QuartzNet15x5

README.md

@@ -23,6 +23,16 @@ tags:
   - _last checkpoint:_ ```quartznet15x5/2023-04-04_02-57-02/QuartzNet15x5--val_loss=1.9543-epoch=34-last.ckpt```
   - _Dataset train:_ **1042635** files totalling **1028.14** hours
   - _Dataset validation:_ **50640** files totalling **50.71** hours
+- 2023-02-23
+  - _language:_ uk-RU
+  - _from_pretrained:_ [stt_en_quartznet15x5](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_quartznet15x5)
+  - _Epoch:_ 7
+  - _Step:_ 217287
+  - _val_loss:_ 1.66200 (best 1.66200)
+  - _model:_ ```quartznet15x5/2023-02-23_11-43-23/stt_uk_ru_quartznet15x5.nemo```
+  - _last checkpoint:_ ```quartznet15x5/2023-02-23_11-43-23/QuartzNet15x5--val_loss=1.9543-epoch=34-last.ckpt```
+  - _Dataset train:_ **1738252** files totalling **1645.74** hours
+  - _Dataset validation:_ **11373** files totalling **12.63** hours
 
 
 ## Jasper10x5

quartznet15x5/2023-02-23_11-43-23/QuartzNet15x5--val_loss=1.6620-epoch=7-last.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15c24c27f9168713d36e02df2213d2b0358d07c05fe0dceb013709f01599ec19
+size 152422731

quartznet15x5/2023-02-23_11-43-23/README.md

@@ -0,0 +1,58 @@
+docker:
+```shell
+docker run --gpus all --ulimit memlock=-1 --ulimit stack=67108864 --shm-size=12gb -p 8888:8888 -p 6006:6006 -it --rm -v ${PWD}/ds:/ds -v ${PWD}/../../datasets:/datasets vadik/nemo-asr-local:1.15
+```
+
+Usage:
+```python
+import pytorch_lightning as pl
+from nemo.core.config import hydra_runner
+from nemo.utils import logging
+from nemo.utils.exp_manager import exp_manager
+from nemo.collections.asr.models import EncDecCTCModel, configs
+from nemo.utils.config_utils import update_model_config
+import nemo.collections.asr as nemo_asr
+import copy
+from omegaconf import DictConfig, OmegaConf
+
+
+@hydra_runner(config_path="conf", config_name="stt_uk_ru_quartznet15x5")
+def main(cfg: DictConfig):
+    """
+    >>> HYDRA_FULL_ERROR=1 python /ds/training_stt_en_quartznet15x5_v4.py
+    """
+    logging.info(f'Конфігурація Hydra')
+
+    model = nemo_asr.models.EncDecCTCModel.from_pretrained(
+        model_name="stt_en_quartznet15x5"
+    )  # EncDecCTCModel
+
+    model.change_vocabulary(
+        new_vocabulary=cfg['labels']
+    )
+
+    new_opt = copy.deepcopy(cfg['model']['optim'])
+    # Use the smaller learning rate we set before
+    model.setup_optimization(optim_config=DictConfig(new_opt))
+
+    model.setup_training_data(train_data_config=cfg['model']['train_ds'])
+    model.setup_validation_data(val_data_config=cfg['model']['validation_ds'])
+
+    # And now we can create a PyTorch Lightning trainer and call `fit` again.
+    trainer = pl.Trainer(**cfg['trainer'])
+
+    exp_manager(trainer, cfg['exp_manager'])
+
+    trainer.fit(model)
+
+    logging.info(f'Done!')
+
+    model.save_to(save_path="/ds/experiments/stt_en_quartznet15x5_v4.nemo")
+
+    logging.info(f'Done!')
+
+
+if __name__ == '__main__':
+    main()
+
+```

quartznet15x5/2023-02-23_11-43-23/lightning_logs.txt

@@ -0,0 +1,57 @@
+Using 16bit native Automatic Mixed Precision (AMP)
+GPU available: True (cuda), used: True
+TPU available: False, using: 0 TPU cores
+IPU available: False, using: 0 IPUs
+HPU available: False, using: 0 HPUs
+----------------------------------------------------------------------------------------------------
+distributed_backend=nccl
+All distributed processes registered. Starting with 1 processes
+----------------------------------------------------------------------------------------------------
+
+LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
+
+  | Name              | Type                              | Params
+------------------------------------------------------------------------
+0 | preprocessor      | AudioToMelSpectrogramPreprocessor | 0     
+1 | encoder           | ConvASREncoder                    | 18.9 M
+2 | spec_augmentation | SpectrogramAugmentation           | 0     
+3 | _wer              | WER                               | 0     
+4 | decoder           | ConvASRDecoder                    | 42.0 K
+5 | loss              | CTCLoss                           | 0     
+------------------------------------------------------------------------
+18.9 M    Trainable params
+0         Non-trainable params
+18.9 M    Total params
+37.873    Total estimated model params size (MB)
+Epoch 0, global step 6790: 'val_loss' reached 17.07648 (best 17.07648), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=17.0765-epoch=0.ckpt' as top 3
+Epoch 0, global step 13580: 'val_loss' reached 10.97846 (best 10.97846), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=10.9785-epoch=0.ckpt' as top 3
+Epoch 0, global step 20370: 'val_loss' reached 9.71839 (best 9.71839), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=9.7184-epoch=0.ckpt' as top 3
+Epoch 0, global step 27160: 'val_loss' reached 7.15682 (best 7.15682), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=7.1568-epoch=0.ckpt' as top 3
+Epoch 1, global step 33951: 'val_loss' reached 6.46626 (best 6.46626), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=6.4663-epoch=1.ckpt' as top 3
+Epoch 1, global step 40741: 'val_loss' reached 5.58261 (best 5.58261), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=5.5826-epoch=1.ckpt' as top 3
+Epoch 1, global step 47531: 'val_loss' reached 5.77446 (best 5.58261), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=5.7745-epoch=1.ckpt' as top 3
+Epoch 1, global step 54321: 'val_loss' reached 4.68531 (best 4.68531), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=4.6853-epoch=1.ckpt' as top 3
+Epoch 2, global step 61112: 'val_loss' reached 4.32403 (best 4.32403), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=4.3240-epoch=2.ckpt' as top 3
+Epoch 2, global step 67902: 'val_loss' reached 3.81663 (best 3.81663), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=3.8166-epoch=2.ckpt' as top 3
+Epoch 2, global step 74692: 'val_loss' reached 4.16412 (best 3.81663), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=4.1641-epoch=2.ckpt' as top 3
+Epoch 2, global step 81482: 'val_loss' reached 3.37959 (best 3.37959), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=3.3796-epoch=2.ckpt' as top 3
+Epoch 3, global step 88273: 'val_loss' reached 3.45001 (best 3.37959), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=3.4500-epoch=3.ckpt' as top 3
+Epoch 3, global step 95063: 'val_loss' reached 2.94031 (best 2.94031), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.9403-epoch=3.ckpt' as top 3
+Epoch 3, global step 101853: 'val_loss' was not in top 3
+Epoch 3, global step 108643: 'val_loss' reached 2.78752 (best 2.78752), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.7875-epoch=3.ckpt' as top 3
+Epoch 4, global step 115434: 'val_loss' reached 2.69196 (best 2.69196), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.6920-epoch=4.ckpt' as top 3
+Epoch 4, global step 122224: 'val_loss' reached 2.59559 (best 2.59559), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.5956-epoch=4.ckpt' as top 3
+Epoch 4, global step 129014: 'val_loss' was not in top 3
+Epoch 4, global step 135804: 'val_loss' reached 2.47542 (best 2.47542), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.4754-epoch=4.ckpt' as top 3
+Epoch 5, global step 142595: 'val_loss' reached 2.34722 (best 2.34722), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.3472-epoch=5.ckpt' as top 3
+Epoch 5, global step 149385: 'val_loss' reached 2.22968 (best 2.22968), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.2297-epoch=5.ckpt' as top 3
+Epoch 5, global step 156175: 'val_loss' was not in top 3
+Epoch 5, global step 162965: 'val_loss' reached 2.00784 (best 2.00784), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.0078-epoch=5.ckpt' as top 3
+Epoch 6, global step 169756: 'val_loss' reached 2.13385 (best 2.00784), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.1338-epoch=6.ckpt' as top 3
+Epoch 6, global step 176546: 'val_loss' reached 1.95768 (best 1.95768), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=1.9577-epoch=6.ckpt' as top 3
+Epoch 6, global step 183336: 'val_loss' was not in top 3
+Epoch 6, global step 190126: 'val_loss' reached 1.91314 (best 1.91314), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=1.9131-epoch=6.ckpt' as top 3
+Epoch 7, global step 196917: 'val_loss' reached 1.90434 (best 1.90434), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=1.9043-epoch=7.ckpt' as top 3
+Epoch 7, global step 203707: 'val_loss' reached 1.77183 (best 1.77183), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=1.7718-epoch=7.ckpt' as top 3
+Epoch 7, global step 210497: 'val_loss' was not in top 3
+Epoch 7, global step 217287: 'val_loss' reached 1.66200 (best 1.66200), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=1.6620-epoch=7.ckpt' as top 3

quartznet15x5/2023-02-23_11-43-23/stt_uk_ru_quartznet15x5.nemo

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:adcfb10e9e02f00acac8ec952906693ac733a76bf30a5616fe4b63e4f7046f71
+size 76400640

quartznet15x5/2023-02-23_11-43-23/stt_uk_ru_quartznet15x5.yaml

@@ -0,0 +1,287 @@
+name: &name "stt_uk_ru_quartznet15x5"
+
+num_classes: &num_classes 40
+sample_rate: &sample_rate 16000
+repeat: &repeat 5
+dropout: &dropout 0.0
+separable: &separable true
+labels: &labels [ " ", "а", "б", "в", "г", "ґ", "д", "е", "є", "ж", "з", "и", "і", "ї", "й", "к", "л", "м", "н",
+                "о", "п", "р", "с", "т", "у", "ф", "х", "ц", "ч", "ш", "щ", "ь", "ю", "я",
+                "ё", "ъ", "ы", "э", "'", "-" ]
+
+model:
+  labels: *labels
+  sample_rate: *sample_rate
+  repeat: *repeat
+  dropout: *dropout
+  separable: *separable
+
+  train_ds:
+    manifest_filepath: /datasets/full_nemo_tarred/tarred_audio_manifest_utf8.json
+    sample_rate: *sample_rate
+    labels: *labels
+    batch_size: 64
+    trim_silence: True
+    max_duration: 9.7
+    min_duration: 1.0
+    shuffle: True
+#    shuffle_n: 2048
+    is_tarred: True
+    tarred_audio_filepaths: /datasets/full_nemo_tarred/audio__OP_0..299_CL_.tar
+    num_workers: 32
+    bucketing_strategy: "synced_randomized"
+    bucketing_batch_size: null
+
+  validation_ds:
+    manifest_filepath:
+      - /datasets/ds/cv_corpus_all_2022_12_31_uk/nemo_validation_manifest_docker.json
+      - /datasets/ds/cv_corpus_all_2022_12_31_ru/nemo_validation_manifest_docker.json
+    sample_rate: *sample_rate
+    labels: *labels
+    shuffle: False
+    batch_size: 32
+
+  preprocessor:
+    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+    normalize: "per_feature"
+    window_size: 0.02
+    sample_rate: *sample_rate
+    window_stride: 0.01
+    window: "hann"
+    features: &n_mels 64
+    n_fft: 512
+    frame_splicing: 1
+    dither: 0.00001
+
+  spec_augment:
+    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
+    rect_freq: 50
+    rect_masks: 5
+    rect_time: 120
+
+  encoder:
+    _target_: nemo.collections.asr.modules.ConvASREncoder
+    feat_in: *n_mels
+    activation: relu
+    conv_mask: true
+
+    jasper:
+      - dilation: [1]
+        dropout: *dropout
+        filters: 256
+        kernel: [33]
+        repeat: 1
+        residual: false
+        separable: *separable
+        stride: [2]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 256
+        kernel: [33]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 256
+        kernel: [33]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 256
+        kernel: [33]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 256
+        kernel: [39]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 256
+        kernel: [39]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 256
+        kernel: [39]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 512
+        kernel: [51]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 512
+        kernel: [51]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 512
+        kernel: [51]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 512
+        kernel: [63]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 512
+        kernel: [63]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 512
+        kernel: [63]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 512
+        kernel: [75]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 512
+        kernel: [75]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: 512
+        kernel: [75]
+        repeat: *repeat
+        residual: true
+        separable: *separable
+        stride: [1]
+
+      - dilation: [2]
+        dropout: *dropout
+        filters: 512
+        kernel: [87]
+        repeat: 1
+        residual: false
+        separable: *separable
+        stride: [1]
+
+      - dilation: [1]
+        dropout: *dropout
+        filters: &enc_filters 1024
+        kernel: [1]
+        repeat: 1
+        residual: false
+        stride: [1]
+
+  decoder:
+    _target_: nemo.collections.asr.modules.ConvASRDecoder
+    feat_in: *enc_filters
+    num_classes: *num_classes
+    vocabulary: *labels
+
+  optim:
+    name: novograd
+    lr: 0.001
+    betas: [0.8, 0.5]
+    weight_decay: 0.0001
+
+    sched:
+      name: CosineAnnealing
+      warmup_steps: null
+      warmup_ratio: null
+      min_lr: 0.0
+      last_epoch: -1
+
+      # pytorch lightning args
+      monitor: val_loss
+      reduce_on_plateau: false
+
+trainer:
+  #  gpus: -1 # number of gpus was configured so validation will run at the end of the training epoch..
+  accelerator: gpu
+  devices: -1 # number of gpus
+  max_epochs: 50
+  max_steps: -1 # computed at runtime if not set
+  num_nodes: 1
+  strategy: ddp
+  accumulate_grad_batches: 1
+  enable_checkpointing: False  # Provided by exp_manager
+  logger: False  # Provided by exp_manager
+  log_every_n_steps: 1  # Interval of logging.
+  val_check_interval: 0.25  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
+  benchmark: false # має бути false для моделей із мовним введенням змінної довжини, оскільки це сповільнює навчання
+  precision: 16 # Should be set to 16 for O1 and O2 to enable the AMP.
+
+exp_manager:
+  exp_dir: "/ds/experiments"
+  name: *name
+  create_tensorboard_logger: False
+  create_checkpoint_callback: True
+#  checkpoint_callback_params:
+#    monitor: "val_wer"
+#    mode: "min"
+  create_wandb_logger: False
+  wandb_logger_kwargs:
+    name: null
+    project: null
+
+hydra:
+  run:
+    dir: .
+  job_logging:
+    root:
+      handlers: null

quartznet15x5/2023-02-23_11-43-23/terminal_output.txt

@@ -0,0 +1,263 @@
+HYDRA_FULL_ERROR=1 python /ds/training_stt_en_quartznet15x5_v4.py
+[NeMo W 2023-02-23 11:38:01 experimental:27] Module <class 'nemo.collections.asr.models.audio_to_audio_model.AudioToAudioModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
+[NeMo W 2023-02-23 11:38:02 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.
+[NeMo W 2023-02-23 11:38:02 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/torch/jit/annotations.py:309: UserWarning: TorchScript will treat type annotations of Tensor dtype-specific subtypes as if they are normal Tensors. dtype constraints are not enforced in compilation either.
+      warnings.warn("TorchScript will treat type annotations of Tensor "
+
+[NeMo W 2023-02-23 11:38:02 experimental:27] Module <class 'nemo.collections.asr.data.audio_to_audio.BaseAudioDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.
+[NeMo W 2023-02-23 11:38:02 experimental:27] Module <class 'nemo.collections.asr.data.audio_to_audio.AudioToTargetDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.
+[NeMo W 2023-02-23 11:38:02 experimental:27] Module <class 'nemo.collections.asr.data.audio_to_audio.AudioToTargetWithReferenceDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.
+[NeMo W 2023-02-23 11:38:02 experimental:27] Module <class 'nemo.collections.asr.data.audio_to_audio.AudioToTargetWithEmbeddingDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.
+[NeMo W 2023-02-23 11:38:02 experimental:27] Module <class 'nemo.collections.asr.models.enhancement_models.EncMaskDecAudioToAudioModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
+[NeMo W 2023-02-23 11:38:02 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
+    See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
+      ret = run_job(
+
+[NeMo I 2023-02-23 11:38:02 training_stt_en_quartznet15x5_v4:21] Конфігурація Hydra
+[NeMo I 2023-02-23 11:38:02 cloud:56] Found existing object /root/.cache/torch/NeMo/NeMo_1.15.0/stt_en_quartznet15x5/16661021d16e679bdfd97a2a03944c49/stt_en_quartznet15x5.nemo.
+[NeMo I 2023-02-23 11:38:02 cloud:62] Re-using file from: /root/.cache/torch/NeMo/NeMo_1.15.0/stt_en_quartznet15x5/16661021d16e679bdfd97a2a03944c49/stt_en_quartznet15x5.nemo
+[NeMo I 2023-02-23 11:38:02 common:913] Instantiating model from pre-trained checkpoint
+[NeMo W 2023-02-23 11:38:03 modelPT:156] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
+    Train config :
+    manifest_filepath: /data2/voices/train_1k.json
+    sample_rate: 16000
+    labels:
+    - ' '
+    - a
+    - b
+    - c
+    - d
+    - e
+    - f
+    - g
+    - h
+    - i
+    - j
+    - k
+    - l
+    - m
+    - 'n'
+    - o
+    - p
+    - q
+    - r
+    - s
+    - t
+    - u
+    - v
+    - w
+    - x
+    - 'y'
+    - z
+    - ''''
+    batch_size: 32
+    trim_silence: true
+    max_duration: 16.7
+    shuffle: true
+    is_tarred: false
+    tarred_audio_filepaths: /asr_set_1.2/train/train_{0..1023}.tar
+    num_workers: 20
+
+[NeMo W 2023-02-23 11:38:03 modelPT:163] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
+    Validation config :
+    manifest_filepath: /data2/voices/train_1k_samp.json
+    sample_rate: 16000
+    labels:
+    - ' '
+    - a
+    - b
+    - c
+    - d
+    - e
+    - f
+    - g
+    - h
+    - i
+    - j
+    - k
+    - l
+    - m
+    - 'n'
+    - o
+    - p
+    - q
+    - r
+    - s
+    - t
+    - u
+    - v
+    - w
+    - x
+    - 'y'
+    - z
+    - ''''
+    batch_size: 32
+    shuffle: false
+
+[NeMo I 2023-02-23 11:38:03 features:267] PADDING: 16
+[NeMo I 2023-02-23 11:38:04 save_restore_connector:243] Model EncDecCTCModel was successfully restored from /root/.cache/torch/NeMo/NeMo_1.15.0/stt_en_quartznet15x5/16661021d16e679bdfd97a2a03944c49/stt_en_quartznet15x5.nemo.
+[NeMo I 2023-02-23 11:38:04 ctc_models:294] Changed decoder to output to [' ', 'а', 'б', 'в', 'г', 'ґ', 'д', 'е', 'є', 'ж', 'з', 'и', 'і', 'ї', 'й', 'к', 'л', 'м', 'н', 'о', 'п', 'р', 'с', 'т', 'у', 'ф', 'х', 'ц', 'ч', 'ш', 'щ', 'ь', 'ю', 'я', 'ё', 'ъ', 'ы', 'э', "'", '-'] vocabulary.
+[NeMo W 2023-02-23 11:38:04 modelPT:505] Trainer wasn't specified in model constructor. Make sure that you really wanted it.
+[NeMo I 2023-02-23 11:38:04 modelPT:616] Optimizer config = Novograd (
+    Parameter Group 0
+        amsgrad: False
+        betas: [0.8, 0.5]
+        eps: 1e-08
+        grad_averaging: False
+        lr: 0.001
+        weight_decay: 0.0001
+    )
+[NeMo W 2023-02-23 11:38:04 lr_scheduler:890] Neither `max_steps` nor `iters_per_batch` were provided to `optim.sched`, cannot compute effective `max_steps` !
+    Scheduler will not be instantiated !
+[NeMo W 2023-02-23 11:38:04 audio_to_text_dataset:56] `labels` is explicitly provided to the data loader, and is different from the `labels` provided at the model level config.
+    If this is incorrect, please set the dataloader's `labels` to None.
+[NeMo I 2023-02-23 11:43:15 collections:193] Dataset loaded with 1738252 files totalling 1645.74 hours
+[NeMo I 2023-02-23 11:43:15 collections:194] 18548 files were filtered totalling 61.11 hours
+[NeMo W 2023-02-23 11:43:15 ctc_models:461] Model Trainer was not set before constructing the dataset, incorrect number of training batches will be used. Please set the trainer and rebuild the dataset.
+[NeMo W 2023-02-23 11:43:15 audio_to_text_dataset:56] `labels` is explicitly provided to the data loader, and is different from the `labels` provided at the model level config.
+    If this is incorrect, please set the dataloader's `labels` to None.
+[NeMo I 2023-02-23 11:43:23 collections:193] Dataset loaded with 11373 files totalling 12.63 hours
+[NeMo I 2023-02-23 11:43:23 collections:194] 0 files were filtered totalling 0.00 hours
+Using 16bit native Automatic Mixed Precision (AMP)
+GPU available: True (cuda), used: True
+TPU available: False, using: 0 TPU cores
+IPU available: False, using: 0 IPUs
+HPU available: False, using: 0 HPUs
+[NeMo I 2023-02-23 11:43:23 exp_manager:361] Experiments will be logged at /ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23
+Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
+----------------------------------------------------------------------------------------------------
+distributed_backend=nccl
+All distributed processes registered. Starting with 1 processes
+----------------------------------------------------------------------------------------------------
+
+LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
+[NeMo I 2023-02-23 11:43:24 modelPT:616] Optimizer config = Novograd (
+    Parameter Group 0
+        amsgrad: False
+        betas: [0.8, 0.5]
+        eps: 1e-08
+        grad_averaging: False
+        lr: 0.001
+        weight_decay: 0.0001
+    )
+[NeMo I 2023-02-23 11:43:24 lr_scheduler:910] Scheduler "<nemo.core.optim.lr_scheduler.CosineAnnealing object at 0x7f6e52320a60>"
+    will be used during training (effective maximum steps = 1358050) -
+    Parameters :
+    (warmup_steps: null
+    warmup_ratio: null
+    min_lr: 0.0
+    last_epoch: -1
+    max_steps: 1358050
+    )
+
+  | Name              | Type                              | Params
+------------------------------------------------------------------------
+0 | preprocessor      | AudioToMelSpectrogramPreprocessor | 0
+1 | encoder           | ConvASREncoder                    | 18.9 M
+2 | spec_augmentation | SpectrogramAugmentation           | 0
+3 | _wer              | WER                               | 0
+4 | decoder           | ConvASRDecoder                    | 42.0 K
+5 | loss              | CTCLoss                           | 0
+------------------------------------------------------------------------
+18.9 M    Trainable params
+0         Non-trainable params
+18.9 M    Total params
+37.873    Total estimated model params size (MB)
+Sanity Checking: 0it [00:00, ?it/s][NeMo W 2023-02-23 11:43:24 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py:224: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 32 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
+      rank_zero_warn(
+
+Sanity Checking DataLoader 0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  2.72it/s][NeMo W 2023-02-23 11:43:25 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:537: PossibleUserWarning: It is recommended to use `self.log('global_step', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.
+      warning_cache.warn(
+
+[NeMo W 2023-02-23 11:43:25 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:537: PossibleUserWarning: It is recommended to use `self.log('val_loss', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.
+      warning_cache.warn(
+
+[NeMo W 2023-02-23 11:43:25 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:537: PossibleUserWarning: It is recommended to use `self.log('val_wer', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.
+      warning_cache.warn(
+
+[NeMo W 2023-02-23 11:43:25 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/lightning_lite/utilities/data.py:63: UserWarning: Your `IterableDataset` has `__len__` defined. In combination with multi-process data loading (when num_workers > 1), `__len__` could be inaccurate if each worker is not configured independently to avoid having duplicate data.
+      rank_zero_warn(
+
+Epoch 0:   0%|                                                                                                                                                                          | 0/28585 [00:00<?, ?it/s][W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
+[NeMo W 2023-02-23 11:43:28 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/torch/optim/lr_scheduler.py:138: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
+      warnings.warn("Detected call of `lr_scheduler.step()` before `optimizer.step()`. "
+
+Epoch 0:  25%|████████████████████████████████████▍                                                                                                             | 7146/28585 [34:16<1:42:51,  3.47it/s, loss=26.1
+Epoch 0, global step 6790: 'val_loss' reached 17.07648 (best 17.07648), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=17.0765-epoch=0.ckpt' as top 3
+Epoch 0:  50%|███████████████████████████████████████████████████████████████████████▍                                                                       | 14292/28585 [1:08:40<1:08:40,  3.47it/s, loss=26.7
+Epoch 0, global step 13580: 'val_loss' reached 10.97846 (best 10.97846), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=10.9785-epoch=0.ckpt' as top 3
+Epoch 0:  55%|███████████████████████████████████████████████████████████████████████████████▋                                                                 | 15719/28585 [1:16:06<1:02:17,  3.44it/s, loss=20]
+Epoch 0:  64%|█████████████████████████████████████████████████████████████████████████████████████████████▎                                                   | 18394/28585 [1:28:51<49:13,  3.45it/s, loss=13.2]
+Epoch 0:  75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                    | 21438/28585 [1:43:49<34:36,  3.44it/s, loss=11.4
+Epoch 0, global step 20370: 'val_loss' reached 9.71839 (best 9.71839), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=9.7184-epoch=0.ckpt' as top 3
+Epoch 0:  75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▉                                    | 21486/28585 [1:44:07<34:24,  3.44it/s, loss=10.2]
+Epoch 0:  91%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎            | 26080/28585 [2:05:48<12:05,  3.45it/s, loss=21.6]
+Epoch 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 28584/28585 [2:18:41<00:00,  3.44it/s, loss=28.5
+Epoch 0, global step 27160: 'val_loss' reached 7.15682 (best 7.15682), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=7.1568-epoch=0.ckpt' as top 3
+Epoch 1:  25%|████████████████████████████████████▍                                                                                                             | 7146/28585 [34:01<1:42:03,  3.50it/s, loss=17.7
+Epoch 1, global step 33951: 'val_loss' reached 6.46626 (best 6.46626), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=6.4663-epoch=1.ckpt' as top 3
+Epoch 1:  50%|███████████████████████████████████████████████████████████████████████▍                                                                       | 14292/28585 [1:08:25<1:08:25,  3.48it/s, loss=20.3
+Epoch 1, global step 40741: 'val_loss' reached 5.58261 (best 5.58261), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=5.5826-epoch=1.ckpt' as top 3
+Epoch 1:  75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                    | 21438/28585 [1:43:30<34:30,  3.45it/s, loss=8.82
+Epoch 1, global step 47531: 'val_loss' reached 5.77446 (best 5.58261), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=5.7745-epoch=1.ckpt' as top 3
+Epoch 1: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 28584/28585 [2:18:16<00:00,  3.45it/s, loss=23.5
+Epoch 1, global step 54321: 'val_loss' reached 4.68531 (best 4.68531), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=4.6853-epoch=1.ckpt' as top 3
+Epoch 2:  25%|████████████████████████████████████▍                                                                                                             | 7146/28585 [34:06<1:42:18,  3.49it/s, loss=15.2
+Epoch 2, global step 61112: 'val_loss' reached 4.32403 (best 4.32403), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=4.3240-epoch=2.ckpt' as top 3
+Epoch 2:  50%|███████████████████████████████████████████████████████████████████████▍                                                                       | 14292/28585 [1:08:31<1:08:32,  3.48it/s, loss=16.2
+Epoch 2, global step 67902: 'val_loss' reached 3.81663 (best 3.81663), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=3.8166-epoch=2.ckpt' as top 3
+Epoch 2:  75%|███████████████████████████████████████████████████████████████████████████���████████████████████████████████▋                                    | 21438/28585 [1:43:50<34:37,  3.44it/s, loss=6.72
+Epoch 2, global step 74692: 'val_loss' reached 4.16412 (best 3.81663), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=4.1641-epoch=2.ckpt' as top 3
+Epoch 2: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 28584/28585 [2:18:39<00:00,  3.44it/s, loss=21.3
+Epoch 2, global step 81482: 'val_loss' reached 3.37959 (best 3.37959), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=3.3796-epoch=2.ckpt' as top 3
+Epoch 3:  25%|████████████████████████████████████▍                                                                                                             | 7146/28585 [33:58<1:41:54,  3.51it/s, loss=13.6
+Epoch 3, global step 88273: 'val_loss' reached 3.45001 (best 3.37959), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=3.4500-epoch=3.ckpt' as top 3
+Epoch 3:  50%|███████████████████████████████████████████████████████████████████████▍                                                                       | 14292/28585 [1:08:41<1:08:41,  3.47it/s, loss=14.3
+Epoch 3, global step 95063: 'val_loss' reached 2.94031 (best 2.94031), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.9403-epoch=3.ckpt' as top 3
+Epoch 3:  75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                    | 21438/28585 [1:43:51<34:37,  3.44it/s, loss=6.29
+Epoch 3, global step 101853: 'val_loss' was not in top 3
+Epoch 3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 28584/28585 [2:18:30<00:00,  3.44it/s, loss=18.6
+Epoch 3, global step 108643: 'val_loss' reached 2.78752 (best 2.78752), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.7875-epoch=3.ckpt' as top 3
+Epoch 4:  25%|████████████████████████████████████▍                                                                                                             | 7146/28585 [34:20<1:43:01,  3.47it/s, loss=12.4
+Epoch 4, global step 115434: 'val_loss' reached 2.69196 (best 2.69196), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.6920-epoch=4.ckpt' as top 3
+Epoch 4:  50%|███████████████████████████████████████████████████████████████████████▍                                                                       | 14292/28585 [1:08:58<1:08:58,  3.45it/s, loss=13.1
+Epoch 4, global step 122224: 'val_loss' reached 2.59559 (best 2.59559), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.5956-epoch=4.ckpt' as top 3
+Epoch 4:  75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                    | 21438/28585 [1:44:16<34:45,  3.43it/s, loss=4.96
+Epoch 4, global step 129014: 'val_loss' was not in top 3
+Epoch 4: 100%|███████████████████████████��████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 28584/28585 [2:18:56<00:00,  3.43it/s, loss=16.7
+Epoch 4, global step 135804: 'val_loss' reached 2.47542 (best 2.47542), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.4754-epoch=4.ckpt' as top 3
+Epoch 5:  25%|████████████████████████████████████▍                                                                                                             | 7146/28585 [34:03<1:42:10,  3.50it/s, loss=11.3
+Epoch 5, global step 142595: 'val_loss' reached 2.34722 (best 2.34722), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.3472-epoch=5.ckpt' as top 3
+Epoch 5:  50%|███████████████████████████████████████████████████████████████████████▍                                                                       | 14292/28585 [1:08:38<1:08:38,  3.47it/s, loss=12.5
+Epoch 5, global step 149385: 'val_loss' reached 2.22968 (best 2.22968), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.2297-epoch=5.ckpt' as top 3
+Epoch 5:  75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                    | 21438/28585 [1:43:57<34:39,  3.44it/s, loss=5.45
+Epoch 5, global step 156175: 'val_loss' was not in top 3
+Epoch 5: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 28584/28585 [2:18:35<00:00,  3.44it/s, loss=17.1
+Epoch 5, global step 162965: 'val_loss' reached 2.00784 (best 2.00784), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.0078-epoch=5.ckpt' as top 3
+Epoch 6:  25%|████████████████████████████████████▍                                                                                                             | 7146/28585 [34:29<1:43:30,  3.45it/s, loss=10.7
+Epoch 6, global step 169756: 'val_loss' reached 2.13385 (best 2.00784), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=2.1338-epoch=6.ckpt' as top 3
+Epoch 6:  50%|███████████████████████████████████████████████████████████████████████▍                                                                       | 14292/28585 [1:08:50<1:08:50,  3.46it/s, loss=11.3
+Epoch 6, global step 176546: 'val_loss' reached 1.95768 (best 1.95768), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=1.9577-epoch=6.ckpt' as top 3
+Epoch 6:  75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                    | 21438/28585 [1:44:03<34:41,  3.43it/s, loss=4.76
+Epoch 6, global step 183336: 'val_loss' was not in top 3
+Epoch 6: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 28584/28585 [2:18:34<00:00,  3.44it/s, loss=16.4
+Epoch 6, global step 190126: 'val_loss' reached 1.91314 (best 1.91314), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=1.9131-epoch=6.ckpt' as top 3
+Epoch 7:  25%|████████████████████████████████████▍                                                                                                             | 7146/28585 [34:35<1:43:47,  3.44it/s, loss=10.9
+Epoch 7, global step 196917: 'val_loss' reached 1.90434 (best 1.90434), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=1.9043-epoch=7.ckpt' as top 3
+Epoch 7:  50%|███████████████████████████████████████████████████████████████████████▍                                                                       | 14292/28585 [1:09:00<1:09:00,  3.45it/s, loss=11.6
+Epoch 7, global step 203707: 'val_loss' reached 1.77183 (best 1.77183), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=1.7718-epoch=7.ckpt' as top 3
+Epoch 7:  58%|███████████████████████████████████████████████████████████████████████████████████▉                                                             | 16559/28585 [1:20:27<58:25,  3.43it/s, loss=10.9]
+Epoch 7:  69%|████████████████████████████████████████████████████████████████████████████████████████████████████▋                                            | 19848/28585 [1:36:38<42:32,  3.42it/s, loss=11.2]
+Epoch 7:  75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                    | 21438/28585 [1:45:22<35:07,  3.39it/s, loss=4.96
+Epoch 7, global step 210497: 'val_loss' was not in top 3
+Epoch 7:  98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████   | 28014/28585 [2:17:05<02:47,  3.41it/s, loss=12.1]
+Epoch 7:  99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 28320/28585 [2:18:51<01:17,  3.40it/s, loss=14.6]
+Epoch 7: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 28584/28585 [2:20:34<00:00,  3.39it/s, loss=14.6
+Epoch 7, global step 217287: 'val_loss' reached 1.66200 (best 1.66200), saving model to '/ds/experiments/stt_en_quartznet15x5_v4/2023-02-23_11-43-23/checkpoints/stt_en_quartznet15x5_v4--val_loss=1.6620-epoch=7.ckpt' as top 3
+Epoch 8:   0%|▋                                                                                                                                                  | 140/28585 [00:37<2:06:59,  3.73it/s, loss=10.4]
+^C[NeMo W 2023-02-24 06:15:00 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/call.py:48: UserWarning: Detected KeyboardInterrupt, attempting graceful shutdown...
+      rank_zero_warn("Detected KeyboardInterrupt, attempting graceful shutdown...")
+
+[NeMo I 2023-02-24 06:15:00 training_stt_en_quartznet15x5_v4:45] Done!
+[NeMo I 2023-02-24 06:15:00 training_stt_en_quartznet15x5_v4:49] Done!
+root@bd72308fa83d:/workspace/nemo#