Hugging Face's logo

u107
/
nemo

NeMo
Ukrainian
Russian
deepspeech
Model card Files
xet
Vadim commited on
Commit
9fba329
·
1 Parent(s): 19f4702

add QuartzNet15x5

Browse files
Files changed (7) hide show
  1. README.md +11 -1
  2. quartznet15x5/2023-04-04_02-57-02/QuartzNet15x5--val_loss=1.9543-epoch=34-last.ckpt +3 -0
  3. quartznet15x5/2023-04-04_02-57-02/README.md +58 -0
  4. quartznet15x5/2023-04-04_02-57-02/lightning_logs.txt +165 -0
  5. quartznet15x5/2023-04-04_02-57-02/stt_uk_quartznet15x5.nemo +3 -0
  6. quartznet15x5/2023-04-04_02-57-02/stt_uk_quartznet15x5.yaml +282 -0
  7. quartznet15x5/2023-04-04_02-57-02/terminal_output.txt +507 -0
README.md CHANGED
@@ -13,6 +13,16 @@ tags:
13
  ## QuartzNet15x5
14
 
15
  ---
 
 
 
 
 
 
 
 
 
 
16
 
17
 
18
  ## Jasper10x5
@@ -23,7 +33,7 @@ tags:
23
  - _from_pretrained:_ [stt_en_jasper10x5dr](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_jasper10x5dr)
24
  - _Epoch:_ 17
25
  - _Step:_ 293256
26
- - _val_loss:_ 2.10919
27
  - _model:_ ```jasper10x5/2023-04-07_05-01-29/stt_uk_jasper10x5.nemo```
28
  - _last checkpoint:_ ```jasper10x5/2023-04-07_05-01-29/Jasper10x5--val_loss=2.1092-epoch=17-last.ckpt```
29
  - _Dataset train:_ **1042635** files totalling **1028.14** hours
 
13
  ## QuartzNet15x5
14
 
15
  ---
16
+ - 2023-04-04_02
17
+ - _language:_ uk
18
+ - _from_pretrained:_ [stt_en_quartznet15x5](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_quartznet15x5)
19
+ - _Epoch:_ 34
20
+ - _Step:_ 566076
21
+ - _val_loss:_ 1.90886 (best 1.89907)
22
+ - _model:_ ```quartznet15x5/2023-04-04_02-57-02/stt_uk_quartznet15x5.nemo```
23
+ - _last checkpoint:_ ```quartznet15x5/2023-04-04_02-57-02/QuartzNet15x5--val_loss=1.9543-epoch=34-last.ckpt```
24
+ - _Dataset train:_ **1042635** files totalling **1028.14** hours
25
+ - _Dataset validation:_ **50640** files totalling **50.71** hours
26
 
27
 
28
  ## Jasper10x5
 
33
  - _from_pretrained:_ [stt_en_jasper10x5dr](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_jasper10x5dr)
34
  - _Epoch:_ 17
35
  - _Step:_ 293256
36
+ - _val_loss:_ 2.10919 (best 2.10919)
37
  - _model:_ ```jasper10x5/2023-04-07_05-01-29/stt_uk_jasper10x5.nemo```
38
  - _last checkpoint:_ ```jasper10x5/2023-04-07_05-01-29/Jasper10x5--val_loss=2.1092-epoch=17-last.ckpt```
39
  - _Dataset train:_ **1042635** files totalling **1028.14** hours
quartznet15x5/2023-04-04_02-57-02/QuartzNet15x5--val_loss=1.9543-epoch=34-last.ckpt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:02ae51524978d9aef862ef525b6bba232f6ad6d7bc0ae06945fe66c29149978e
3
+ size 152366091
quartznet15x5/2023-04-04_02-57-02/README.md ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ docker:
2
+ ```shell
3
+ docker run --gpus all --ulimit memlock=-1 --ulimit stack=67108864 --shm-size=18gb -it --rm -v ${PWD}/ds:/ds -v ${PWD}/../../datasets:/datasets nvcr.io/nvidia/nemo:23.02
4
+ ```
5
+
6
+ Usage:
7
+ ```python
8
+ import pytorch_lightning as pl
9
+ from nemo.core.config import hydra_runner
10
+ from nemo.utils import logging
11
+ from nemo.utils.exp_manager import exp_manager
12
+ from nemo.collections.asr.models import EncDecCTCModel, configs
13
+ from nemo.utils.config_utils import update_model_config
14
+ import nemo.collections.asr as nemo_asr
15
+ import copy
16
+ from omegaconf import DictConfig, OmegaConf
17
+
18
+
19
+ @hydra_runner(config_path="conf", config_name="stt_en_quartznet15x5_v1")
20
+ def main(cfg: DictConfig):
21
+ """
22
+ >>> HYDRA_FULL_ERROR=1 python /ds/training_stt_en_quartznet15x5_v1.py
23
+ """
24
+ logging.info(f'Конфігурація Hydra')
25
+
26
+ model = nemo_asr.models.EncDecCTCModel.from_pretrained(
27
+ model_name="stt_en_quartznet15x5"
28
+ ) # EncDecCTCModel
29
+
30
+ model.change_vocabulary(
31
+ new_vocabulary=cfg['labels']
32
+ )
33
+
34
+ new_opt = copy.deepcopy(cfg['model']['optim'])
35
+ # Use the smaller learning rate we set before
36
+ model.setup_optimization(optim_config=DictConfig(new_opt))
37
+
38
+ model.setup_training_data(train_data_config=cfg['model']['train_ds'])
39
+ model.setup_validation_data(val_data_config=cfg['model']['validation_ds'])
40
+
41
+ # And now we can create a PyTorch Lightning trainer and call `fit` again.
42
+ trainer = pl.Trainer(**cfg['trainer'])
43
+
44
+ exp_manager(trainer, cfg['exp_manager'])
45
+
46
+ trainer.fit(model)
47
+
48
+ logging.info(f'fit Done!')
49
+
50
+ model.save_to(save_path="/ds/experiments/stt_en_quartznet15x5_v1.nemo")
51
+
52
+ logging.info(f'Done!')
53
+
54
+
55
+ if __name__ == '__main__':
56
+ main()
57
+
58
+ ```
quartznet15x5/2023-04-04_02-57-02/lightning_logs.txt ADDED
@@ -0,0 +1,165 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Using 16bit None Automatic Mixed Precision (AMP)
2
+ GPU available: True (cuda), used: True
3
+ TPU available: False, using: 0 TPU cores
4
+ IPU available: False, using: 0 IPUs
5
+ HPU available: False, using: 0 HPUs
6
+ ----------------------------------------------------------------------------------------------------
7
+ distributed_backend=nccl
8
+ All distributed processes registered. Starting with 1 processes
9
+ ----------------------------------------------------------------------------------------------------
10
+
11
+ LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
12
+
13
+ | Name | Type | Params
14
+ ------------------------------------------------------------------------
15
+ 0 | preprocessor | AudioToMelSpectrogramPreprocessor | 0
16
+ 1 | encoder | ConvASREncoder | 18.9 M
17
+ 2 | spec_augmentation | SpectrogramAugmentation | 0
18
+ 3 | _wer | WER | 0
19
+ 4 | decoder | ConvASRDecoder | 37.9 K
20
+ 5 | loss | CTCLoss | 0
21
+ ------------------------------------------------------------------------
22
+ 18.9 M Trainable params
23
+ 0 Non-trainable params
24
+ 18.9 M Total params
25
+ 37.865 Total estimated model params size (MB)
26
+ Epoch 0, global step 4072: 'val_loss' reached 14.59636 (best 14.59636), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=14.5964-epoch=0.ckpt' as top 3
27
+ Epoch 0, global step 8144: 'val_loss' reached 10.81774 (best 10.81774), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=10.8177-epoch=0.ckpt' as top 3
28
+ Epoch 0, global step 12216: 'val_loss' reached 8.56820 (best 8.56820), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=8.5682-epoch=0.ckpt' as top 3
29
+ Epoch 0, global step 16288: 'val_loss' reached 7.18057 (best 7.18057), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=7.1806-epoch=0.ckpt' as top 3
30
+ Epoch 1, global step 20362: 'val_loss' reached 6.77190 (best 6.77190), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=6.7719-epoch=1.ckpt' as top 3
31
+ Epoch 1, global step 24434: 'val_loss' reached 6.46053 (best 6.46053), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=6.4605-epoch=1.ckpt' as top 3
32
+ Epoch 1, global step 28506: 'val_loss' reached 5.61906 (best 5.61906), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=5.6191-epoch=1.ckpt' as top 3
33
+ Epoch 1, global step 32578: 'val_loss' reached 5.02838 (best 5.02838), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=5.0284-epoch=1.ckpt' as top 3
34
+ Epoch 2, global step 36652: 'val_loss' reached 5.03870 (best 5.02838), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=5.0387-epoch=2.ckpt' as top 3
35
+ Epoch 2, global step 40724: 'val_loss' reached 5.12578 (best 5.02838), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=5.1258-epoch=2.ckpt' as top 3
36
+ Epoch 2, global step 44796: 'val_loss' reached 4.47569 (best 4.47569), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=4.4757-epoch=2.ckpt' as top 3
37
+ Epoch 2, global step 48868: 'val_loss' reached 4.21310 (best 4.21310), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=4.2131-epoch=2.ckpt' as top 3
38
+ Epoch 3, global step 52942: 'val_loss' reached 4.17443 (best 4.17443), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=4.1744-epoch=3.ckpt' as top 3
39
+ Epoch 3, global step 57014: 'val_loss' reached 4.28295 (best 4.17443), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=4.2830-epoch=3.ckpt' as top 3
40
+ Epoch 3, global step 61086: 'val_loss' reached 3.92967 (best 3.92967), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.9297-epoch=3.ckpt' as top 3
41
+ Epoch 3, global step 65158: 'val_loss' reached 3.70603 (best 3.70603), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.7060-epoch=3.ckpt' as top 3
42
+ Epoch 4, global step 69232: 'val_loss' reached 3.68972 (best 3.68972), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.6897-epoch=4.ckpt' as top 3
43
+ Epoch 4, global step 73304: 'val_loss' reached 3.87810 (best 3.68972), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.8781-epoch=4.ckpt' as top 3
44
+ Epoch 4, global step 77376: 'val_loss' reached 3.52081 (best 3.52081), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.5208-epoch=4.ckpt' as top 3
45
+ Epoch 4, global step 81448: 'val_loss' reached 3.35277 (best 3.35277), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.3528-epoch=4.ckpt' as top 3
46
+ Epoch 5, global step 85522: 'val_loss' reached 3.36439 (best 3.35277), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.3644-epoch=5.ckpt' as top 3
47
+ Epoch 5, global step 89594: 'val_loss' was not in top 3
48
+ Epoch 5, global step 93666: 'val_loss' reached 3.24885 (best 3.24885), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.2488-epoch=5.ckpt' as top 3
49
+ Epoch 5, global step 97738: 'val_loss' reached 3.16221 (best 3.16221), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.1622-epoch=5.ckpt' as top 3
50
+ Epoch 6, global step 101812: 'val_loss' reached 3.16146 (best 3.16146), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.1615-epoch=6.ckpt' as top 3
51
+ Epoch 6, global step 105884: 'val_loss' was not in top 3
52
+ Epoch 6, global step 109956: 'val_loss' reached 3.04043 (best 3.04043), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.0404-epoch=6.ckpt' as top 3
53
+ Epoch 6, global step 114028: 'val_loss' reached 3.00790 (best 3.00790), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.0079-epoch=6.ckpt' as top 3
54
+ Epoch 7, global step 118102: 'val_loss' reached 2.99897 (best 2.99897), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.9990-epoch=7.ckpt' as top 3
55
+ Epoch 7, global step 122174: 'val_loss' was not in top 3
56
+ Epoch 7, global step 126246: 'val_loss' reached 2.97048 (best 2.97048), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.9705-epoch=7.ckpt' as top 3
57
+ Epoch 7, global step 130318: 'val_loss' reached 2.82341 (best 2.82341), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.8234-epoch=7.ckpt' as top 3
58
+ Epoch 8, global step 134392: 'val_loss' reached 2.83983 (best 2.82341), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.8398-epoch=8.ckpt' as top 3
59
+ Epoch 8, global step 138464: 'val_loss' was not in top 3
60
+ Epoch 8, global step 142536: 'val_loss' reached 2.80415 (best 2.80415), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.8042-epoch=8.ckpt' as top 3
61
+ Epoch 8, global step 146608: 'val_loss' reached 2.70087 (best 2.70087), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.7009-epoch=8.ckpt' as top 3
62
+ Epoch 9, global step 150682: 'val_loss' reached 2.76297 (best 2.70087), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.7630-epoch=9.ckpt' as top 3
63
+ Epoch 9, global step 154754: 'val_loss' was not in top 3
64
+ Epoch 9, global step 158826: 'val_loss' reached 2.67927 (best 2.67927), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.6793-epoch=9.ckpt' as top 3
65
+ Epoch 9, global step 162898: 'val_loss' reached 2.61542 (best 2.61542), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.6154-epoch=9.ckpt' as top 3
66
+ Epoch 10, global step 166972: 'val_loss' reached 2.68566 (best 2.61542), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.6857-epoch=10.ckpt' as top 3
67
+ Epoch 10, global step 171044: 'val_loss' was not in top 3
68
+ Epoch 10, global step 175116: 'val_loss' reached 2.60802 (best 2.60802), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.6080-epoch=10.ckpt' as top 3
69
+ Epoch 10, global step 179188: 'val_loss' reached 2.57574 (best 2.57574), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.5757-epoch=10.ckpt' as top 3
70
+ Epoch 11, global step 183262: 'val_loss' reached 2.60414 (best 2.57574), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.6041-epoch=11.ckpt' as top 3
71
+ Epoch 11, global step 187334: 'val_loss' was not in top 3
72
+ Epoch 11, global step 191406: 'val_loss' reached 2.56114 (best 2.56114), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.5611-epoch=11.ckpt' as top 3
73
+ Epoch 11, global step 195478: 'val_loss' reached 2.46534 (best 2.46534), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.4653-epoch=11.ckpt' as top 3
74
+ Epoch 12, global step 199552: 'val_loss' reached 2.54478 (best 2.46534), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.5448-epoch=12.ckpt' as top 3
75
+ Epoch 12, global step 203624: 'val_loss' was not in top 3
76
+ Epoch 12, global step 207696: 'val_loss' reached 2.48122 (best 2.46534), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.4812-epoch=12.ckpt' as top 3
77
+ Epoch 12, global step 211768: 'val_loss' reached 2.39888 (best 2.39888), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3989-epoch=12.ckpt' as top 3
78
+ Epoch 13, global step 215842: 'val_loss' reached 2.44645 (best 2.39888), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.4464-epoch=13.ckpt' as top 3
79
+ Epoch 13, global step 219914: 'val_loss' was not in top 3
80
+ Epoch 13, global step 223986: 'val_loss' reached 2.44335 (best 2.39888), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.4433-epoch=13.ckpt' as top 3
81
+ Epoch 13, global step 228058: 'val_loss' reached 2.40012 (best 2.39888), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.4001-epoch=13.ckpt' as top 3
82
+ Epoch 14, global step 232132: 'val_loss' reached 2.42736 (best 2.39888), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.4274-epoch=14.ckpt' as top 3
83
+ Epoch 14, global step 236204: 'val_loss' was not in top 3
84
+ Epoch 14, global step 240276: 'val_loss' reached 2.38463 (best 2.38463), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3846-epoch=14.ckpt' as top 3
85
+ Epoch 14, global step 244348: 'val_loss' reached 2.32972 (best 2.32972), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3297-epoch=14.ckpt' as top 3
86
+ Epoch 15, global step 248422: 'val_loss' reached 2.36627 (best 2.32972), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3663-epoch=15.ckpt' as top 3
87
+ Epoch 15, global step 252494: 'val_loss' was not in top 3
88
+ Epoch 15, global step 256566: 'val_loss' reached 2.34547 (best 2.32972), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3455-epoch=15.ckpt' as top 3
89
+ Epoch 15, global step 260638: 'val_loss' reached 2.32556 (best 2.32556), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3256-epoch=15.ckpt' as top 3
90
+ Epoch 16, global step 264712: 'val_loss' was not in top 3
91
+ Epoch 16, global step 268784: 'val_loss' was not in top 3
92
+ Epoch 16, global step 272856: 'val_loss' reached 2.24361 (best 2.24361), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2436-epoch=16.ckpt' as top 3
93
+ Epoch 16, global step 276928: 'val_loss' reached 2.28272 (best 2.24361), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2827-epoch=16.ckpt' as top 3
94
+ Epoch 17, global step 281002: 'val_loss' reached 2.30270 (best 2.24361), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3027-epoch=17.ckpt' as top 3
95
+ Epoch 17, global step 285074: 'val_loss' was not in top 3
96
+ Epoch 17, global step 289146: 'val_loss' reached 2.24171 (best 2.24171), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2417-epoch=17.ckpt' as top 3
97
+ Epoch 17, global step 293218: 'val_loss' reached 2.20842 (best 2.20842), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2084-epoch=17.ckpt' as top 3
98
+ Epoch 18, global step 297292: 'val_loss' reached 2.22932 (best 2.20842), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2293-epoch=18.ckpt' as top 3
99
+ Epoch 18, global step 301364: 'val_loss' was not in top 3
100
+ Epoch 18, global step 305436: 'val_loss' reached 2.22272 (best 2.20842), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2227-epoch=18.ckpt' as top 3
101
+ Epoch 18, global step 309508: 'val_loss' was not in top 3
102
+ Epoch 19, global step 313582: 'val_loss' was not in top 3
103
+ Epoch 19, global step 317654: 'val_loss' was not in top 3
104
+ Epoch 19, global step 321726: 'val_loss' reached 2.20790 (best 2.20790), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2079-epoch=19.ckpt' as top 3
105
+ Epoch 19, global step 325798: 'val_loss' reached 2.18583 (best 2.18583), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1858-epoch=19.ckpt' as top 3
106
+ Epoch 20, global step 329872: 'val_loss' reached 2.16728 (best 2.16728), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1673-epoch=20.ckpt' as top 3
107
+ Epoch 20, global step 333944: 'val_loss' was not in top 3
108
+ Epoch 20, global step 338016: 'val_loss' reached 2.15001 (best 2.15001), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1500-epoch=20.ckpt' as top 3
109
+ Epoch 20, global step 342088: 'val_loss' reached 2.15241 (best 2.15001), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1524-epoch=20.ckpt' as top 3
110
+ Epoch 21, global step 346162: 'val_loss' reached 2.16201 (best 2.15001), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1620-epoch=21.ckpt' as top 3
111
+ Epoch 21, global step 350234: 'val_loss' was not in top 3
112
+ Epoch 21, global step 354306: 'val_loss' reached 2.11477 (best 2.11477), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1148-epoch=21.ckpt' as top 3
113
+ Epoch 21, global step 358378: 'val_loss' reached 2.10107 (best 2.10107), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1011-epoch=21.ckpt' as top 3
114
+ Epoch 22, global step 362452: 'val_loss' reached 2.11003 (best 2.10107), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1100-epoch=22.ckpt' as top 3
115
+ Epoch 22, global step 366524: 'val_loss' was not in top 3
116
+ Epoch 22, global step 370596: 'val_loss' reached 2.09616 (best 2.09616), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0962-epoch=22.ckpt' as top 3
117
+ Epoch 22, global step 374668: 'val_loss' was not in top 3
118
+ Epoch 23, global step 378742: 'val_loss' reached 2.09999 (best 2.09616), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1000-epoch=23.ckpt' as top 3
119
+ Epoch 23, global step 382814: 'val_loss' was not in top 3
120
+ Epoch 23, global step 386886: 'val_loss' reached 2.02523 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0252-epoch=23.ckpt' as top 3
121
+ Epoch 23, global step 390958: 'val_loss' reached 2.04520 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0452-epoch=23.ckpt' as top 3
122
+ Epoch 24, global step 395032: 'val_loss' reached 2.09024 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0902-epoch=24.ckpt' as top 3
123
+ Epoch 24, global step 399104: 'val_loss' was not in top 3
124
+ Epoch 24, global step 403176: 'val_loss' reached 2.06819 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0682-epoch=24.ckpt' as top 3
125
+ Epoch 24, global step 407248: 'val_loss' reached 2.04502 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0450-epoch=24.ckpt' as top 3
126
+ Epoch 25, global step 411322: 'val_loss' was not in top 3
127
+ Epoch 25, global step 415394: 'val_loss' was not in top 3
128
+ Epoch 25, global step 419466: 'val_loss' reached 2.04345 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0434-epoch=25.ckpt' as top 3
129
+ Epoch 25, global step 423538: 'val_loss' was not in top 3
130
+ Epoch 26, global step 427612: 'val_loss' reached 2.02936 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0294-epoch=26.ckpt' as top 3
131
+ Epoch 26, global step 431684: 'val_loss' was not in top 3
132
+ Epoch 26, global step 435756: 'val_loss' reached 1.98137 (best 1.98137), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9814-epoch=26.ckpt' as top 3
133
+ Epoch 26, global step 439828: 'val_loss' reached 1.99461 (best 1.98137), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9946-epoch=26.ckpt' as top 3
134
+ Epoch 27, global step 443902: 'val_loss' was not in top 3
135
+ Epoch 27, global step 447974: 'val_loss' was not in top 3
136
+ Epoch 27, global step 452046: 'val_loss' reached 2.02101 (best 1.98137), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0210-epoch=27.ckpt' as top 3
137
+ Epoch 27, global step 456118: 'val_loss' reached 1.99866 (best 1.98137), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9987-epoch=27.ckpt' as top 3
138
+ Epoch 28, global step 460192: 'val_loss' reached 1.98746 (best 1.98137), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9875-epoch=28.ckpt' as top 3
139
+ Epoch 28, global step 464264: 'val_loss' was not in top 3
140
+ Epoch 28, global step 468336: 'val_loss' reached 1.98728 (best 1.98137), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9873-epoch=28.ckpt' as top 3
141
+ Epoch 28, global step 472408: 'val_loss' was not in top 3
142
+ Epoch 29, global step 476482: 'val_loss' reached 1.97654 (best 1.97654), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9765-epoch=29.ckpt' as top 3
143
+ Epoch 29, global step 480554: 'val_loss' was not in top 3
144
+ Epoch 29, global step 484626: 'val_loss' reached 1.95432 (best 1.95432), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9543-epoch=29.ckpt' as top 3
145
+ Epoch 29, global step 488698: 'val_loss' reached 1.97343 (best 1.95432), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9734-epoch=29.ckpt' as top 3
146
+ Epoch 30, global step 492772: 'val_loss' reached 1.96907 (best 1.95432), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9691-epoch=30.ckpt' as top 3
147
+ Epoch 30, global step 496844: 'val_loss' was not in top 3
148
+ Epoch 30, global step 500916: 'val_loss' reached 1.95860 (best 1.95432), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9586-epoch=30.ckpt' as top 3
149
+ Epoch 30, global step 504988: 'val_loss' was not in top 3
150
+ Epoch 31, global step 509062: 'val_loss' reached 1.96284 (best 1.95432), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9628-epoch=31.ckpt' as top 3
151
+ Epoch 31, global step 513134: 'val_loss' was not in top 3
152
+ Epoch 31, global step 517206: 'val_loss' reached 1.94839 (best 1.94839), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9484-epoch=31.ckpt' as top 3
153
+ Epoch 31, global step 521278: 'val_loss' was not in top 3
154
+ Epoch 32, global step 525352: 'val_loss' was not in top 3
155
+ Epoch 32, global step 529424: 'val_loss' was not in top 3
156
+ Epoch 32, global step 533496: 'val_loss' reached 1.93692 (best 1.93692), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9369-epoch=32.ckpt' as top 3
157
+ Epoch 32, global step 537568: 'val_loss' reached 1.94155 (best 1.93692), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9416-epoch=32.ckpt' as top 3
158
+ Epoch 33, global step 541642: 'val_loss' reached 1.94692 (best 1.93692), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9469-epoch=33.ckpt' as top 3
159
+ Epoch 33, global step 545714: 'val_loss' was not in top 3
160
+ Epoch 33, global step 549786: 'val_loss' reached 1.89907 (best 1.89907), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.8991-epoch=33.ckpt' as top 3
161
+ Epoch 33, global step 553858: 'val_loss' was not in top 3
162
+ Epoch 34, global step 557932: 'val_loss' reached 1.91820 (best 1.89907), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9182-epoch=34.ckpt' as top 3
163
+ Epoch 34, global step 562004: 'val_loss' was not in top 3
164
+ Epoch 34, global step 566076: 'val_loss' reached 1.90886 (best 1.89907), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9089-epoch=34.ckpt' as top 3
165
+ Epoch 34, global step 570148: 'val_loss' was not in top 3
quartznet15x5/2023-04-04_02-57-02/stt_uk_quartznet15x5.nemo ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe04ae1512ef0256c7420f3e1de287641e07725d67fd701bca8ac77c79c00a21
3
+ size 76369920
quartznet15x5/2023-04-04_02-57-02/stt_uk_quartznet15x5.yaml ADDED
@@ -0,0 +1,282 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ name: &name "stt_uk_quartznet15x5"
2
+
3
+ num_classes: &num_classes 36
4
+ sample_rate: &sample_rate 16000
5
+ repeat: &repeat 5
6
+ dropout: &dropout 0.0
7
+ separable: &separable true
8
+ labels: &labels [ " ", "а", "б", "в", "г", "ґ", "д", "е", "є", "ж", "з", "и", "і", "ї", "й", "к", "л", "м", "н",
9
+ "о", "п", "р", "с", "т", "у", "ф", "х", "ц", "ч", "ш", "щ", "ь", "ю", "я", "'", "-" ]
10
+
11
+ model:
12
+ labels: *labels
13
+ sample_rate: *sample_rate
14
+ repeat: *repeat
15
+ dropout: *dropout
16
+ separable: *separable
17
+
18
+ train_ds:
19
+ manifest_filepath: /datasets/nemo_tarred_uk/tarred_audio_manifest.json
20
+ sample_rate: *sample_rate
21
+ labels: *labels
22
+ batch_size: 64
23
+ trim_silence: True
24
+ max_duration: 9.7
25
+ min_duration: 1.0
26
+ shuffle: True
27
+ # shuffle_n: 2048
28
+ is_tarred: True
29
+ tarred_audio_filepaths: /datasets/nemo_tarred_uk/audio__OP_0..199_CL_.tar
30
+ num_workers: 32
31
+ bucketing_strategy: "synced_randomized"
32
+ bucketing_batch_size: null
33
+
34
+ validation_ds:
35
+ manifest_filepath:
36
+ - /datasets/nemo_tarred_uk/docker_tarred_validation_manifest.json
37
+ sample_rate: *sample_rate
38
+ labels: *labels
39
+ shuffle: False
40
+ batch_size: 32
41
+
42
+ preprocessor:
43
+ _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
44
+ normalize: "per_feature"
45
+ window_size: 0.02
46
+ sample_rate: *sample_rate
47
+ window_stride: 0.01
48
+ window: "hann"
49
+ features: &n_mels 64
50
+ n_fft: 512
51
+ frame_splicing: 1
52
+ dither: 0.00001
53
+
54
+ spec_augment:
55
+ _target_: nemo.collections.asr.modules.SpectrogramAugmentation
56
+ rect_freq: 50
57
+ rect_masks: 5
58
+ rect_time: 120
59
+
60
+ encoder:
61
+ _target_: nemo.collections.asr.modules.ConvASREncoder
62
+ feat_in: *n_mels
63
+ activation: relu
64
+ conv_mask: true
65
+
66
+ jasper:
67
+ - dilation: [1]
68
+ dropout: *dropout
69
+ filters: 256
70
+ kernel: [33]
71
+ repeat: 1
72
+ residual: false
73
+ separable: *separable
74
+ stride: [2]
75
+
76
+ - dilation: [1]
77
+ dropout: *dropout
78
+ filters: 256
79
+ kernel: [33]
80
+ repeat: *repeat
81
+ residual: true
82
+ separable: *separable
83
+ stride: [1]
84
+
85
+ - dilation: [1]
86
+ dropout: *dropout
87
+ filters: 256
88
+ kernel: [33]
89
+ repeat: *repeat
90
+ residual: true
91
+ separable: *separable
92
+ stride: [1]
93
+
94
+ - dilation: [1]
95
+ dropout: *dropout
96
+ filters: 256
97
+ kernel: [33]
98
+ repeat: *repeat
99
+ residual: true
100
+ separable: *separable
101
+ stride: [1]
102
+
103
+ - dilation: [1]
104
+ dropout: *dropout
105
+ filters: 256
106
+ kernel: [39]
107
+ repeat: *repeat
108
+ residual: true
109
+ separable: *separable
110
+ stride: [1]
111
+
112
+ - dilation: [1]
113
+ dropout: *dropout
114
+ filters: 256
115
+ kernel: [39]
116
+ repeat: *repeat
117
+ residual: true
118
+ separable: *separable
119
+ stride: [1]
120
+
121
+ - dilation: [1]
122
+ dropout: *dropout
123
+ filters: 256
124
+ kernel: [39]
125
+ repeat: *repeat
126
+ residual: true
127
+ separable: *separable
128
+ stride: [1]
129
+
130
+ - dilation: [1]
131
+ dropout: *dropout
132
+ filters: 512
133
+ kernel: [51]
134
+ repeat: *repeat
135
+ residual: true
136
+ separable: *separable
137
+ stride: [1]
138
+
139
+ - dilation: [1]
140
+ dropout: *dropout
141
+ filters: 512
142
+ kernel: [51]
143
+ repeat: *repeat
144
+ residual: true
145
+ separable: *separable
146
+ stride: [1]
147
+
148
+ - dilation: [1]
149
+ dropout: *dropout
150
+ filters: 512
151
+ kernel: [51]
152
+ repeat: *repeat
153
+ residual: true
154
+ separable: *separable
155
+ stride: [1]
156
+
157
+ - dilation: [1]
158
+ dropout: *dropout
159
+ filters: 512
160
+ kernel: [63]
161
+ repeat: *repeat
162
+ residual: true
163
+ separable: *separable
164
+ stride: [1]
165
+
166
+ - dilation: [1]
167
+ dropout: *dropout
168
+ filters: 512
169
+ kernel: [63]
170
+ repeat: *repeat
171
+ residual: true
172
+ separable: *separable
173
+ stride: [1]
174
+
175
+ - dilation: [1]
176
+ dropout: *dropout
177
+ filters: 512
178
+ kernel: [63]
179
+ repeat: *repeat
180
+ residual: true
181
+ separable: *separable
182
+ stride: [1]
183
+
184
+ - dilation: [1]
185
+ dropout: *dropout
186
+ filters: 512
187
+ kernel: [75]
188
+ repeat: *repeat
189
+ residual: true
190
+ separable: *separable
191
+ stride: [1]
192
+
193
+ - dilation: [1]
194
+ dropout: *dropout
195
+ filters: 512
196
+ kernel: [75]
197
+ repeat: *repeat
198
+ residual: true
199
+ separable: *separable
200
+ stride: [1]
201
+
202
+ - dilation: [1]
203
+ dropout: *dropout
204
+ filters: 512
205
+ kernel: [75]
206
+ repeat: *repeat
207
+ residual: true
208
+ separable: *separable
209
+ stride: [1]
210
+
211
+ - dilation: [2]
212
+ dropout: *dropout
213
+ filters: 512
214
+ kernel: [87]
215
+ repeat: 1
216
+ residual: false
217
+ separable: *separable
218
+ stride: [1]
219
+
220
+ - dilation: [1]
221
+ dropout: *dropout
222
+ filters: &enc_filters 1024
223
+ kernel: [1]
224
+ repeat: 1
225
+ residual: false
226
+ stride: [1]
227
+
228
+ decoder:
229
+ _target_: nemo.collections.asr.modules.ConvASRDecoder
230
+ feat_in: *enc_filters
231
+ num_classes: *num_classes
232
+ vocabulary: *labels
233
+
234
+ optim:
235
+ name: novograd
236
+ lr: 0.001
237
+ betas: [0.8, 0.5]
238
+ weight_decay: 0.0001
239
+
240
+ sched:
241
+ name: CosineAnnealing
242
+ warmup_steps: null
243
+ warmup_ratio: null
244
+ min_lr: 0.0
245
+ last_epoch: -1
246
+
247
+ # pytorch lightning args
248
+ monitor: val_loss
249
+ reduce_on_plateau: false
250
+
251
+ trainer:
252
+ # gpus: -1 # number of gpus was configured so validation will run at the end of the training epoch..
253
+ accelerator: gpu
254
+ devices: -1 # number of gpus
255
+ max_epochs: 50
256
+ max_steps: -1 # computed at runtime if not set
257
+ num_nodes: 1
258
+ strategy: ddp
259
+ accumulate_grad_batches: 1
260
+ enable_checkpointing: False # Provided by exp_manager
261
+ logger: False # Provided by exp_manager
262
+ log_every_n_steps: 1 # Interval of logging.
263
+ val_check_interval: 0.25 # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
264
+ benchmark: false # має бути false для моделей із мовним введенням змінної довжини, оскільки це сповільнює навчання
265
+ precision: 16 # Should be set to 16 for O1 and O2 to enable the AMP.
266
+
267
+ exp_manager:
268
+ exp_dir: "/ds/experiments"
269
+ name: *name
270
+ create_tensorboard_logger: False
271
+ create_checkpoint_callback: True
272
+ create_wandb_logger: False
273
+ wandb_logger_kwargs:
274
+ name: null
275
+ project: null
276
+
277
+ hydra:
278
+ run:
279
+ dir: .
280
+ job_logging:
281
+ root:
282
+ handlers: null
quartznet15x5/2023-04-04_02-57-02/terminal_output.txt ADDED
@@ -0,0 +1,507 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ docker run --gpus all --ulimit memlock=-1 --ulimit stack=67108864 --shm-size=18gb -p 8888:8888 -p 6006:6006 -it --rm -v ${PWD}/ds:/ds -v ${PWD}/../../datasets:/datasets nvcr.io/nvidia/nemo:23.01
2
+
3
+ =============
4
+ == PyTorch ==
5
+ =============
6
+
7
+ NVIDIA Release 23.01 (build 52269074)
8
+ PyTorch Version 1.14.0a0+44dac51
9
+
10
+ Container image Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
11
+
12
+ Copyright (c) 2014-2023 Facebook Inc.
13
+ Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
14
+ Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
15
+ Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
16
+ Copyright (c) 2011-2013 NYU (Clement Farabet)
17
+ Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
18
+ Copyright (c) 2006 Idiap Research Institute (Samy Bengio)
19
+ Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
20
+ Copyright (c) 2015 Google Inc.
21
+ Copyright (c) 2015 Yangqing Jia
22
+ Copyright (c) 2013-2016 The Caffe contributors
23
+ All rights reserved.
24
+
25
+ Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.
26
+
27
+ This container image and its contents are governed by the NVIDIA Deep Learning Container License.
28
+ By pulling and using the container, you accept the terms and conditions of this license:
29
+ https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
30
+
31
+ root@e0f8def33916:/workspace/nemo# тмсс -М
32
+ bash: тмсс: command not found
33
+ root@e0f8def33916:/workspace/nemo# nvcc -V
34
+ nvcc: NVIDIA (R) Cuda compiler driver
35
+ Copyright (c) 2005-2023 NVIDIA Corporation
36
+ Built on Fri_Jan__6_16:45:21_PST_2023
37
+ Cuda compilation tools, release 12.0, V12.0.140
38
+ Build cuda_12.0.r12.0/compiler.32267302_0
39
+ root@e0f8def33916:/workspace/nemo# HYDRA_FULL_ERROR=1 python /ds/training_stt_en_quartznet15x5_v1.py
40
+ [NeMo W 2023-04-04 02:52:07 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.
41
+ [NeMo W 2023-04-04 02:52:08 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
42
+ See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
43
+ ret = run_job(
44
+
45
+ [NeMo I 2023-04-04 02:52:08 training_stt_en_quartznet15x5_v1:21] Конфігурація Hydra
46
+ [NeMo I 2023-04-04 02:52:08 cloud:66] Downloading from: https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_quartznet15x5/versions/1.0.0rc1/files/stt_en_quartznet15x5.nemo to /root/.cache/torch/NeMo/NeMo_1.16.0/stt_en_quartznet15x5/16661021d16e679bdfd97a2a03944c49/stt_en_quartznet15x5.nemo
47
+ 100% [........................................................................] 70993538 / 70993538[NeMo I 2023-04-04 02:52:21 common:913] Instantiating model from pre-trained checkpoint
48
+ [NeMo W 2023-04-04 02:52:22 modelPT:161] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
49
+ Train config :
50
+ manifest_filepath: /data2/voices/train_1k.json
51
+ sample_rate: 16000
52
+ labels:
53
+ - ' '
54
+ - a
55
+ - b
56
+ - c
57
+ - d
58
+ - e
59
+ - f
60
+ - g
61
+ - h
62
+ - i
63
+ - j
64
+ - k
65
+ - l
66
+ - m
67
+ - 'n'
68
+ - o
69
+ - p
70
+ - q
71
+ - r
72
+ - s
73
+ - t
74
+ - u
75
+ - v
76
+ - w
77
+ - x
78
+ - 'y'
79
+ - z
80
+ - ''''
81
+ batch_size: 32
82
+ trim_silence: true
83
+ max_duration: 16.7
84
+ shuffle: true
85
+ is_tarred: false
86
+ tarred_audio_filepaths: /asr_set_1.2/train/train_{0..1023}.tar
87
+ num_workers: 20
88
+
89
+ [NeMo W 2023-04-04 02:52:22 modelPT:168] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
90
+ Validation config :
91
+ manifest_filepath: /data2/voices/train_1k_samp.json
92
+ sample_rate: 16000
93
+ labels:
94
+ - ' '
95
+ - a
96
+ - b
97
+ - c
98
+ - d
99
+ - e
100
+ - f
101
+ - g
102
+ - h
103
+ - i
104
+ - j
105
+ - k
106
+ - l
107
+ - m
108
+ - 'n'
109
+ - o
110
+ - p
111
+ - q
112
+ - r
113
+ - s
114
+ - t
115
+ - u
116
+ - v
117
+ - w
118
+ - x
119
+ - 'y'
120
+ - z
121
+ - ''''
122
+ batch_size: 32
123
+ shuffle: false
124
+
125
+ [NeMo I 2023-04-04 02:52:22 features:286] PADDING: 16
126
+ [NeMo I 2023-04-04 02:52:23 save_restore_connector:247] Model EncDecCTCModel was successfully restored from /root/.cache/torch/NeMo/NeMo_1.16.0/stt_en_quartznet15x5/16661021d16e679bdfd97a2a03944c49/stt_en_quartznet15x5.nemo.
127
+ [NeMo I 2023-04-04 02:52:23 ctc_models:300] Changed decoder to output to [' ', 'а', 'б', 'в', 'г', 'ґ', 'д', 'е', 'є', 'ж', 'з', 'и', 'і', 'ї', 'й', 'к', 'л', 'м', 'н', 'о', 'п', 'р', 'с', 'т', 'у', 'ф', 'х', 'ц', 'ч', 'ш', 'щ', 'ь', 'ю', 'я', "'", '-'] vocabulary.
128
+ [NeMo W 2023-04-04 02:52:23 modelPT:609] Trainer wasn't specified in model constructor. Make sure that you really wanted it.
129
+ [NeMo I 2023-04-04 02:52:23 modelPT:720] Optimizer config = Novograd (
130
+ Parameter Group 0
131
+ amsgrad: False
132
+ betas: [0.8, 0.5]
133
+ eps: 1e-08
134
+ grad_averaging: False
135
+ lr: 0.001
136
+ weight_decay: 0.0001
137
+ )
138
+ [NeMo W 2023-04-04 02:52:23 lr_scheduler:890] Neither `max_steps` nor `iters_per_batch` were provided to `optim.sched`, cannot compute effective `max_steps` !
139
+ Scheduler will not be instantiated !
140
+ [NeMo W 2023-04-04 02:52:23 audio_to_text_dataset:58] `labels` is explicitly provided to the data loader, and is different from the `labels` provided at the model level config.
141
+ If this is incorrect, please set the dataloader's `labels` to None.
142
+ [NeMo I 2023-04-04 02:55:52 collections:193] Dataset loaded with 1042543 files totalling 1028.12 hours
143
+ [NeMo I 2023-04-04 02:55:52 collections:194] 9857 files were filtered totalling 32.16 hours
144
+ [NeMo W 2023-04-04 02:55:52 ctc_models:409] Model Trainer was not set before constructing the dataset, incorrect number of training batches will be used. Please set the trainer and rebuild the dataset.
145
+ [NeMo W 2023-04-04 02:55:52 audio_to_text_dataset:58] `labels` is explicitly provided to the data loader, and is different from the `labels` provided at the model level config.
146
+ If this is incorrect, please set the dataloader's `labels` to None.
147
+ [NeMo I 2023-04-04 02:57:02 collections:193] Dataset loaded with 50645 files totalling 50.72 hours
148
+ [NeMo I 2023-04-04 02:57:02 collections:194] 0 files were filtered totalling 0.00 hours
149
+ Using 16bit None Automatic Mixed Precision (AMP)
150
+ GPU available: True (cuda), used: True
151
+ TPU available: False, using: 0 TPU cores
152
+ IPU available: False, using: 0 IPUs
153
+ HPU available: False, using: 0 HPUs
154
+ [NeMo I 2023-04-04 02:57:02 exp_manager:383] Experiments will be logged at /ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02
155
+ Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
156
+ ----------------------------------------------------------------------------------------------------
157
+ distributed_backend=nccl
158
+ All distributed processes registered. Starting with 1 processes
159
+ ----------------------------------------------------------------------------------------------------
160
+
161
+ You are using a CUDA device ('NVIDIA GeForce RTX 3090 Ti') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
162
+ LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
163
+ [NeMo I 2023-04-04 02:57:03 modelPT:720] Optimizer config = Novograd (
164
+ Parameter Group 0
165
+ amsgrad: False
166
+ betas: [0.8, 0.5]
167
+ eps: 1e-08
168
+ grad_averaging: False
169
+ lr: 0.001
170
+ weight_decay: 0.0001
171
+ )
172
+ [NeMo I 2023-04-04 02:57:03 lr_scheduler:910] Scheduler "<nemo.core.optim.lr_scheduler.CosineAnnealing object at 0x7f0c81da4f70>"
173
+ will be used during training (effective maximum steps = 814500) -
174
+ Parameters :
175
+ (warmup_steps: null
176
+ warmup_ratio: null
177
+ min_lr: 0.0
178
+ last_epoch: -1
179
+ max_steps: 814500
180
+ )
181
+
182
+ | Name | Type | Params
183
+ ------------------------------------------------------------------------
184
+ 0 | preprocessor | AudioToMelSpectrogramPreprocessor | 0
185
+ 1 | encoder | ConvASREncoder | 18.9 M
186
+ 2 | spec_augmentation | SpectrogramAugmentation | 0
187
+ 3 | _wer | WER | 0
188
+ 4 | decoder | ConvASRDecoder | 37.9 K
189
+ 5 | loss | CTCLoss | 0
190
+ ------------------------------------------------------------------------
191
+ 18.9 M Trainable params
192
+ 0 Non-trainable params
193
+ 18.9 M Total params
194
+ 37.865 Total estimated model params size (MB)
195
+ Sanity Checking: 0it [00:00, ?it/s][NeMo W 2023-04-04 02:57:03 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py:224: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 32 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
196
+ rank_zero_warn(
197
+
198
+ Sanity Checking DataLoader 0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00, 1.01it/s][NeMo W 2023-04-04 02:57:06 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:536: PossibleUserWarning: It is recommended to use `self.log('global_step', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.
199
+ warning_cache.warn(
200
+
201
+ [NeMo W 2023-04-04 02:57:06 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:536: PossibleUserWarning: It is recommended to use `self.log('val_loss', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.
202
+ warning_cache.warn(
203
+
204
+ [NeMo W 2023-04-04 02:57:06 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:536: PossibleUserWarning: It is recommended to use `self.log('val_wer', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.
205
+ warning_cache.warn(
206
+
207
+ [NeMo W 2023-04-04 02:57:06 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/lightning_fabric/utilities/data.py:63: UserWarning: Your `IterableDataset` has `__len__` defined. In combination with multi-process data loading (when num_workers > 1), `__len__` could be inaccurate if each worker is not configured independently to avoid having duplicate data.
208
+ rank_zero_warn(
209
+
210
+ Epoch 0: 0%| | 0/22622 [00:00<?, ?it/s][W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
211
+ [NeMo W 2023-04-04 02:57:09 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/torch/optim/lr_scheduler.py:138: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
212
+ warnings.warn("Detected call of `lr_scheduler.step()` before `optimizer.step()`. "
213
+
214
+ Epoch 0: 25%|████████████████████████████████████▍ | 5655/22622 [31:39<1:35:00, 2.98it/s, loss=26.2
215
+ Epoch 0, global step 4072: 'val_loss' reached 14.59636 (best 14.59636), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=14.5964-epoch=0.ckpt' as top 3
216
+ Epoch 0: 26%|█████████████████████████████████████▋ | 5843/22622 [32:35<1:33:34, 2.99it/s, loss=27.3]
217
+ Epoch 0: 35%|██████████████████████████████████████████████████▍ | 7819/22622 [42:56<1:21:17, 3.04it/s, loss=20.7]
218
+ Epoch 0: 37%|█████████████████████████████████████████████████████▉ | 8352/22622 [45:21<1:17:30, 3.07it/s, loss=22.9]
219
+ Epoch 0: 50%|███████████████████████████████████████████████████████████████████████▍ | 11310/22622 [1:03:42<1:03:43, 2.96it/s, loss=10.4
220
+ Epoch 0, global step 8144: 'val_loss' reached 10.81774 (best 10.81774), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=10.8177-epoch=0.ckpt' as top 3
221
+ Epoch 0: 75%|██████████████████████████████████████████████████████████████████████████████████���█████████████████████████▋ | 16965/22622 [1:33:47<31:16, 3.01it/s, loss=19.6
222
+ Epoch 0, global step 12216: 'val_loss' reached 8.56820 (best 8.56820), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=8.5682-epoch=0.ckpt' as top 3
223
+ Epoch 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:04:41<00:00, 3.02it/s, loss=28.2
224
+ Epoch 0, global step 16288: 'val_loss' reached 7.18057 (best 7.18057), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=7.1806-epoch=0.ckpt' as top 3
225
+ Epoch 1: 25%|████████████████████████████████████▍ | 5655/22622 [29:50<1:29:32, 3.16it/s, loss=15.4
226
+ Epoch 1, global step 20362: 'val_loss' reached 6.77190 (best 6.77190), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=6.7719-epoch=1.ckpt' as top 3
227
+ Epoch 1: 50%|███████████████████████████████████████████████████████████████████████▍ | 11310/22622 [1:01:42<1:01:43, 3.05it/s, loss=7.48
228
+ Epoch 1, global step 24434: 'val_loss' reached 6.46053 (best 6.46053), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=6.4605-epoch=1.ckpt' as top 3
229
+ Epoch 1: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 16965/22622 [1:33:05<31:02, 3.04it/s, loss=14.3
230
+ Epoch 1, global step 28506: 'val_loss' reached 5.61906 (best 5.61906), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=5.6191-epoch=1.ckpt' as top 3
231
+ Epoch 1: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:04:46<00:00, 3.02it/s, loss=21.2
232
+ Epoch 1, global step 32578: 'val_loss' reached 5.02838 (best 5.02838), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=5.0284-epoch=1.ckpt' as top 3
233
+ Epoch 2: 25%|████████████████████████████████████▉ | 5655/22622 [30:07<1:30:22, 3.13it/s, loss=12
234
+ Epoch 2, global step 36652: 'val_loss' reached 5.03870 (best 5.02838), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=5.0387-epoch=2.ckpt' as top 3
235
+ Epoch 2: 50%|███████████████████████████████████████████████████████████████████████▍ | 11310/22622 [1:01:31<1:01:31, 3.06it/s, loss=6.09
236
+ Epoch 2, global step 40724: 'val_loss' reached 5.12578 (best 5.02838), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=5.1258-epoch=2.ckpt' as top 3
237
+ Epoch 2: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 16965/22622 [1:31:45<30:35, 3.08it/s, loss=11.6
238
+ Epoch 2, global step 44796: 'val_loss' reached 4.47569 (best 4.47569), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=4.4757-epoch=2.ckpt' as top 3
239
+ Epoch 2: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:03:00<00:00, 3.06it/s, loss=16.9
240
+ Epoch 2, global step 48868: 'val_loss' reached 4.21310 (best 4.21310), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=4.2131-epoch=2.ckpt' as top 3
241
+ Epoch 3: 25%|████████████████████████████████████▍ | 5655/22622 [30:11<1:30:35, 3.12it/s, loss=10.4
242
+ Epoch 3, global step 52942: 'val_loss' reached 4.17443 (best 4.17443), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=4.1744-epoch=3.ckpt' as top 3
243
+ Epoch 3: 50%|███████████████████████████████████████████████████████████████████████▍ | 11310/22622 [1:01:34<1:01:34, 3.06it/s, loss=4.76
244
+ Epoch 3, global step 57014: 'val_loss' reached 4.28295 (best 4.17443), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=4.2830-epoch=3.ckpt' as top 3
245
+ Epoch 3: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 16965/22622 [1:32:09<30:43, 3.07it/s, loss=12.3
246
+ Epoch 3, global step 61086: 'val_loss' reached 3.92967 (best 3.92967), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.9297-epoch=3.ckpt' as top 3
247
+ Epoch 3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:03:36<00:00, 3.05it/s, loss=17.1
248
+ Epoch 3, global step 65158: 'val_loss' reached 3.70603 (best 3.70603), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.7060-epoch=3.ckpt' as top 3
249
+ Epoch 4: 25%|████████████████████████████████████▍ | 5655/22622 [30:17<1:30:53, 3.11it/s, loss=9.81
250
+ Epoch 4, global step 69232: 'val_loss' reached 3.68972 (best 3.68972), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.6897-epoch=4.ckpt' as top 3
251
+ Epoch 4: 50%|███████████████████████████████████████████████████████████████████████▍ | 11310/22622 [1:01:32<1:01:33, 3.06it/s, loss=4.82
252
+ Epoch 4, global step 73304: 'val_loss' reached 3.87810 (best 3.68972), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.8781-epoch=4.ckpt' as top 3
253
+ Epoch 4: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 16965/22622 [1:31:48<30:36, 3.08it/s, loss=10.3
254
+ Epoch 4, global step 77376: 'val_loss' reached 3.52081 (best 3.52081), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.5208-epoch=4.ckpt' as top 3
255
+ Epoch 4: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:02:56<00:00, 3.07it/s, loss=14.7
256
+ Epoch 4, global step 81448: 'val_loss' reached 3.35277 (best 3.35277), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.3528-epoch=4.ckpt' as top 3
257
+ Epoch 5: 25%|████████████████████████████████████▍ | 5655/22622 [29:57<1:29:52, 3.15it/s, loss=8.93
258
+ Epoch 5, global step 85522: 'val_loss' reached 3.36439 (best 3.35277), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.3644-epoch=5.ckpt' as top 3
259
+ Epoch 5: 50%|███████████████████████████████████████████████████████████████████████▍ | 11310/22622 [1:01:12<1:01:12, 3.08it/s, loss=3.07
260
+ Epoch 5, global step 89594: 'val_loss' was not in top 3
261
+ Epoch 5: 73%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 16624/22622 [1:29:46<32:23, 3.09it/s, loss=9.43]
262
+ Epoch 5: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 16965/22622 [1:31:54<30:38, 3.08it/s, loss=9.43
263
+ Epoch 5, global step 93666: 'val_loss' reached 3.24885 (best 3.24885), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.2488-epoch=5.ckpt' as top 3
264
+ Epoch 5: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:03:37<00:00, 3.05it/s, loss=12.8
265
+ Epoch 5, global step 97738: 'val_loss' reached 3.16221 (best 3.16221), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.1622-epoch=5.ckpt' as top 3
266
+ Epoch 6: 25%|████████████████████████████████████▍ | 5655/22622 [30:13<1:30:39, 3.12it/s, loss=8.47
267
+ Epoch 6, global step 101812: 'val_loss' reached 3.16146 (best 3.16146), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.1615-epoch=6.ckpt' as top 3
268
+ Epoch 6: 50%|███████████████████████████████████████████████████████████████████████▍ | 11310/22622 [1:01:36<1:01:36, 3.06it/s, loss=3.62
269
+ Epoch 6, global step 105884: 'val_loss' was not in top 3
270
+ Epoch 6: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 16965/22622 [1:32:08<30:43, 3.07it/s, loss=8.57
271
+ Epoch 6, global step 109956: 'val_loss' reached 3.04043 (best 3.04043), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.0404-epoch=6.ckpt' as top 3
272
+ Epoch 6: 100%|█████████████���██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:03:20<00:00, 3.06it/s, loss=14.9
273
+ Epoch 6, global step 114028: 'val_loss' reached 3.00790 (best 3.00790), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=3.0079-epoch=6.ckpt' as top 3
274
+ Epoch 7: 25%|████████████████████████████████████▍ | 5655/22622 [29:53<1:29:42, 3.15it/s, loss=8.51
275
+ Epoch 7, global step 118102: 'val_loss' reached 2.99897 (best 2.99897), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.9990-epoch=7.ckpt' as top 3
276
+ Epoch 7: 50%|███████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:01:14<1:01:15, 3.08it/s, loss=2.9
277
+ Epoch 7, global step 122174: 'val_loss' was not in top 3
278
+ Epoch 7: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 16965/22622 [1:31:31<30:31, 3.09it/s, loss=7.97
279
+ Epoch 7, global step 126246: 'val_loss' reached 2.97048 (best 2.97048), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.9705-epoch=7.ckpt' as top 3
280
+ Epoch 7: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:02:42<00:00, 3.07it/s, loss=12.1
281
+ Epoch 7, global step 130318: 'val_loss' reached 2.82341 (best 2.82341), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.8234-epoch=7.ckpt' as top 3
282
+ Epoch 8: 25%|████████████████████████████████████▍ | 5655/22622 [29:51<1:29:34, 3.16it/s, loss=7.38
283
+ Epoch 8, global step 134392: 'val_loss' reached 2.83983 (best 2.82341), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.8398-epoch=8.ckpt' as top 3
284
+ Epoch 8: 50%|███████████████████████████████████████████████████████████████████████▍ | 11310/22622 [1:01:10<1:01:10, 3.08it/s, loss=3.59
285
+ Epoch 8, global step 138464: 'val_loss' was not in top 3
286
+ Epoch 8: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 16965/22622 [1:31:18<30:26, 3.10it/s, loss=7.66
287
+ Epoch 8, global step 142536: 'val_loss' reached 2.80415 (best 2.80415), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.8042-epoch=8.ckpt' as top 3
288
+ Epoch 8: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:02:23<00:00, 3.08it/s, loss=12
289
+ Epoch 8, global step 146608: 'val_loss' reached 2.70087 (best 2.70087), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.7009-epoch=8.ckpt' as top 3
290
+ Epoch 9: 25%|█████████████████████████████████████▏ | 5655/22622 [29:41<1:29:03, 3.18it/s, loss=7
291
+ Epoch 9, global step 150682: 'val_loss' reached 2.76297 (best 2.70087), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.7630-epoch=9.ckpt' as top 3
292
+ Epoch 9: 50%|███████████████████████████████████████████████████████████████████████▍ | 11310/22622 [1:00:55<1:00:56, 3.09it/s, loss=3.23
293
+ Epoch 9, global step 154754: 'val_loss' was not in top 3
294
+ Epoch 9: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 16965/22622 [1:31:26<30:29, 3.09it/s, loss=7.16
295
+ Epoch 9, global step 158826: 'val_loss' reached 2.67927 (best 2.67927), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.6793-epoch=9.ckpt' as top 3
296
+ Epoch 9: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:02:27<00:00, 3.08it/s, loss=12.3
297
+ Epoch 9, global step 162898: 'val_loss' reached 2.61542 (best 2.61542), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.6154-epoch=9.ckpt' as top 3
298
+ Epoch 10: 25%|████████████████████████████████████▏ | 5655/22622 [29:57<1:29:53, 3.15it/s, loss=6.68
299
+ Epoch 10, global step 166972: 'val_loss' reached 2.68566 (best 2.61542), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.6857-epoch=10.ckpt' as top 3
300
+ Epoch 10: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:01:16<1:01:16, 3.08it/s, loss=3.96
301
+ Epoch 10, global step 171044: 'val_loss' was not in top 3
302
+ Epoch 10: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:31:34<30:32, 3.09it/s, loss=7.13
303
+ Epoch 10, global step 175116: 'val_loss' reached 2.60802 (best 2.60802), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.6080-epoch=10.ckpt' as top 3
304
+ Epoch 10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:02:46<00:00, 3.07it/s, loss=11.8
305
+ Epoch 10, global step 179188: 'val_loss' reached 2.57574 (best 2.57574), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.5757-epoch=10.ckpt' as top 3
306
+ Epoch 11: 25%|████████████████████████████████████▏ | 5655/22622 [30:09<1:30:28, 3.13it/s, loss=6.46
307
+ Epoch 11, global step 183262: 'val_loss' reached 2.60414 (best 2.57574), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.6041-epoch=11.ckpt' as top 3
308
+ Epoch 11: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:01:20<1:01:21, 3.07it/s, loss=4.65
309
+ Epoch 11, global step 187334: 'val_loss' was not in top 3
310
+ Epoch 11: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:32:18<30:46, 3.06it/s, loss=6.55
311
+ Epoch 11, global step 191406: 'val_loss' reached 2.56114 (best 2.56114), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.5611-epoch=11.ckpt' as top 3
312
+ Epoch 11: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:03:49<00:00, 3.04it/s, loss=10.7
313
+ Epoch 11, global step 195478: 'val_loss' reached 2.46534 (best 2.46534), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.4653-epoch=11.ckpt' as top 3
314
+ Epoch 12: 25%|████████████████████████████████████▏ | 5655/22622 [30:00<1:30:02, 3.14it/s, loss=6.37
315
+ Epoch 12, global step 199552: 'val_loss' reached 2.54478 (best 2.46534), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.5448-epoch=12.ckpt' as top 3
316
+ Epoch 12: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:01:36<1:01:37, 3.06it/s, loss=2.53
317
+ Epoch 12, global step 203624: 'val_loss' was not in top 3
318
+ Epoch 12: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:32:09<30:43, 3.07it/s, loss=7.49
319
+ Epoch 12, global step 207696: 'val_loss' reached 2.48122 (best 2.46534), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.4812-epoch=12.ckpt' as top 3
320
+ Epoch 12: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:05:13<00:00, 3.01it/s, loss=12.1
321
+ Epoch 12, global step 211768: 'val_loss' reached 2.39888 (best 2.39888), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3989-epoch=12.ckpt' as top 3
322
+ Epoch 13: 25%|████████████████████████████████████▏ | 5655/22622 [31:37<1:34:51, 2.98it/s, loss=5.86
323
+ Epoch 13, global step 215842: 'val_loss' reached 2.44645 (best 2.39888), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.4464-epoch=13.ckpt' as top 3
324
+ Epoch 13: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:04:08<1:04:09, 2.94it/s, loss=3.38
325
+ Epoch 13, global step 219914: 'val_loss' was not in top 3
326
+ Epoch 13: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:36:21<32:07, 2.93it/s, loss=6.51
327
+ Epoch 13, global step 223986: 'val_loss' reached 2.44335 (best 2.39888), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.4433-epoch=13.ckpt' as top 3
328
+ Epoch 13: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:09:10<00:00, 2.92it/s, loss=10.7
329
+ Epoch 13, global step 228058: 'val_loss' reached 2.40012 (best 2.39888), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.4001-epoch=13.ckpt' as top 3
330
+ Epoch 14: 25%|████████████████████████████████████▏ | 5655/22622 [30:50<1:32:33, 3.06it/s, loss=5.95
331
+ Epoch 14, global step 232132: 'val_loss' reached 2.42736 (best 2.39888), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.4274-epoch=14.ckpt' as top 3
332
+ Epoch 14: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:42<1:02:43, 3.01it/s, loss=2.75
333
+ Epoch 14, global step 236204: 'val_loss' was not in top 3
334
+ Epoch 14: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:31<31:11, 3.02it/s, loss=6.76
335
+ Epoch 14, global step 240276: 'val_loss' reached 2.38463 (best 2.38463), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3846-epoch=14.ckpt' as top 3
336
+ Epoch 14: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:05:40<00:00, 3.00it/s, loss=10.1
337
+ Epoch 14, global step 244348: 'val_loss' reached 2.32972 (best 2.32972), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3297-epoch=14.ckpt' as top 3
338
+ Epoch 15: 25%|████████████████████████████████████▏ | 5655/22622 [31:04<1:33:15, 3.03it/s, loss=5.91
339
+ Epoch 15, global step 248422: 'val_loss' reached 2.36627 (best 2.32972), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3663-epoch=15.ckpt' as top 3
340
+ Epoch 15: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:03:02<1:03:02, 2.99it/s, loss=3.37
341
+ Epoch 15, global step 252494: 'val_loss' was not in top 3
342
+ Epoch 15: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 16965/22622 [1:34:49<31:37, 2.98it/s, loss=6.3
343
+ Epoch 15, global step 256566: 'val_loss' reached 2.34547 (best 2.32972), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3455-epoch=15.ckpt' as top 3
344
+ Epoch 15: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:07:02<00:00, 2.97it/s, loss=10.2
345
+ Epoch 15, global step 260638: 'val_loss' reached 2.32556 (best 2.32556), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3256-epoch=15.ckpt' as top 3
346
+ Epoch 16: 25%|████████████████████████████████████▏ | 5655/22622 [30:39<1:31:59, 3.07it/s, loss=5.68
347
+ Epoch 16, global step 264712: 'val_loss' was not in top 3
348
+ Epoch 16: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:24<1:02:25, 3.02it/s, loss=2.59
349
+ Epoch 16, global step 268784: 'val_loss' was not in top 3
350
+ Epoch 16: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:13<31:05, 3.03it/s, loss=6.23
351
+ Epoch 16, global step 272856: 'val_loss' reached 2.24361 (best 2.24361), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2436-epoch=16.ckpt' as top 3
352
+ Epoch 16: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:04:43<00:00, 3.02it/s, loss=8.8
353
+ Epoch 16, global step 276928: 'val_loss' reached 2.28272 (best 2.24361), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2827-epoch=16.ckpt' as top 3
354
+ Epoch 17: 25%|████████████████████████████████████▏ | 5655/22622 [30:34<1:31:44, 3.08it/s, loss=5.51
355
+ Epoch 17, global step 281002: 'val_loss' reached 2.30270 (best 2.24361), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.3027-epoch=17.ckpt' as top 3
356
+ Epoch 17: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:31<1:02:32, 3.01it/s, loss=2.94
357
+ Epoch 17, global step 285074: 'val_loss' was not in top 3
358
+ Epoch 17: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:27<31:09, 3.03it/s, loss=6.12
359
+ Epoch 17, global step 289146: 'val_loss' reached 2.24171 (best 2.24171), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2417-epoch=17.ckpt' as top 3
360
+ Epoch 17: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████���█▉| 22620/22622 [2:05:53<00:00, 2.99it/s, loss=9.28
361
+ Epoch 17, global step 293218: 'val_loss' reached 2.20842 (best 2.20842), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2084-epoch=17.ckpt' as top 3
362
+ Epoch 18: 25%|████████████████████████████████████▍ | 5655/22622 [30:57<1:32:51, 3.05it/s, loss=5.1
363
+ Epoch 18, global step 297292: 'val_loss' reached 2.22932 (best 2.20842), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2293-epoch=18.ckpt' as top 3
364
+ Epoch 18: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:43<1:02:44, 3.00it/s, loss=2.83
365
+ Epoch 18, global step 301364: 'val_loss' was not in top 3
366
+ Epoch 18: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:32<31:11, 3.02it/s, loss=5.32
367
+ Epoch 18, global step 305436: 'val_loss' reached 2.22272 (best 2.20842), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2227-epoch=18.ckpt' as top 3
368
+ Epoch 18: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:05:18<00:00, 3.01it/s, loss=8
369
+ Epoch 18, global step 309508: 'val_loss' was not in top 3
370
+ Epoch 19: 25%|████████████████████████████████████▏ | 5655/22622 [30:32<1:31:36, 3.09it/s, loss=5.53
371
+ Epoch 19, global step 313582: 'val_loss' was not in top 3
372
+ Epoch 19: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:31<1:02:32, 3.01it/s, loss=2.19
373
+ Epoch 19, global step 317654: 'val_loss' was not in top 3
374
+ Epoch 19: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:20<31:07, 3.03it/s, loss=5.25
375
+ Epoch 19, global step 321726: 'val_loss' reached 2.20790 (best 2.20790), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.2079-epoch=19.ckpt' as top 3
376
+ Epoch 19: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:04:53<00:00, 3.02it/s, loss=8.85
377
+ Epoch 19, global step 325798: 'val_loss' reached 2.18583 (best 2.18583), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1858-epoch=19.ckpt' as top 3
378
+ Epoch 20: 25%|████████████████████████████████████▏ | 5655/22622 [30:33<1:31:41, 3.08it/s, loss=5.18
379
+ Epoch 20, global step 329872: 'val_loss' reached 2.16728 (best 2.16728), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1673-epoch=20.ckpt' as top 3
380
+ Epoch 20: 50%|███████���██████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:31<1:02:32, 3.01it/s, loss=2.19
381
+ Epoch 20, global step 333944: 'val_loss' was not in top 3
382
+ Epoch 20: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:23<31:08, 3.03it/s, loss=4.86
383
+ Epoch 20, global step 338016: 'val_loss' reached 2.15001 (best 2.15001), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1500-epoch=20.ckpt' as top 3
384
+ Epoch 20: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:05:05<00:00, 3.01it/s, loss=8.18
385
+ Epoch 20, global step 342088: 'val_loss' reached 2.15241 (best 2.15001), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1524-epoch=20.ckpt' as top 3
386
+ Epoch 21: 25%|████████████████████████████████████▏ | 5655/22622 [30:40<1:32:01, 3.07it/s, loss=4.87
387
+ Epoch 21, global step 346162: 'val_loss' reached 2.16201 (best 2.15001), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1620-epoch=21.ckpt' as top 3
388
+ Epoch 21: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:40<1:02:40, 3.01it/s, loss=1.77
389
+ Epoch 21, global step 350234: 'val_loss' was not in top 3
390
+ Epoch 21: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 16965/22622 [1:33:40<31:14, 3.02it/s, loss=4.9
391
+ Epoch 21, global step 354306: 'val_loss' reached 2.11477 (best 2.11477), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1148-epoch=21.ckpt' as top 3
392
+ Epoch 21: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:05:24<00:00, 3.01it/s, loss=8.89
393
+ Epoch 21, global step 358378: 'val_loss' reached 2.10107 (best 2.10107), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1011-epoch=21.ckpt' as top 3
394
+ Epoch 22: 25%|████████████████████████████████████▍ | 5655/22622 [30:35<1:31:47, 3.08it/s, loss=4.2
395
+ Epoch 22, global step 362452: 'val_loss' reached 2.11003 (best 2.10107), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1100-epoch=22.ckpt' as top 3
396
+ Epoch 22: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:34<1:02:34, 3.01it/s, loss=2.22
397
+ Epoch 22, global step 366524: 'val_loss' was not in top 3
398
+ Epoch 22: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:43<31:15, 3.02it/s, loss=4.85
399
+ Epoch 22, global step 370596: 'val_loss' reached 2.09616 (best 2.09616), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0962-epoch=22.ckpt' as top 3
400
+ Epoch 22: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:05:29<00:00, 3.00it/s, loss=7.2
401
+ Epoch 22, global step 374668: 'val_loss' was not in top 3
402
+ Epoch 23: 25%|████████████████████████████████████▏ | 5655/22622 [31:12<1:33:39, 3.02it/s, loss=4.72
403
+ Epoch 23, global step 378742: 'val_loss' reached 2.09999 (best 2.09616), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.1000-epoch=23.ckpt' as top 3
404
+ Epoch 23: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:03:08<1:03:09, 2.99it/s, loss=2.14
405
+ Epoch 23, global step 382814: 'val_loss' was not in top 3
406
+ Epoch 23: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:59<31:20, 3.01it/s, loss=4.96
407
+ Epoch 23, global step 386886: 'val_loss' reached 2.02523 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0252-epoch=23.ckpt' as top 3
408
+ Epoch 23: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:06:04<00:00, 2.99it/s, loss=7.46
409
+ Epoch 23, global step 390958: 'val_loss' reached 2.04520 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0452-epoch=23.ckpt' as top 3
410
+ Epoch 24: 25%|████████████████████████████████████▏ | 5655/22622 [30:53<1:32:40, 3.05it/s, loss=4.86
411
+ Epoch 24, global step 395032: 'val_loss' reached 2.09024 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0902-epoch=24.ckpt' as top 3
412
+ Epoch 24: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:03:19<1:03:19, 2.98it/s, loss=2.01
413
+ Epoch 24, global step 399104: 'val_loss' was not in top 3
414
+ Epoch 24: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:35:04<31:42, 2.97it/s, loss=4.56
415
+ Epoch 24, global step 403176: 'val_loss' reached 2.06819 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0682-epoch=24.ckpt' as top 3
416
+ Epoch 24: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:07:24<00:00, 2.96it/s, loss=7.63
417
+ Epoch 24, global step 407248: 'val_loss' reached 2.04502 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0450-epoch=24.ckpt' as top 3
418
+ Epoch 25: 25%|████████████████████████████████████▏ | 5655/22622 [32:22<1:37:07, 2.91it/s, loss=4.52
419
+ Epoch 25, global step 411322: 'val_loss' was not in top 3
420
+ Epoch 25: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:06:21<1:06:22, 2.84it/s, loss=2.36
421
+ Epoch 25, global step 415394: 'val_loss' was not in top 3
422
+ Epoch 25: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:39:16<33:06, 2.85it/s, loss=4.45
423
+ Epoch 25, global step 419466: 'val_loss' reached 2.04345 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0434-epoch=25.ckpt' as top 3
424
+ Epoch 25: 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 20500/22622 [1:57:18<12:08, 2.91it/s, loss=2.57]
425
+ Epoch 25: 91%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 20583/22622 [1:57:41<11:39, 2.91it/s, loss=4.38]
426
+ Epoch 25: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:10:55<00:00, 2.88it/s, loss=7.73
427
+ Epoch 25, global step 423538: 'val_loss' was not in top 3
428
+ Epoch 26: 25%|████████████████████████████████████▏ | 5655/22622 [30:41<1:32:06, 3.07it/s, loss=4.11
429
+ Epoch 26, global step 427612: 'val_loss' reached 2.02936 (best 2.02523), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0294-epoch=26.ckpt' as top 3
430
+ Epoch 26: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:37<1:02:38, 3.01it/s, loss=2.64
431
+ Epoch 26, global step 431684: 'val_loss' was not in top 3
432
+ Epoch 26: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:25<31:09, 3.03it/s, loss=4.69
433
+ Epoch 26, global step 435756: 'val_loss' reached 1.98137 (best 1.98137), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9814-epoch=26.ckpt' as top 3
434
+ Epoch 26: 100%|█████████���█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:05:00<00:00, 3.02it/s, loss=6.87
435
+ Epoch 26, global step 439828: 'val_loss' reached 1.99461 (best 1.98137), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9946-epoch=26.ckpt' as top 3
436
+ Epoch 27: 25%|████████████████████████████████████▏ | 5655/22622 [30:33<1:31:39, 3.09it/s, loss=4.36
437
+ Epoch 27, global step 443902: 'val_loss' was not in top 3
438
+ Epoch 27: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:29<1:02:30, 3.02it/s, loss=1.89
439
+ Epoch 27, global step 447974: 'val_loss' was not in top 3
440
+ Epoch 27: 69%|████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 15646/22622 [1:25:03<37:55, 3.07it/s, loss=4.3]
441
+ Epoch 27: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 16965/22622 [1:35:45<31:55, 2.95it/s, loss=4.3
442
+ Epoch 27, global step 452046: 'val_loss' reached 2.02101 (best 1.98137), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=2.0210-epoch=27.ckpt' as top 3
443
+ Epoch 27: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:08:18<00:00, 2.94it/s, loss=7.41
444
+ Epoch 27, global step 456118: 'val_loss' reached 1.99866 (best 1.98137), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9987-epoch=27.ckpt' as top 3
445
+ Epoch 28: 25%|████████████████████████████████████▏ | 5655/22622 [31:37<1:34:53, 2.98it/s, loss=4.11
446
+ Epoch 28, global step 460192: 'val_loss' reached 1.98746 (best 1.98137), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9875-epoch=28.ckpt' as top 3
447
+ Epoch 28: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:04:29<1:04:29, 2.92it/s, loss=1.21
448
+ Epoch 28, global step 464264: 'val_loss' was not in top 3
449
+ Epoch 28: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:36:08<32:03, 2.94it/s, loss=4.24
450
+ Epoch 28, global step 468336: 'val_loss' reached 1.98728 (best 1.98137), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9873-epoch=28.ckpt' as top 3
451
+ Epoch 28: 100%|███████████████████████████████████████████████████████████████████████████████████████████████��███████████████████████████████████████████████▉| 22620/22622 [2:07:59<00:00, 2.95it/s, loss=7.27
452
+ Epoch 28, global step 472408: 'val_loss' was not in top 3
453
+ Epoch 29: 25%|████████████████████████████████████▏ | 5655/22622 [30:34<1:31:45, 3.08it/s, loss=3.93
454
+ Epoch 29, global step 476482: 'val_loss' reached 1.97654 (best 1.97654), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9765-epoch=29.ckpt' as top 3
455
+ Epoch 29: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:29<1:02:29, 3.02it/s, loss=2.08
456
+ Epoch 29, global step 480554: 'val_loss' was not in top 3
457
+ Epoch 29: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:28<31:10, 3.03it/s, loss=4.02
458
+ Epoch 29, global step 484626: 'val_loss' reached 1.95432 (best 1.95432), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9543-epoch=29.ckpt' as top 3
459
+ Epoch 29: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:05:10<00:00, 3.01it/s, loss=6.43
460
+ Epoch 29, global step 488698: 'val_loss' reached 1.97343 (best 1.95432), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9734-epoch=29.ckpt' as top 3
461
+ Epoch 30: 25%|████████████████████████████████████▏ | 5655/22622 [30:17<1:30:53, 3.11it/s, loss=4.03
462
+ Epoch 30, global step 492772: 'val_loss' reached 1.96907 (best 1.95432), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9691-epoch=30.ckpt' as top 3
463
+ Epoch 30: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:05<1:02:06, 3.04it/s, loss=2.11
464
+ Epoch 30, global step 496844: 'val_loss' was not in top 3
465
+ Epoch 30: 75%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 16965/22622 [1:32:56<30:59, 3.04it/s, loss=4
466
+ Epoch 30, global step 500916: 'val_loss' reached 1.95860 (best 1.95432), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9586-epoch=30.ckpt' as top 3
467
+ Epoch 30: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:04:29<00:00, 3.03it/s, loss=7.02
468
+ Epoch 30, global step 504988: 'val_loss' was not in top 3
469
+ Epoch 31: 25%|████████████████████████████████████▏ | 5655/22622 [30:30<1:31:30, 3.09it/s, loss=4.05
470
+ Epoch 31, global step 509062: 'val_loss' reached 1.96284 (best 1.95432), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9628-epoch=31.ckpt' as top 3
471
+ Epoch 31: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:25<1:02:26, 3.02it/s, loss=1.28
472
+ Epoch 31, global step 513134: 'val_loss' was not in top 3
473
+ Epoch 31: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:12<31:04, 3.03it/s, loss=4.27
474
+ Epoch 31, global step 517206: 'val_loss' reached 1.94839 (best 1.94839), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9484-epoch=31.ckpt' as top 3
475
+ Epoch 31: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:04:43<00:00, 3.02it/s, loss=6.85
476
+ Epoch 31, global step 521278: 'val_loss' was not in top 3
477
+ Epoch 32: 25%|████████████████████████████████████▏ | 5655/22622 [30:31<1:31:33, 3.09it/s, loss=3.87
478
+ Epoch 32, global step 525352: 'val_loss' was not in top 3
479
+ Epoch 32: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:15<1:02:16, 3.03it/s, loss=1.58
480
+ Epoch 32, global step 529424: 'val_loss' was not in top 3
481
+ Epoch 32: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:11<31:04, 3.03it/s, loss=4.01
482
+ Epoch 32, global step 533496: 'val_loss' reached 1.93692 (best 1.93692), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9369-epoch=32.ckpt' as top 3
483
+ Epoch 32: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:04:54<00:00, 3.02it/s, loss=6.91
484
+ Epoch 32, global step 537568: 'val_loss' reached 1.94155 (best 1.93692), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9416-epoch=32.ckpt' as top 3
485
+ Epoch 33: 25%|████████████████████████████████████▏ | 5655/22622 [31:17<1:33:53, 3.01it/s, loss=3.65
486
+ Epoch 33, global step 541642: 'val_loss' reached 1.94692 (best 1.93692), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9469-epoch=33.ckpt' as top 3
487
+ Epoch 33: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:03:17<1:03:18, 2.98it/s, loss=1.37
488
+ Epoch 33, global step 545714: 'val_loss' was not in top 3
489
+ Epoch 33: 75%|█████████████████████████████████████████████████████████████████████████████████████████████████████████��█▉ | 16965/22622 [1:34:14<31:25, 3.00it/s, loss=3.67
490
+ Epoch 33, global step 549786: 'val_loss' reached 1.89907 (best 1.89907), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.8991-epoch=33.ckpt' as top 3
491
+ Epoch 33: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:05:55<00:00, 2.99it/s, loss=6.16
492
+ Epoch 33, global step 553858: 'val_loss' was not in top 3
493
+ Epoch 34: 25%|████████████████████████████████████▏ | 5655/22622 [30:34<1:31:44, 3.08it/s, loss=3.99
494
+ Epoch 34, global step 557932: 'val_loss' reached 1.91820 (best 1.89907), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9182-epoch=34.ckpt' as top 3
495
+ Epoch 34: 50%|██████████████████████████████████████████████████████████████████████▉ | 11310/22622 [1:02:22<1:02:23, 3.02it/s, loss=1.44
496
+ Epoch 34, global step 562004: 'val_loss' was not in top 3
497
+ Epoch 34: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 16965/22622 [1:33:32<31:11, 3.02it/s, loss=3.65
498
+ Epoch 34, global step 566076: 'val_loss' reached 1.90886 (best 1.89907), saving model to '/ds/experiments/stt_en_quartznet15x5_v1/2023-04-04_02-57-02/checkpoints/stt_en_quartznet15x5_v1--val_loss=1.9089-epoch=34.ckpt' as top 3
499
+ Epoch 34: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 22620/22622 [2:05:38<00:00, 3.00it/s, loss=6.1
500
+ Epoch 34, global step 570148: 'val_loss' was not in top 3
501
+ Epoch 35: 0%|▏ | 20/22622 [00:08<2:42:53, 2.31it/s, loss=5.5]
502
+ ^C[NeMo W 2023-04-07 03:58:41 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/call.py:48: UserWarning: Detected KeyboardInterrupt, attempting graceful shutdown...
503
+ rank_zero_warn("Detected KeyboardInterrupt, attempting graceful shutdown...")
504
+
505
+ [NeMo I 2023-04-07 03:58:41 training_stt_en_quartznet15x5_v1:45] fit Done!
506
+ [NeMo I 2023-04-07 03:58:42 training_stt_en_quartznet15x5_v1:49] Done!
507
+ root@e0f8def33916:/workspace/nemo#