change file location
Browse files- cpu_jit.pt → exp/cpu_jit.pt +0 -0
- exp/log/log-train-2022-08-08-11-57-24-0 +122 -0
- exp/log/log-train-2022-08-08-11-57-24-1 +70 -0
- exp/log/log-train-2022-08-08-11-57-24-2 +70 -0
- exp/log/log-train-2022-08-08-14-49-56-0 +0 -0
- exp/log/log-train-2022-08-08-14-49-56-1 +0 -0
- exp/log/log-train-2022-08-08-14-49-56-2 +0 -0
- exp/log/log-train-2022-08-09-16-45-14-0 +208 -0
- exp/log/log-train-2022-08-09-16-45-14-1 +158 -0
- exp/log/log-train-2022-08-09-16-45-14-2 +158 -0
- exp/log/log-train-2022-08-11-08-52-44-0 +278 -0
- exp/log/log-train-2022-08-11-08-52-44-1 +225 -0
- exp/log/log-train-2022-08-11-08-52-44-2 +225 -0
- exp/log/log-train-2022-08-11-18-53-52-0 +344 -0
- exp/log/log-train-2022-08-11-18-53-52-1 +290 -0
- exp/log/log-train-2022-08-11-18-53-52-2 +290 -0
- pretrained.pt → exp/pretrained_epoch17_avg2.pt +0 -0
- {tensorboard_log → exp/tensorboard}/events.out.tfevents.1659925381.dgxs-01.cluster.thspzh.com.18736.0 +0 -0
- {tensorboard_log → exp/tensorboard}/events.out.tfevents.1659926695.dgxs-01.cluster.thspzh.com.27410.0 +0 -0
- {tensorboard_log → exp/tensorboard}/events.out.tfevents.1659927762.dgxs-01.cluster.thspzh.com.34561.0 +0 -0
- {tensorboard_log → exp/tensorboard}/events.out.tfevents.1659928526.dgxs-01.cluster.thspzh.com.39902.0 +0 -0
- {tensorboard_log → exp/tensorboard}/events.out.tfevents.1659929206.dgxs-01.cluster.thspzh.com.4352.0 +0 -0
- {tensorboard_log → exp/tensorboard}/events.out.tfevents.1659930110.dgxs-01.cluster.thspzh.com.10312.0 +0 -0
- {tensorboard_log → exp/tensorboard}/events.out.tfevents.1659931044.dgxs-01.cluster.thspzh.com.16831.0 +0 -0
- {tensorboard_log → exp/tensorboard}/events.out.tfevents.1659940194.dgxs-01.cluster.thspzh.com.37638.0 +0 -0
- {tensorboard_log → exp/tensorboard}/events.out.tfevents.1659941396.dgxs-01.cluster.thspzh.com.5637.0 +0 -0
- {tensorboard_log → exp/tensorboard}/events.out.tfevents.1660034714.dgxs-01.cluster.thspzh.com.3516.0 +0 -0
- {tensorboard_log → exp/tensorboard}/events.out.tfevents.1660179164.dgxs-01.cluster.thspzh.com.14444.0 +0 -0
- {tensorboard_log → exp/tensorboard}/events.out.tfevents.1660215232.dgxs-01.cluster.thspzh.com.3323.0 +0 -0
cpu_jit.pt → exp/cpu_jit.pt
RENAMED
|
File without changes
|
exp/log/log-train-2022-08-08-11-57-24-0
ADDED
|
@@ -0,0 +1,122 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2022-08-08 11:57:24,907 INFO [train.py:786] (0/3) Training started
|
| 2 |
+
2022-08-08 11:57:24,908 INFO [train.py:787] (0/3) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'env_info': {'k2-version': '1.17', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '7dcabf85e8bf06984c4abab0400ef1322b5ff3df', 'k2-git-date': 'Tue Aug 2 21:22:39 2022', 'lhotse-version': '1.4.0.dev+git.59318ed.clean', 'torch-version': '1.11.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.3', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '6af45c3-dirty', 'icefall-git-date': 'Tue Aug 2 14:47:09 2022', 'icefall-path': '/thspfs/home/acct-hpc/thspzh-hpc1/git/icefall_dev', 'k2-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/k2-1.17.dev20220805+cuda11.3.torch1.11.0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'dgxs-01.cluster.thspzh.com', 'IP address': '192.168.200.101'}, 'world_size': 3, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 10, 'start_epoch': 1, 'start_batch': 0, 'exp_dir': PosixPath('cnn_ctc/exp_phone'), 'lang_dir': 'data/lang_phone', 'bpe_model': 'data/lang_bpe_500/bpe.model', 'initial_lr': 0.003, 'lr_batches': 5000, 'lr_epochs': 6, 'seed': 42, 'print_diagnostics': False, 'save_every_n': 8000, 'keep_last_k': 20, 'average_period': 100, 'use_fp16': False, 'conv_blocks': 3, 'full_libri': True, 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 200.0, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'input_strategy': 'PrecomputedFeatures'}
|
| 3 |
+
2022-08-08 11:57:25,284 INFO [lexicon.py:176] (0/3) Loading pre-compiled data/lang_phone/Linv.pt
|
| 4 |
+
2022-08-08 11:57:36,661 INFO [train.py:824] (0/3) About to create model
|
| 5 |
+
2022-08-08 11:57:36,684 INFO [train.py:834] (0/3) Number of model parameters: 1766904
|
| 6 |
+
2022-08-08 11:57:42,077 INFO [train.py:838] (0/3) ==========================================================================================
|
| 7 |
+
Layer (type:depth-idx) Output Shape Param #
|
| 8 |
+
==========================================================================================
|
| 9 |
+
MobileNetS4 [1, 25, 72] --
|
| 10 |
+
├─Sequential: 1-1 [1, 32, 25, 20] --
|
| 11 |
+
│ └─Conv2d: 2-1 [1, 8, 100, 80] 80
|
| 12 |
+
│ └─BatchNorm2d: 2-2 [1, 8, 100, 80] 16
|
| 13 |
+
│ └─ReLU6: 2-3 [1, 8, 100, 80] --
|
| 14 |
+
│ └─Conv2d: 2-4 [1, 16, 50, 40] 1,168
|
| 15 |
+
│ └─BatchNorm2d: 2-5 [1, 16, 50, 40] 32
|
| 16 |
+
│ └─ReLU6: 2-6 [1, 16, 50, 40] --
|
| 17 |
+
│ └─Conv2d: 2-7 [1, 32, 25, 20] 4,640
|
| 18 |
+
│ └─BatchNorm2d: 2-8 [1, 32, 25, 20] 64
|
| 19 |
+
│ └─ReLU6: 2-9 [1, 32, 25, 20] --
|
| 20 |
+
├─Sequential: 1-2 [1, 256, 25, 3] --
|
| 21 |
+
│ └─Bottleneck: 2-10 [1, 64, 25, 10] --
|
| 22 |
+
│ │ └─Sequential: 3-1 [1, 64, 25, 10] 14,400
|
| 23 |
+
│ └─Bottleneck: 2-11 [1, 64, 25, 10] --
|
| 24 |
+
│ │ └─Sequential: 3-2 [1, 64, 25, 10] 36,800
|
| 25 |
+
│ └─Bottleneck: 2-12 [1, 64, 25, 10] --
|
| 26 |
+
│ │ └─Sequential: 3-3 [1, 64, 25, 10] 36,800
|
| 27 |
+
│ └─Bottleneck: 2-13 [1, 128, 25, 5] --
|
| 28 |
+
│ │ └─Sequential: 3-4 [1, 128, 25, 5] 53,376
|
| 29 |
+
│ └─Bottleneck: 2-14 [1, 128, 25, 5] --
|
| 30 |
+
│ │ └─Sequential: 3-5 [1, 128, 25, 5] 139,136
|
| 31 |
+
│ └─Bottleneck: 2-15 [1, 128, 25, 5] --
|
| 32 |
+
│ │ └─Sequential: 3-6 [1, 128, 25, 5] 139,136
|
| 33 |
+
│ └─Bottleneck: 2-16 [1, 256, 25, 3] --
|
| 34 |
+
│ │ └─Sequential: 3-7 [1, 256, 25, 3] 205,056
|
| 35 |
+
│ └─Bottleneck: 2-17 [1, 256, 25, 3] --
|
| 36 |
+
│ │ └─Sequential: 3-8 [1, 256, 25, 3] 540,416
|
| 37 |
+
│ └─Bottleneck: 2-18 [1, 256, 25, 3] --
|
| 38 |
+
│ │ └─Sequential: 3-9 [1, 256, 25, 3] 540,416
|
| 39 |
+
├─Linear: 1-3 [1, 25, 72] 55,368
|
| 40 |
+
==========================================================================================
|
| 41 |
+
Total params: 1,766,904
|
| 42 |
+
Trainable params: 1,766,904
|
| 43 |
+
Non-trainable params: 0
|
| 44 |
+
Total mult-adds (M): 169.35
|
| 45 |
+
==========================================================================================
|
| 46 |
+
Input size (MB): 0.03
|
| 47 |
+
Forward/backward pass size (MB): 22.08
|
| 48 |
+
Params size (MB): 7.07
|
| 49 |
+
Estimated Total Size (MB): 29.18
|
| 50 |
+
==========================================================================================
|
| 51 |
+
2022-08-08 11:57:42,137 INFO [train.py:855] (0/3) Using DDP
|
| 52 |
+
2022-08-08 11:57:42,267 INFO [asr_datamodule.py:409] (0/3) About to get train-clean-100 cuts
|
| 53 |
+
2022-08-08 11:57:42,318 INFO [asr_datamodule.py:416] (0/3) About to get train-clean-360 cuts
|
| 54 |
+
2022-08-08 11:57:42,339 INFO [asr_datamodule.py:423] (0/3) About to get train-other-500 cuts
|
| 55 |
+
2022-08-08 11:57:42,362 INFO [asr_datamodule.py:225] (0/3) Enable MUSAN
|
| 56 |
+
2022-08-08 11:57:42,363 INFO [asr_datamodule.py:226] (0/3) About to get Musan cuts
|
| 57 |
+
2022-08-08 11:57:44,447 INFO [asr_datamodule.py:254] (0/3) Enable SpecAugment
|
| 58 |
+
2022-08-08 11:57:44,447 INFO [asr_datamodule.py:255] (0/3) Time warp factor: 80
|
| 59 |
+
2022-08-08 11:57:44,447 INFO [asr_datamodule.py:267] (0/3) Num frame mask: 10
|
| 60 |
+
2022-08-08 11:57:44,447 INFO [asr_datamodule.py:280] (0/3) About to create train dataset
|
| 61 |
+
2022-08-08 11:57:44,447 INFO [asr_datamodule.py:309] (0/3) Using DynamicBucketingSampler.
|
| 62 |
+
2022-08-08 12:00:03,359 INFO [asr_datamodule.py:324] (0/3) About to create train dataloader
|
| 63 |
+
2022-08-08 12:00:03,378 INFO [asr_datamodule.py:430] (0/3) About to get dev-clean cuts
|
| 64 |
+
2022-08-08 12:00:03,633 INFO [asr_datamodule.py:437] (0/3) About to get dev-other cuts
|
| 65 |
+
2022-08-08 12:00:03,654 INFO [asr_datamodule.py:355] (0/3) About to create dev dataset
|
| 66 |
+
2022-08-08 12:00:03,967 INFO [asr_datamodule.py:374] (0/3) About to create dev dataloader
|
| 67 |
+
2022-08-08 12:03:06,077 INFO [train.py:718] (0/3) Epoch 1, batch 0, loss[loss=3.5, over 4808.00 frames; over 0.00 utterances.], tot_loss[loss=3.5, over 4808.00 frames; over 0.00 utterances.], batch size: 11, lr: 3.00e-03
|
| 68 |
+
2022-08-08 12:03:06,234 INFO [distributed.py:948] (0/3) Reducer buckets have been rebuilt in this iteration.
|
| 69 |
+
2022-08-08 12:04:07,322 INFO [train.py:718] (0/3) Epoch 1, batch 100, loss[loss=0.8795, over 4686.00 frames; over 0.00 utterances.], tot_loss[loss=1.046, over 370422.90 frames; over 0.00 utterances.], batch size: 18, lr: 3.00e-03
|
| 70 |
+
2022-08-08 12:05:07,016 INFO [train.py:718] (0/3) Epoch 1, batch 200, loss[loss=0.7045, over 4858.00 frames; over 0.00 utterances.], tot_loss[loss=0.8583, over 587433.55 frames; over 0.00 utterances.], batch size: 14, lr: 3.00e-03
|
| 71 |
+
2022-08-08 12:06:05,036 INFO [train.py:718] (0/3) Epoch 1, batch 300, loss[loss=0.5934, over 4719.00 frames; over 0.00 utterances.], tot_loss[loss=0.7477, over 722634.32 frames; over 0.00 utterances.], batch size: 13, lr: 3.00e-03
|
| 72 |
+
2022-08-08 12:07:03,654 INFO [train.py:718] (0/3) Epoch 1, batch 400, loss[loss=0.5681, over 4577.00 frames; over 0.00 utterances.], tot_loss[loss=0.6813, over 799954.49 frames; over 0.00 utterances.], batch size: 20, lr: 3.00e-03
|
| 73 |
+
2022-08-08 12:08:02,702 INFO [train.py:718] (0/3) Epoch 1, batch 500, loss[loss=0.5124, over 4603.00 frames; over 0.00 utterances.], tot_loss[loss=0.6319, over 853201.95 frames; over 0.00 utterances.], batch size: 16, lr: 2.99e-03
|
| 74 |
+
2022-08-08 12:09:01,211 INFO [train.py:718] (0/3) Epoch 1, batch 600, loss[loss=0.4919, over 4599.00 frames; over 0.00 utterances.], tot_loss[loss=0.5932, over 886419.63 frames; over 0.00 utterances.], batch size: 13, lr: 2.99e-03
|
| 75 |
+
2022-08-08 12:09:58,146 INFO [train.py:718] (0/3) Epoch 1, batch 700, loss[loss=0.5246, over 4740.00 frames; over 0.00 utterances.], tot_loss[loss=0.5661, over 904289.39 frames; over 0.00 utterances.], batch size: 12, lr: 2.99e-03
|
| 76 |
+
2022-08-08 12:10:42,889 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-1.pt
|
| 77 |
+
2022-08-08 12:13:45,526 INFO [train.py:718] (0/3) Epoch 2, batch 0, loss[loss=0.5289, over 4602.00 frames; over 0.00 utterances.], tot_loss[loss=0.5289, over 4602.00 frames; over 0.00 utterances.], batch size: 20, lr: 2.96e-03
|
| 78 |
+
2022-08-08 12:14:42,205 INFO [train.py:718] (0/3) Epoch 2, batch 100, loss[loss=0.5258, over 4804.00 frames; over 0.00 utterances.], tot_loss[loss=0.4971, over 367238.76 frames; over 0.00 utterances.], batch size: 12, lr: 2.96e-03
|
| 79 |
+
2022-08-08 12:15:36,069 INFO [train.py:718] (0/3) Epoch 2, batch 200, loss[loss=0.4961, over 4812.00 frames; over 0.00 utterances.], tot_loss[loss=0.4929, over 587646.82 frames; over 0.00 utterances.], batch size: 15, lr: 2.95e-03
|
| 80 |
+
2022-08-08 12:16:29,347 INFO [train.py:718] (0/3) Epoch 2, batch 300, loss[loss=0.4743, over 4686.00 frames; over 0.00 utterances.], tot_loss[loss=0.4901, over 723194.07 frames; over 0.00 utterances.], batch size: 12, lr: 2.95e-03
|
| 81 |
+
2022-08-08 12:17:17,407 INFO [train.py:718] (0/3) Epoch 2, batch 400, loss[loss=0.4773, over 4352.00 frames; over 0.00 utterances.], tot_loss[loss=0.488, over 805919.45 frames; over 0.00 utterances.], batch size: 23, lr: 2.94e-03
|
| 82 |
+
2022-08-08 12:18:04,574 INFO [train.py:718] (0/3) Epoch 2, batch 500, loss[loss=0.4655, over 4286.00 frames; over 0.00 utterances.], tot_loss[loss=0.4837, over 854414.10 frames; over 0.00 utterances.], batch size: 38, lr: 2.93e-03
|
| 83 |
+
2022-08-08 12:18:51,651 INFO [train.py:718] (0/3) Epoch 2, batch 600, loss[loss=0.556, over 4882.00 frames; over 0.00 utterances.], tot_loss[loss=0.4776, over 884533.73 frames; over 0.00 utterances.], batch size: 12, lr: 2.93e-03
|
| 84 |
+
2022-08-08 12:19:42,987 INFO [train.py:718] (0/3) Epoch 2, batch 700, loss[loss=0.4943, over 4621.00 frames; over 0.00 utterances.], tot_loss[loss=0.4754, over 903358.22 frames; over 0.00 utterances.], batch size: 20, lr: 2.92e-03
|
| 85 |
+
2022-08-08 12:20:22,464 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-2.pt
|
| 86 |
+
2022-08-08 12:23:25,732 INFO [train.py:718] (0/3) Epoch 3, batch 0, loss[loss=0.4261, over 4807.00 frames; over 0.00 utterances.], tot_loss[loss=0.4261, over 4807.00 frames; over 0.00 utterances.], batch size: 11, lr: 2.86e-03
|
| 87 |
+
2022-08-08 12:24:24,646 INFO [train.py:718] (0/3) Epoch 3, batch 100, loss[loss=0.4667, over 4383.00 frames; over 0.00 utterances.], tot_loss[loss=0.4439, over 372391.82 frames; over 0.00 utterances.], batch size: 29, lr: 2.85e-03
|
| 88 |
+
2022-08-08 12:25:22,419 INFO [train.py:718] (0/3) Epoch 3, batch 200, loss[loss=0.4083, over 4729.00 frames; over 0.00 utterances.], tot_loss[loss=0.4403, over 592320.37 frames; over 0.00 utterances.], batch size: 18, lr: 2.84e-03
|
| 89 |
+
2022-08-08 12:26:16,407 INFO [train.py:718] (0/3) Epoch 3, batch 300, loss[loss=0.412, over 4929.00 frames; over 0.00 utterances.], tot_loss[loss=0.4439, over 728048.17 frames; over 0.00 utterances.], batch size: 12, lr: 2.83e-03
|
| 90 |
+
2022-08-08 12:27:11,962 INFO [train.py:718] (0/3) Epoch 3, batch 400, loss[loss=0.4107, over 4669.00 frames; over 0.00 utterances.], tot_loss[loss=0.4459, over 807137.73 frames; over 0.00 utterances.], batch size: 12, lr: 2.82e-03
|
| 91 |
+
2022-08-08 12:28:07,045 INFO [train.py:718] (0/3) Epoch 3, batch 500, loss[loss=0.4801, over 4685.00 frames; over 0.00 utterances.], tot_loss[loss=0.4475, over 851289.46 frames; over 0.00 utterances.], batch size: 13, lr: 2.81e-03
|
| 92 |
+
2022-08-08 12:28:57,341 INFO [train.py:718] (0/3) Epoch 3, batch 600, loss[loss=0.4326, over 4808.00 frames; over 0.00 utterances.], tot_loss[loss=0.4449, over 883406.23 frames; over 0.00 utterances.], batch size: 18, lr: 2.80e-03
|
| 93 |
+
2022-08-08 12:29:50,234 INFO [train.py:718] (0/3) Epoch 3, batch 700, loss[loss=0.4539, over 4630.00 frames; over 0.00 utterances.], tot_loss[loss=0.4458, over 898356.10 frames; over 0.00 utterances.], batch size: 16, lr: 2.79e-03
|
| 94 |
+
2022-08-08 12:30:35,724 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-3.pt
|
| 95 |
+
2022-08-08 12:33:28,615 INFO [train.py:718] (0/3) Epoch 4, batch 0, loss[loss=0.4819, over 4598.00 frames; over 0.00 utterances.], tot_loss[loss=0.4819, over 4598.00 frames; over 0.00 utterances.], batch size: 15, lr: 2.70e-03
|
| 96 |
+
2022-08-08 12:34:26,047 INFO [train.py:718] (0/3) Epoch 4, batch 100, loss[loss=0.3584, over 4810.00 frames; over 0.00 utterances.], tot_loss[loss=0.4205, over 369984.00 frames; over 0.00 utterances.], batch size: 15, lr: 2.69e-03
|
| 97 |
+
2022-08-08 12:35:21,128 INFO [train.py:718] (0/3) Epoch 4, batch 200, loss[loss=0.5133, over 4316.00 frames; over 0.00 utterances.], tot_loss[loss=0.4287, over 587659.26 frames; over 0.00 utterances.], batch size: 38, lr: 2.68e-03
|
| 98 |
+
2022-08-08 12:36:12,971 INFO [train.py:718] (0/3) Epoch 4, batch 300, loss[loss=0.4205, over 4806.00 frames; over 0.00 utterances.], tot_loss[loss=0.4311, over 725391.42 frames; over 0.00 utterances.], batch size: 15, lr: 2.67e-03
|
| 99 |
+
2022-08-08 12:37:09,704 INFO [train.py:718] (0/3) Epoch 4, batch 400, loss[loss=0.5384, over 3490.00 frames; over 0.00 utterances.], tot_loss[loss=0.4308, over 805910.34 frames; over 0.00 utterances.], batch size: 55, lr: 2.66e-03
|
| 100 |
+
2022-08-08 12:38:00,656 INFO [train.py:718] (0/3) Epoch 4, batch 500, loss[loss=0.4212, over 4630.00 frames; over 0.00 utterances.], tot_loss[loss=0.4303, over 851251.29 frames; over 0.00 utterances.], batch size: 14, lr: 2.65e-03
|
| 101 |
+
2022-08-08 12:38:57,938 INFO [train.py:718] (0/3) Epoch 4, batch 600, loss[loss=0.4084, over 4630.00 frames; over 0.00 utterances.], tot_loss[loss=0.4295, over 879648.95 frames; over 0.00 utterances.], batch size: 20, lr: 2.64e-03
|
| 102 |
+
2022-08-08 12:39:46,641 INFO [train.py:718] (0/3) Epoch 4, batch 700, loss[loss=0.3916, over 4930.00 frames; over 0.00 utterances.], tot_loss[loss=0.4311, over 899080.12 frames; over 0.00 utterances.], batch size: 12, lr: 2.63e-03
|
| 103 |
+
2022-08-08 12:40:23,639 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-4.pt
|
| 104 |
+
2022-08-08 12:43:24,557 INFO [train.py:718] (0/3) Epoch 5, batch 0, loss[loss=0.4231, over 4898.00 frames; over 0.00 utterances.], tot_loss[loss=0.4231, over 4898.00 frames; over 0.00 utterances.], batch size: 16, lr: 2.52e-03
|
| 105 |
+
2022-08-08 12:44:22,006 INFO [train.py:718] (0/3) Epoch 5, batch 100, loss[loss=0.4225, over 4812.00 frames; over 0.00 utterances.], tot_loss[loss=0.4053, over 369414.86 frames; over 0.00 utterances.], batch size: 11, lr: 2.51e-03
|
| 106 |
+
2022-08-08 12:45:13,939 INFO [train.py:718] (0/3) Epoch 5, batch 200, loss[loss=0.4483, over 4867.00 frames; over 0.00 utterances.], tot_loss[loss=0.4061, over 591385.76 frames; over 0.00 utterances.], batch size: 11, lr: 2.50e-03
|
| 107 |
+
2022-08-08 12:46:05,145 INFO [train.py:718] (0/3) Epoch 5, batch 300, loss[loss=0.3985, over 4604.00 frames; over 0.00 utterances.], tot_loss[loss=0.4147, over 726709.58 frames; over 0.00 utterances.], batch size: 16, lr: 2.49e-03
|
| 108 |
+
2022-08-08 12:47:03,456 INFO [train.py:718] (0/3) Epoch 5, batch 400, loss[loss=0.4628, over 4140.00 frames; over 0.00 utterances.], tot_loss[loss=0.4156, over 807365.38 frames; over 0.00 utterances.], batch size: 38, lr: 2.48e-03
|
| 109 |
+
2022-08-08 12:47:56,601 INFO [train.py:718] (0/3) Epoch 5, batch 500, loss[loss=0.39, over 4568.00 frames; over 0.00 utterances.], tot_loss[loss=0.4171, over 856420.95 frames; over 0.00 utterances.], batch size: 10, lr: 2.47e-03
|
| 110 |
+
2022-08-08 12:48:47,970 INFO [train.py:718] (0/3) Epoch 5, batch 600, loss[loss=0.3454, over 4645.00 frames; over 0.00 utterances.], tot_loss[loss=0.412, over 886820.53 frames; over 0.00 utterances.], batch size: 14, lr: 2.46e-03
|
| 111 |
+
2022-08-08 12:49:40,855 INFO [train.py:718] (0/3) Epoch 5, batch 700, loss[loss=0.4502, over 4607.00 frames; over 0.00 utterances.], tot_loss[loss=0.4153, over 906053.87 frames; over 0.00 utterances.], batch size: 13, lr: 2.44e-03
|
| 112 |
+
2022-08-08 12:50:21,757 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-5.pt
|
| 113 |
+
2022-08-08 12:53:11,632 INFO [train.py:718] (0/3) Epoch 6, batch 0, loss[loss=0.3713, over 4731.00 frames; over 0.00 utterances.], tot_loss[loss=0.3713, over 4731.00 frames; over 0.00 utterances.], batch size: 12, lr: 2.34e-03
|
| 114 |
+
2022-08-08 12:54:08,888 INFO [train.py:718] (0/3) Epoch 6, batch 100, loss[loss=0.4342, over 4922.00 frames; over 0.00 utterances.], tot_loss[loss=0.3915, over 373908.88 frames; over 0.00 utterances.], batch size: 13, lr: 2.33e-03
|
| 115 |
+
2022-08-08 12:55:02,075 INFO [train.py:718] (0/3) Epoch 6, batch 200, loss[loss=0.3864, over 4862.00 frames; over 0.00 utterances.], tot_loss[loss=0.3966, over 598135.14 frames; over 0.00 utterances.], batch size: 13, lr: 2.32e-03
|
| 116 |
+
2022-08-08 12:55:53,429 INFO [train.py:718] (0/3) Epoch 6, batch 300, loss[loss=0.3674, over 4612.00 frames; over 0.00 utterances.], tot_loss[loss=0.3984, over 726369.26 frames; over 0.00 utterances.], batch size: 15, lr: 2.31e-03
|
| 117 |
+
2022-08-08 12:56:42,191 INFO [train.py:718] (0/3) Epoch 6, batch 400, loss[loss=0.3642, over 4881.00 frames; over 0.00 utterances.], tot_loss[loss=0.3992, over 806882.34 frames; over 0.00 utterances.], batch size: 16, lr: 2.30e-03
|
| 118 |
+
2022-08-08 12:57:35,165 INFO [train.py:718] (0/3) Epoch 6, batch 500, loss[loss=0.4078, over 4576.00 frames; over 0.00 utterances.], tot_loss[loss=0.3995, over 855020.05 frames; over 0.00 utterances.], batch size: 11, lr: 2.28e-03
|
| 119 |
+
2022-08-08 12:58:33,295 INFO [train.py:718] (0/3) Epoch 6, batch 600, loss[loss=0.4676, over 4164.00 frames; over 0.00 utterances.], tot_loss[loss=0.4062, over 877378.84 frames; over 0.00 utterances.], batch size: 38, lr: 2.27e-03
|
| 120 |
+
2022-08-08 12:59:23,136 INFO [train.py:718] (0/3) Epoch 6, batch 700, loss[loss=0.4103, over 4160.00 frames; over 0.00 utterances.], tot_loss[loss=0.4067, over 898417.62 frames; over 0.00 utterances.], batch size: 38, lr: 2.26e-03
|
| 121 |
+
2022-08-08 13:00:09,617 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-6.pt
|
| 122 |
+
2022-08-08 13:03:10,378 INFO [train.py:718] (0/3) Epoch 7, batch 0, loss[loss=0.3404, over 4688.00 frames; over 0.00 utterances.], tot_loss[loss=0.3404, over 4688.00 frames; over 0.00 utterances.], batch size: 11, lr: 2.16e-03
|
exp/log/log-train-2022-08-08-11-57-24-1
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2022-08-08 11:57:24,912 INFO [train.py:786] (1/3) Training started
|
| 2 |
+
2022-08-08 11:57:24,912 INFO [train.py:787] (1/3) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'env_info': {'k2-version': '1.17', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '7dcabf85e8bf06984c4abab0400ef1322b5ff3df', 'k2-git-date': 'Tue Aug 2 21:22:39 2022', 'lhotse-version': '1.4.0.dev+git.59318ed.clean', 'torch-version': '1.11.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.3', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '6af45c3-dirty', 'icefall-git-date': 'Tue Aug 2 14:47:09 2022', 'icefall-path': '/thspfs/home/acct-hpc/thspzh-hpc1/git/icefall_dev', 'k2-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/k2-1.17.dev20220805+cuda11.3.torch1.11.0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'dgxs-01.cluster.thspzh.com', 'IP address': '192.168.200.101'}, 'world_size': 3, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 10, 'start_epoch': 1, 'start_batch': 0, 'exp_dir': PosixPath('cnn_ctc/exp_phone'), 'lang_dir': 'data/lang_phone', 'bpe_model': 'data/lang_bpe_500/bpe.model', 'initial_lr': 0.003, 'lr_batches': 5000, 'lr_epochs': 6, 'seed': 42, 'print_diagnostics': False, 'save_every_n': 8000, 'keep_last_k': 20, 'average_period': 100, 'use_fp16': False, 'conv_blocks': 3, 'full_libri': True, 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 200.0, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'input_strategy': 'PrecomputedFeatures'}
|
| 3 |
+
2022-08-08 11:57:25,283 INFO [lexicon.py:176] (1/3) Loading pre-compiled data/lang_phone/Linv.pt
|
| 4 |
+
2022-08-08 11:57:36,661 INFO [train.py:824] (1/3) About to create model
|
| 5 |
+
2022-08-08 11:57:36,692 INFO [train.py:855] (1/3) Using DDP
|
| 6 |
+
2022-08-08 11:57:42,267 INFO [asr_datamodule.py:409] (1/3) About to get train-clean-100 cuts
|
| 7 |
+
2022-08-08 11:57:42,318 INFO [asr_datamodule.py:416] (1/3) About to get train-clean-360 cuts
|
| 8 |
+
2022-08-08 11:57:42,339 INFO [asr_datamodule.py:423] (1/3) About to get train-other-500 cuts
|
| 9 |
+
2022-08-08 11:57:42,362 INFO [asr_datamodule.py:225] (1/3) Enable MUSAN
|
| 10 |
+
2022-08-08 11:57:42,363 INFO [asr_datamodule.py:226] (1/3) About to get Musan cuts
|
| 11 |
+
2022-08-08 11:57:44,410 INFO [asr_datamodule.py:254] (1/3) Enable SpecAugment
|
| 12 |
+
2022-08-08 11:57:44,410 INFO [asr_datamodule.py:255] (1/3) Time warp factor: 80
|
| 13 |
+
2022-08-08 11:57:44,410 INFO [asr_datamodule.py:267] (1/3) Num frame mask: 10
|
| 14 |
+
2022-08-08 11:57:44,411 INFO [asr_datamodule.py:280] (1/3) About to create train dataset
|
| 15 |
+
2022-08-08 11:57:44,411 INFO [asr_datamodule.py:309] (1/3) Using DynamicBucketingSampler.
|
| 16 |
+
2022-08-08 12:00:05,542 INFO [asr_datamodule.py:324] (1/3) About to create train dataloader
|
| 17 |
+
2022-08-08 12:00:05,543 INFO [asr_datamodule.py:430] (1/3) About to get dev-clean cuts
|
| 18 |
+
2022-08-08 12:00:05,550 INFO [asr_datamodule.py:437] (1/3) About to get dev-other cuts
|
| 19 |
+
2022-08-08 12:00:05,552 INFO [asr_datamodule.py:355] (1/3) About to create dev dataset
|
| 20 |
+
2022-08-08 12:00:05,876 INFO [asr_datamodule.py:374] (1/3) About to create dev dataloader
|
| 21 |
+
2022-08-08 12:03:06,080 INFO [train.py:718] (1/3) Epoch 1, batch 0, loss[loss=3.532, over 4610.00 frames; over 0.00 utterances.], tot_loss[loss=3.532, over 4610.00 frames; over 0.00 utterances.], batch size: 13, lr: 3.00e-03
|
| 22 |
+
2022-08-08 12:03:06,235 INFO [distributed.py:948] (1/3) Reducer buckets have been rebuilt in this iteration.
|
| 23 |
+
2022-08-08 12:04:07,322 INFO [train.py:718] (1/3) Epoch 1, batch 100, loss[loss=0.8491, over 3466.00 frames; over 0.00 utterances.], tot_loss[loss=1.039, over 371124.22 frames; over 0.00 utterances.], batch size: 55, lr: 3.00e-03
|
| 24 |
+
2022-08-08 12:05:07,013 INFO [train.py:718] (1/3) Epoch 1, batch 200, loss[loss=0.7121, over 4904.00 frames; over 0.00 utterances.], tot_loss[loss=0.8526, over 594715.52 frames; over 0.00 utterances.], batch size: 16, lr: 3.00e-03
|
| 25 |
+
2022-08-08 12:06:05,038 INFO [train.py:718] (1/3) Epoch 1, batch 300, loss[loss=0.6198, over 4925.00 frames; over 0.00 utterances.], tot_loss[loss=0.7411, over 730064.95 frames; over 0.00 utterances.], batch size: 13, lr: 3.00e-03
|
| 26 |
+
2022-08-08 12:07:03,653 INFO [train.py:718] (1/3) Epoch 1, batch 400, loss[loss=0.5577, over 4596.00 frames; over 0.00 utterances.], tot_loss[loss=0.6726, over 809390.92 frames; over 0.00 utterances.], batch size: 12, lr: 3.00e-03
|
| 27 |
+
2022-08-08 12:08:02,704 INFO [train.py:718] (1/3) Epoch 1, batch 500, loss[loss=0.5752, over 4647.00 frames; over 0.00 utterances.], tot_loss[loss=0.6269, over 855875.57 frames; over 0.00 utterances.], batch size: 14, lr: 2.99e-03
|
| 28 |
+
2022-08-08 12:09:01,212 INFO [train.py:718] (1/3) Epoch 1, batch 600, loss[loss=0.4919, over 4873.00 frames; over 0.00 utterances.], tot_loss[loss=0.5936, over 883451.95 frames; over 0.00 utterances.], batch size: 14, lr: 2.99e-03
|
| 29 |
+
2022-08-08 12:09:58,147 INFO [train.py:718] (1/3) Epoch 1, batch 700, loss[loss=0.5183, over 4179.00 frames; over 0.00 utterances.], tot_loss[loss=0.5682, over 896695.63 frames; over 0.00 utterances.], batch size: 38, lr: 2.99e-03
|
| 30 |
+
2022-08-08 12:13:45,527 INFO [train.py:718] (1/3) Epoch 2, batch 0, loss[loss=0.5091, over 4668.00 frames; over 0.00 utterances.], tot_loss[loss=0.5091, over 4668.00 frames; over 0.00 utterances.], batch size: 12, lr: 2.96e-03
|
| 31 |
+
2022-08-08 12:14:42,206 INFO [train.py:718] (1/3) Epoch 2, batch 100, loss[loss=0.6025, over 3607.00 frames; over 0.00 utterances.], tot_loss[loss=0.495, over 369326.08 frames; over 0.00 utterances.], batch size: 55, lr: 2.96e-03
|
| 32 |
+
2022-08-08 12:15:36,070 INFO [train.py:718] (1/3) Epoch 2, batch 200, loss[loss=0.4696, over 4609.00 frames; over 0.00 utterances.], tot_loss[loss=0.4927, over 588345.73 frames; over 0.00 utterances.], batch size: 12, lr: 2.95e-03
|
| 33 |
+
2022-08-08 12:16:29,348 INFO [train.py:718] (1/3) Epoch 2, batch 300, loss[loss=0.4679, over 4878.00 frames; over 0.00 utterances.], tot_loss[loss=0.4888, over 725261.29 frames; over 0.00 utterances.], batch size: 11, lr: 2.95e-03
|
| 34 |
+
2022-08-08 12:17:17,409 INFO [train.py:718] (1/3) Epoch 2, batch 400, loss[loss=0.4821, over 4749.00 frames; over 0.00 utterances.], tot_loss[loss=0.4835, over 806972.02 frames; over 0.00 utterances.], batch size: 12, lr: 2.94e-03
|
| 35 |
+
2022-08-08 12:18:04,575 INFO [train.py:718] (1/3) Epoch 2, batch 500, loss[loss=0.4728, over 4921.00 frames; over 0.00 utterances.], tot_loss[loss=0.4804, over 853853.37 frames; over 0.00 utterances.], batch size: 13, lr: 2.93e-03
|
| 36 |
+
2022-08-08 12:18:51,652 INFO [train.py:718] (1/3) Epoch 2, batch 600, loss[loss=0.5185, over 4927.00 frames; over 0.00 utterances.], tot_loss[loss=0.4768, over 886779.05 frames; over 0.00 utterances.], batch size: 13, lr: 2.93e-03
|
| 37 |
+
2022-08-08 12:19:42,987 INFO [train.py:718] (1/3) Epoch 2, batch 700, loss[loss=0.4512, over 4933.00 frames; over 0.00 utterances.], tot_loss[loss=0.4766, over 903855.32 frames; over 0.00 utterances.], batch size: 12, lr: 2.92e-03
|
| 38 |
+
2022-08-08 12:23:25,733 INFO [train.py:718] (1/3) Epoch 3, batch 0, loss[loss=0.4455, over 4801.00 frames; over 0.00 utterances.], tot_loss[loss=0.4455, over 4801.00 frames; over 0.00 utterances.], batch size: 11, lr: 2.86e-03
|
| 39 |
+
2022-08-08 12:24:24,647 INFO [train.py:718] (1/3) Epoch 3, batch 100, loss[loss=0.4917, over 4679.00 frames; over 0.00 utterances.], tot_loss[loss=0.4426, over 370462.15 frames; over 0.00 utterances.], batch size: 12, lr: 2.85e-03
|
| 40 |
+
2022-08-08 12:25:22,418 INFO [train.py:718] (1/3) Epoch 3, batch 200, loss[loss=0.4172, over 4741.00 frames; over 0.00 utterances.], tot_loss[loss=0.4425, over 593484.39 frames; over 0.00 utterances.], batch size: 11, lr: 2.84e-03
|
| 41 |
+
2022-08-08 12:26:16,407 INFO [train.py:718] (1/3) Epoch 3, batch 300, loss[loss=0.445, over 4893.00 frames; over 0.00 utterances.], tot_loss[loss=0.4456, over 729145.21 frames; over 0.00 utterances.], batch size: 16, lr: 2.83e-03
|
| 42 |
+
2022-08-08 12:27:11,963 INFO [train.py:718] (1/3) Epoch 3, batch 400, loss[loss=0.3559, over 4865.00 frames; over 0.00 utterances.], tot_loss[loss=0.4394, over 811220.44 frames; over 0.00 utterances.], batch size: 13, lr: 2.82e-03
|
| 43 |
+
2022-08-08 12:28:07,046 INFO [train.py:718] (1/3) Epoch 3, batch 500, loss[loss=0.3872, over 4539.00 frames; over 0.00 utterances.], tot_loss[loss=0.4397, over 860300.02 frames; over 0.00 utterances.], batch size: 20, lr: 2.81e-03
|
| 44 |
+
2022-08-08 12:28:57,342 INFO [train.py:718] (1/3) Epoch 3, batch 600, loss[loss=0.4714, over 4589.00 frames; over 0.00 utterances.], tot_loss[loss=0.439, over 889326.77 frames; over 0.00 utterances.], batch size: 20, lr: 2.80e-03
|
| 45 |
+
2022-08-08 12:29:50,235 INFO [train.py:718] (1/3) Epoch 3, batch 700, loss[loss=0.4935, over 4825.00 frames; over 0.00 utterances.], tot_loss[loss=0.4424, over 905503.87 frames; over 0.00 utterances.], batch size: 15, lr: 2.79e-03
|
| 46 |
+
2022-08-08 12:33:28,617 INFO [train.py:718] (1/3) Epoch 4, batch 0, loss[loss=0.3909, over 4679.00 frames; over 0.00 utterances.], tot_loss[loss=0.3909, over 4679.00 frames; over 0.00 utterances.], batch size: 12, lr: 2.70e-03
|
| 47 |
+
2022-08-08 12:34:26,049 INFO [train.py:718] (1/3) Epoch 4, batch 100, loss[loss=0.4315, over 4911.00 frames; over 0.00 utterances.], tot_loss[loss=0.4196, over 369630.96 frames; over 0.00 utterances.], batch size: 16, lr: 2.69e-03
|
| 48 |
+
2022-08-08 12:35:21,129 INFO [train.py:718] (1/3) Epoch 4, batch 200, loss[loss=0.3723, over 4628.00 frames; over 0.00 utterances.], tot_loss[loss=0.4228, over 591644.49 frames; over 0.00 utterances.], batch size: 11, lr: 2.68e-03
|
| 49 |
+
2022-08-08 12:36:12,972 INFO [train.py:718] (1/3) Epoch 4, batch 300, loss[loss=0.4609, over 4819.00 frames; over 0.00 utterances.], tot_loss[loss=0.425, over 727231.47 frames; over 0.00 utterances.], batch size: 15, lr: 2.67e-03
|
| 50 |
+
2022-08-08 12:37:09,705 INFO [train.py:718] (1/3) Epoch 4, batch 400, loss[loss=0.4306, over 4359.00 frames; over 0.00 utterances.], tot_loss[loss=0.4277, over 805898.26 frames; over 0.00 utterances.], batch size: 23, lr: 2.66e-03
|
| 51 |
+
2022-08-08 12:38:00,657 INFO [train.py:718] (1/3) Epoch 4, batch 500, loss[loss=0.439, over 4568.00 frames; over 0.00 utterances.], tot_loss[loss=0.4249, over 856894.68 frames; over 0.00 utterances.], batch size: 10, lr: 2.65e-03
|
| 52 |
+
2022-08-08 12:38:57,939 INFO [train.py:718] (1/3) Epoch 4, batch 600, loss[loss=0.3632, over 4614.00 frames; over 0.00 utterances.], tot_loss[loss=0.4293, over 882385.99 frames; over 0.00 utterances.], batch size: 13, lr: 2.64e-03
|
| 53 |
+
2022-08-08 12:39:46,641 INFO [train.py:718] (1/3) Epoch 4, batch 700, loss[loss=0.4291, over 4506.00 frames; over 0.00 utterances.], tot_loss[loss=0.4276, over 901202.46 frames; over 0.00 utterances.], batch size: 20, lr: 2.63e-03
|
| 54 |
+
2022-08-08 12:43:24,560 INFO [train.py:718] (1/3) Epoch 5, batch 0, loss[loss=0.4785, over 4642.00 frames; over 0.00 utterances.], tot_loss[loss=0.4785, over 4642.00 frames; over 0.00 utterances.], batch size: 14, lr: 2.52e-03
|
| 55 |
+
2022-08-08 12:44:22,007 INFO [train.py:718] (1/3) Epoch 5, batch 100, loss[loss=0.5254, over 4803.00 frames; over 0.00 utterances.], tot_loss[loss=0.4059, over 371742.20 frames; over 0.00 utterances.], batch size: 15, lr: 2.51e-03
|
| 56 |
+
2022-08-08 12:45:13,940 INFO [train.py:718] (1/3) Epoch 5, batch 200, loss[loss=0.4181, over 4611.00 frames; over 0.00 utterances.], tot_loss[loss=0.4106, over 591975.17 frames; over 0.00 utterances.], batch size: 12, lr: 2.50e-03
|
| 57 |
+
2022-08-08 12:46:05,146 INFO [train.py:718] (1/3) Epoch 5, batch 300, loss[loss=0.4941, over 4812.00 frames; over 0.00 utterances.], tot_loss[loss=0.4133, over 725705.00 frames; over 0.00 utterances.], batch size: 15, lr: 2.49e-03
|
| 58 |
+
2022-08-08 12:47:03,457 INFO [train.py:718] (1/3) Epoch 5, batch 400, loss[loss=0.4415, over 3600.00 frames; over 0.00 utterances.], tot_loss[loss=0.4126, over 802192.92 frames; over 0.00 utterances.], batch size: 55, lr: 2.48e-03
|
| 59 |
+
2022-08-08 12:47:56,601 INFO [train.py:718] (1/3) Epoch 5, batch 500, loss[loss=0.4981, over 4582.00 frames; over 0.00 utterances.], tot_loss[loss=0.4138, over 851326.57 frames; over 0.00 utterances.], batch size: 11, lr: 2.47e-03
|
| 60 |
+
2022-08-08 12:48:47,971 INFO [train.py:718] (1/3) Epoch 5, batch 600, loss[loss=0.3675, over 4606.00 frames; over 0.00 utterances.], tot_loss[loss=0.4126, over 884229.65 frames; over 0.00 utterances.], batch size: 13, lr: 2.46e-03
|
| 61 |
+
2022-08-08 12:49:40,854 INFO [train.py:718] (1/3) Epoch 5, batch 700, loss[loss=0.4025, over 4917.00 frames; over 0.00 utterances.], tot_loss[loss=0.4154, over 903964.43 frames; over 0.00 utterances.], batch size: 13, lr: 2.44e-03
|
| 62 |
+
2022-08-08 12:53:11,633 INFO [train.py:718] (1/3) Epoch 6, batch 0, loss[loss=0.3945, over 4794.00 frames; over 0.00 utterances.], tot_loss[loss=0.3945, over 4794.00 frames; over 0.00 utterances.], batch size: 15, lr: 2.34e-03
|
| 63 |
+
2022-08-08 12:54:08,890 INFO [train.py:718] (1/3) Epoch 6, batch 100, loss[loss=0.3527, over 4794.00 frames; over 0.00 utterances.], tot_loss[loss=0.3886, over 369210.55 frames; over 0.00 utterances.], batch size: 12, lr: 2.33e-03
|
| 64 |
+
2022-08-08 12:55:02,076 INFO [train.py:718] (1/3) Epoch 6, batch 200, loss[loss=0.3977, over 4581.00 frames; over 0.00 utterances.], tot_loss[loss=0.398, over 589448.24 frames; over 0.00 utterances.], batch size: 10, lr: 2.32e-03
|
| 65 |
+
2022-08-08 12:55:53,429 INFO [train.py:718] (1/3) Epoch 6, batch 300, loss[loss=0.4582, over 4799.00 frames; over 0.00 utterances.], tot_loss[loss=0.4041, over 724906.99 frames; over 0.00 utterances.], batch size: 18, lr: 2.31e-03
|
| 66 |
+
2022-08-08 12:56:42,192 INFO [train.py:718] (1/3) Epoch 6, batch 400, loss[loss=0.4434, over 4925.00 frames; over 0.00 utterances.], tot_loss[loss=0.4041, over 800750.64 frames; over 0.00 utterances.], batch size: 13, lr: 2.30e-03
|
| 67 |
+
2022-08-08 12:57:35,165 INFO [train.py:718] (1/3) Epoch 6, batch 500, loss[loss=0.3364, over 4642.00 frames; over 0.00 utterances.], tot_loss[loss=0.4047, over 851039.63 frames; over 0.00 utterances.], batch size: 14, lr: 2.28e-03
|
| 68 |
+
2022-08-08 12:58:33,297 INFO [train.py:718] (1/3) Epoch 6, batch 600, loss[loss=0.4223, over 4681.00 frames; over 0.00 utterances.], tot_loss[loss=0.4073, over 883168.56 frames; over 0.00 utterances.], batch size: 11, lr: 2.27e-03
|
| 69 |
+
2022-08-08 12:59:23,136 INFO [train.py:718] (1/3) Epoch 6, batch 700, loss[loss=0.5367, over 4805.00 frames; over 0.00 utterances.], tot_loss[loss=0.4084, over 900155.76 frames; over 0.00 utterances.], batch size: 11, lr: 2.26e-03
|
| 70 |
+
2022-08-08 13:03:10,380 INFO [train.py:718] (1/3) Epoch 7, batch 0, loss[loss=0.3459, over 4681.00 frames; over 0.00 utterances.], tot_loss[loss=0.3459, over 4681.00 frames; over 0.00 utterances.], batch size: 12, lr: 2.16e-03
|
exp/log/log-train-2022-08-08-11-57-24-2
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2022-08-08 11:57:24,902 INFO [train.py:786] (2/3) Training started
|
| 2 |
+
2022-08-08 11:57:24,902 INFO [train.py:787] (2/3) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'env_info': {'k2-version': '1.17', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '7dcabf85e8bf06984c4abab0400ef1322b5ff3df', 'k2-git-date': 'Tue Aug 2 21:22:39 2022', 'lhotse-version': '1.4.0.dev+git.59318ed.clean', 'torch-version': '1.11.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.3', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '6af45c3-dirty', 'icefall-git-date': 'Tue Aug 2 14:47:09 2022', 'icefall-path': '/thspfs/home/acct-hpc/thspzh-hpc1/git/icefall_dev', 'k2-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/k2-1.17.dev20220805+cuda11.3.torch1.11.0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'dgxs-01.cluster.thspzh.com', 'IP address': '192.168.200.101'}, 'world_size': 3, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 10, 'start_epoch': 1, 'start_batch': 0, 'exp_dir': PosixPath('cnn_ctc/exp_phone'), 'lang_dir': 'data/lang_phone', 'bpe_model': 'data/lang_bpe_500/bpe.model', 'initial_lr': 0.003, 'lr_batches': 5000, 'lr_epochs': 6, 'seed': 42, 'print_diagnostics': False, 'save_every_n': 8000, 'keep_last_k': 20, 'average_period': 100, 'use_fp16': False, 'conv_blocks': 3, 'full_libri': True, 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 200.0, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'input_strategy': 'PrecomputedFeatures'}
|
| 3 |
+
2022-08-08 11:57:25,279 INFO [lexicon.py:176] (2/3) Loading pre-compiled data/lang_phone/Linv.pt
|
| 4 |
+
2022-08-08 11:57:36,661 INFO [train.py:824] (2/3) About to create model
|
| 5 |
+
2022-08-08 11:57:36,692 INFO [train.py:855] (2/3) Using DDP
|
| 6 |
+
2022-08-08 11:57:42,267 INFO [asr_datamodule.py:409] (2/3) About to get train-clean-100 cuts
|
| 7 |
+
2022-08-08 11:57:42,318 INFO [asr_datamodule.py:416] (2/3) About to get train-clean-360 cuts
|
| 8 |
+
2022-08-08 11:57:42,339 INFO [asr_datamodule.py:423] (2/3) About to get train-other-500 cuts
|
| 9 |
+
2022-08-08 11:57:42,363 INFO [asr_datamodule.py:225] (2/3) Enable MUSAN
|
| 10 |
+
2022-08-08 11:57:42,363 INFO [asr_datamodule.py:226] (2/3) About to get Musan cuts
|
| 11 |
+
2022-08-08 11:57:44,439 INFO [asr_datamodule.py:254] (2/3) Enable SpecAugment
|
| 12 |
+
2022-08-08 11:57:44,440 INFO [asr_datamodule.py:255] (2/3) Time warp factor: 80
|
| 13 |
+
2022-08-08 11:57:44,440 INFO [asr_datamodule.py:267] (2/3) Num frame mask: 10
|
| 14 |
+
2022-08-08 11:57:44,440 INFO [asr_datamodule.py:280] (2/3) About to create train dataset
|
| 15 |
+
2022-08-08 11:57:44,440 INFO [asr_datamodule.py:309] (2/3) Using DynamicBucketingSampler.
|
| 16 |
+
2022-08-08 12:00:04,638 INFO [asr_datamodule.py:324] (2/3) About to create train dataloader
|
| 17 |
+
2022-08-08 12:00:04,639 INFO [asr_datamodule.py:430] (2/3) About to get dev-clean cuts
|
| 18 |
+
2022-08-08 12:00:04,642 INFO [asr_datamodule.py:437] (2/3) About to get dev-other cuts
|
| 19 |
+
2022-08-08 12:00:04,647 INFO [asr_datamodule.py:355] (2/3) About to create dev dataset
|
| 20 |
+
2022-08-08 12:00:04,961 INFO [asr_datamodule.py:374] (2/3) About to create dev dataloader
|
| 21 |
+
2022-08-08 12:03:06,080 INFO [train.py:718] (2/3) Epoch 1, batch 0, loss[loss=3.52, over 4627.00 frames; over 0.00 utterances.], tot_loss[loss=3.52, over 4627.00 frames; over 0.00 utterances.], batch size: 11, lr: 3.00e-03
|
| 22 |
+
2022-08-08 12:03:06,237 INFO [distributed.py:948] (2/3) Reducer buckets have been rebuilt in this iteration.
|
| 23 |
+
2022-08-08 12:04:07,321 INFO [train.py:718] (2/3) Epoch 1, batch 100, loss[loss=0.9056, over 4852.00 frames; over 0.00 utterances.], tot_loss[loss=1.038, over 367492.03 frames; over 0.00 utterances.], batch size: 13, lr: 3.00e-03
|
| 24 |
+
2022-08-08 12:05:07,012 INFO [train.py:718] (2/3) Epoch 1, batch 200, loss[loss=0.7441, over 4800.00 frames; over 0.00 utterances.], tot_loss[loss=0.8531, over 588522.14 frames; over 0.00 utterances.], batch size: 15, lr: 3.00e-03
|
| 25 |
+
2022-08-08 12:06:05,037 INFO [train.py:718] (2/3) Epoch 1, batch 300, loss[loss=0.6336, over 4629.00 frames; over 0.00 utterances.], tot_loss[loss=0.7428, over 723969.69 frames; over 0.00 utterances.], batch size: 14, lr: 3.00e-03
|
| 26 |
+
2022-08-08 12:07:03,653 INFO [train.py:718] (2/3) Epoch 1, batch 400, loss[loss=0.5298, over 4870.00 frames; over 0.00 utterances.], tot_loss[loss=0.676, over 806004.55 frames; over 0.00 utterances.], batch size: 12, lr: 3.00e-03
|
| 27 |
+
2022-08-08 12:08:02,702 INFO [train.py:718] (2/3) Epoch 1, batch 500, loss[loss=0.574, over 4864.00 frames; over 0.00 utterances.], tot_loss[loss=0.6295, over 854663.37 frames; over 0.00 utterances.], batch size: 11, lr: 2.99e-03
|
| 28 |
+
2022-08-08 12:09:01,211 INFO [train.py:718] (2/3) Epoch 1, batch 600, loss[loss=0.5943, over 4862.00 frames; over 0.00 utterances.], tot_loss[loss=0.5954, over 880652.22 frames; over 0.00 utterances.], batch size: 13, lr: 2.99e-03
|
| 29 |
+
2022-08-08 12:09:58,146 INFO [train.py:718] (2/3) Epoch 1, batch 700, loss[loss=0.4683, over 4605.00 frames; over 0.00 utterances.], tot_loss[loss=0.5701, over 899329.78 frames; over 0.00 utterances.], batch size: 13, lr: 2.99e-03
|
| 30 |
+
2022-08-08 12:13:45,526 INFO [train.py:718] (2/3) Epoch 2, batch 0, loss[loss=0.5374, over 4811.00 frames; over 0.00 utterances.], tot_loss[loss=0.5374, over 4811.00 frames; over 0.00 utterances.], batch size: 11, lr: 2.96e-03
|
| 31 |
+
2022-08-08 12:14:42,205 INFO [train.py:718] (2/3) Epoch 2, batch 100, loss[loss=0.4851, over 4615.00 frames; over 0.00 utterances.], tot_loss[loss=0.4942, over 369287.45 frames; over 0.00 utterances.], batch size: 13, lr: 2.96e-03
|
| 32 |
+
2022-08-08 12:15:36,069 INFO [train.py:718] (2/3) Epoch 2, batch 200, loss[loss=0.4895, over 4808.00 frames; over 0.00 utterances.], tot_loss[loss=0.4898, over 589882.85 frames; over 0.00 utterances.], batch size: 11, lr: 2.95e-03
|
| 33 |
+
2022-08-08 12:16:29,347 INFO [train.py:718] (2/3) Epoch 2, batch 300, loss[loss=0.5137, over 4607.00 frames; over 0.00 utterances.], tot_loss[loss=0.4829, over 724804.83 frames; over 0.00 utterances.], batch size: 13, lr: 2.95e-03
|
| 34 |
+
2022-08-08 12:17:17,407 INFO [train.py:718] (2/3) Epoch 2, batch 400, loss[loss=0.5288, over 4936.00 frames; over 0.00 utterances.], tot_loss[loss=0.4814, over 809282.73 frames; over 0.00 utterances.], batch size: 12, lr: 2.94e-03
|
| 35 |
+
2022-08-08 12:18:04,574 INFO [train.py:718] (2/3) Epoch 2, batch 500, loss[loss=0.4069, over 4856.00 frames; over 0.00 utterances.], tot_loss[loss=0.4787, over 856924.96 frames; over 0.00 utterances.], batch size: 13, lr: 2.93e-03
|
| 36 |
+
2022-08-08 12:18:51,651 INFO [train.py:718] (2/3) Epoch 2, batch 600, loss[loss=0.4771, over 4864.00 frames; over 0.00 utterances.], tot_loss[loss=0.4796, over 884316.15 frames; over 0.00 utterances.], batch size: 13, lr: 2.93e-03
|
| 37 |
+
2022-08-08 12:19:42,986 INFO [train.py:718] (2/3) Epoch 2, batch 700, loss[loss=0.4944, over 4928.00 frames; over 0.00 utterances.], tot_loss[loss=0.4793, over 900370.17 frames; over 0.00 utterances.], batch size: 12, lr: 2.92e-03
|
| 38 |
+
2022-08-08 12:23:25,732 INFO [train.py:718] (2/3) Epoch 3, batch 0, loss[loss=0.4557, over 4182.00 frames; over 0.00 utterances.], tot_loss[loss=0.4557, over 4182.00 frames; over 0.00 utterances.], batch size: 38, lr: 2.86e-03
|
| 39 |
+
2022-08-08 12:24:24,645 INFO [train.py:718] (2/3) Epoch 3, batch 100, loss[loss=0.4599, over 4587.00 frames; over 0.00 utterances.], tot_loss[loss=0.4461, over 366667.28 frames; over 0.00 utterances.], batch size: 15, lr: 2.85e-03
|
| 40 |
+
2022-08-08 12:25:22,417 INFO [train.py:718] (2/3) Epoch 3, batch 200, loss[loss=0.487, over 4465.00 frames; over 0.00 utterances.], tot_loss[loss=0.4453, over 587677.03 frames; over 0.00 utterances.], batch size: 24, lr: 2.84e-03
|
| 41 |
+
2022-08-08 12:26:16,406 INFO [train.py:718] (2/3) Epoch 3, batch 300, loss[loss=0.3603, over 4579.00 frames; over 0.00 utterances.], tot_loss[loss=0.4421, over 723394.66 frames; over 0.00 utterances.], batch size: 11, lr: 2.83e-03
|
| 42 |
+
2022-08-08 12:27:11,962 INFO [train.py:718] (2/3) Epoch 3, batch 400, loss[loss=0.4463, over 4777.00 frames; over 0.00 utterances.], tot_loss[loss=0.4435, over 805233.43 frames; over 0.00 utterances.], batch size: 13, lr: 2.82e-03
|
| 43 |
+
2022-08-08 12:28:07,045 INFO [train.py:718] (2/3) Epoch 3, batch 500, loss[loss=0.4025, over 4788.00 frames; over 0.00 utterances.], tot_loss[loss=0.4448, over 853254.68 frames; over 0.00 utterances.], batch size: 13, lr: 2.81e-03
|
| 44 |
+
2022-08-08 12:28:57,342 INFO [train.py:718] (2/3) Epoch 3, batch 600, loss[loss=0.3724, over 4677.00 frames; over 0.00 utterances.], tot_loss[loss=0.446, over 883555.83 frames; over 0.00 utterances.], batch size: 11, lr: 2.80e-03
|
| 45 |
+
2022-08-08 12:29:50,234 INFO [train.py:718] (2/3) Epoch 3, batch 700, loss[loss=0.4832, over 4688.00 frames; over 0.00 utterances.], tot_loss[loss=0.4455, over 901478.88 frames; over 0.00 utterances.], batch size: 11, lr: 2.79e-03
|
| 46 |
+
2022-08-08 12:33:28,616 INFO [train.py:718] (2/3) Epoch 4, batch 0, loss[loss=0.3924, over 4614.00 frames; over 0.00 utterances.], tot_loss[loss=0.3924, over 4614.00 frames; over 0.00 utterances.], batch size: 12, lr: 2.70e-03
|
| 47 |
+
2022-08-08 12:34:26,047 INFO [train.py:718] (2/3) Epoch 4, batch 100, loss[loss=0.4796, over 4693.00 frames; over 0.00 utterances.], tot_loss[loss=0.4225, over 370707.55 frames; over 0.00 utterances.], batch size: 11, lr: 2.69e-03
|
| 48 |
+
2022-08-08 12:35:21,128 INFO [train.py:718] (2/3) Epoch 4, batch 200, loss[loss=0.4826, over 4675.00 frames; over 0.00 utterances.], tot_loss[loss=0.4301, over 589510.11 frames; over 0.00 utterances.], batch size: 12, lr: 2.68e-03
|
| 49 |
+
2022-08-08 12:36:12,971 INFO [train.py:718] (2/3) Epoch 4, batch 300, loss[loss=0.4491, over 4617.00 frames; over 0.00 utterances.], tot_loss[loss=0.4269, over 728299.09 frames; over 0.00 utterances.], batch size: 13, lr: 2.67e-03
|
| 50 |
+
2022-08-08 12:37:09,704 INFO [train.py:718] (2/3) Epoch 4, batch 400, loss[loss=0.4789, over 3548.00 frames; over 0.00 utterances.], tot_loss[loss=0.4312, over 804370.46 frames; over 0.00 utterances.], batch size: 56, lr: 2.66e-03
|
| 51 |
+
2022-08-08 12:38:00,655 INFO [train.py:718] (2/3) Epoch 4, batch 500, loss[loss=0.3839, over 4705.00 frames; over 0.00 utterances.], tot_loss[loss=0.4281, over 854746.27 frames; over 0.00 utterances.], batch size: 13, lr: 2.65e-03
|
| 52 |
+
2022-08-08 12:38:57,939 INFO [train.py:718] (2/3) Epoch 4, batch 600, loss[loss=0.4514, over 4856.00 frames; over 0.00 utterances.], tot_loss[loss=0.4293, over 886213.41 frames; over 0.00 utterances.], batch size: 14, lr: 2.64e-03
|
| 53 |
+
2022-08-08 12:39:46,641 INFO [train.py:718] (2/3) Epoch 4, batch 700, loss[loss=0.3684, over 4933.00 frames; over 0.00 utterances.], tot_loss[loss=0.4292, over 901772.90 frames; over 0.00 utterances.], batch size: 12, lr: 2.63e-03
|
| 54 |
+
2022-08-08 12:43:24,558 INFO [train.py:718] (2/3) Epoch 5, batch 0, loss[loss=0.3807, over 4351.00 frames; over 0.00 utterances.], tot_loss[loss=0.3807, over 4351.00 frames; over 0.00 utterances.], batch size: 24, lr: 2.52e-03
|
| 55 |
+
2022-08-08 12:44:22,006 INFO [train.py:718] (2/3) Epoch 5, batch 100, loss[loss=0.3665, over 4813.00 frames; over 0.00 utterances.], tot_loss[loss=0.4063, over 370090.67 frames; over 0.00 utterances.], batch size: 12, lr: 2.51e-03
|
| 56 |
+
2022-08-08 12:45:13,939 INFO [train.py:718] (2/3) Epoch 5, batch 200, loss[loss=0.3861, over 4789.00 frames; over 0.00 utterances.], tot_loss[loss=0.401, over 594722.26 frames; over 0.00 utterances.], batch size: 13, lr: 2.50e-03
|
| 57 |
+
2022-08-08 12:46:05,145 INFO [train.py:718] (2/3) Epoch 5, batch 300, loss[loss=0.4284, over 4795.00 frames; over 0.00 utterances.], tot_loss[loss=0.4053, over 727093.69 frames; over 0.00 utterances.], batch size: 15, lr: 2.49e-03
|
| 58 |
+
2022-08-08 12:47:03,455 INFO [train.py:718] (2/3) Epoch 5, batch 400, loss[loss=0.4859, over 4874.00 frames; over 0.00 utterances.], tot_loss[loss=0.4102, over 809046.13 frames; over 0.00 utterances.], batch size: 14, lr: 2.48e-03
|
| 59 |
+
2022-08-08 12:47:56,600 INFO [train.py:718] (2/3) Epoch 5, batch 500, loss[loss=0.3605, over 4584.00 frames; over 0.00 utterances.], tot_loss[loss=0.4108, over 859369.39 frames; over 0.00 utterances.], batch size: 11, lr: 2.47e-03
|
| 60 |
+
2022-08-08 12:48:47,970 INFO [train.py:718] (2/3) Epoch 5, batch 600, loss[loss=0.3857, over 4691.00 frames; over 0.00 utterances.], tot_loss[loss=0.4114, over 887384.66 frames; over 0.00 utterances.], batch size: 11, lr: 2.46e-03
|
| 61 |
+
2022-08-08 12:49:40,853 INFO [train.py:718] (2/3) Epoch 5, batch 700, loss[loss=0.3836, over 4802.00 frames; over 0.00 utterances.], tot_loss[loss=0.414, over 905360.41 frames; over 0.00 utterances.], batch size: 12, lr: 2.44e-03
|
| 62 |
+
2022-08-08 12:53:11,632 INFO [train.py:718] (2/3) Epoch 6, batch 0, loss[loss=0.4213, over 4583.00 frames; over 0.00 utterances.], tot_loss[loss=0.4213, over 4583.00 frames; over 0.00 utterances.], batch size: 11, lr: 2.34e-03
|
| 63 |
+
2022-08-08 12:54:08,888 INFO [train.py:718] (2/3) Epoch 6, batch 100, loss[loss=0.3298, over 4866.00 frames; over 0.00 utterances.], tot_loss[loss=0.3945, over 369966.91 frames; over 0.00 utterances.], batch size: 12, lr: 2.33e-03
|
| 64 |
+
2022-08-08 12:55:02,075 INFO [train.py:718] (2/3) Epoch 6, batch 200, loss[loss=0.3972, over 4657.00 frames; over 0.00 utterances.], tot_loss[loss=0.3973, over 589796.87 frames; over 0.00 utterances.], batch size: 14, lr: 2.32e-03
|
| 65 |
+
2022-08-08 12:55:53,429 INFO [train.py:718] (2/3) Epoch 6, batch 300, loss[loss=0.4018, over 4874.00 frames; over 0.00 utterances.], tot_loss[loss=0.4003, over 724293.99 frames; over 0.00 utterances.], batch size: 12, lr: 2.31e-03
|
| 66 |
+
2022-08-08 12:56:42,192 INFO [train.py:718] (2/3) Epoch 6, batch 400, loss[loss=0.3538, over 4617.00 frames; over 0.00 utterances.], tot_loss[loss=0.402, over 805022.54 frames; over 0.00 utterances.], batch size: 13, lr: 2.30e-03
|
| 67 |
+
2022-08-08 12:57:35,164 INFO [train.py:718] (2/3) Epoch 6, batch 500, loss[loss=0.498, over 3566.00 frames; over 0.00 utterances.], tot_loss[loss=0.4042, over 854035.70 frames; over 0.00 utterances.], batch size: 56, lr: 2.28e-03
|
| 68 |
+
2022-08-08 12:58:33,295 INFO [train.py:718] (2/3) Epoch 6, batch 600, loss[loss=0.4679, over 4326.00 frames; over 0.00 utterances.], tot_loss[loss=0.4073, over 881586.02 frames; over 0.00 utterances.], batch size: 29, lr: 2.27e-03
|
| 69 |
+
2022-08-08 12:59:23,135 INFO [train.py:718] (2/3) Epoch 6, batch 700, loss[loss=0.356, over 4801.00 frames; over 0.00 utterances.], tot_loss[loss=0.4066, over 903842.28 frames; over 0.00 utterances.], batch size: 11, lr: 2.26e-03
|
| 70 |
+
2022-08-08 13:03:10,379 INFO [train.py:718] (2/3) Epoch 7, batch 0, loss[loss=0.3448, over 4742.00 frames; over 0.00 utterances.], tot_loss[loss=0.3448, over 4742.00 frames; over 0.00 utterances.], batch size: 11, lr: 2.16e-03
|
exp/log/log-train-2022-08-08-14-49-56-0
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
exp/log/log-train-2022-08-08-14-49-56-1
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
exp/log/log-train-2022-08-08-14-49-56-2
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
exp/log/log-train-2022-08-09-16-45-14-0
ADDED
|
@@ -0,0 +1,208 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2022-08-09 16:45:14,173 INFO [train.py:805] (0/3) Training started
|
| 2 |
+
2022-08-09 16:45:14,173 INFO [train.py:806] (0/3) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'env_info': {'k2-version': '1.17', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '7dcabf85e8bf06984c4abab0400ef1322b5ff3df', 'k2-git-date': 'Tue Aug 2 21:22:39 2022', 'lhotse-version': '1.4.0.dev+git.59318ed.clean', 'torch-version': '1.11.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.3', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '6af45c3-dirty', 'icefall-git-date': 'Tue Aug 2 14:47:09 2022', 'icefall-path': '/thspfs/home/acct-hpc/thspzh-hpc1/git/icefall_dev', 'k2-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/k2-1.17.dev20220805+cuda11.3.torch1.11.0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'dgxs-01.cluster.thspzh.com', 'IP address': '192.168.200.101'}, 'world_size': 3, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 20, 'start_epoch': 11, 'start_batch': 0, 'exp_dir': PosixPath('cnn_ctc/exp_phone'), 'lang_dir': 'data/lang_phone', 'bpe_model': 'data/lang_bpe_500/bpe.model', 'initial_lr': 0.003, 'lr_batches': 5000, 'lr_epochs': 6, 'seed': 42, 'print_diagnostics': False, 'save_every_n': 8000, 'keep_last_k': 20, 'average_period': 100, 'use_fp16': False, 'first_out_channels': 8, 'conv_blocks': 3, 'rnn_dim': 0, 'full_libri': True, 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 600, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'input_strategy': 'PrecomputedFeatures'}
|
| 3 |
+
2022-08-09 16:45:14,474 INFO [lexicon.py:176] (0/3) Loading pre-compiled data/lang_phone/Linv.pt
|
| 4 |
+
2022-08-09 16:45:28,309 INFO [train.py:843] (0/3) About to create model
|
| 5 |
+
2022-08-09 16:45:28,333 INFO [train.py:855] (0/3) Number of model parameters: 1766904
|
| 6 |
+
2022-08-09 16:45:36,111 INFO [train.py:859] (0/3) ==========================================================================================
|
| 7 |
+
Layer (type:depth-idx) Output Shape Param #
|
| 8 |
+
==========================================================================================
|
| 9 |
+
MobileNetS4 [1, 25, 72] --
|
| 10 |
+
├─Sequential: 1-1 [1, 32, 25, 20] --
|
| 11 |
+
│ └─Conv2d: 2-1 [1, 8, 100, 80] 80
|
| 12 |
+
│ └─BatchNorm2d: 2-2 [1, 8, 100, 80] 16
|
| 13 |
+
│ └─ReLU6: 2-3 [1, 8, 100, 80] --
|
| 14 |
+
│ └─Conv2d: 2-4 [1, 16, 50, 40] 1,168
|
| 15 |
+
│ └─BatchNorm2d: 2-5 [1, 16, 50, 40] 32
|
| 16 |
+
│ └─ReLU6: 2-6 [1, 16, 50, 40] --
|
| 17 |
+
│ └─Conv2d: 2-7 [1, 32, 25, 20] 4,640
|
| 18 |
+
│ └─BatchNorm2d: 2-8 [1, 32, 25, 20] 64
|
| 19 |
+
│ └─ReLU6: 2-9 [1, 32, 25, 20] --
|
| 20 |
+
├─Sequential: 1-2 [1, 256, 25, 3] --
|
| 21 |
+
│ └─Bottleneck: 2-10 [1, 64, 25, 10] --
|
| 22 |
+
│ │ └─Sequential: 3-1 [1, 64, 25, 10] 14,400
|
| 23 |
+
│ └─Bottleneck: 2-11 [1, 64, 25, 10] --
|
| 24 |
+
│ │ └─Sequential: 3-2 [1, 64, 25, 10] 36,800
|
| 25 |
+
│ └─Bottleneck: 2-12 [1, 64, 25, 10] --
|
| 26 |
+
│ │ └─Sequential: 3-3 [1, 64, 25, 10] 36,800
|
| 27 |
+
│ └─Bottleneck: 2-13 [1, 128, 25, 5] --
|
| 28 |
+
│ │ └─Sequential: 3-4 [1, 128, 25, 5] 53,376
|
| 29 |
+
│ └─Bottleneck: 2-14 [1, 128, 25, 5] --
|
| 30 |
+
│ │ └─Sequential: 3-5 [1, 128, 25, 5] 139,136
|
| 31 |
+
│ └─Bottleneck: 2-15 [1, 128, 25, 5] --
|
| 32 |
+
│ │ └─Sequential: 3-6 [1, 128, 25, 5] 139,136
|
| 33 |
+
│ └─Bottleneck: 2-16 [1, 256, 25, 3] --
|
| 34 |
+
│ │ └─Sequential: 3-7 [1, 256, 25, 3] 205,056
|
| 35 |
+
│ └─Bottleneck: 2-17 [1, 256, 25, 3] --
|
| 36 |
+
│ │ └─Sequential: 3-8 [1, 256, 25, 3] 540,416
|
| 37 |
+
│ └─Bottleneck: 2-18 [1, 256, 25, 3] --
|
| 38 |
+
│ │ └─Sequential: 3-9 [1, 256, 25, 3] 540,416
|
| 39 |
+
├─Linear: 1-3 [1, 25, 72] 55,368
|
| 40 |
+
==========================================================================================
|
| 41 |
+
Total params: 1,766,904
|
| 42 |
+
Trainable params: 1,766,904
|
| 43 |
+
Non-trainable params: 0
|
| 44 |
+
Total mult-adds (M): 169.35
|
| 45 |
+
==========================================================================================
|
| 46 |
+
Input size (MB): 0.03
|
| 47 |
+
Forward/backward pass size (MB): 22.08
|
| 48 |
+
Params size (MB): 7.07
|
| 49 |
+
Estimated Total Size (MB): 29.18
|
| 50 |
+
==========================================================================================
|
| 51 |
+
2022-08-09 16:45:36,158 INFO [checkpoint.py:112] (0/3) Loading checkpoint from cnn_ctc/exp_phone/epoch-10.pt
|
| 52 |
+
2022-08-09 16:45:37,272 INFO [checkpoint.py:131] (0/3) Loading averaged model
|
| 53 |
+
2022-08-09 16:45:37,284 INFO [train.py:876] (0/3) Using DDP
|
| 54 |
+
2022-08-09 16:45:37,468 INFO [train.py:884] (0/3) Loading optimizer state dict
|
| 55 |
+
2022-08-09 16:45:37,518 INFO [train.py:892] (0/3) Loading scheduler state dict
|
| 56 |
+
2022-08-09 16:45:37,518 INFO [asr_datamodule.py:409] (0/3) About to get train-clean-100 cuts
|
| 57 |
+
2022-08-09 16:45:37,618 INFO [asr_datamodule.py:416] (0/3) About to get train-clean-360 cuts
|
| 58 |
+
2022-08-09 16:45:37,630 INFO [asr_datamodule.py:423] (0/3) About to get train-other-500 cuts
|
| 59 |
+
2022-08-09 16:45:37,724 INFO [asr_datamodule.py:225] (0/3) Enable MUSAN
|
| 60 |
+
2022-08-09 16:45:37,724 INFO [asr_datamodule.py:226] (0/3) About to get Musan cuts
|
| 61 |
+
2022-08-09 16:45:39,834 INFO [asr_datamodule.py:254] (0/3) Enable SpecAugment
|
| 62 |
+
2022-08-09 16:45:39,834 INFO [asr_datamodule.py:255] (0/3) Time warp factor: 80
|
| 63 |
+
2022-08-09 16:45:39,834 INFO [asr_datamodule.py:267] (0/3) Num frame mask: 10
|
| 64 |
+
2022-08-09 16:45:39,834 INFO [asr_datamodule.py:280] (0/3) About to create train dataset
|
| 65 |
+
2022-08-09 16:45:39,834 INFO [asr_datamodule.py:309] (0/3) Using DynamicBucketingSampler.
|
| 66 |
+
2022-08-09 16:45:42,414 INFO [asr_datamodule.py:324] (0/3) About to create train dataloader
|
| 67 |
+
2022-08-09 16:45:42,414 INFO [asr_datamodule.py:430] (0/3) About to get dev-clean cuts
|
| 68 |
+
2022-08-09 16:45:42,419 INFO [asr_datamodule.py:437] (0/3) About to get dev-other cuts
|
| 69 |
+
2022-08-09 16:45:42,420 INFO [asr_datamodule.py:355] (0/3) About to create dev dataset
|
| 70 |
+
2022-08-09 16:45:42,726 INFO [asr_datamodule.py:374] (0/3) About to create dev dataloader
|
| 71 |
+
2022-08-09 16:45:42,727 INFO [train.py:957] (0/3) Loading grad scaler state dict
|
| 72 |
+
2022-08-09 16:46:43,614 INFO [train.py:737] (0/3) Epoch 11, batch 0, loss[loss=0.3849, over 14559.00 frames; utt_duration=1495 frames, utt_pad_proportion=0.007577, over 39.00 utterances.], tot_loss[loss=0.3849, over 14559.00 frames; utt_duration=1495 frames, utt_pad_proportion=0.007577, over 39.00 utterances.], batch size: 39, lr: 4.33e-04
|
| 73 |
+
2022-08-09 16:46:43,648 INFO [distributed.py:948] (0/3) Reducer buckets have been rebuilt in this iteration.
|
| 74 |
+
2022-08-09 16:49:50,756 INFO [train.py:737] (0/3) Epoch 11, batch 100, loss[loss=0.3559, over 14597.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005447, over 41.00 utterances.], tot_loss[loss=0.3769, over 1148378.26 frames; utt_duration=1325 frames, utt_pad_proportion=0.03351, over 3471.16 utterances.], batch size: 41, lr: 4.32e-04
|
| 75 |
+
2022-08-09 16:53:45,380 INFO [train.py:737] (0/3) Epoch 11, batch 200, loss[loss=0.3462, over 14752.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.005102, over 43.00 utterances.], tot_loss[loss=0.3717, over 1831109.13 frames; utt_duration=1331 frames, utt_pad_proportion=0.03767, over 5510.33 utterances.], batch size: 43, lr: 4.32e-04
|
| 76 |
+
2022-08-09 16:57:48,954 INFO [train.py:737] (0/3) Epoch 11, batch 300, loss[loss=0.3407, over 14579.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005892, over 40.00 utterances.], tot_loss[loss=0.3695, over 2232120.17 frames; utt_duration=1291 frames, utt_pad_proportion=0.04968, over 6922.76 utterances.], batch size: 40, lr: 4.32e-04
|
| 77 |
+
2022-08-09 17:01:14,227 INFO [train.py:737] (0/3) Epoch 11, batch 400, loss[loss=0.3201, over 14564.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006982, over 40.00 utterances.], tot_loss[loss=0.3613, over 2490250.99 frames; utt_duration=1331 frames, utt_pad_proportion=0.04474, over 7494.23 utterances.], batch size: 40, lr: 4.32e-04
|
| 78 |
+
2022-08-09 17:06:23,850 INFO [train.py:737] (0/3) Epoch 11, batch 500, loss[loss=0.3716, over 14637.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.008021, over 44.00 utterances.], tot_loss[loss=0.3606, over 2649111.44 frames; utt_duration=1334 frames, utt_pad_proportion=0.038, over 7954.84 utterances.], batch size: 44, lr: 4.32e-04
|
| 79 |
+
2022-08-09 17:24:40,904 INFO [train.py:737] (0/3) Epoch 11, batch 600, loss[loss=0.2936, over 14480.00 frames; utt_duration=1757 frames, utt_pad_proportion=0.009246, over 33.00 utterances.], tot_loss[loss=0.362, over 2744132.61 frames; utt_duration=1323 frames, utt_pad_proportion=0.03649, over 8307.56 utterances.], batch size: 33, lr: 4.31e-04
|
| 80 |
+
2022-08-09 17:31:07,970 INFO [train.py:737] (0/3) Epoch 11, batch 700, loss[loss=0.2926, over 14047.00 frames; utt_duration=1814 frames, utt_pad_proportion=0.03353, over 31.00 utterances.], tot_loss[loss=0.3678, over 2797513.48 frames; utt_duration=1313 frames, utt_pad_proportion=0.0376, over 8529.25 utterances.], batch size: 31, lr: 4.31e-04
|
| 81 |
+
2022-08-09 17:38:02,125 INFO [train.py:737] (0/3) Epoch 11, batch 800, loss[loss=0.3416, over 14386.00 frames; utt_duration=1087 frames, utt_pad_proportion=0.03449, over 53.00 utterances.], tot_loss[loss=0.3678, over 2834051.40 frames; utt_duration=1322 frames, utt_pad_proportion=0.03367, over 8582.94 utterances.], batch size: 53, lr: 4.31e-04
|
| 82 |
+
2022-08-09 17:51:06,850 INFO [train.py:737] (0/3) Epoch 11, batch 900, loss[loss=0.4119, over 14497.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.0174, over 48.00 utterances.], tot_loss[loss=0.3708, over 2850796.86 frames; utt_duration=1316 frames, utt_pad_proportion=0.03736, over 8674.17 utterances.], batch size: 48, lr: 4.31e-04
|
| 83 |
+
2022-08-09 18:11:34,572 INFO [train.py:737] (0/3) Epoch 11, batch 1000, loss[loss=0.3935, over 14581.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.006178, over 41.00 utterances.], tot_loss[loss=0.3712, over 2870516.86 frames; utt_duration=1326 frames, utt_pad_proportion=0.03205, over 8669.47 utterances.], batch size: 41, lr: 4.31e-04
|
| 84 |
+
2022-08-09 18:30:17,060 INFO [train.py:737] (0/3) Epoch 11, batch 1100, loss[loss=0.3399, over 14790.00 frames; utt_duration=1600 frames, utt_pad_proportion=0.005909, over 37.00 utterances.], tot_loss[loss=0.3717, over 2870389.85 frames; utt_duration=1307 frames, utt_pad_proportion=0.03804, over 8793.02 utterances.], batch size: 37, lr: 4.31e-04
|
| 85 |
+
2022-08-09 18:45:42,693 INFO [train.py:737] (0/3) Epoch 11, batch 1200, loss[loss=0.4166, over 14480.00 frames; utt_duration=1160 frames, utt_pad_proportion=0.02389, over 50.00 utterances.], tot_loss[loss=0.3725, over 2868983.46 frames; utt_duration=1305 frames, utt_pad_proportion=0.04144, over 8805.37 utterances.], batch size: 50, lr: 4.30e-04
|
| 86 |
+
2022-08-09 19:00:31,129 INFO [train.py:737] (0/3) Epoch 11, batch 1300, loss[loss=0.4315, over 14740.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.006147, over 43.00 utterances.], tot_loss[loss=0.3704, over 2861395.51 frames; utt_duration=1287 frames, utt_pad_proportion=0.04813, over 8904.21 utterances.], batch size: 43, lr: 4.30e-04
|
| 87 |
+
2022-08-09 19:12:14,041 INFO [train.py:737] (0/3) Epoch 11, batch 1400, loss[loss=0.3711, over 13942.00 frames; utt_duration=962.8 frames, utt_pad_proportion=0.06432, over 58.00 utterances.], tot_loss[loss=0.3699, over 2874624.84 frames; utt_duration=1311 frames, utt_pad_proportion=0.03979, over 8779.69 utterances.], batch size: 58, lr: 4.30e-04
|
| 88 |
+
2022-08-09 19:24:33,673 INFO [train.py:737] (0/3) Epoch 11, batch 1500, loss[loss=0.376, over 14750.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006376, over 40.00 utterances.], tot_loss[loss=0.3701, over 2876251.57 frames; utt_duration=1311 frames, utt_pad_proportion=0.04009, over 8782.86 utterances.], batch size: 40, lr: 4.30e-04
|
| 89 |
+
2022-08-09 19:36:04,609 INFO [train.py:737] (0/3) Epoch 11, batch 1600, loss[loss=0.3502, over 14081.00 frames; utt_duration=1819 frames, utt_pad_proportion=0.03411, over 31.00 utterances.], tot_loss[loss=0.3729, over 2866043.97 frames; utt_duration=1290 frames, utt_pad_proportion=0.04762, over 8897.56 utterances.], batch size: 31, lr: 4.30e-04
|
| 90 |
+
2022-08-09 19:45:58,873 INFO [train.py:737] (0/3) Epoch 11, batch 1700, loss[loss=0.3236, over 14768.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.00687, over 37.00 utterances.], tot_loss[loss=0.3702, over 2868417.28 frames; utt_duration=1297 frames, utt_pad_proportion=0.04732, over 8853.65 utterances.], batch size: 37, lr: 4.30e-04
|
| 91 |
+
2022-08-09 19:55:10,363 INFO [train.py:737] (0/3) Epoch 11, batch 1800, loss[loss=0.4165, over 14757.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005252, over 41.00 utterances.], tot_loss[loss=0.3708, over 2847090.08 frames; utt_duration=1246 frames, utt_pad_proportion=0.0606, over 9152.05 utterances.], batch size: 41, lr: 4.29e-04
|
| 92 |
+
2022-08-09 20:04:21,776 INFO [train.py:737] (0/3) Epoch 11, batch 1900, loss[loss=0.3482, over 14560.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006237, over 40.00 utterances.], tot_loss[loss=0.3705, over 2867983.37 frames; utt_duration=1285 frames, utt_pad_proportion=0.04847, over 8939.81 utterances.], batch size: 40, lr: 4.29e-04
|
| 93 |
+
2022-08-09 20:14:15,006 INFO [train.py:737] (0/3) Epoch 11, batch 2000, loss[loss=0.3509, over 14725.00 frames; utt_duration=1371 frames, utt_pad_proportion=0.007157, over 43.00 utterances.], tot_loss[loss=0.3679, over 2878313.89 frames; utt_duration=1317 frames, utt_pad_proportion=0.04209, over 8751.42 utterances.], batch size: 43, lr: 4.29e-04
|
| 94 |
+
2022-08-09 20:25:06,833 INFO [train.py:737] (0/3) Epoch 11, batch 2100, loss[loss=0.2979, over 14462.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01086, over 33.00 utterances.], tot_loss[loss=0.3687, over 2880688.76 frames; utt_duration=1329 frames, utt_pad_proportion=0.04088, over 8676.99 utterances.], batch size: 33, lr: 4.29e-04
|
| 95 |
+
2022-08-09 20:35:11,572 INFO [train.py:737] (0/3) Epoch 11, batch 2200, loss[loss=0.3794, over 13765.00 frames; utt_duration=967.3 frames, utt_pad_proportion=0.06633, over 57.00 utterances.], tot_loss[loss=0.3673, over 2867016.80 frames; utt_duration=1296 frames, utt_pad_proportion=0.04923, over 8859.93 utterances.], batch size: 57, lr: 4.29e-04
|
| 96 |
+
2022-08-09 20:43:57,459 INFO [train.py:737] (0/3) Epoch 11, batch 2300, loss[loss=0.4016, over 14581.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.006315, over 41.00 utterances.], tot_loss[loss=0.3698, over 2847219.98 frames; utt_duration=1251 frames, utt_pad_proportion=0.06034, over 9117.83 utterances.], batch size: 41, lr: 4.29e-04
|
| 97 |
+
2022-08-09 20:54:17,080 INFO [train.py:737] (0/3) Epoch 11, batch 2400, loss[loss=0.341, over 12876.00 frames; utt_duration=770.3 frames, utt_pad_proportion=0.1315, over 67.00 utterances.], tot_loss[loss=0.3695, over 2863792.64 frames; utt_duration=1286 frames, utt_pad_proportion=0.0502, over 8921.00 utterances.], batch size: 67, lr: 4.28e-04
|
| 98 |
+
2022-08-09 21:04:37,869 INFO [train.py:737] (0/3) Epoch 11, batch 2500, loss[loss=0.3829, over 14581.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006314, over 42.00 utterances.], tot_loss[loss=0.3709, over 2864333.18 frames; utt_duration=1294 frames, utt_pad_proportion=0.05113, over 8863.82 utterances.], batch size: 42, lr: 4.28e-04
|
| 99 |
+
2022-08-09 21:14:14,370 INFO [train.py:737] (0/3) Epoch 11, batch 2600, loss[loss=0.3485, over 14752.00 frames; utt_duration=1687 frames, utt_pad_proportion=0.008645, over 35.00 utterances.], tot_loss[loss=0.3692, over 2864601.06 frames; utt_duration=1290 frames, utt_pad_proportion=0.05072, over 8895.27 utterances.], batch size: 35, lr: 4.28e-04
|
| 100 |
+
2022-08-09 21:22:52,904 INFO [train.py:737] (0/3) Epoch 11, batch 2700, loss[loss=0.3885, over 13076.00 frames; utt_duration=782.2 frames, utt_pad_proportion=0.1181, over 67.00 utterances.], tot_loss[loss=0.3696, over 2847841.36 frames; utt_duration=1257 frames, utt_pad_proportion=0.06114, over 9075.44 utterances.], batch size: 67, lr: 4.28e-04
|
| 101 |
+
2022-08-09 21:29:45,369 INFO [train.py:737] (0/3) Epoch 11, batch 2800, loss[loss=0.3612, over 14590.00 frames; utt_duration=1623 frames, utt_pad_proportion=0.00815, over 36.00 utterances.], tot_loss[loss=0.369, over 2861857.60 frames; utt_duration=1286 frames, utt_pad_proportion=0.05267, over 8912.35 utterances.], batch size: 36, lr: 4.28e-04
|
| 102 |
+
2022-08-09 21:38:58,795 INFO [train.py:737] (0/3) Epoch 11, batch 2900, loss[loss=0.3996, over 14769.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.004578, over 41.00 utterances.], tot_loss[loss=0.3869, over 2866214.76 frames; utt_duration=1287 frames, utt_pad_proportion=0.04894, over 8916.57 utterances.], batch size: 41, lr: 4.28e-04
|
| 103 |
+
2022-08-09 21:47:56,448 INFO [train.py:737] (0/3) Epoch 11, batch 3000, loss[loss=0.4351, over 14580.00 frames; utt_duration=1621 frames, utt_pad_proportion=0.008948, over 36.00 utterances.], tot_loss[loss=0.4028, over 2854412.61 frames; utt_duration=1258 frames, utt_pad_proportion=0.05571, over 9085.12 utterances.], batch size: 36, lr: 4.27e-04
|
| 104 |
+
2022-08-09 21:47:56,449 INFO [train.py:761] (0/3) Computing validation loss
|
| 105 |
+
2022-08-09 21:50:31,850 INFO [train.py:770] (0/3) Epoch 11, validation: loss=0.2999, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 106 |
+
2022-08-09 21:58:13,828 INFO [train.py:737] (0/3) Epoch 11, batch 3100, loss[loss=0.3698, over 14770.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.006436, over 38.00 utterances.], tot_loss[loss=0.4173, over 2852350.20 frames; utt_duration=1258 frames, utt_pad_proportion=0.05722, over 9080.88 utterances.], batch size: 38, lr: 4.27e-04
|
| 107 |
+
2022-08-09 22:06:18,344 INFO [train.py:737] (0/3) Epoch 11, batch 3200, loss[loss=0.4846, over 9553.00 frames; utt_duration=416.8 frames, utt_pad_proportion=0.3607, over 92.00 utterances.], tot_loss[loss=0.4263, over 2843490.77 frames; utt_duration=1235 frames, utt_pad_proportion=0.06198, over 9222.63 utterances.], batch size: 92, lr: 4.27e-04
|
| 108 |
+
2022-08-09 22:15:31,408 INFO [train.py:737] (0/3) Epoch 11, batch 3300, loss[loss=0.4615, over 14729.00 frames; utt_duration=1282 frames, utt_pad_proportion=0.009904, over 46.00 utterances.], tot_loss[loss=0.4336, over 2844434.29 frames; utt_duration=1236 frames, utt_pad_proportion=0.06125, over 9219.68 utterances.], batch size: 46, lr: 4.27e-04
|
| 109 |
+
2022-08-09 22:23:12,453 INFO [train.py:737] (0/3) Epoch 11, batch 3400, loss[loss=0.3898, over 14771.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.006033, over 38.00 utterances.], tot_loss[loss=0.4409, over 2839907.04 frames; utt_duration=1230 frames, utt_pad_proportion=0.06458, over 9243.62 utterances.], batch size: 38, lr: 4.27e-04
|
| 110 |
+
2022-08-09 22:31:24,029 INFO [train.py:737] (0/3) Epoch 11, batch 3500, loss[loss=0.3988, over 14604.00 frames; utt_duration=1624 frames, utt_pad_proportion=0.006575, over 36.00 utterances.], tot_loss[loss=0.4459, over 2837895.60 frames; utt_duration=1228 frames, utt_pad_proportion=0.06635, over 9254.25 utterances.], batch size: 36, lr: 4.27e-04
|
| 111 |
+
2022-08-09 22:40:34,203 INFO [train.py:737] (0/3) Epoch 11, batch 3600, loss[loss=0.5184, over 14640.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.007886, over 44.00 utterances.], tot_loss[loss=0.4484, over 2823829.25 frames; utt_duration=1201 frames, utt_pad_proportion=0.07522, over 9417.62 utterances.], batch size: 44, lr: 4.26e-04
|
| 112 |
+
2022-08-09 22:48:24,926 INFO [train.py:737] (0/3) Epoch 11, batch 3700, loss[loss=0.504, over 14710.00 frames; utt_duration=1309 frames, utt_pad_proportion=0.008148, over 45.00 utterances.], tot_loss[loss=0.4515, over 2820252.41 frames; utt_duration=1191 frames, utt_pad_proportion=0.07682, over 9480.95 utterances.], batch size: 45, lr: 4.26e-04
|
| 113 |
+
2022-08-09 22:55:40,793 INFO [train.py:737] (0/3) Epoch 11, batch 3800, loss[loss=0.4624, over 13084.00 frames; utt_duration=782.9 frames, utt_pad_proportion=0.1174, over 67.00 utterances.], tot_loss[loss=0.4516, over 2818931.44 frames; utt_duration=1195 frames, utt_pad_proportion=0.07626, over 9445.81 utterances.], batch size: 67, lr: 4.26e-04
|
| 114 |
+
2022-08-09 23:03:07,339 INFO [train.py:737] (0/3) Epoch 11, batch 3900, loss[loss=0.3446, over 14603.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01009, over 34.00 utterances.], tot_loss[loss=0.4493, over 2822558.29 frames; utt_duration=1201 frames, utt_pad_proportion=0.07505, over 9410.15 utterances.], batch size: 34, lr: 4.26e-04
|
| 115 |
+
2022-08-09 23:09:27,268 INFO [train.py:737] (0/3) Epoch 11, batch 4000, loss[loss=0.3774, over 14569.00 frames; utt_duration=1495 frames, utt_pad_proportion=0.006389, over 39.00 utterances.], tot_loss[loss=0.4473, over 2826509.22 frames; utt_duration=1216 frames, utt_pad_proportion=0.07194, over 9307.75 utterances.], batch size: 39, lr: 4.26e-04
|
| 116 |
+
2022-08-09 23:16:24,246 INFO [train.py:737] (0/3) Epoch 11, batch 4100, loss[loss=0.3976, over 14590.00 frames; utt_duration=1623 frames, utt_pad_proportion=0.008116, over 36.00 utterances.], tot_loss[loss=0.4485, over 2834902.07 frames; utt_duration=1231 frames, utt_pad_proportion=0.06684, over 9225.92 utterances.], batch size: 36, lr: 4.26e-04
|
| 117 |
+
2022-08-09 23:23:18,412 INFO [train.py:737] (0/3) Epoch 11, batch 4200, loss[loss=0.4384, over 14585.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.005399, over 41.00 utterances.], tot_loss[loss=0.4485, over 2833727.97 frames; utt_duration=1232 frames, utt_pad_proportion=0.0656, over 9209.07 utterances.], batch size: 41, lr: 4.25e-04
|
| 118 |
+
2022-08-09 23:32:50,143 INFO [train.py:737] (0/3) Epoch 11, batch 4300, loss[loss=0.3585, over 14561.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.006914, over 37.00 utterances.], tot_loss[loss=0.4457, over 2836358.53 frames; utt_duration=1248 frames, utt_pad_proportion=0.06487, over 9103.20 utterances.], batch size: 37, lr: 4.25e-04
|
| 119 |
+
2022-08-09 23:36:57,313 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/checkpoint-128000.pt
|
| 120 |
+
2022-08-09 23:43:42,059 INFO [train.py:737] (0/3) Epoch 11, batch 4400, loss[loss=0.408, over 14756.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005521, over 41.00 utterances.], tot_loss[loss=0.4455, over 2839566.61 frames; utt_duration=1249 frames, utt_pad_proportion=0.06344, over 9104.16 utterances.], batch size: 41, lr: 4.25e-04
|
| 121 |
+
2022-08-09 23:55:06,950 INFO [train.py:737] (0/3) Epoch 11, batch 4500, loss[loss=0.3838, over 14773.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.006134, over 38.00 utterances.], tot_loss[loss=0.4466, over 2833625.08 frames; utt_duration=1230 frames, utt_pad_proportion=0.06742, over 9222.63 utterances.], batch size: 38, lr: 4.25e-04
|
| 122 |
+
2022-08-10 00:03:22,382 INFO [train.py:737] (0/3) Epoch 11, batch 4600, loss[loss=0.5755, over 9644.00 frames; utt_duration=420.9 frames, utt_pad_proportion=0.3495, over 92.00 utterances.], tot_loss[loss=0.4475, over 2827303.16 frames; utt_duration=1214 frames, utt_pad_proportion=0.07109, over 9326.93 utterances.], batch size: 92, lr: 4.25e-04
|
| 123 |
+
2022-08-10 00:11:18,338 INFO [train.py:737] (0/3) Epoch 11, batch 4700, loss[loss=0.3995, over 14762.00 frames; utt_duration=1478 frames, utt_pad_proportion=0.006254, over 40.00 utterances.], tot_loss[loss=0.4452, over 2838246.62 frames; utt_duration=1229 frames, utt_pad_proportion=0.06476, over 9246.93 utterances.], batch size: 40, lr: 4.25e-04
|
| 124 |
+
2022-08-10 00:19:24,287 INFO [train.py:737] (0/3) Epoch 11, batch 4800, loss[loss=0.4217, over 14765.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006284, over 39.00 utterances.], tot_loss[loss=0.4444, over 2833834.56 frames; utt_duration=1227 frames, utt_pad_proportion=0.06805, over 9252.47 utterances.], batch size: 39, lr: 4.24e-04
|
| 125 |
+
2022-08-10 00:31:46,381 INFO [train.py:737] (0/3) Epoch 11, batch 4900, loss[loss=0.51, over 12868.00 frames; utt_duration=769.6 frames, utt_pad_proportion=0.1294, over 67.00 utterances.], tot_loss[loss=0.4434, over 2834292.03 frames; utt_duration=1216 frames, utt_pad_proportion=0.06755, over 9333.71 utterances.], batch size: 67, lr: 4.24e-04
|
| 126 |
+
2022-08-10 00:40:42,901 INFO [train.py:737] (0/3) Epoch 11, batch 5000, loss[loss=0.5448, over 9776.00 frames; utt_duration=421.9 frames, utt_pad_proportion=0.3429, over 93.00 utterances.], tot_loss[loss=0.4471, over 2822457.06 frames; utt_duration=1197 frames, utt_pad_proportion=0.07381, over 9443.65 utterances.], batch size: 93, lr: 4.24e-04
|
| 127 |
+
2022-08-10 00:48:51,915 INFO [train.py:737] (0/3) Epoch 11, batch 5100, loss[loss=0.3568, over 14589.00 frames; utt_duration=1623 frames, utt_pad_proportion=0.007577, over 36.00 utterances.], tot_loss[loss=0.4478, over 2821433.00 frames; utt_duration=1198 frames, utt_pad_proportion=0.07621, over 9430.87 utterances.], batch size: 36, lr: 4.24e-04
|
| 128 |
+
2022-08-10 00:56:58,915 INFO [train.py:737] (0/3) Epoch 11, batch 5200, loss[loss=0.4451, over 14883.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.006353, over 44.00 utterances.], tot_loss[loss=0.4479, over 2805794.62 frames; utt_duration=1173 frames, utt_pad_proportion=0.08455, over 9582.19 utterances.], batch size: 44, lr: 4.24e-04
|
| 129 |
+
2022-08-10 01:03:13,744 INFO [train.py:737] (0/3) Epoch 11, batch 5300, loss[loss=0.4703, over 14754.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.004902, over 41.00 utterances.], tot_loss[loss=0.4464, over 2828618.19 frames; utt_duration=1224 frames, utt_pad_proportion=0.07261, over 9251.53 utterances.], batch size: 41, lr: 4.24e-04
|
| 130 |
+
2022-08-10 01:11:10,721 INFO [train.py:737] (0/3) Epoch 11, batch 5400, loss[loss=0.5211, over 13212.00 frames; utt_duration=778.6 frames, utt_pad_proportion=0.1172, over 68.00 utterances.], tot_loss[loss=0.4468, over 2826109.05 frames; utt_duration=1212 frames, utt_pad_proportion=0.07306, over 9341.20 utterances.], batch size: 68, lr: 4.23e-04
|
| 131 |
+
2022-08-10 01:19:04,626 INFO [train.py:737] (0/3) Epoch 11, batch 5500, loss[loss=0.4306, over 14699.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01112, over 46.00 utterances.], tot_loss[loss=0.4449, over 2832179.60 frames; utt_duration=1222 frames, utt_pad_proportion=0.06943, over 9283.53 utterances.], batch size: 46, lr: 4.23e-04
|
| 132 |
+
2022-08-10 01:27:52,229 INFO [train.py:737] (0/3) Epoch 11, batch 5600, loss[loss=0.4231, over 14592.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.006732, over 38.00 utterances.], tot_loss[loss=0.4484, over 2818682.36 frames; utt_duration=1202 frames, utt_pad_proportion=0.07774, over 9393.80 utterances.], batch size: 38, lr: 4.23e-04
|
| 133 |
+
2022-08-10 01:37:26,308 INFO [train.py:737] (0/3) Epoch 11, batch 5700, loss[loss=0.5167, over 14681.00 frames; utt_duration=1251 frames, utt_pad_proportion=0.01115, over 47.00 utterances.], tot_loss[loss=0.449, over 2825661.45 frames; utt_duration=1205 frames, utt_pad_proportion=0.07191, over 9393.71 utterances.], batch size: 47, lr: 4.23e-04
|
| 134 |
+
2022-08-10 01:44:25,843 INFO [train.py:737] (0/3) Epoch 11, batch 5800, loss[loss=0.3884, over 14563.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.006846, over 37.00 utterances.], tot_loss[loss=0.4507, over 2821940.75 frames; utt_duration=1197 frames, utt_pad_proportion=0.07361, over 9439.33 utterances.], batch size: 37, lr: 4.23e-04
|
| 135 |
+
2022-08-10 01:54:30,355 INFO [train.py:737] (0/3) Epoch 11, batch 5900, loss[loss=0.5525, over 14594.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.005514, over 42.00 utterances.], tot_loss[loss=0.4506, over 2811327.32 frames; utt_duration=1180 frames, utt_pad_proportion=0.08117, over 9539.19 utterances.], batch size: 42, lr: 4.23e-04
|
| 136 |
+
2022-08-10 02:02:21,713 INFO [train.py:737] (0/3) Epoch 11, batch 6000, loss[loss=0.406, over 14441.00 frames; utt_duration=1652 frames, utt_pad_proportion=0.00893, over 35.00 utterances.], tot_loss[loss=0.449, over 2799205.88 frames; utt_duration=1165 frames, utt_pad_proportion=0.0871, over 9626.66 utterances.], batch size: 35, lr: 4.22e-04
|
| 137 |
+
2022-08-10 02:02:21,713 INFO [train.py:761] (0/3) Computing validation loss
|
| 138 |
+
2022-08-10 02:03:17,900 INFO [train.py:770] (0/3) Epoch 11, validation: loss=0.3454, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 139 |
+
2022-08-10 02:09:36,991 INFO [train.py:737] (0/3) Epoch 11, batch 6100, loss[loss=0.5381, over 9640.00 frames; utt_duration=416.2 frames, utt_pad_proportion=0.3538, over 93.00 utterances.], tot_loss[loss=0.4498, over 2752089.21 frames; utt_duration=1100 frames, utt_pad_proportion=0.1137, over 10019.45 utterances.], batch size: 93, lr: 4.22e-04
|
| 140 |
+
2022-08-10 02:13:09,550 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-11.pt
|
| 141 |
+
2022-08-10 02:19:32,340 INFO [train.py:737] (0/3) Epoch 12, batch 0, loss[loss=0.4075, over 14586.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.005501, over 41.00 utterances.], tot_loss[loss=0.4075, over 14586.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.005501, over 41.00 utterances.], batch size: 41, lr: 4.07e-04
|
| 142 |
+
2022-08-10 02:29:04,932 INFO [train.py:737] (0/3) Epoch 12, batch 100, loss[loss=0.3275, over 14619.00 frames; utt_duration=1330 frames, utt_pad_proportion=0.009409, over 44.00 utterances.], tot_loss[loss=0.3683, over 1135723.49 frames; utt_duration=1281 frames, utt_pad_proportion=0.05206, over 3551.62 utterances.], batch size: 44, lr: 4.07e-04
|
| 143 |
+
2022-08-10 02:38:10,101 INFO [train.py:737] (0/3) Epoch 12, batch 200, loss[loss=0.3475, over 14778.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.007365, over 37.00 utterances.], tot_loss[loss=0.3622, over 1819555.20 frames; utt_duration=1306 frames, utt_pad_proportion=0.04796, over 5579.29 utterances.], batch size: 37, lr: 4.07e-04
|
| 144 |
+
2022-08-10 02:50:17,050 INFO [train.py:737] (0/3) Epoch 12, batch 300, loss[loss=0.4479, over 14687.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009815, over 45.00 utterances.], tot_loss[loss=0.3606, over 2238040.94 frames; utt_duration=1329 frames, utt_pad_proportion=0.04417, over 6743.96 utterances.], batch size: 45, lr: 4.07e-04
|
| 145 |
+
2022-08-10 02:59:35,400 INFO [train.py:737] (0/3) Epoch 12, batch 400, loss[loss=0.3606, over 14580.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.005688, over 42.00 utterances.], tot_loss[loss=0.3612, over 2500393.29 frames; utt_duration=1345 frames, utt_pad_proportion=0.03529, over 7446.31 utterances.], batch size: 42, lr: 4.07e-04
|
| 146 |
+
2022-08-10 03:10:03,633 INFO [train.py:737] (0/3) Epoch 12, batch 500, loss[loss=0.4267, over 14418.00 frames; utt_duration=1090 frames, utt_pad_proportion=0.03053, over 53.00 utterances.], tot_loss[loss=0.3641, over 2634257.64 frames; utt_duration=1297 frames, utt_pad_proportion=0.04808, over 8133.63 utterances.], batch size: 53, lr: 4.07e-04
|
| 147 |
+
2022-08-10 03:22:54,031 INFO [train.py:737] (0/3) Epoch 12, batch 600, loss[loss=0.4178, over 14701.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01173, over 46.00 utterances.], tot_loss[loss=0.3628, over 2739600.49 frames; utt_duration=1324 frames, utt_pad_proportion=0.04016, over 8287.75 utterances.], batch size: 46, lr: 4.06e-04
|
| 148 |
+
2022-08-10 03:34:44,749 INFO [train.py:737] (0/3) Epoch 12, batch 700, loss[loss=0.3542, over 14591.00 frames; utt_duration=1498 frames, utt_pad_proportion=0.005448, over 39.00 utterances.], tot_loss[loss=0.3636, over 2788030.14 frames; utt_duration=1295 frames, utt_pad_proportion=0.04495, over 8621.57 utterances.], batch size: 39, lr: 4.06e-04
|
| 149 |
+
2022-08-10 03:46:11,194 INFO [train.py:737] (0/3) Epoch 12, batch 800, loss[loss=0.4034, over 14689.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.008795, over 45.00 utterances.], tot_loss[loss=0.3675, over 2818994.03 frames; utt_duration=1286 frames, utt_pad_proportion=0.04521, over 8781.34 utterances.], batch size: 45, lr: 4.06e-04
|
| 150 |
+
2022-08-10 03:56:27,268 INFO [train.py:737] (0/3) Epoch 12, batch 900, loss[loss=0.4036, over 14571.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.006267, over 40.00 utterances.], tot_loss[loss=0.3674, over 2828447.79 frames; utt_duration=1275 frames, utt_pad_proportion=0.0523, over 8884.63 utterances.], batch size: 40, lr: 4.06e-04
|
| 151 |
+
2022-08-10 04:06:59,878 INFO [train.py:737] (0/3) Epoch 12, batch 1000, loss[loss=0.4173, over 10310.00 frames; utt_duration=449.6 frames, utt_pad_proportion=0.3094, over 92.00 utterances.], tot_loss[loss=0.3676, over 2842258.23 frames; utt_duration=1262 frames, utt_pad_proportion=0.05077, over 9020.35 utterances.], batch size: 92, lr: 4.06e-04
|
| 152 |
+
2022-08-10 04:16:28,351 INFO [train.py:737] (0/3) Epoch 12, batch 1100, loss[loss=0.3399, over 14465.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01098, over 33.00 utterances.], tot_loss[loss=0.3688, over 2849595.41 frames; utt_duration=1266 frames, utt_pad_proportion=0.05051, over 9014.62 utterances.], batch size: 33, lr: 4.06e-04
|
| 153 |
+
2022-08-10 04:27:14,725 INFO [train.py:737] (0/3) Epoch 12, batch 1200, loss[loss=0.3799, over 14569.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006471, over 40.00 utterances.], tot_loss[loss=0.3668, over 2854648.38 frames; utt_duration=1268 frames, utt_pad_proportion=0.05021, over 9015.12 utterances.], batch size: 40, lr: 4.05e-04
|
| 154 |
+
2022-08-10 04:37:22,609 INFO [train.py:737] (0/3) Epoch 12, batch 1300, loss[loss=0.332, over 14774.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.005847, over 39.00 utterances.], tot_loss[loss=0.3671, over 2858078.77 frames; utt_duration=1269 frames, utt_pad_proportion=0.04917, over 9020.40 utterances.], batch size: 39, lr: 4.05e-04
|
| 155 |
+
2022-08-10 04:46:39,486 INFO [train.py:737] (0/3) Epoch 12, batch 1400, loss[loss=0.3417, over 14351.00 frames; utt_duration=1085 frames, utt_pad_proportion=0.03401, over 53.00 utterances.], tot_loss[loss=0.3669, over 2855335.10 frames; utt_duration=1271 frames, utt_pad_proportion=0.05018, over 8995.03 utterances.], batch size: 53, lr: 4.05e-04
|
| 156 |
+
2022-08-10 04:57:14,243 INFO [train.py:737] (0/3) Epoch 12, batch 1500, loss[loss=0.5502, over 10111.00 frames; utt_duration=441.1 frames, utt_pad_proportion=0.3214, over 92.00 utterances.], tot_loss[loss=0.3696, over 2848092.42 frames; utt_duration=1253 frames, utt_pad_proportion=0.05821, over 9103.19 utterances.], batch size: 92, lr: 4.05e-04
|
| 157 |
+
2022-08-10 05:09:29,811 INFO [train.py:737] (0/3) Epoch 12, batch 1600, loss[loss=0.4023, over 14625.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.009054, over 44.00 utterances.], tot_loss[loss=0.3689, over 2858972.82 frames; utt_duration=1282 frames, utt_pad_proportion=0.05075, over 8933.49 utterances.], batch size: 44, lr: 4.05e-04
|
| 158 |
+
2022-08-10 05:20:38,269 INFO [train.py:737] (0/3) Epoch 12, batch 1700, loss[loss=0.343, over 14582.00 frames; utt_duration=1536 frames, utt_pad_proportion=0.007514, over 38.00 utterances.], tot_loss[loss=0.3689, over 2866328.86 frames; utt_duration=1288 frames, utt_pad_proportion=0.04695, over 8915.20 utterances.], batch size: 38, lr: 4.05e-04
|
| 159 |
+
2022-08-10 05:31:51,598 INFO [train.py:737] (0/3) Epoch 12, batch 1800, loss[loss=0.332, over 14559.00 frames; utt_duration=1575 frames, utt_pad_proportion=0.007289, over 37.00 utterances.], tot_loss[loss=0.3669, over 2877957.53 frames; utt_duration=1309 frames, utt_pad_proportion=0.04034, over 8803.50 utterances.], batch size: 37, lr: 4.05e-04
|
| 160 |
+
2022-08-10 05:42:58,981 INFO [train.py:737] (0/3) Epoch 12, batch 1900, loss[loss=0.3967, over 14678.00 frames; utt_duration=1251 frames, utt_pad_proportion=0.01213, over 47.00 utterances.], tot_loss[loss=0.368, over 2873598.19 frames; utt_duration=1303 frames, utt_pad_proportion=0.04207, over 8832.15 utterances.], batch size: 47, lr: 4.04e-04
|
| 161 |
+
2022-08-10 05:52:47,051 INFO [train.py:737] (0/3) Epoch 12, batch 2000, loss[loss=0.3522, over 14712.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.01095, over 46.00 utterances.], tot_loss[loss=0.3671, over 2873458.50 frames; utt_duration=1302 frames, utt_pad_proportion=0.04418, over 8838.71 utterances.], batch size: 46, lr: 4.04e-04
|
| 162 |
+
2022-08-10 06:03:23,203 INFO [train.py:737] (0/3) Epoch 12, batch 2100, loss[loss=0.3437, over 14570.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006608, over 37.00 utterances.], tot_loss[loss=0.3675, over 2857964.93 frames; utt_duration=1272 frames, utt_pad_proportion=0.05436, over 8999.62 utterances.], batch size: 37, lr: 4.04e-04
|
| 163 |
+
2022-08-10 06:13:20,122 INFO [train.py:737] (0/3) Epoch 12, batch 2200, loss[loss=0.3625, over 14741.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.006849, over 43.00 utterances.], tot_loss[loss=0.3672, over 2859699.95 frames; utt_duration=1266 frames, utt_pad_proportion=0.05335, over 9045.78 utterances.], batch size: 43, lr: 4.04e-04
|
| 164 |
+
2022-08-10 06:24:03,209 INFO [train.py:737] (0/3) Epoch 12, batch 2300, loss[loss=0.3199, over 14575.00 frames; utt_duration=1536 frames, utt_pad_proportion=0.007008, over 38.00 utterances.], tot_loss[loss=0.3637, over 2864530.58 frames; utt_duration=1285 frames, utt_pad_proportion=0.04999, over 8925.99 utterances.], batch size: 38, lr: 4.04e-04
|
| 165 |
+
2022-08-10 06:36:03,152 INFO [train.py:737] (0/3) Epoch 12, batch 2400, loss[loss=0.3556, over 14667.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.01272, over 47.00 utterances.], tot_loss[loss=0.3647, over 2861211.40 frames; utt_duration=1278 frames, utt_pad_proportion=0.05193, over 8968.48 utterances.], batch size: 47, lr: 4.04e-04
|
| 166 |
+
2022-08-10 06:48:05,000 INFO [train.py:737] (0/3) Epoch 12, batch 2500, loss[loss=0.3832, over 13182.00 frames; utt_duration=776.8 frames, utt_pad_proportion=0.1162, over 68.00 utterances.], tot_loss[loss=0.3643, over 2863957.46 frames; utt_duration=1290 frames, utt_pad_proportion=0.04973, over 8893.87 utterances.], batch size: 68, lr: 4.03e-04
|
| 167 |
+
2022-08-10 07:00:47,848 INFO [train.py:737] (0/3) Epoch 12, batch 2600, loss[loss=0.3368, over 14734.00 frames; utt_duration=1685 frames, utt_pad_proportion=0.009753, over 35.00 utterances.], tot_loss[loss=0.3615, over 2867616.36 frames; utt_duration=1298 frames, utt_pad_proportion=0.04821, over 8845.40 utterances.], batch size: 35, lr: 4.03e-04
|
| 168 |
+
2022-08-10 07:12:58,139 INFO [train.py:737] (0/3) Epoch 12, batch 2700, loss[loss=0.3359, over 14750.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.00626, over 43.00 utterances.], tot_loss[loss=0.3648, over 2866084.96 frames; utt_duration=1291 frames, utt_pad_proportion=0.04917, over 8890.10 utterances.], batch size: 43, lr: 4.03e-04
|
| 169 |
+
2022-08-10 07:21:23,110 INFO [train.py:737] (0/3) Epoch 12, batch 2800, loss[loss=0.3988, over 14418.00 frames; utt_duration=1155 frames, utt_pad_proportion=0.02481, over 50.00 utterances.], tot_loss[loss=0.3662, over 2853297.57 frames; utt_duration=1262 frames, utt_pad_proportion=0.0567, over 9051.55 utterances.], batch size: 50, lr: 4.03e-04
|
| 170 |
+
2022-08-10 07:30:52,941 INFO [train.py:737] (0/3) Epoch 12, batch 2900, loss[loss=0.4441, over 13053.00 frames; utt_duration=769.3 frames, utt_pad_proportion=0.1267, over 68.00 utterances.], tot_loss[loss=0.3841, over 2856901.98 frames; utt_duration=1256 frames, utt_pad_proportion=0.05441, over 9106.22 utterances.], batch size: 68, lr: 4.03e-04
|
| 171 |
+
2022-08-10 07:40:15,528 INFO [train.py:737] (0/3) Epoch 12, batch 3000, loss[loss=0.4534, over 14577.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.005401, over 39.00 utterances.], tot_loss[loss=0.4001, over 2848638.56 frames; utt_duration=1251 frames, utt_pad_proportion=0.05925, over 9118.05 utterances.], batch size: 39, lr: 4.03e-04
|
| 172 |
+
2022-08-10 07:40:15,528 INFO [train.py:761] (0/3) Computing validation loss
|
| 173 |
+
2022-08-10 07:43:16,492 INFO [train.py:770] (0/3) Epoch 12, validation: loss=0.297, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 174 |
+
2022-08-10 07:53:07,546 INFO [train.py:737] (0/3) Epoch 12, batch 3100, loss[loss=0.4514, over 13777.00 frames; utt_duration=968.2 frames, utt_pad_proportion=0.06817, over 57.00 utterances.], tot_loss[loss=0.4135, over 2826071.43 frames; utt_duration=1208 frames, utt_pad_proportion=0.07221, over 9371.72 utterances.], batch size: 57, lr: 4.03e-04
|
| 175 |
+
2022-08-10 08:03:53,493 INFO [train.py:737] (0/3) Epoch 12, batch 3200, loss[loss=0.3644, over 14703.00 frames; utt_duration=1682 frames, utt_pad_proportion=0.01135, over 35.00 utterances.], tot_loss[loss=0.4199, over 2836405.79 frames; utt_duration=1234 frames, utt_pad_proportion=0.06638, over 9202.93 utterances.], batch size: 35, lr: 4.02e-04
|
| 176 |
+
2022-08-10 08:15:52,838 INFO [train.py:737] (0/3) Epoch 12, batch 3300, loss[loss=0.5045, over 14642.00 frames; utt_duration=1248 frames, utt_pad_proportion=0.01369, over 47.00 utterances.], tot_loss[loss=0.4306, over 2829758.26 frames; utt_duration=1218 frames, utt_pad_proportion=0.07063, over 9302.54 utterances.], batch size: 47, lr: 4.02e-04
|
| 177 |
+
2022-08-10 08:24:14,593 INFO [train.py:737] (0/3) Epoch 12, batch 3400, loss[loss=0.4619, over 14685.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009249, over 45.00 utterances.], tot_loss[loss=0.4359, over 2836860.65 frames; utt_duration=1228 frames, utt_pad_proportion=0.0658, over 9250.96 utterances.], batch size: 45, lr: 4.02e-04
|
| 178 |
+
2022-08-10 08:32:25,018 INFO [train.py:737] (0/3) Epoch 12, batch 3500, loss[loss=0.5394, over 14882.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.007115, over 44.00 utterances.], tot_loss[loss=0.4419, over 2841068.20 frames; utt_duration=1238 frames, utt_pad_proportion=0.06368, over 9192.54 utterances.], batch size: 44, lr: 4.02e-04
|
| 179 |
+
2022-08-10 08:43:20,504 INFO [train.py:737] (0/3) Epoch 12, batch 3600, loss[loss=0.4123, over 14579.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005164, over 40.00 utterances.], tot_loss[loss=0.4442, over 2826549.61 frames; utt_duration=1210 frames, utt_pad_proportion=0.07295, over 9357.35 utterances.], batch size: 40, lr: 4.02e-04
|
| 180 |
+
2022-08-10 08:54:44,668 INFO [train.py:737] (0/3) Epoch 12, batch 3700, loss[loss=0.542, over 10065.00 frames; utt_duration=444.1 frames, utt_pad_proportion=0.32, over 91.00 utterances.], tot_loss[loss=0.4436, over 2815428.64 frames; utt_duration=1197 frames, utt_pad_proportion=0.0786, over 9418.18 utterances.], batch size: 91, lr: 4.02e-04
|
| 181 |
+
2022-08-10 09:03:23,104 INFO [train.py:737] (0/3) Epoch 12, batch 3800, loss[loss=0.4598, over 14583.00 frames; utt_duration=1536 frames, utt_pad_proportion=0.006872, over 38.00 utterances.], tot_loss[loss=0.4473, over 2815392.82 frames; utt_duration=1196 frames, utt_pad_proportion=0.08031, over 9424.94 utterances.], batch size: 38, lr: 4.01e-04
|
| 182 |
+
2022-08-10 09:12:39,487 INFO [train.py:737] (0/3) Epoch 12, batch 3900, loss[loss=0.5462, over 9361.00 frames; utt_duration=403.9 frames, utt_pad_proportion=0.3708, over 93.00 utterances.], tot_loss[loss=0.4494, over 2806306.91 frames; utt_duration=1175 frames, utt_pad_proportion=0.08527, over 9565.98 utterances.], batch size: 93, lr: 4.01e-04
|
| 183 |
+
2022-08-10 09:22:00,877 INFO [train.py:737] (0/3) Epoch 12, batch 4000, loss[loss=0.4068, over 14518.00 frames; utt_duration=1212 frames, utt_pad_proportion=0.01504, over 48.00 utterances.], tot_loss[loss=0.4508, over 2819831.67 frames; utt_duration=1196 frames, utt_pad_proportion=0.07682, over 9440.20 utterances.], batch size: 48, lr: 4.01e-04
|
| 184 |
+
2022-08-10 09:32:08,725 INFO [train.py:737] (0/3) Epoch 12, batch 4100, loss[loss=0.4366, over 14779.00 frames; utt_duration=1479 frames, utt_pad_proportion=0.005077, over 40.00 utterances.], tot_loss[loss=0.4462, over 2815929.10 frames; utt_duration=1199 frames, utt_pad_proportion=0.07832, over 9407.61 utterances.], batch size: 40, lr: 4.01e-04
|
| 185 |
+
2022-08-10 09:41:30,765 INFO [train.py:737] (0/3) Epoch 12, batch 4200, loss[loss=0.4674, over 14626.00 frames; utt_duration=1627 frames, utt_pad_proportion=0.005671, over 36.00 utterances.], tot_loss[loss=0.4454, over 2832115.02 frames; utt_duration=1224 frames, utt_pad_proportion=0.06966, over 9269.32 utterances.], batch size: 36, lr: 4.01e-04
|
| 186 |
+
2022-08-10 09:51:28,127 INFO [train.py:737] (0/3) Epoch 12, batch 4300, loss[loss=0.5501, over 14471.00 frames; utt_duration=1159 frames, utt_pad_proportion=0.02258, over 50.00 utterances.], tot_loss[loss=0.4433, over 2837674.58 frames; utt_duration=1240 frames, utt_pad_proportion=0.06562, over 9165.78 utterances.], batch size: 50, lr: 4.01e-04
|
| 187 |
+
2022-08-10 10:02:08,396 INFO [train.py:737] (0/3) Epoch 12, batch 4400, loss[loss=0.5572, over 14629.00 frames; utt_duration=1247 frames, utt_pad_proportion=0.01372, over 47.00 utterances.], tot_loss[loss=0.4439, over 2830568.92 frames; utt_duration=1222 frames, utt_pad_proportion=0.07024, over 9276.63 utterances.], batch size: 47, lr: 4.01e-04
|
| 188 |
+
2022-08-10 10:12:01,452 INFO [train.py:737] (0/3) Epoch 12, batch 4500, loss[loss=0.5027, over 13034.00 frames; utt_duration=779.7 frames, utt_pad_proportion=0.12, over 67.00 utterances.], tot_loss[loss=0.4451, over 2821792.31 frames; utt_duration=1201 frames, utt_pad_proportion=0.07467, over 9409.57 utterances.], batch size: 67, lr: 4.00e-04
|
| 189 |
+
2022-08-10 10:22:19,557 INFO [train.py:737] (0/3) Epoch 12, batch 4600, loss[loss=0.4053, over 14691.00 frames; utt_duration=1680 frames, utt_pad_proportion=0.01209, over 35.00 utterances.], tot_loss[loss=0.4452, over 2820162.18 frames; utt_duration=1209 frames, utt_pad_proportion=0.07648, over 9344.79 utterances.], batch size: 35, lr: 4.00e-04
|
| 190 |
+
2022-08-10 10:30:47,462 INFO [train.py:737] (0/3) Epoch 12, batch 4700, loss[loss=0.4423, over 14571.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005607, over 40.00 utterances.], tot_loss[loss=0.4481, over 2810364.48 frames; utt_duration=1186 frames, utt_pad_proportion=0.08194, over 9493.35 utterances.], batch size: 40, lr: 4.00e-04
|
| 191 |
+
2022-08-10 10:43:44,061 INFO [train.py:737] (0/3) Epoch 12, batch 4800, loss[loss=0.5407, over 14724.00 frames; utt_duration=1282 frames, utt_pad_proportion=0.009122, over 46.00 utterances.], tot_loss[loss=0.4473, over 2806134.74 frames; utt_duration=1173 frames, utt_pad_proportion=0.08371, over 9577.73 utterances.], batch size: 46, lr: 4.00e-04
|
| 192 |
+
2022-08-10 10:56:42,545 INFO [train.py:737] (0/3) Epoch 12, batch 4900, loss[loss=0.3667, over 14578.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.006231, over 39.00 utterances.], tot_loss[loss=0.4488, over 2793325.11 frames; utt_duration=1157 frames, utt_pad_proportion=0.09145, over 9673.23 utterances.], batch size: 39, lr: 4.00e-04
|
| 193 |
+
2022-08-10 11:12:32,188 INFO [train.py:737] (0/3) Epoch 12, batch 5000, loss[loss=0.5192, over 14783.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.00531, over 38.00 utterances.], tot_loss[loss=0.4463, over 2826421.23 frames; utt_duration=1214 frames, utt_pad_proportion=0.07359, over 9320.68 utterances.], batch size: 38, lr: 4.00e-04
|
| 194 |
+
2022-08-10 11:47:05,509 INFO [train.py:737] (0/3) Epoch 12, batch 5100, loss[loss=0.3745, over 14423.00 frames; utt_duration=1650 frames, utt_pad_proportion=0.01027, over 35.00 utterances.], tot_loss[loss=0.4452, over 2839166.28 frames; utt_duration=1226 frames, utt_pad_proportion=0.0649, over 9271.81 utterances.], batch size: 35, lr: 4.00e-04
|
| 195 |
+
2022-08-10 12:13:52,155 INFO [train.py:737] (0/3) Epoch 12, batch 5200, loss[loss=0.456, over 14444.00 frames; utt_duration=1652 frames, utt_pad_proportion=0.00893, over 35.00 utterances.], tot_loss[loss=0.446, over 2840259.98 frames; utt_duration=1229 frames, utt_pad_proportion=0.06281, over 9255.23 utterances.], batch size: 35, lr: 3.99e-04
|
| 196 |
+
2022-08-10 12:46:45,458 INFO [train.py:737] (0/3) Epoch 12, batch 5300, loss[loss=0.3723, over 14710.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.0108, over 35.00 utterances.], tot_loss[loss=0.4436, over 2826111.33 frames; utt_duration=1213 frames, utt_pad_proportion=0.07059, over 9330.51 utterances.], batch size: 35, lr: 3.99e-04
|
| 197 |
+
2022-08-10 13:18:04,045 INFO [train.py:737] (0/3) Epoch 12, batch 5400, loss[loss=0.51, over 14454.00 frames; utt_duration=1158 frames, utt_pad_proportion=0.02554, over 50.00 utterances.], tot_loss[loss=0.4435, over 2825918.57 frames; utt_duration=1214 frames, utt_pad_proportion=0.07078, over 9319.58 utterances.], batch size: 50, lr: 3.99e-04
|
| 198 |
+
2022-08-10 13:50:20,062 INFO [train.py:737] (0/3) Epoch 12, batch 5500, loss[loss=0.486, over 14699.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.0111, over 46.00 utterances.], tot_loss[loss=0.4448, over 2829580.80 frames; utt_duration=1215 frames, utt_pad_proportion=0.06994, over 9330.82 utterances.], batch size: 46, lr: 3.99e-04
|
| 199 |
+
2022-08-10 14:17:39,805 INFO [train.py:737] (0/3) Epoch 12, batch 5600, loss[loss=0.4768, over 14517.00 frames; utt_duration=1211 frames, utt_pad_proportion=0.01601, over 48.00 utterances.], tot_loss[loss=0.4482, over 2823620.91 frames; utt_duration=1201 frames, utt_pad_proportion=0.07409, over 9416.88 utterances.], batch size: 48, lr: 3.99e-04
|
| 200 |
+
2022-08-10 14:45:43,325 INFO [train.py:737] (0/3) Epoch 12, batch 5700, loss[loss=0.535, over 14690.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009646, over 45.00 utterances.], tot_loss[loss=0.447, over 2826305.12 frames; utt_duration=1210 frames, utt_pad_proportion=0.07308, over 9356.06 utterances.], batch size: 45, lr: 3.99e-04
|
| 201 |
+
2022-08-10 15:11:33,155 INFO [train.py:737] (0/3) Epoch 12, batch 5800, loss[loss=0.4497, over 13644.00 frames; utt_duration=959.1 frames, utt_pad_proportion=0.07604, over 57.00 utterances.], tot_loss[loss=0.4484, over 2821174.35 frames; utt_duration=1201 frames, utt_pad_proportion=0.0753, over 9408.97 utterances.], batch size: 57, lr: 3.99e-04
|
| 202 |
+
2022-08-10 15:40:12,810 INFO [train.py:737] (0/3) Epoch 12, batch 5900, loss[loss=0.4384, over 14552.00 frames; utt_duration=1457 frames, utt_pad_proportion=0.007425, over 40.00 utterances.], tot_loss[loss=0.4472, over 2816481.08 frames; utt_duration=1193 frames, utt_pad_proportion=0.07939, over 9451.92 utterances.], batch size: 40, lr: 3.98e-04
|
| 203 |
+
2022-08-10 16:07:46,683 INFO [train.py:737] (0/3) Epoch 12, batch 6000, loss[loss=0.4337, over 14742.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.005928, over 43.00 utterances.], tot_loss[loss=0.4452, over 2825284.74 frames; utt_duration=1206 frames, utt_pad_proportion=0.07295, over 9384.51 utterances.], batch size: 43, lr: 3.98e-04
|
| 204 |
+
2022-08-10 16:07:46,683 INFO [train.py:761] (0/3) Computing validation loss
|
| 205 |
+
2022-08-10 16:20:31,487 INFO [train.py:770] (0/3) Epoch 12, validation: loss=0.2954, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 206 |
+
2022-08-10 16:47:18,865 INFO [train.py:737] (0/3) Epoch 12, batch 6100, loss[loss=0.5719, over 10047.00 frames; utt_duration=438.1 frames, utt_pad_proportion=0.3281, over 92.00 utterances.], tot_loss[loss=0.4493, over 2769123.96 frames; utt_duration=1111 frames, utt_pad_proportion=0.1048, over 9979.29 utterances.], batch size: 92, lr: 3.98e-04
|
| 207 |
+
2022-08-10 16:54:07,613 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-12.pt
|
| 208 |
+
2022-08-10 17:14:26,425 INFO [train.py:737] (0/3) Epoch 13, batch 0, loss[loss=0.4239, over 14762.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.005151, over 41.00 utterances.], tot_loss[loss=0.4239, over 14762.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.005151, over 41.00 utterances.], batch size: 41, lr: 3.85e-04
|
exp/log/log-train-2022-08-09-16-45-14-1
ADDED
|
@@ -0,0 +1,158 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2022-08-09 16:45:14,168 INFO [train.py:805] (1/3) Training started
|
| 2 |
+
2022-08-09 16:45:14,168 INFO [train.py:806] (1/3) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'env_info': {'k2-version': '1.17', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '7dcabf85e8bf06984c4abab0400ef1322b5ff3df', 'k2-git-date': 'Tue Aug 2 21:22:39 2022', 'lhotse-version': '1.4.0.dev+git.59318ed.clean', 'torch-version': '1.11.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.3', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '6af45c3-dirty', 'icefall-git-date': 'Tue Aug 2 14:47:09 2022', 'icefall-path': '/thspfs/home/acct-hpc/thspzh-hpc1/git/icefall_dev', 'k2-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/k2-1.17.dev20220805+cuda11.3.torch1.11.0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'dgxs-01.cluster.thspzh.com', 'IP address': '192.168.200.101'}, 'world_size': 3, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 20, 'start_epoch': 11, 'start_batch': 0, 'exp_dir': PosixPath('cnn_ctc/exp_phone'), 'lang_dir': 'data/lang_phone', 'bpe_model': 'data/lang_bpe_500/bpe.model', 'initial_lr': 0.003, 'lr_batches': 5000, 'lr_epochs': 6, 'seed': 42, 'print_diagnostics': False, 'save_every_n': 8000, 'keep_last_k': 20, 'average_period': 100, 'use_fp16': False, 'first_out_channels': 8, 'conv_blocks': 3, 'rnn_dim': 0, 'full_libri': True, 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 600, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'input_strategy': 'PrecomputedFeatures'}
|
| 3 |
+
2022-08-09 16:45:14,475 INFO [lexicon.py:176] (1/3) Loading pre-compiled data/lang_phone/Linv.pt
|
| 4 |
+
2022-08-09 16:45:28,307 INFO [train.py:843] (1/3) About to create model
|
| 5 |
+
2022-08-09 16:45:28,330 INFO [checkpoint.py:112] (1/3) Loading checkpoint from cnn_ctc/exp_phone/epoch-10.pt
|
| 6 |
+
2022-08-09 16:45:29,329 INFO [train.py:876] (1/3) Using DDP
|
| 7 |
+
2022-08-09 16:45:37,468 INFO [train.py:884] (1/3) Loading optimizer state dict
|
| 8 |
+
2022-08-09 16:45:37,519 INFO [train.py:892] (1/3) Loading scheduler state dict
|
| 9 |
+
2022-08-09 16:45:37,520 INFO [asr_datamodule.py:409] (1/3) About to get train-clean-100 cuts
|
| 10 |
+
2022-08-09 16:45:37,624 INFO [asr_datamodule.py:416] (1/3) About to get train-clean-360 cuts
|
| 11 |
+
2022-08-09 16:45:37,630 INFO [asr_datamodule.py:423] (1/3) About to get train-other-500 cuts
|
| 12 |
+
2022-08-09 16:45:37,723 INFO [asr_datamodule.py:225] (1/3) Enable MUSAN
|
| 13 |
+
2022-08-09 16:45:37,723 INFO [asr_datamodule.py:226] (1/3) About to get Musan cuts
|
| 14 |
+
2022-08-09 16:45:39,743 INFO [asr_datamodule.py:254] (1/3) Enable SpecAugment
|
| 15 |
+
2022-08-09 16:45:39,743 INFO [asr_datamodule.py:255] (1/3) Time warp factor: 80
|
| 16 |
+
2022-08-09 16:45:39,743 INFO [asr_datamodule.py:267] (1/3) Num frame mask: 10
|
| 17 |
+
2022-08-09 16:45:39,743 INFO [asr_datamodule.py:280] (1/3) About to create train dataset
|
| 18 |
+
2022-08-09 16:45:39,743 INFO [asr_datamodule.py:309] (1/3) Using DynamicBucketingSampler.
|
| 19 |
+
2022-08-09 16:45:42,083 INFO [asr_datamodule.py:324] (1/3) About to create train dataloader
|
| 20 |
+
2022-08-09 16:45:42,109 INFO [asr_datamodule.py:430] (1/3) About to get dev-clean cuts
|
| 21 |
+
2022-08-09 16:45:42,137 INFO [asr_datamodule.py:437] (1/3) About to get dev-other cuts
|
| 22 |
+
2022-08-09 16:45:42,160 INFO [asr_datamodule.py:355] (1/3) About to create dev dataset
|
| 23 |
+
2022-08-09 16:45:42,496 INFO [asr_datamodule.py:374] (1/3) About to create dev dataloader
|
| 24 |
+
2022-08-09 16:45:42,497 INFO [train.py:957] (1/3) Loading grad scaler state dict
|
| 25 |
+
2022-08-09 16:46:43,615 INFO [train.py:737] (1/3) Epoch 11, batch 0, loss[loss=0.4721, over 12951.00 frames; utt_duration=775 frames, utt_pad_proportion=0.1273, over 67.00 utterances.], tot_loss[loss=0.4721, over 12951.00 frames; utt_duration=775 frames, utt_pad_proportion=0.1273, over 67.00 utterances.], batch size: 67, lr: 4.33e-04
|
| 26 |
+
2022-08-09 16:46:43,651 INFO [distributed.py:948] (1/3) Reducer buckets have been rebuilt in this iteration.
|
| 27 |
+
2022-08-09 16:49:50,756 INFO [train.py:737] (1/3) Epoch 11, batch 100, loss[loss=0.3181, over 14091.00 frames; utt_duration=1819 frames, utt_pad_proportion=0.03426, over 31.00 utterances.], tot_loss[loss=0.3765, over 1152037.92 frames; utt_duration=1366 frames, utt_pad_proportion=0.0259, over 3377.52 utterances.], batch size: 31, lr: 4.32e-04
|
| 28 |
+
2022-08-09 16:53:45,380 INFO [train.py:737] (1/3) Epoch 11, batch 200, loss[loss=0.3661, over 14566.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.00639, over 37.00 utterances.], tot_loss[loss=0.3673, over 1837228.75 frames; utt_duration=1361 frames, utt_pad_proportion=0.0291, over 5404.36 utterances.], batch size: 37, lr: 4.32e-04
|
| 29 |
+
2022-08-09 16:57:48,955 INFO [train.py:737] (1/3) Epoch 11, batch 300, loss[loss=0.4304, over 14539.00 frames; utt_duration=1213 frames, utt_pad_proportion=0.01541, over 48.00 utterances.], tot_loss[loss=0.3677, over 2242116.05 frames; utt_duration=1321 frames, utt_pad_proportion=0.03864, over 6797.91 utterances.], batch size: 48, lr: 4.32e-04
|
| 30 |
+
2022-08-09 17:01:14,227 INFO [train.py:737] (1/3) Epoch 11, batch 400, loss[loss=0.4106, over 13771.00 frames; utt_duration=967.9 frames, utt_pad_proportion=0.06661, over 57.00 utterances.], tot_loss[loss=0.3673, over 2482391.01 frames; utt_duration=1295 frames, utt_pad_proportion=0.04547, over 7676.04 utterances.], batch size: 57, lr: 4.32e-04
|
| 31 |
+
2022-08-09 17:06:23,850 INFO [train.py:737] (1/3) Epoch 11, batch 500, loss[loss=0.3946, over 12596.00 frames; utt_duration=753.3 frames, utt_pad_proportion=0.1507, over 67.00 utterances.], tot_loss[loss=0.3658, over 2623037.51 frames; utt_duration=1274 frames, utt_pad_proportion=0.0544, over 8246.77 utterances.], batch size: 67, lr: 4.32e-04
|
| 32 |
+
2022-08-09 17:24:40,905 INFO [train.py:737] (1/3) Epoch 11, batch 600, loss[loss=0.4021, over 13024.00 frames; utt_duration=779.1 frames, utt_pad_proportion=0.1196, over 67.00 utterances.], tot_loss[loss=0.3669, over 2710875.27 frames; utt_duration=1268 frames, utt_pad_proportion=0.05625, over 8562.98 utterances.], batch size: 67, lr: 4.31e-04
|
| 33 |
+
2022-08-09 17:31:07,970 INFO [train.py:737] (1/3) Epoch 11, batch 700, loss[loss=0.3673, over 14594.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.006647, over 38.00 utterances.], tot_loss[loss=0.3681, over 2776722.03 frames; utt_duration=1289 frames, utt_pad_proportion=0.04973, over 8625.87 utterances.], batch size: 38, lr: 4.31e-04
|
| 34 |
+
2022-08-09 17:38:02,126 INFO [train.py:737] (1/3) Epoch 11, batch 800, loss[loss=0.3679, over 14635.00 frames; utt_duration=1247 frames, utt_pad_proportion=0.01342, over 47.00 utterances.], tot_loss[loss=0.3691, over 2811320.32 frames; utt_duration=1290 frames, utt_pad_proportion=0.04886, over 8729.73 utterances.], batch size: 47, lr: 4.31e-04
|
| 35 |
+
2022-08-09 17:51:06,850 INFO [train.py:737] (1/3) Epoch 11, batch 900, loss[loss=0.4396, over 14511.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.01747, over 48.00 utterances.], tot_loss[loss=0.369, over 2835401.38 frames; utt_duration=1301 frames, utt_pad_proportion=0.0465, over 8729.20 utterances.], batch size: 48, lr: 4.31e-04
|
| 36 |
+
2022-08-09 18:11:34,573 INFO [train.py:737] (1/3) Epoch 11, batch 1000, loss[loss=0.3481, over 14472.00 frames; utt_duration=1207 frames, utt_pad_proportion=0.02012, over 48.00 utterances.], tot_loss[loss=0.3687, over 2842353.82 frames; utt_duration=1285 frames, utt_pad_proportion=0.05041, over 8855.39 utterances.], batch size: 48, lr: 4.31e-04
|
| 37 |
+
2022-08-09 18:30:17,061 INFO [train.py:737] (1/3) Epoch 11, batch 1100, loss[loss=0.3113, over 14620.00 frames; utt_duration=1626 frames, utt_pad_proportion=0.006316, over 36.00 utterances.], tot_loss[loss=0.3698, over 2852979.22 frames; utt_duration=1291 frames, utt_pad_proportion=0.0489, over 8849.19 utterances.], batch size: 36, lr: 4.31e-04
|
| 38 |
+
2022-08-09 18:45:42,693 INFO [train.py:737] (1/3) Epoch 11, batch 1200, loss[loss=0.4435, over 14527.00 frames; utt_duration=1212 frames, utt_pad_proportion=0.01618, over 48.00 utterances.], tot_loss[loss=0.3702, over 2863969.29 frames; utt_duration=1300 frames, utt_pad_proportion=0.04611, over 8822.43 utterances.], batch size: 48, lr: 4.30e-04
|
| 39 |
+
2022-08-09 19:00:31,129 INFO [train.py:737] (1/3) Epoch 11, batch 1300, loss[loss=0.4138, over 14868.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.006691, over 44.00 utterances.], tot_loss[loss=0.3715, over 2869277.83 frames; utt_duration=1296 frames, utt_pad_proportion=0.04328, over 8864.76 utterances.], batch size: 44, lr: 4.30e-04
|
| 40 |
+
2022-08-09 19:12:14,042 INFO [train.py:737] (1/3) Epoch 11, batch 1400, loss[loss=0.3602, over 14797.00 frames; utt_duration=1559 frames, utt_pad_proportion=0.004487, over 38.00 utterances.], tot_loss[loss=0.3708, over 2871102.92 frames; utt_duration=1298 frames, utt_pad_proportion=0.04425, over 8860.15 utterances.], batch size: 38, lr: 4.30e-04
|
| 41 |
+
2022-08-09 19:24:33,673 INFO [train.py:737] (1/3) Epoch 11, batch 1500, loss[loss=0.3682, over 14582.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.007344, over 38.00 utterances.], tot_loss[loss=0.3713, over 2867252.80 frames; utt_duration=1288 frames, utt_pad_proportion=0.048, over 8911.86 utterances.], batch size: 38, lr: 4.30e-04
|
| 42 |
+
2022-08-09 19:36:04,609 INFO [train.py:737] (1/3) Epoch 11, batch 1600, loss[loss=0.3436, over 14580.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.006078, over 39.00 utterances.], tot_loss[loss=0.3698, over 2855523.89 frames; utt_duration=1264 frames, utt_pad_proportion=0.05409, over 9050.34 utterances.], batch size: 39, lr: 4.30e-04
|
| 43 |
+
2022-08-09 19:45:58,873 INFO [train.py:737] (1/3) Epoch 11, batch 1700, loss[loss=0.4373, over 14473.00 frames; utt_duration=1094 frames, utt_pad_proportion=0.02777, over 53.00 utterances.], tot_loss[loss=0.371, over 2865901.90 frames; utt_duration=1287 frames, utt_pad_proportion=0.04788, over 8918.29 utterances.], batch size: 53, lr: 4.30e-04
|
| 44 |
+
2022-08-09 19:55:10,363 INFO [train.py:737] (1/3) Epoch 11, batch 1800, loss[loss=0.3627, over 14879.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.006553, over 44.00 utterances.], tot_loss[loss=0.3681, over 2872530.39 frames; utt_duration=1308 frames, utt_pad_proportion=0.04434, over 8796.58 utterances.], batch size: 44, lr: 4.29e-04
|
| 45 |
+
2022-08-09 20:04:21,775 INFO [train.py:737] (1/3) Epoch 11, batch 1900, loss[loss=0.3217, over 14767.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.005835, over 38.00 utterances.], tot_loss[loss=0.3677, over 2855636.83 frames; utt_duration=1270 frames, utt_pad_proportion=0.05424, over 9002.38 utterances.], batch size: 38, lr: 4.29e-04
|
| 46 |
+
2022-08-09 20:14:15,006 INFO [train.py:737] (1/3) Epoch 11, batch 2000, loss[loss=0.3861, over 14682.00 frames; utt_duration=1251 frames, utt_pad_proportion=0.0111, over 47.00 utterances.], tot_loss[loss=0.3713, over 2852401.51 frames; utt_duration=1257 frames, utt_pad_proportion=0.05618, over 9085.67 utterances.], batch size: 47, lr: 4.29e-04
|
| 47 |
+
2022-08-09 20:25:06,834 INFO [train.py:737] (1/3) Epoch 11, batch 2100, loss[loss=0.3964, over 13834.00 frames; utt_duration=972.1 frames, utt_pad_proportion=0.06437, over 57.00 utterances.], tot_loss[loss=0.371, over 2848205.38 frames; utt_duration=1245 frames, utt_pad_proportion=0.05828, over 9162.76 utterances.], batch size: 57, lr: 4.29e-04
|
| 48 |
+
2022-08-09 20:35:11,572 INFO [train.py:737] (1/3) Epoch 11, batch 2200, loss[loss=0.3349, over 14430.00 frames; utt_duration=1650 frames, utt_pad_proportion=0.009907, over 35.00 utterances.], tot_loss[loss=0.3712, over 2840678.55 frames; utt_duration=1241 frames, utt_pad_proportion=0.06297, over 9166.08 utterances.], batch size: 35, lr: 4.29e-04
|
| 49 |
+
2022-08-09 20:43:57,459 INFO [train.py:737] (1/3) Epoch 11, batch 2300, loss[loss=0.2736, over 14479.00 frames; utt_duration=1757 frames, utt_pad_proportion=0.009771, over 33.00 utterances.], tot_loss[loss=0.37, over 2860704.47 frames; utt_duration=1284 frames, utt_pad_proportion=0.05173, over 8922.42 utterances.], batch size: 33, lr: 4.29e-04
|
| 50 |
+
2022-08-09 20:54:17,081 INFO [train.py:737] (1/3) Epoch 11, batch 2400, loss[loss=0.3078, over 14601.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.006171, over 38.00 utterances.], tot_loss[loss=0.3679, over 2854977.27 frames; utt_duration=1272 frames, utt_pad_proportion=0.05506, over 8989.90 utterances.], batch size: 38, lr: 4.28e-04
|
| 51 |
+
2022-08-09 21:04:37,869 INFO [train.py:737] (1/3) Epoch 11, batch 2500, loss[loss=0.3552, over 14602.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.009352, over 34.00 utterances.], tot_loss[loss=0.3695, over 2852077.55 frames; utt_duration=1248 frames, utt_pad_proportion=0.05607, over 9154.20 utterances.], batch size: 34, lr: 4.28e-04
|
| 52 |
+
2022-08-09 21:14:14,370 INFO [train.py:737] (1/3) Epoch 11, batch 2600, loss[loss=0.3711, over 14857.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.007687, over 44.00 utterances.], tot_loss[loss=0.3687, over 2850260.68 frames; utt_duration=1254 frames, utt_pad_proportion=0.05771, over 9102.01 utterances.], batch size: 44, lr: 4.28e-04
|
| 53 |
+
2022-08-09 21:22:52,904 INFO [train.py:737] (1/3) Epoch 11, batch 2700, loss[loss=0.3101, over 14422.00 frames; utt_duration=1804 frames, utt_pad_proportion=0.031, over 32.00 utterances.], tot_loss[loss=0.3675, over 2847871.60 frames; utt_duration=1253 frames, utt_pad_proportion=0.05957, over 9100.86 utterances.], batch size: 32, lr: 4.28e-04
|
| 54 |
+
2022-08-09 21:29:45,369 INFO [train.py:737] (1/3) Epoch 11, batch 2800, loss[loss=0.3155, over 14834.00 frames; utt_duration=1650 frames, utt_pad_proportion=0.008597, over 36.00 utterances.], tot_loss[loss=0.3682, over 2856146.53 frames; utt_duration=1269 frames, utt_pad_proportion=0.05347, over 9016.82 utterances.], batch size: 36, lr: 4.28e-04
|
| 55 |
+
2022-08-09 21:38:58,795 INFO [train.py:737] (1/3) Epoch 11, batch 2900, loss[loss=0.4106, over 14565.00 frames; utt_duration=1535 frames, utt_pad_proportion=0.007336, over 38.00 utterances.], tot_loss[loss=0.3819, over 2833702.52 frames; utt_duration=1236 frames, utt_pad_proportion=0.06925, over 9184.37 utterances.], batch size: 38, lr: 4.28e-04
|
| 56 |
+
2022-08-09 21:47:56,448 INFO [train.py:737] (1/3) Epoch 11, batch 3000, loss[loss=0.4523, over 14776.00 frames; utt_duration=1409 frames, utt_pad_proportion=0.005229, over 42.00 utterances.], tot_loss[loss=0.4029, over 2843967.53 frames; utt_duration=1254 frames, utt_pad_proportion=0.06134, over 9082.70 utterances.], batch size: 42, lr: 4.27e-04
|
| 57 |
+
2022-08-09 21:47:56,448 INFO [train.py:761] (1/3) Computing validation loss
|
| 58 |
+
2022-08-09 21:50:31,852 INFO [train.py:770] (1/3) Epoch 11, validation: loss=0.2999, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 59 |
+
2022-08-09 21:58:13,829 INFO [train.py:737] (1/3) Epoch 11, batch 3100, loss[loss=0.4217, over 14591.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005668, over 41.00 utterances.], tot_loss[loss=0.4161, over 2837662.64 frames; utt_duration=1243 frames, utt_pad_proportion=0.06476, over 9139.12 utterances.], batch size: 41, lr: 4.27e-04
|
| 60 |
+
2022-08-09 22:06:18,344 INFO [train.py:737] (1/3) Epoch 11, batch 3200, loss[loss=0.4331, over 10815.00 frames; utt_duration=477 frames, utt_pad_proportion=0.2695, over 91.00 utterances.], tot_loss[loss=0.4263, over 2817231.10 frames; utt_duration=1197 frames, utt_pad_proportion=0.07767, over 9426.83 utterances.], batch size: 91, lr: 4.27e-04
|
| 61 |
+
2022-08-09 22:15:31,409 INFO [train.py:737] (1/3) Epoch 11, batch 3300, loss[loss=0.4442, over 14767.00 frames; utt_duration=1478 frames, utt_pad_proportion=0.005918, over 40.00 utterances.], tot_loss[loss=0.4347, over 2824033.83 frames; utt_duration=1211 frames, utt_pad_proportion=0.07505, over 9336.68 utterances.], batch size: 40, lr: 4.27e-04
|
| 62 |
+
2022-08-09 22:23:12,453 INFO [train.py:737] (1/3) Epoch 11, batch 3400, loss[loss=0.3622, over 14738.00 frames; utt_duration=1686 frames, utt_pad_proportion=0.008835, over 35.00 utterances.], tot_loss[loss=0.4389, over 2833898.82 frames; utt_duration=1225 frames, utt_pad_proportion=0.06963, over 9261.33 utterances.], batch size: 35, lr: 4.27e-04
|
| 63 |
+
2022-08-09 22:31:24,029 INFO [train.py:737] (1/3) Epoch 11, batch 3500, loss[loss=0.4041, over 14717.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.01053, over 35.00 utterances.], tot_loss[loss=0.4416, over 2835493.10 frames; utt_duration=1226 frames, utt_pad_proportion=0.06698, over 9259.93 utterances.], batch size: 35, lr: 4.27e-04
|
| 64 |
+
2022-08-09 22:40:34,203 INFO [train.py:737] (1/3) Epoch 11, batch 3600, loss[loss=0.4763, over 14755.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005504, over 41.00 utterances.], tot_loss[loss=0.4459, over 2824300.17 frames; utt_duration=1209 frames, utt_pad_proportion=0.07323, over 9356.43 utterances.], batch size: 41, lr: 4.26e-04
|
| 65 |
+
2022-08-09 22:48:24,926 INFO [train.py:737] (1/3) Epoch 11, batch 3700, loss[loss=0.4508, over 12901.00 frames; utt_duration=771.9 frames, utt_pad_proportion=0.1298, over 67.00 utterances.], tot_loss[loss=0.447, over 2827313.59 frames; utt_duration=1222 frames, utt_pad_proportion=0.07144, over 9262.40 utterances.], batch size: 67, lr: 4.26e-04
|
| 66 |
+
2022-08-09 22:55:40,794 INFO [train.py:737] (1/3) Epoch 11, batch 3800, loss[loss=0.4022, over 14517.00 frames; utt_duration=1211 frames, utt_pad_proportion=0.01677, over 48.00 utterances.], tot_loss[loss=0.4503, over 2805924.78 frames; utt_duration=1186 frames, utt_pad_proportion=0.08511, over 9474.97 utterances.], batch size: 48, lr: 4.26e-04
|
| 67 |
+
2022-08-09 23:03:07,340 INFO [train.py:737] (1/3) Epoch 11, batch 3900, loss[loss=0.4733, over 14438.00 frames; utt_duration=1157 frames, utt_pad_proportion=0.02539, over 50.00 utterances.], tot_loss[loss=0.4515, over 2813139.05 frames; utt_duration=1188 frames, utt_pad_proportion=0.08011, over 9483.23 utterances.], batch size: 50, lr: 4.26e-04
|
| 68 |
+
2022-08-09 23:09:27,269 INFO [train.py:737] (1/3) Epoch 11, batch 4000, loss[loss=0.4575, over 14748.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005824, over 41.00 utterances.], tot_loss[loss=0.4528, over 2816843.89 frames; utt_duration=1191 frames, utt_pad_proportion=0.07711, over 9471.12 utterances.], batch size: 41, lr: 4.26e-04
|
| 69 |
+
2022-08-09 23:16:24,246 INFO [train.py:737] (1/3) Epoch 11, batch 4100, loss[loss=0.3586, over 14761.00 frames; utt_duration=1597 frames, utt_pad_proportion=0.007424, over 37.00 utterances.], tot_loss[loss=0.4533, over 2823173.02 frames; utt_duration=1198 frames, utt_pad_proportion=0.07298, over 9434.22 utterances.], batch size: 37, lr: 4.26e-04
|
| 70 |
+
2022-08-09 23:23:18,413 INFO [train.py:737] (1/3) Epoch 11, batch 4200, loss[loss=0.5294, over 9985.00 frames; utt_duration=440.3 frames, utt_pad_proportion=0.3257, over 91.00 utterances.], tot_loss[loss=0.4511, over 2819277.87 frames; utt_duration=1195 frames, utt_pad_proportion=0.07708, over 9447.09 utterances.], batch size: 91, lr: 4.25e-04
|
| 71 |
+
2022-08-09 23:32:50,143 INFO [train.py:737] (1/3) Epoch 11, batch 4300, loss[loss=0.4713, over 12804.00 frames; utt_duration=765.9 frames, utt_pad_proportion=0.1365, over 67.00 utterances.], tot_loss[loss=0.4512, over 2804814.04 frames; utt_duration=1177 frames, utt_pad_proportion=0.08552, over 9543.26 utterances.], batch size: 67, lr: 4.25e-04
|
| 72 |
+
2022-08-09 23:43:42,059 INFO [train.py:737] (1/3) Epoch 11, batch 4400, loss[loss=0.3914, over 14577.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.00608, over 37.00 utterances.], tot_loss[loss=0.4502, over 2819977.62 frames; utt_duration=1197 frames, utt_pad_proportion=0.0762, over 9433.21 utterances.], batch size: 37, lr: 4.25e-04
|
| 73 |
+
2022-08-09 23:55:06,951 INFO [train.py:737] (1/3) Epoch 11, batch 4500, loss[loss=0.5383, over 9681.00 frames; utt_duration=422.5 frames, utt_pad_proportion=0.348, over 92.00 utterances.], tot_loss[loss=0.4504, over 2791354.39 frames; utt_duration=1156 frames, utt_pad_proportion=0.09244, over 9667.79 utterances.], batch size: 92, lr: 4.25e-04
|
| 74 |
+
2022-08-10 00:03:22,382 INFO [train.py:737] (1/3) Epoch 11, batch 4600, loss[loss=0.4482, over 14712.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.01065, over 35.00 utterances.], tot_loss[loss=0.4487, over 2810131.56 frames; utt_duration=1188 frames, utt_pad_proportion=0.08191, over 9476.16 utterances.], batch size: 35, lr: 4.25e-04
|
| 75 |
+
2022-08-10 00:11:18,338 INFO [train.py:737] (1/3) Epoch 11, batch 4700, loss[loss=0.4355, over 14499.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.01743, over 48.00 utterances.], tot_loss[loss=0.4479, over 2807655.29 frames; utt_duration=1181 frames, utt_pad_proportion=0.08268, over 9523.88 utterances.], batch size: 48, lr: 4.25e-04
|
| 76 |
+
2022-08-10 00:19:24,287 INFO [train.py:737] (1/3) Epoch 11, batch 4800, loss[loss=0.4678, over 14774.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.007034, over 37.00 utterances.], tot_loss[loss=0.4469, over 2821169.84 frames; utt_duration=1202 frames, utt_pad_proportion=0.07542, over 9397.40 utterances.], batch size: 37, lr: 4.24e-04
|
| 77 |
+
2022-08-10 00:31:46,381 INFO [train.py:737] (1/3) Epoch 11, batch 4900, loss[loss=0.5239, over 9410.00 frames; utt_duration=415.1 frames, utt_pad_proportion=0.3643, over 91.00 utterances.], tot_loss[loss=0.4455, over 2813366.46 frames; utt_duration=1198 frames, utt_pad_proportion=0.07982, over 9402.23 utterances.], batch size: 91, lr: 4.24e-04
|
| 78 |
+
2022-08-10 00:40:42,901 INFO [train.py:737] (1/3) Epoch 11, batch 5000, loss[loss=0.3399, over 14448.00 frames; utt_duration=1653 frames, utt_pad_proportion=0.008518, over 35.00 utterances.], tot_loss[loss=0.4438, over 2832486.21 frames; utt_duration=1225 frames, utt_pad_proportion=0.06865, over 9259.99 utterances.], batch size: 35, lr: 4.24e-04
|
| 79 |
+
2022-08-10 00:48:51,915 INFO [train.py:737] (1/3) Epoch 11, batch 5100, loss[loss=0.4125, over 14581.00 frames; utt_duration=1460 frames, utt_pad_proportion=0.005688, over 40.00 utterances.], tot_loss[loss=0.4467, over 2828809.78 frames; utt_duration=1215 frames, utt_pad_proportion=0.07042, over 9326.74 utterances.], batch size: 40, lr: 4.24e-04
|
| 80 |
+
2022-08-10 00:56:58,916 INFO [train.py:737] (1/3) Epoch 11, batch 5200, loss[loss=0.4764, over 14762.00 frames; utt_duration=1555 frames, utt_pad_proportion=0.006755, over 38.00 utterances.], tot_loss[loss=0.447, over 2834401.28 frames; utt_duration=1231 frames, utt_pad_proportion=0.06658, over 9218.86 utterances.], batch size: 38, lr: 4.24e-04
|
| 81 |
+
2022-08-10 01:03:13,744 INFO [train.py:737] (1/3) Epoch 11, batch 5300, loss[loss=0.4055, over 14761.00 frames; utt_duration=1478 frames, utt_pad_proportion=0.006355, over 40.00 utterances.], tot_loss[loss=0.447, over 2821257.69 frames; utt_duration=1209 frames, utt_pad_proportion=0.07428, over 9348.51 utterances.], batch size: 40, lr: 4.24e-04
|
| 82 |
+
2022-08-10 01:11:10,722 INFO [train.py:737] (1/3) Epoch 11, batch 5400, loss[loss=0.3777, over 14051.00 frames; utt_duration=1814 frames, utt_pad_proportion=0.04355, over 31.00 utterances.], tot_loss[loss=0.443, over 2841384.44 frames; utt_duration=1248 frames, utt_pad_proportion=0.06298, over 9119.32 utterances.], batch size: 31, lr: 4.23e-04
|
| 83 |
+
2022-08-10 01:19:04,627 INFO [train.py:737] (1/3) Epoch 11, batch 5500, loss[loss=0.4061, over 14618.00 frames; utt_duration=1330 frames, utt_pad_proportion=0.009341, over 44.00 utterances.], tot_loss[loss=0.4452, over 2844016.92 frames; utt_duration=1256 frames, utt_pad_proportion=0.06158, over 9071.05 utterances.], batch size: 44, lr: 4.23e-04
|
| 84 |
+
2022-08-10 01:27:52,230 INFO [train.py:737] (1/3) Epoch 11, batch 5600, loss[loss=0.5586, over 14702.00 frames; utt_duration=1309 frames, utt_pad_proportion=0.008636, over 45.00 utterances.], tot_loss[loss=0.4474, over 2839291.96 frames; utt_duration=1236 frames, utt_pad_proportion=0.06442, over 9200.84 utterances.], batch size: 45, lr: 4.23e-04
|
| 85 |
+
2022-08-10 01:37:26,308 INFO [train.py:737] (1/3) Epoch 11, batch 5700, loss[loss=0.3664, over 14633.00 frames; utt_duration=1723 frames, utt_pad_proportion=0.008868, over 34.00 utterances.], tot_loss[loss=0.4452, over 2831890.19 frames; utt_duration=1223 frames, utt_pad_proportion=0.07037, over 9275.50 utterances.], batch size: 34, lr: 4.23e-04
|
| 86 |
+
2022-08-10 01:44:25,844 INFO [train.py:737] (1/3) Epoch 11, batch 5800, loss[loss=0.4964, over 9907.00 frames; utt_duration=432 frames, utt_pad_proportion=0.3374, over 92.00 utterances.], tot_loss[loss=0.4438, over 2842618.99 frames; utt_duration=1249 frames, utt_pad_proportion=0.06485, over 9117.27 utterances.], batch size: 92, lr: 4.23e-04
|
| 87 |
+
2022-08-10 01:54:30,355 INFO [train.py:737] (1/3) Epoch 11, batch 5900, loss[loss=0.4534, over 14591.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.004871, over 41.00 utterances.], tot_loss[loss=0.4456, over 2830246.04 frames; utt_duration=1222 frames, utt_pad_proportion=0.07057, over 9272.29 utterances.], batch size: 41, lr: 4.23e-04
|
| 88 |
+
2022-08-10 02:02:21,713 INFO [train.py:737] (1/3) Epoch 11, batch 6000, loss[loss=0.4861, over 12579.00 frames; utt_duration=752.7 frames, utt_pad_proportion=0.1486, over 67.00 utterances.], tot_loss[loss=0.4473, over 2832593.15 frames; utt_duration=1217 frames, utt_pad_proportion=0.06837, over 9320.22 utterances.], batch size: 67, lr: 4.22e-04
|
| 89 |
+
2022-08-10 02:02:21,713 INFO [train.py:761] (1/3) Computing validation loss
|
| 90 |
+
2022-08-10 02:03:17,903 INFO [train.py:770] (1/3) Epoch 11, validation: loss=0.3454, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 91 |
+
2022-08-10 02:09:36,991 INFO [train.py:737] (1/3) Epoch 11, batch 6100, loss[loss=0.5523, over 9539.00 frames; utt_duration=416.3 frames, utt_pad_proportion=0.3605, over 92.00 utterances.], tot_loss[loss=0.4527, over 2765421.39 frames; utt_duration=1107 frames, utt_pad_proportion=0.1066, over 10009.34 utterances.], batch size: 92, lr: 4.22e-04
|
| 92 |
+
2022-08-10 02:19:32,341 INFO [train.py:737] (1/3) Epoch 12, batch 0, loss[loss=0.3663, over 14596.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01108, over 34.00 utterances.], tot_loss[loss=0.3663, over 14596.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01108, over 34.00 utterances.], batch size: 34, lr: 4.07e-04
|
| 93 |
+
2022-08-10 02:29:04,933 INFO [train.py:737] (1/3) Epoch 12, batch 100, loss[loss=0.4063, over 14877.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.006486, over 44.00 utterances.], tot_loss[loss=0.3746, over 1133310.89 frames; utt_duration=1240 frames, utt_pad_proportion=0.05208, over 3661.19 utterances.], batch size: 44, lr: 4.07e-04
|
| 94 |
+
2022-08-10 02:38:10,101 INFO [train.py:737] (1/3) Epoch 12, batch 200, loss[loss=0.4205, over 14738.00 frames; utt_duration=1439 frames, utt_pad_proportion=0.006834, over 41.00 utterances.], tot_loss[loss=0.3684, over 1810534.56 frames; utt_duration=1260 frames, utt_pad_proportion=0.05533, over 5752.40 utterances.], batch size: 41, lr: 4.07e-04
|
| 95 |
+
2022-08-10 02:50:17,051 INFO [train.py:737] (1/3) Epoch 12, batch 300, loss[loss=0.3856, over 14630.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.008546, over 44.00 utterances.], tot_loss[loss=0.3669, over 2227845.68 frames; utt_duration=1273 frames, utt_pad_proportion=0.05128, over 7007.56 utterances.], batch size: 44, lr: 4.07e-04
|
| 96 |
+
2022-08-10 02:59:35,400 INFO [train.py:737] (1/3) Epoch 12, batch 400, loss[loss=0.3389, over 14850.00 frames; utt_duration=1652 frames, utt_pad_proportion=0.009148, over 36.00 utterances.], tot_loss[loss=0.3625, over 2482965.04 frames; utt_duration=1287 frames, utt_pad_proportion=0.04814, over 7725.56 utterances.], batch size: 36, lr: 4.07e-04
|
| 97 |
+
2022-08-10 03:10:03,633 INFO [train.py:737] (1/3) Epoch 12, batch 500, loss[loss=0.3335, over 14437.00 frames; utt_duration=1651 frames, utt_pad_proportion=0.009324, over 35.00 utterances.], tot_loss[loss=0.3618, over 2641747.29 frames; utt_duration=1301 frames, utt_pad_proportion=0.04247, over 8134.66 utterances.], batch size: 35, lr: 4.07e-04
|
| 98 |
+
2022-08-10 03:22:54,032 INFO [train.py:737] (1/3) Epoch 12, batch 600, loss[loss=0.3095, over 14571.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.005913, over 37.00 utterances.], tot_loss[loss=0.3632, over 2735563.62 frames; utt_duration=1301 frames, utt_pad_proportion=0.04172, over 8418.25 utterances.], batch size: 37, lr: 4.06e-04
|
| 99 |
+
2022-08-10 03:34:44,750 INFO [train.py:737] (1/3) Epoch 12, batch 700, loss[loss=0.3733, over 14490.00 frames; utt_duration=1161 frames, utt_pad_proportion=0.02116, over 50.00 utterances.], tot_loss[loss=0.3624, over 2793464.02 frames; utt_duration=1314 frames, utt_pad_proportion=0.04056, over 8516.49 utterances.], batch size: 50, lr: 4.06e-04
|
| 100 |
+
2022-08-10 03:46:11,194 INFO [train.py:737] (1/3) Epoch 12, batch 800, loss[loss=0.3028, over 14603.00 frames; utt_duration=1624 frames, utt_pad_proportion=0.007352, over 36.00 utterances.], tot_loss[loss=0.3674, over 2828351.72 frames; utt_duration=1310 frames, utt_pad_proportion=0.03895, over 8649.37 utterances.], batch size: 36, lr: 4.06e-04
|
| 101 |
+
2022-08-10 03:56:27,269 INFO [train.py:737] (1/3) Epoch 12, batch 900, loss[loss=0.36, over 14781.00 frames; utt_duration=1409 frames, utt_pad_proportion=0.004691, over 42.00 utterances.], tot_loss[loss=0.3681, over 2839787.72 frames; utt_duration=1293 frames, utt_pad_proportion=0.04586, over 8793.60 utterances.], batch size: 42, lr: 4.06e-04
|
| 102 |
+
2022-08-10 04:06:59,877 INFO [train.py:737] (1/3) Epoch 12, batch 1000, loss[loss=0.3006, over 14717.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.01091, over 35.00 utterances.], tot_loss[loss=0.3673, over 2854561.28 frames; utt_duration=1297 frames, utt_pad_proportion=0.04396, over 8811.18 utterances.], batch size: 35, lr: 4.06e-04
|
| 103 |
+
2022-08-10 04:16:28,351 INFO [train.py:737] (1/3) Epoch 12, batch 1100, loss[loss=0.3876, over 14686.00 frames; utt_duration=1279 frames, utt_pad_proportion=0.01272, over 46.00 utterances.], tot_loss[loss=0.3662, over 2854249.57 frames; utt_duration=1278 frames, utt_pad_proportion=0.04777, over 8941.78 utterances.], batch size: 46, lr: 4.06e-04
|
| 104 |
+
2022-08-10 04:27:14,725 INFO [train.py:737] (1/3) Epoch 12, batch 1200, loss[loss=0.3606, over 14767.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006385, over 39.00 utterances.], tot_loss[loss=0.3673, over 2867999.50 frames; utt_duration=1311 frames, utt_pad_proportion=0.04152, over 8757.85 utterances.], batch size: 39, lr: 4.05e-04
|
| 105 |
+
2022-08-10 04:37:22,609 INFO [train.py:737] (1/3) Epoch 12, batch 1300, loss[loss=0.3308, over 14835.00 frames; utt_duration=1650 frames, utt_pad_proportion=0.009487, over 36.00 utterances.], tot_loss[loss=0.3661, over 2853951.61 frames; utt_duration=1285 frames, utt_pad_proportion=0.05229, over 8891.30 utterances.], batch size: 36, lr: 4.05e-04
|
| 106 |
+
2022-08-10 04:46:39,487 INFO [train.py:737] (1/3) Epoch 12, batch 1400, loss[loss=0.3638, over 14607.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.009906, over 34.00 utterances.], tot_loss[loss=0.3667, over 2848149.92 frames; utt_duration=1274 frames, utt_pad_proportion=0.05776, over 8951.43 utterances.], batch size: 34, lr: 4.05e-04
|
| 107 |
+
2022-08-10 04:57:14,243 INFO [train.py:737] (1/3) Epoch 12, batch 1500, loss[loss=0.4289, over 10557.00 frames; utt_duration=455.8 frames, utt_pad_proportion=0.2879, over 93.00 utterances.], tot_loss[loss=0.3671, over 2845670.95 frames; utt_duration=1266 frames, utt_pad_proportion=0.06069, over 9004.32 utterances.], batch size: 93, lr: 4.05e-04
|
| 108 |
+
2022-08-10 05:09:29,811 INFO [train.py:737] (1/3) Epoch 12, batch 1600, loss[loss=0.3924, over 14760.00 frames; utt_duration=1375 frames, utt_pad_proportion=0.005419, over 43.00 utterances.], tot_loss[loss=0.3678, over 2848296.16 frames; utt_duration=1262 frames, utt_pad_proportion=0.05875, over 9040.48 utterances.], batch size: 43, lr: 4.05e-04
|
| 109 |
+
2022-08-10 05:20:38,269 INFO [train.py:737] (1/3) Epoch 12, batch 1700, loss[loss=0.3698, over 13782.00 frames; utt_duration=968.6 frames, utt_pad_proportion=0.06774, over 57.00 utterances.], tot_loss[loss=0.368, over 2861429.30 frames; utt_duration=1278 frames, utt_pad_proportion=0.05151, over 8966.43 utterances.], batch size: 57, lr: 4.05e-04
|
| 110 |
+
2022-08-10 05:31:51,599 INFO [train.py:737] (1/3) Epoch 12, batch 1800, loss[loss=0.4239, over 12815.00 frames; utt_duration=766.5 frames, utt_pad_proportion=0.132, over 67.00 utterances.], tot_loss[loss=0.3695, over 2846319.05 frames; utt_duration=1241 frames, utt_pad_proportion=0.06108, over 9182.22 utterances.], batch size: 67, lr: 4.05e-04
|
| 111 |
+
2022-08-10 05:42:58,981 INFO [train.py:737] (1/3) Epoch 12, batch 1900, loss[loss=0.4091, over 12939.00 frames; utt_duration=774.1 frames, utt_pad_proportion=0.1263, over 67.00 utterances.], tot_loss[loss=0.3678, over 2848054.65 frames; utt_duration=1256 frames, utt_pad_proportion=0.06058, over 9084.54 utterances.], batch size: 67, lr: 4.04e-04
|
| 112 |
+
2022-08-10 05:52:47,052 INFO [train.py:737] (1/3) Epoch 12, batch 2000, loss[loss=0.3236, over 14569.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.006642, over 37.00 utterances.], tot_loss[loss=0.3671, over 2848285.31 frames; utt_duration=1249 frames, utt_pad_proportion=0.05937, over 9134.72 utterances.], batch size: 37, lr: 4.04e-04
|
| 113 |
+
2022-08-10 06:03:23,203 INFO [train.py:737] (1/3) Epoch 12, batch 2100, loss[loss=0.3607, over 14702.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.008754, over 45.00 utterances.], tot_loss[loss=0.3661, over 2856997.61 frames; utt_duration=1267 frames, utt_pad_proportion=0.05335, over 9031.67 utterances.], batch size: 45, lr: 4.04e-04
|
| 114 |
+
2022-08-10 06:13:20,123 INFO [train.py:737] (1/3) Epoch 12, batch 2200, loss[loss=0.4412, over 9927.00 frames; utt_duration=432.8 frames, utt_pad_proportion=0.3351, over 92.00 utterances.], tot_loss[loss=0.3642, over 2851537.76 frames; utt_duration=1273 frames, utt_pad_proportion=0.05718, over 8970.37 utterances.], batch size: 92, lr: 4.04e-04
|
| 115 |
+
2022-08-10 06:24:03,209 INFO [train.py:737] (1/3) Epoch 12, batch 2300, loss[loss=0.3864, over 14769.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.004812, over 42.00 utterances.], tot_loss[loss=0.3664, over 2853304.07 frames; utt_duration=1272 frames, utt_pad_proportion=0.05476, over 8985.39 utterances.], batch size: 42, lr: 4.04e-04
|
| 116 |
+
2022-08-10 06:36:03,152 INFO [train.py:737] (1/3) Epoch 12, batch 2400, loss[loss=0.4545, over 12915.00 frames; utt_duration=772.6 frames, utt_pad_proportion=0.129, over 67.00 utterances.], tot_loss[loss=0.3654, over 2853099.43 frames; utt_duration=1272 frames, utt_pad_proportion=0.05612, over 8984.51 utterances.], batch size: 67, lr: 4.04e-04
|
| 117 |
+
2022-08-10 06:48:05,001 INFO [train.py:737] (1/3) Epoch 12, batch 2500, loss[loss=0.3452, over 14589.00 frames; utt_duration=1718 frames, utt_pad_proportion=0.01045, over 34.00 utterances.], tot_loss[loss=0.3694, over 2841067.57 frames; utt_duration=1238 frames, utt_pad_proportion=0.06361, over 9194.17 utterances.], batch size: 34, lr: 4.03e-04
|
| 118 |
+
2022-08-10 07:00:47,848 INFO [train.py:737] (1/3) Epoch 12, batch 2600, loss[loss=0.3661, over 14658.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01308, over 47.00 utterances.], tot_loss[loss=0.3687, over 2823038.15 frames; utt_duration=1202 frames, utt_pad_proportion=0.07542, over 9402.88 utterances.], batch size: 47, lr: 4.03e-04
|
| 119 |
+
2022-08-10 07:12:58,139 INFO [train.py:737] (1/3) Epoch 12, batch 2700, loss[loss=0.3231, over 14587.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005889, over 41.00 utterances.], tot_loss[loss=0.3669, over 2840694.28 frames; utt_duration=1234 frames, utt_pad_proportion=0.06368, over 9219.57 utterances.], batch size: 41, lr: 4.03e-04
|
| 120 |
+
2022-08-10 07:21:23,111 INFO [train.py:737] (1/3) Epoch 12, batch 2800, loss[loss=0.402, over 14555.00 frames; utt_duration=1457 frames, utt_pad_proportion=0.006851, over 40.00 utterances.], tot_loss[loss=0.3671, over 2838593.70 frames; utt_duration=1241 frames, utt_pad_proportion=0.06562, over 9161.87 utterances.], batch size: 40, lr: 4.03e-04
|
| 121 |
+
2022-08-10 07:30:52,940 INFO [train.py:737] (1/3) Epoch 12, batch 2900, loss[loss=0.4057, over 14579.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.005569, over 42.00 utterances.], tot_loss[loss=0.3789, over 2833186.66 frames; utt_duration=1238 frames, utt_pad_proportion=0.06807, over 9168.41 utterances.], batch size: 42, lr: 4.03e-04
|
| 122 |
+
2022-08-10 07:40:15,528 INFO [train.py:737] (1/3) Epoch 12, batch 3000, loss[loss=0.4134, over 14587.00 frames; utt_duration=1578 frames, utt_pad_proportion=0.005501, over 37.00 utterances.], tot_loss[loss=0.3972, over 2837953.29 frames; utt_duration=1244 frames, utt_pad_proportion=0.06613, over 9139.23 utterances.], batch size: 37, lr: 4.03e-04
|
| 123 |
+
2022-08-10 07:40:15,528 INFO [train.py:761] (1/3) Computing validation loss
|
| 124 |
+
2022-08-10 07:43:16,495 INFO [train.py:770] (1/3) Epoch 12, validation: loss=0.297, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 125 |
+
2022-08-10 07:53:07,546 INFO [train.py:737] (1/3) Epoch 12, batch 3100, loss[loss=0.4362, over 14786.00 frames; utt_duration=1518 frames, utt_pad_proportion=0.005192, over 39.00 utterances.], tot_loss[loss=0.4124, over 2854227.91 frames; utt_duration=1270 frames, utt_pad_proportion=0.05661, over 9002.14 utterances.], batch size: 39, lr: 4.03e-04
|
| 126 |
+
2022-08-10 08:03:53,494 INFO [train.py:737] (1/3) Epoch 12, batch 3200, loss[loss=0.4077, over 14092.00 frames; utt_duration=1820 frames, utt_pad_proportion=0.04407, over 31.00 utterances.], tot_loss[loss=0.423, over 2823538.31 frames; utt_duration=1209 frames, utt_pad_proportion=0.07402, over 9350.64 utterances.], batch size: 31, lr: 4.02e-04
|
| 127 |
+
2022-08-10 08:15:52,838 INFO [train.py:737] (1/3) Epoch 12, batch 3300, loss[loss=0.4397, over 14457.00 frames; utt_duration=1158 frames, utt_pad_proportion=0.02438, over 50.00 utterances.], tot_loss[loss=0.4292, over 2836500.68 frames; utt_duration=1242 frames, utt_pad_proportion=0.06541, over 9149.19 utterances.], batch size: 50, lr: 4.02e-04
|
| 128 |
+
2022-08-10 08:24:14,594 INFO [train.py:737] (1/3) Epoch 12, batch 3400, loss[loss=0.4313, over 14519.00 frames; utt_duration=1211 frames, utt_pad_proportion=0.01679, over 48.00 utterances.], tot_loss[loss=0.4378, over 2806758.85 frames; utt_duration=1180 frames, utt_pad_proportion=0.0821, over 9526.38 utterances.], batch size: 48, lr: 4.02e-04
|
| 129 |
+
2022-08-10 08:32:25,017 INFO [train.py:737] (1/3) Epoch 12, batch 3500, loss[loss=0.5324, over 14625.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.008834, over 44.00 utterances.], tot_loss[loss=0.4389, over 2828635.78 frames; utt_duration=1219 frames, utt_pad_proportion=0.06973, over 9296.68 utterances.], batch size: 44, lr: 4.02e-04
|
| 130 |
+
2022-08-10 08:43:20,505 INFO [train.py:737] (1/3) Epoch 12, batch 3600, loss[loss=0.4058, over 14590.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005651, over 41.00 utterances.], tot_loss[loss=0.4433, over 2807848.30 frames; utt_duration=1184 frames, utt_pad_proportion=0.08113, over 9499.41 utterances.], batch size: 41, lr: 4.02e-04
|
| 131 |
+
2022-08-10 08:54:44,669 INFO [train.py:737] (1/3) Epoch 12, batch 3700, loss[loss=0.3847, over 14750.00 frames; utt_duration=1476 frames, utt_pad_proportion=0.006393, over 40.00 utterances.], tot_loss[loss=0.4452, over 2819220.09 frames; utt_duration=1200 frames, utt_pad_proportion=0.0742, over 9406.70 utterances.], batch size: 40, lr: 4.02e-04
|
| 132 |
+
2022-08-10 09:03:23,103 INFO [train.py:737] (1/3) Epoch 12, batch 3800, loss[loss=0.4954, over 14716.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.009996, over 46.00 utterances.], tot_loss[loss=0.4441, over 2834548.26 frames; utt_duration=1232 frames, utt_pad_proportion=0.06672, over 9210.95 utterances.], batch size: 46, lr: 4.01e-04
|
| 133 |
+
2022-08-10 09:12:39,487 INFO [train.py:737] (1/3) Epoch 12, batch 3900, loss[loss=0.5334, over 14757.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005403, over 41.00 utterances.], tot_loss[loss=0.4448, over 2820402.48 frames; utt_duration=1205 frames, utt_pad_proportion=0.07498, over 9373.33 utterances.], batch size: 41, lr: 4.01e-04
|
| 134 |
+
2022-08-10 09:22:00,878 INFO [train.py:737] (1/3) Epoch 12, batch 4000, loss[loss=0.5627, over 9756.00 frames; utt_duration=421.2 frames, utt_pad_proportion=0.344, over 93.00 utterances.], tot_loss[loss=0.4447, over 2819544.38 frames; utt_duration=1215 frames, utt_pad_proportion=0.07605, over 9296.41 utterances.], batch size: 93, lr: 4.01e-04
|
| 135 |
+
2022-08-10 09:32:08,726 INFO [train.py:737] (1/3) Epoch 12, batch 4100, loss[loss=0.5008, over 12704.00 frames; utt_duration=760.1 frames, utt_pad_proportion=0.1421, over 67.00 utterances.], tot_loss[loss=0.4468, over 2834287.31 frames; utt_duration=1229 frames, utt_pad_proportion=0.06777, over 9235.44 utterances.], batch size: 67, lr: 4.01e-04
|
| 136 |
+
2022-08-10 09:41:30,766 INFO [train.py:737] (1/3) Epoch 12, batch 4200, loss[loss=0.506, over 14481.00 frames; utt_duration=1160 frames, utt_pad_proportion=0.0236, over 50.00 utterances.], tot_loss[loss=0.4451, over 2840492.41 frames; utt_duration=1249 frames, utt_pad_proportion=0.06305, over 9107.29 utterances.], batch size: 50, lr: 4.01e-04
|
| 137 |
+
2022-08-10 09:51:28,127 INFO [train.py:737] (1/3) Epoch 12, batch 4300, loss[loss=0.5309, over 14430.00 frames; utt_duration=1156 frames, utt_pad_proportion=0.02531, over 50.00 utterances.], tot_loss[loss=0.448, over 2811042.78 frames; utt_duration=1190 frames, utt_pad_proportion=0.08044, over 9460.38 utterances.], batch size: 50, lr: 4.01e-04
|
| 138 |
+
2022-08-10 10:02:08,397 INFO [train.py:737] (1/3) Epoch 12, batch 4400, loss[loss=0.5097, over 14549.00 frames; utt_duration=1456 frames, utt_pad_proportion=0.007936, over 40.00 utterances.], tot_loss[loss=0.4506, over 2798724.83 frames; utt_duration=1167 frames, utt_pad_proportion=0.08738, over 9606.82 utterances.], batch size: 40, lr: 4.01e-04
|
| 139 |
+
2022-08-10 10:12:01,452 INFO [train.py:737] (1/3) Epoch 12, batch 4500, loss[loss=0.4076, over 14623.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.00902, over 44.00 utterances.], tot_loss[loss=0.4458, over 2816400.34 frames; utt_duration=1199 frames, utt_pad_proportion=0.07647, over 9410.75 utterances.], batch size: 44, lr: 4.00e-04
|
| 140 |
+
2022-08-10 10:22:19,558 INFO [train.py:737] (1/3) Epoch 12, batch 4600, loss[loss=0.3952, over 14608.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.009855, over 34.00 utterances.], tot_loss[loss=0.4458, over 2834788.68 frames; utt_duration=1224 frames, utt_pad_proportion=0.06595, over 9275.96 utterances.], batch size: 34, lr: 4.00e-04
|
| 141 |
+
2022-08-10 10:30:47,462 INFO [train.py:737] (1/3) Epoch 12, batch 4700, loss[loss=0.4155, over 14731.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.006635, over 43.00 utterances.], tot_loss[loss=0.4453, over 2840777.41 frames; utt_duration=1236 frames, utt_pad_proportion=0.06197, over 9207.50 utterances.], batch size: 43, lr: 4.00e-04
|
| 142 |
+
2022-08-10 10:43:44,061 INFO [train.py:737] (1/3) Epoch 12, batch 4800, loss[loss=0.4365, over 14592.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.0056, over 41.00 utterances.], tot_loss[loss=0.4437, over 2842012.46 frames; utt_duration=1241 frames, utt_pad_proportion=0.06262, over 9174.12 utterances.], batch size: 41, lr: 4.00e-04
|
| 143 |
+
2022-08-10 10:56:42,545 INFO [train.py:737] (1/3) Epoch 12, batch 4900, loss[loss=0.5221, over 14296.00 frames; utt_duration=1080 frames, utt_pad_proportion=0.0397, over 53.00 utterances.], tot_loss[loss=0.4408, over 2862368.71 frames; utt_duration=1273 frames, utt_pad_proportion=0.04968, over 9008.00 utterances.], batch size: 53, lr: 4.00e-04
|
| 144 |
+
2022-08-10 11:12:32,188 INFO [train.py:737] (1/3) Epoch 12, batch 5000, loss[loss=0.434, over 14760.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005184, over 41.00 utterances.], tot_loss[loss=0.4436, over 2828702.78 frames; utt_duration=1212 frames, utt_pad_proportion=0.0712, over 9343.67 utterances.], batch size: 41, lr: 4.00e-04
|
| 145 |
+
2022-08-10 11:47:05,509 INFO [train.py:737] (1/3) Epoch 12, batch 5100, loss[loss=0.5088, over 9393.00 frames; utt_duration=405.6 frames, utt_pad_proportion=0.3682, over 93.00 utterances.], tot_loss[loss=0.4435, over 2833920.71 frames; utt_duration=1221 frames, utt_pad_proportion=0.06894, over 9295.49 utterances.], batch size: 93, lr: 4.00e-04
|
| 146 |
+
2022-08-10 12:13:52,156 INFO [train.py:737] (1/3) Epoch 12, batch 5200, loss[loss=0.4292, over 14771.00 frames; utt_duration=1479 frames, utt_pad_proportion=0.005346, over 40.00 utterances.], tot_loss[loss=0.443, over 2838731.66 frames; utt_duration=1239 frames, utt_pad_proportion=0.06605, over 9177.37 utterances.], batch size: 40, lr: 3.99e-04
|
| 147 |
+
2022-08-10 12:46:45,459 INFO [train.py:737] (1/3) Epoch 12, batch 5300, loss[loss=0.479, over 14614.00 frames; utt_duration=1330 frames, utt_pad_proportion=0.008203, over 44.00 utterances.], tot_loss[loss=0.4436, over 2839055.58 frames; utt_duration=1235 frames, utt_pad_proportion=0.06559, over 9202.74 utterances.], batch size: 44, lr: 3.99e-04
|
| 148 |
+
2022-08-10 13:18:04,044 INFO [train.py:737] (1/3) Epoch 12, batch 5400, loss[loss=0.3848, over 14015.00 frames; utt_duration=1810 frames, utt_pad_proportion=0.03795, over 31.00 utterances.], tot_loss[loss=0.445, over 2832847.33 frames; utt_duration=1219 frames, utt_pad_proportion=0.06915, over 9303.97 utterances.], batch size: 31, lr: 3.99e-04
|
| 149 |
+
2022-08-10 13:50:20,064 INFO [train.py:737] (1/3) Epoch 12, batch 5500, loss[loss=0.5262, over 9187.00 frames; utt_duration=392.4 frames, utt_pad_proportion=0.385, over 94.00 utterances.], tot_loss[loss=0.4457, over 2827241.18 frames; utt_duration=1218 frames, utt_pad_proportion=0.07241, over 9299.96 utterances.], batch size: 94, lr: 3.99e-04
|
| 150 |
+
2022-08-10 14:17:39,806 INFO [train.py:737] (1/3) Epoch 12, batch 5600, loss[loss=0.3637, over 14756.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.005755, over 42.00 utterances.], tot_loss[loss=0.4455, over 2841665.09 frames; utt_duration=1237 frames, utt_pad_proportion=0.06314, over 9198.27 utterances.], batch size: 42, lr: 3.99e-04
|
| 151 |
+
2022-08-10 14:45:43,326 INFO [train.py:737] (1/3) Epoch 12, batch 5700, loss[loss=0.3954, over 14575.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006012, over 37.00 utterances.], tot_loss[loss=0.4462, over 2806939.07 frames; utt_duration=1175 frames, utt_pad_proportion=0.08399, over 9568.12 utterances.], batch size: 37, lr: 3.99e-04
|
| 152 |
+
2022-08-10 15:11:33,155 INFO [train.py:737] (1/3) Epoch 12, batch 5800, loss[loss=0.3987, over 14700.00 frames; utt_duration=1681 frames, utt_pad_proportion=0.01106, over 35.00 utterances.], tot_loss[loss=0.4449, over 2814664.21 frames; utt_duration=1194 frames, utt_pad_proportion=0.07953, over 9440.06 utterances.], batch size: 35, lr: 3.99e-04
|
| 153 |
+
2022-08-10 15:40:12,810 INFO [train.py:737] (1/3) Epoch 12, batch 5900, loss[loss=0.4362, over 14678.00 frames; utt_duration=1251 frames, utt_pad_proportion=0.0121, over 47.00 utterances.], tot_loss[loss=0.4435, over 2818531.71 frames; utt_duration=1200 frames, utt_pad_proportion=0.07707, over 9404.19 utterances.], batch size: 47, lr: 3.98e-04
|
| 154 |
+
2022-08-10 16:07:46,683 INFO [train.py:737] (1/3) Epoch 12, batch 6000, loss[loss=0.4118, over 14475.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.01021, over 33.00 utterances.], tot_loss[loss=0.4442, over 2812705.96 frames; utt_duration=1200 frames, utt_pad_proportion=0.08021, over 9390.56 utterances.], batch size: 33, lr: 3.98e-04
|
| 155 |
+
2022-08-10 16:07:46,683 INFO [train.py:761] (1/3) Computing validation loss
|
| 156 |
+
2022-08-10 16:20:31,490 INFO [train.py:770] (1/3) Epoch 12, validation: loss=0.2954, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 157 |
+
2022-08-10 16:47:18,865 INFO [train.py:737] (1/3) Epoch 12, batch 6100, loss[loss=0.5231, over 9916.00 frames; utt_duration=432.5 frames, utt_pad_proportion=0.3356, over 92.00 utterances.], tot_loss[loss=0.4493, over 2753285.17 frames; utt_duration=1105 frames, utt_pad_proportion=0.1141, over 9975.92 utterances.], batch size: 92, lr: 3.98e-04
|
| 158 |
+
2022-08-10 17:14:26,426 INFO [train.py:737] (1/3) Epoch 13, batch 0, loss[loss=0.4348, over 14678.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.009671, over 45.00 utterances.], tot_loss[loss=0.4348, over 14678.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.009671, over 45.00 utterances.], batch size: 45, lr: 3.85e-04
|
exp/log/log-train-2022-08-09-16-45-14-2
ADDED
|
@@ -0,0 +1,158 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2022-08-09 16:45:14,147 INFO [train.py:805] (2/3) Training started
|
| 2 |
+
2022-08-09 16:45:14,147 INFO [train.py:806] (2/3) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'env_info': {'k2-version': '1.17', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '7dcabf85e8bf06984c4abab0400ef1322b5ff3df', 'k2-git-date': 'Tue Aug 2 21:22:39 2022', 'lhotse-version': '1.4.0.dev+git.59318ed.clean', 'torch-version': '1.11.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.3', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '6af45c3-dirty', 'icefall-git-date': 'Tue Aug 2 14:47:09 2022', 'icefall-path': '/thspfs/home/acct-hpc/thspzh-hpc1/git/icefall_dev', 'k2-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/k2-1.17.dev20220805+cuda11.3.torch1.11.0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'dgxs-01.cluster.thspzh.com', 'IP address': '192.168.200.101'}, 'world_size': 3, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 20, 'start_epoch': 11, 'start_batch': 0, 'exp_dir': PosixPath('cnn_ctc/exp_phone'), 'lang_dir': 'data/lang_phone', 'bpe_model': 'data/lang_bpe_500/bpe.model', 'initial_lr': 0.003, 'lr_batches': 5000, 'lr_epochs': 6, 'seed': 42, 'print_diagnostics': False, 'save_every_n': 8000, 'keep_last_k': 20, 'average_period': 100, 'use_fp16': False, 'first_out_channels': 8, 'conv_blocks': 3, 'rnn_dim': 0, 'full_libri': True, 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 600, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'input_strategy': 'PrecomputedFeatures'}
|
| 3 |
+
2022-08-09 16:45:14,473 INFO [lexicon.py:176] (2/3) Loading pre-compiled data/lang_phone/Linv.pt
|
| 4 |
+
2022-08-09 16:45:28,307 INFO [train.py:843] (2/3) About to create model
|
| 5 |
+
2022-08-09 16:45:28,330 INFO [checkpoint.py:112] (2/3) Loading checkpoint from cnn_ctc/exp_phone/epoch-10.pt
|
| 6 |
+
2022-08-09 16:45:29,328 INFO [train.py:876] (2/3) Using DDP
|
| 7 |
+
2022-08-09 16:45:37,468 INFO [train.py:884] (2/3) Loading optimizer state dict
|
| 8 |
+
2022-08-09 16:45:37,518 INFO [train.py:892] (2/3) Loading scheduler state dict
|
| 9 |
+
2022-08-09 16:45:37,519 INFO [asr_datamodule.py:409] (2/3) About to get train-clean-100 cuts
|
| 10 |
+
2022-08-09 16:45:37,619 INFO [asr_datamodule.py:416] (2/3) About to get train-clean-360 cuts
|
| 11 |
+
2022-08-09 16:45:37,630 INFO [asr_datamodule.py:423] (2/3) About to get train-other-500 cuts
|
| 12 |
+
2022-08-09 16:45:37,723 INFO [asr_datamodule.py:225] (2/3) Enable MUSAN
|
| 13 |
+
2022-08-09 16:45:37,723 INFO [asr_datamodule.py:226] (2/3) About to get Musan cuts
|
| 14 |
+
2022-08-09 16:45:39,759 INFO [asr_datamodule.py:254] (2/3) Enable SpecAugment
|
| 15 |
+
2022-08-09 16:45:39,759 INFO [asr_datamodule.py:255] (2/3) Time warp factor: 80
|
| 16 |
+
2022-08-09 16:45:39,759 INFO [asr_datamodule.py:267] (2/3) Num frame mask: 10
|
| 17 |
+
2022-08-09 16:45:39,759 INFO [asr_datamodule.py:280] (2/3) About to create train dataset
|
| 18 |
+
2022-08-09 16:45:39,759 INFO [asr_datamodule.py:309] (2/3) Using DynamicBucketingSampler.
|
| 19 |
+
2022-08-09 16:45:42,130 INFO [asr_datamodule.py:324] (2/3) About to create train dataloader
|
| 20 |
+
2022-08-09 16:45:42,130 INFO [asr_datamodule.py:430] (2/3) About to get dev-clean cuts
|
| 21 |
+
2022-08-09 16:45:42,136 INFO [asr_datamodule.py:437] (2/3) About to get dev-other cuts
|
| 22 |
+
2022-08-09 16:45:42,154 INFO [asr_datamodule.py:355] (2/3) About to create dev dataset
|
| 23 |
+
2022-08-09 16:45:42,499 INFO [asr_datamodule.py:374] (2/3) About to create dev dataloader
|
| 24 |
+
2022-08-09 16:45:42,499 INFO [train.py:957] (2/3) Loading grad scaler state dict
|
| 25 |
+
2022-08-09 16:46:43,617 INFO [train.py:737] (2/3) Epoch 11, batch 0, loss[loss=0.4351, over 14776.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005297, over 42.00 utterances.], tot_loss[loss=0.4351, over 14776.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005297, over 42.00 utterances.], batch size: 42, lr: 4.33e-04
|
| 26 |
+
2022-08-09 16:46:43,658 INFO [distributed.py:948] (2/3) Reducer buckets have been rebuilt in this iteration.
|
| 27 |
+
2022-08-09 16:49:50,758 INFO [train.py:737] (2/3) Epoch 11, batch 100, loss[loss=0.3844, over 14575.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006518, over 42.00 utterances.], tot_loss[loss=0.3784, over 1143630.67 frames; utt_duration=1315 frames, utt_pad_proportion=0.03978, over 3481.91 utterances.], batch size: 42, lr: 4.32e-04
|
| 28 |
+
2022-08-09 16:53:45,381 INFO [train.py:737] (2/3) Epoch 11, batch 200, loss[loss=0.3957, over 14542.00 frames; utt_duration=1456 frames, utt_pad_proportion=0.007839, over 40.00 utterances.], tot_loss[loss=0.371, over 1828073.46 frames; utt_duration=1317 frames, utt_pad_proportion=0.03826, over 5559.44 utterances.], batch size: 40, lr: 4.32e-04
|
| 29 |
+
2022-08-09 16:57:48,956 INFO [train.py:737] (2/3) Epoch 11, batch 300, loss[loss=0.326, over 14054.00 frames; utt_duration=1815 frames, utt_pad_proportion=0.03695, over 31.00 utterances.], tot_loss[loss=0.3673, over 2241392.54 frames; utt_duration=1319 frames, utt_pad_proportion=0.03848, over 6806.82 utterances.], batch size: 31, lr: 4.32e-04
|
| 30 |
+
2022-08-09 17:01:14,229 INFO [train.py:737] (2/3) Epoch 11, batch 400, loss[loss=0.3892, over 14500.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.01801, over 48.00 utterances.], tot_loss[loss=0.3648, over 2489382.39 frames; utt_duration=1314 frames, utt_pad_proportion=0.04202, over 7588.13 utterances.], batch size: 48, lr: 4.32e-04
|
| 31 |
+
2022-08-09 17:06:23,851 INFO [train.py:737] (2/3) Epoch 11, batch 500, loss[loss=0.3489, over 14581.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006314, over 42.00 utterances.], tot_loss[loss=0.3656, over 2636089.87 frames; utt_duration=1299 frames, utt_pad_proportion=0.04665, over 8125.14 utterances.], batch size: 42, lr: 4.32e-04
|
| 32 |
+
2022-08-09 17:24:40,907 INFO [train.py:737] (2/3) Epoch 11, batch 600, loss[loss=0.3521, over 14774.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.006916, over 37.00 utterances.], tot_loss[loss=0.3644, over 2732203.56 frames; utt_duration=1317 frames, utt_pad_proportion=0.0422, over 8305.71 utterances.], batch size: 37, lr: 4.31e-04
|
| 33 |
+
2022-08-09 17:31:07,972 INFO [train.py:737] (2/3) Epoch 11, batch 700, loss[loss=0.4112, over 14736.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.006517, over 43.00 utterances.], tot_loss[loss=0.3686, over 2792876.97 frames; utt_duration=1323 frames, utt_pad_proportion=0.03941, over 8456.68 utterances.], batch size: 43, lr: 4.31e-04
|
| 34 |
+
2022-08-09 17:38:02,127 INFO [train.py:737] (2/3) Epoch 11, batch 800, loss[loss=0.3584, over 14758.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005504, over 41.00 utterances.], tot_loss[loss=0.3685, over 2824110.86 frames; utt_duration=1323 frames, utt_pad_proportion=0.04181, over 8548.31 utterances.], batch size: 41, lr: 4.31e-04
|
| 35 |
+
2022-08-09 17:51:06,853 INFO [train.py:737] (2/3) Epoch 11, batch 900, loss[loss=0.3472, over 14724.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.01058, over 35.00 utterances.], tot_loss[loss=0.37, over 2836633.50 frames; utt_duration=1297 frames, utt_pad_proportion=0.0487, over 8759.91 utterances.], batch size: 35, lr: 4.31e-04
|
| 36 |
+
2022-08-09 18:11:34,574 INFO [train.py:737] (2/3) Epoch 11, batch 1000, loss[loss=0.3221, over 14610.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.009804, over 34.00 utterances.], tot_loss[loss=0.3687, over 2841946.45 frames; utt_duration=1286 frames, utt_pad_proportion=0.05118, over 8851.50 utterances.], batch size: 34, lr: 4.31e-04
|
| 37 |
+
2022-08-09 18:30:17,063 INFO [train.py:737] (2/3) Epoch 11, batch 1100, loss[loss=0.4216, over 13123.00 frames; utt_duration=784.9 frames, utt_pad_proportion=0.1141, over 67.00 utterances.], tot_loss[loss=0.3709, over 2855091.13 frames; utt_duration=1294 frames, utt_pad_proportion=0.0484, over 8832.98 utterances.], batch size: 67, lr: 4.31e-04
|
| 38 |
+
2022-08-09 18:45:42,695 INFO [train.py:737] (2/3) Epoch 11, batch 1200, loss[loss=0.3664, over 13733.00 frames; utt_duration=965.2 frames, utt_pad_proportion=0.07099, over 57.00 utterances.], tot_loss[loss=0.3727, over 2851668.13 frames; utt_duration=1273 frames, utt_pad_proportion=0.05216, over 8972.84 utterances.], batch size: 57, lr: 4.30e-04
|
| 39 |
+
2022-08-09 19:00:31,131 INFO [train.py:737] (2/3) Epoch 11, batch 1300, loss[loss=0.3823, over 14563.00 frames; utt_duration=1215 frames, utt_pad_proportion=0.0136, over 48.00 utterances.], tot_loss[loss=0.3716, over 2859464.46 frames; utt_duration=1284 frames, utt_pad_proportion=0.04994, over 8917.24 utterances.], batch size: 48, lr: 4.30e-04
|
| 40 |
+
2022-08-09 19:12:14,043 INFO [train.py:737] (2/3) Epoch 11, batch 1400, loss[loss=0.3684, over 14780.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.004543, over 38.00 utterances.], tot_loss[loss=0.3705, over 2867349.44 frames; utt_duration=1299 frames, utt_pad_proportion=0.04518, over 8840.28 utterances.], batch size: 38, lr: 4.30e-04
|
| 41 |
+
2022-08-09 19:24:33,675 INFO [train.py:737] (2/3) Epoch 11, batch 1500, loss[loss=0.341, over 14051.00 frames; utt_duration=1814 frames, utt_pad_proportion=0.05419, over 31.00 utterances.], tot_loss[loss=0.368, over 2858513.41 frames; utt_duration=1284 frames, utt_pad_proportion=0.05257, over 8916.03 utterances.], batch size: 31, lr: 4.30e-04
|
| 42 |
+
2022-08-09 19:36:04,611 INFO [train.py:737] (2/3) Epoch 11, batch 1600, loss[loss=0.3061, over 14128.00 frames; utt_duration=1824 frames, utt_pad_proportion=0.04637, over 31.00 utterances.], tot_loss[loss=0.3663, over 2867265.73 frames; utt_duration=1300 frames, utt_pad_proportion=0.04587, over 8835.52 utterances.], batch size: 31, lr: 4.30e-04
|
| 43 |
+
2022-08-09 19:45:58,874 INFO [train.py:737] (2/3) Epoch 11, batch 1700, loss[loss=0.3976, over 14471.00 frames; utt_duration=1159 frames, utt_pad_proportion=0.02344, over 50.00 utterances.], tot_loss[loss=0.3684, over 2864857.01 frames; utt_duration=1280 frames, utt_pad_proportion=0.04789, over 8963.91 utterances.], batch size: 50, lr: 4.30e-04
|
| 44 |
+
2022-08-09 19:55:10,365 INFO [train.py:737] (2/3) Epoch 11, batch 1800, loss[loss=0.3946, over 14604.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01009, over 34.00 utterances.], tot_loss[loss=0.3718, over 2838716.16 frames; utt_duration=1224 frames, utt_pad_proportion=0.06402, over 9287.79 utterances.], batch size: 34, lr: 4.29e-04
|
| 45 |
+
2022-08-09 20:04:21,777 INFO [train.py:737] (2/3) Epoch 11, batch 1900, loss[loss=0.324, over 14768.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.005734, over 38.00 utterances.], tot_loss[loss=0.3713, over 2851273.43 frames; utt_duration=1259 frames, utt_pad_proportion=0.05646, over 9066.37 utterances.], batch size: 38, lr: 4.29e-04
|
| 46 |
+
2022-08-09 20:14:15,007 INFO [train.py:737] (2/3) Epoch 11, batch 2000, loss[loss=0.3544, over 14864.00 frames; utt_duration=1352 frames, utt_pad_proportion=0.00772, over 44.00 utterances.], tot_loss[loss=0.3682, over 2851885.99 frames; utt_duration=1268 frames, utt_pad_proportion=0.0565, over 9010.07 utterances.], batch size: 44, lr: 4.29e-04
|
| 47 |
+
2022-08-09 20:25:06,835 INFO [train.py:737] (2/3) Epoch 11, batch 2100, loss[loss=0.356, over 14583.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.005194, over 42.00 utterances.], tot_loss[loss=0.3694, over 2849919.40 frames; utt_duration=1257 frames, utt_pad_proportion=0.05735, over 9080.72 utterances.], batch size: 42, lr: 4.29e-04
|
| 48 |
+
2022-08-09 20:35:11,574 INFO [train.py:737] (2/3) Epoch 11, batch 2200, loss[loss=0.3106, over 14774.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.005364, over 38.00 utterances.], tot_loss[loss=0.3678, over 2866112.91 frames; utt_duration=1288 frames, utt_pad_proportion=0.04661, over 8911.25 utterances.], batch size: 38, lr: 4.29e-04
|
| 49 |
+
2022-08-09 20:43:57,461 INFO [train.py:737] (2/3) Epoch 11, batch 2300, loss[loss=0.3569, over 14662.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01324, over 47.00 utterances.], tot_loss[loss=0.3666, over 2864240.49 frames; utt_duration=1285 frames, utt_pad_proportion=0.04911, over 8925.42 utterances.], batch size: 47, lr: 4.29e-04
|
| 50 |
+
2022-08-09 20:54:17,082 INFO [train.py:737] (2/3) Epoch 11, batch 2400, loss[loss=0.4519, over 14684.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009848, over 45.00 utterances.], tot_loss[loss=0.3698, over 2849066.28 frames; utt_duration=1252 frames, utt_pad_proportion=0.05771, over 9111.85 utterances.], batch size: 45, lr: 4.28e-04
|
| 51 |
+
2022-08-09 21:04:37,871 INFO [train.py:737] (2/3) Epoch 11, batch 2500, loss[loss=0.3553, over 14884.00 frames; utt_duration=1355 frames, utt_pad_proportion=0.006003, over 44.00 utterances.], tot_loss[loss=0.367, over 2846737.57 frames; utt_duration=1259 frames, utt_pad_proportion=0.05999, over 9052.41 utterances.], batch size: 44, lr: 4.28e-04
|
| 52 |
+
2022-08-09 21:14:14,371 INFO [train.py:737] (2/3) Epoch 11, batch 2600, loss[loss=0.33, over 14748.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.006127, over 41.00 utterances.], tot_loss[loss=0.3664, over 2858249.34 frames; utt_duration=1274 frames, utt_pad_proportion=0.05357, over 8981.97 utterances.], batch size: 41, lr: 4.28e-04
|
| 53 |
+
2022-08-09 21:22:52,906 INFO [train.py:737] (2/3) Epoch 11, batch 2700, loss[loss=0.4067, over 14669.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.01103, over 47.00 utterances.], tot_loss[loss=0.3683, over 2851599.59 frames; utt_duration=1262 frames, utt_pad_proportion=0.05782, over 9047.45 utterances.], batch size: 47, lr: 4.28e-04
|
| 54 |
+
2022-08-09 21:29:45,370 INFO [train.py:737] (2/3) Epoch 11, batch 2800, loss[loss=0.422, over 14689.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.008997, over 45.00 utterances.], tot_loss[loss=0.3724, over 2849881.62 frames; utt_duration=1248 frames, utt_pad_proportion=0.05831, over 9146.62 utterances.], batch size: 45, lr: 4.28e-04
|
| 55 |
+
2022-08-09 21:38:58,798 INFO [train.py:737] (2/3) Epoch 11, batch 2900, loss[loss=0.3935, over 14434.00 frames; utt_duration=1651 frames, utt_pad_proportion=0.009667, over 35.00 utterances.], tot_loss[loss=0.3881, over 2850837.15 frames; utt_duration=1243 frames, utt_pad_proportion=0.05665, over 9187.21 utterances.], batch size: 35, lr: 4.28e-04
|
| 56 |
+
2022-08-09 21:47:56,450 INFO [train.py:737] (2/3) Epoch 11, batch 3000, loss[loss=0.4409, over 14533.00 frames; utt_duration=1213 frames, utt_pad_proportion=0.01493, over 48.00 utterances.], tot_loss[loss=0.4006, over 2843070.58 frames; utt_duration=1237 frames, utt_pad_proportion=0.06269, over 9208.00 utterances.], batch size: 48, lr: 4.27e-04
|
| 57 |
+
2022-08-09 21:47:56,450 INFO [train.py:761] (2/3) Computing validation loss
|
| 58 |
+
2022-08-09 21:50:31,852 INFO [train.py:770] (2/3) Epoch 11, validation: loss=0.2999, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 59 |
+
2022-08-09 21:58:13,830 INFO [train.py:737] (2/3) Epoch 11, batch 3100, loss[loss=0.4453, over 14500.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.01809, over 48.00 utterances.], tot_loss[loss=0.4136, over 2839257.48 frames; utt_duration=1225 frames, utt_pad_proportion=0.06544, over 9284.19 utterances.], batch size: 48, lr: 4.27e-04
|
| 60 |
+
2022-08-09 22:06:18,346 INFO [train.py:737] (2/3) Epoch 11, batch 3200, loss[loss=0.4256, over 14581.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.005367, over 39.00 utterances.], tot_loss[loss=0.4218, over 2839959.14 frames; utt_duration=1238 frames, utt_pad_proportion=0.06431, over 9188.72 utterances.], batch size: 39, lr: 4.27e-04
|
| 61 |
+
2022-08-09 22:15:31,411 INFO [train.py:737] (2/3) Epoch 11, batch 3300, loss[loss=0.5235, over 12851.00 frames; utt_duration=768.8 frames, utt_pad_proportion=0.1333, over 67.00 utterances.], tot_loss[loss=0.431, over 2837502.82 frames; utt_duration=1234 frames, utt_pad_proportion=0.06406, over 9209.03 utterances.], batch size: 67, lr: 4.27e-04
|
| 62 |
+
2022-08-09 22:23:12,455 INFO [train.py:737] (2/3) Epoch 11, batch 3400, loss[loss=0.4732, over 14626.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.008884, over 44.00 utterances.], tot_loss[loss=0.4387, over 2827371.98 frames; utt_duration=1214 frames, utt_pad_proportion=0.06964, over 9326.49 utterances.], batch size: 44, lr: 4.27e-04
|
| 63 |
+
2022-08-09 22:31:24,032 INFO [train.py:737] (2/3) Epoch 11, batch 3500, loss[loss=0.4131, over 13611.00 frames; utt_duration=1816 frames, utt_pad_proportion=0.07144, over 30.00 utterances.], tot_loss[loss=0.4405, over 2845636.82 frames; utt_duration=1249 frames, utt_pad_proportion=0.05926, over 9126.65 utterances.], batch size: 30, lr: 4.27e-04
|
| 64 |
+
2022-08-09 22:40:34,205 INFO [train.py:737] (2/3) Epoch 11, batch 3600, loss[loss=0.5347, over 9809.00 frames; utt_duration=428 frames, utt_pad_proportion=0.3416, over 92.00 utterances.], tot_loss[loss=0.4455, over 2835144.41 frames; utt_duration=1222 frames, utt_pad_proportion=0.06535, over 9291.07 utterances.], batch size: 92, lr: 4.26e-04
|
| 65 |
+
2022-08-09 22:48:24,927 INFO [train.py:737] (2/3) Epoch 11, batch 3700, loss[loss=0.4651, over 14864.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.008214, over 44.00 utterances.], tot_loss[loss=0.4491, over 2828896.07 frames; utt_duration=1210 frames, utt_pad_proportion=0.06919, over 9360.03 utterances.], batch size: 44, lr: 4.26e-04
|
| 66 |
+
2022-08-09 22:55:40,795 INFO [train.py:737] (2/3) Epoch 11, batch 3800, loss[loss=0.4967, over 14514.00 frames; utt_duration=1211 frames, utt_pad_proportion=0.0154, over 48.00 utterances.], tot_loss[loss=0.4516, over 2837871.65 frames; utt_duration=1223 frames, utt_pad_proportion=0.06478, over 9296.81 utterances.], batch size: 48, lr: 4.26e-04
|
| 67 |
+
2022-08-09 23:03:07,341 INFO [train.py:737] (2/3) Epoch 11, batch 3900, loss[loss=0.5565, over 14365.00 frames; utt_duration=1086 frames, utt_pad_proportion=0.03337, over 53.00 utterances.], tot_loss[loss=0.4489, over 2849302.30 frames; utt_duration=1250 frames, utt_pad_proportion=0.05844, over 9127.97 utterances.], batch size: 53, lr: 4.26e-04
|
| 68 |
+
2022-08-09 23:09:27,270 INFO [train.py:737] (2/3) Epoch 11, batch 4000, loss[loss=0.4394, over 14719.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.009944, over 35.00 utterances.], tot_loss[loss=0.4471, over 2830800.12 frames; utt_duration=1219 frames, utt_pad_proportion=0.07119, over 9297.79 utterances.], batch size: 35, lr: 4.26e-04
|
| 69 |
+
2022-08-09 23:16:24,248 INFO [train.py:737] (2/3) Epoch 11, batch 4100, loss[loss=0.4381, over 14338.00 frames; utt_duration=1084 frames, utt_pad_proportion=0.03666, over 53.00 utterances.], tot_loss[loss=0.4486, over 2822280.94 frames; utt_duration=1199 frames, utt_pad_proportion=0.07595, over 9426.67 utterances.], batch size: 53, lr: 4.26e-04
|
| 70 |
+
2022-08-09 23:23:18,415 INFO [train.py:737] (2/3) Epoch 11, batch 4200, loss[loss=0.3761, over 14585.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.005874, over 39.00 utterances.], tot_loss[loss=0.4482, over 2831003.52 frames; utt_duration=1216 frames, utt_pad_proportion=0.07068, over 9323.16 utterances.], batch size: 39, lr: 4.25e-04
|
| 71 |
+
2022-08-09 23:32:50,145 INFO [train.py:737] (2/3) Epoch 11, batch 4300, loss[loss=0.4322, over 14880.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.00637, over 44.00 utterances.], tot_loss[loss=0.4466, over 2828968.05 frames; utt_duration=1208 frames, utt_pad_proportion=0.07186, over 9375.59 utterances.], batch size: 44, lr: 4.25e-04
|
| 72 |
+
2022-08-09 23:43:42,061 INFO [train.py:737] (2/3) Epoch 11, batch 4400, loss[loss=0.4485, over 14575.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006757, over 42.00 utterances.], tot_loss[loss=0.4485, over 2834294.25 frames; utt_duration=1219 frames, utt_pad_proportion=0.06682, over 9311.67 utterances.], batch size: 42, lr: 4.25e-04
|
| 73 |
+
2022-08-09 23:55:06,952 INFO [train.py:737] (2/3) Epoch 11, batch 4500, loss[loss=0.5046, over 12766.00 frames; utt_duration=763.6 frames, utt_pad_proportion=0.1372, over 67.00 utterances.], tot_loss[loss=0.4474, over 2843883.84 frames; utt_duration=1241 frames, utt_pad_proportion=0.06107, over 9174.52 utterances.], batch size: 67, lr: 4.25e-04
|
| 74 |
+
2022-08-10 00:03:22,384 INFO [train.py:737] (2/3) Epoch 11, batch 4600, loss[loss=0.5018, over 14780.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.005461, over 38.00 utterances.], tot_loss[loss=0.4479, over 2827314.58 frames; utt_duration=1211 frames, utt_pad_proportion=0.07051, over 9350.68 utterances.], batch size: 38, lr: 4.25e-04
|
| 75 |
+
2022-08-10 00:11:18,339 INFO [train.py:737] (2/3) Epoch 11, batch 4700, loss[loss=0.4719, over 14580.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006314, over 42.00 utterances.], tot_loss[loss=0.4493, over 2846447.28 frames; utt_duration=1247 frames, utt_pad_proportion=0.05988, over 9144.90 utterances.], batch size: 42, lr: 4.25e-04
|
| 76 |
+
2022-08-10 00:19:24,289 INFO [train.py:737] (2/3) Epoch 11, batch 4800, loss[loss=0.3795, over 14610.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.008929, over 34.00 utterances.], tot_loss[loss=0.45, over 2829305.87 frames; utt_duration=1217 frames, utt_pad_proportion=0.07117, over 9308.03 utterances.], batch size: 34, lr: 4.24e-04
|
| 77 |
+
2022-08-10 00:31:46,383 INFO [train.py:737] (2/3) Epoch 11, batch 4900, loss[loss=0.4214, over 14759.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.00639, over 42.00 utterances.], tot_loss[loss=0.4488, over 2831420.84 frames; utt_duration=1220 frames, utt_pad_proportion=0.0696, over 9296.17 utterances.], batch size: 42, lr: 4.24e-04
|
| 78 |
+
2022-08-10 00:40:42,903 INFO [train.py:737] (2/3) Epoch 11, batch 5000, loss[loss=0.4945, over 14576.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006216, over 37.00 utterances.], tot_loss[loss=0.448, over 2824673.21 frames; utt_duration=1205 frames, utt_pad_proportion=0.07397, over 9386.14 utterances.], batch size: 37, lr: 4.24e-04
|
| 79 |
+
2022-08-10 00:48:51,917 INFO [train.py:737] (2/3) Epoch 11, batch 5100, loss[loss=0.4448, over 14574.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.005521, over 37.00 utterances.], tot_loss[loss=0.4477, over 2830193.10 frames; utt_duration=1221 frames, utt_pad_proportion=0.07001, over 9284.98 utterances.], batch size: 37, lr: 4.24e-04
|
| 80 |
+
2022-08-10 00:56:58,917 INFO [train.py:737] (2/3) Epoch 11, batch 5200, loss[loss=0.4809, over 14581.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.006417, over 41.00 utterances.], tot_loss[loss=0.4494, over 2823588.92 frames; utt_duration=1206 frames, utt_pad_proportion=0.07441, over 9373.23 utterances.], batch size: 41, lr: 4.24e-04
|
| 81 |
+
2022-08-10 01:03:13,746 INFO [train.py:737] (2/3) Epoch 11, batch 5300, loss[loss=0.4234, over 14780.00 frames; utt_duration=1479 frames, utt_pad_proportion=0.005111, over 40.00 utterances.], tot_loss[loss=0.451, over 2801413.34 frames; utt_duration=1159 frames, utt_pad_proportion=0.08455, over 9679.28 utterances.], batch size: 40, lr: 4.24e-04
|
| 82 |
+
2022-08-10 01:11:10,724 INFO [train.py:737] (2/3) Epoch 11, batch 5400, loss[loss=0.4337, over 14574.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.0063, over 39.00 utterances.], tot_loss[loss=0.45, over 2810668.79 frames; utt_duration=1177 frames, utt_pad_proportion=0.08032, over 9565.97 utterances.], batch size: 39, lr: 4.23e-04
|
| 83 |
+
2022-08-10 01:19:04,629 INFO [train.py:737] (2/3) Epoch 11, batch 5500, loss[loss=0.3879, over 14757.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.005151, over 41.00 utterances.], tot_loss[loss=0.4527, over 2821357.89 frames; utt_duration=1190 frames, utt_pad_proportion=0.07414, over 9497.48 utterances.], batch size: 41, lr: 4.23e-04
|
| 84 |
+
2022-08-10 01:27:52,232 INFO [train.py:737] (2/3) Epoch 11, batch 5600, loss[loss=0.4672, over 14574.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.006114, over 40.00 utterances.], tot_loss[loss=0.4511, over 2819758.89 frames; utt_duration=1196 frames, utt_pad_proportion=0.07529, over 9439.10 utterances.], batch size: 40, lr: 4.23e-04
|
| 85 |
+
2022-08-10 01:37:26,310 INFO [train.py:737] (2/3) Epoch 11, batch 5700, loss[loss=0.4354, over 14452.00 frames; utt_duration=1754 frames, utt_pad_proportion=0.0104, over 33.00 utterances.], tot_loss[loss=0.4473, over 2818535.02 frames; utt_duration=1199 frames, utt_pad_proportion=0.07545, over 9411.43 utterances.], batch size: 33, lr: 4.23e-04
|
| 86 |
+
2022-08-10 01:44:25,845 INFO [train.py:737] (2/3) Epoch 11, batch 5800, loss[loss=0.4319, over 14688.00 frames; utt_duration=1279 frames, utt_pad_proportion=0.01264, over 46.00 utterances.], tot_loss[loss=0.4459, over 2830633.36 frames; utt_duration=1216 frames, utt_pad_proportion=0.068, over 9324.16 utterances.], batch size: 46, lr: 4.23e-04
|
| 87 |
+
2022-08-10 01:54:30,357 INFO [train.py:737] (2/3) Epoch 11, batch 5900, loss[loss=0.4408, over 13154.00 frames; utt_duration=786.7 frames, utt_pad_proportion=0.1101, over 67.00 utterances.], tot_loss[loss=0.444, over 2828372.01 frames; utt_duration=1220 frames, utt_pad_proportion=0.07023, over 9287.14 utterances.], batch size: 67, lr: 4.23e-04
|
| 88 |
+
2022-08-10 02:02:21,714 INFO [train.py:737] (2/3) Epoch 11, batch 6000, loss[loss=0.4223, over 14571.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.006301, over 40.00 utterances.], tot_loss[loss=0.4428, over 2853774.18 frames; utt_duration=1267 frames, utt_pad_proportion=0.05489, over 9018.00 utterances.], batch size: 40, lr: 4.22e-04
|
| 89 |
+
2022-08-10 02:02:21,715 INFO [train.py:761] (2/3) Computing validation loss
|
| 90 |
+
2022-08-10 02:03:17,903 INFO [train.py:770] (2/3) Epoch 11, validation: loss=0.3454, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 91 |
+
2022-08-10 02:09:36,993 INFO [train.py:737] (2/3) Epoch 11, batch 6100, loss[loss=0.5179, over 9848.00 frames; utt_duration=429.7 frames, utt_pad_proportion=0.3379, over 92.00 utterances.], tot_loss[loss=0.4521, over 2764019.70 frames; utt_duration=1107 frames, utt_pad_proportion=0.108, over 10000.14 utterances.], batch size: 92, lr: 4.22e-04
|
| 92 |
+
2022-08-10 02:19:32,344 INFO [train.py:737] (2/3) Epoch 12, batch 0, loss[loss=0.4113, over 14753.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006107, over 40.00 utterances.], tot_loss[loss=0.4113, over 14753.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006107, over 40.00 utterances.], batch size: 40, lr: 4.07e-04
|
| 93 |
+
2022-08-10 02:29:04,935 INFO [train.py:737] (2/3) Epoch 12, batch 100, loss[loss=0.3677, over 14418.00 frames; utt_duration=1155 frames, utt_pad_proportion=0.02773, over 50.00 utterances.], tot_loss[loss=0.3743, over 1128062.16 frames; utt_duration=1236 frames, utt_pad_proportion=0.06235, over 3655.76 utterances.], batch size: 50, lr: 4.07e-04
|
| 94 |
+
2022-08-10 02:38:10,104 INFO [train.py:737] (2/3) Epoch 12, batch 200, loss[loss=0.4001, over 14572.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005675, over 40.00 utterances.], tot_loss[loss=0.3665, over 1823519.52 frames; utt_duration=1317 frames, utt_pad_proportion=0.0442, over 5542.84 utterances.], batch size: 40, lr: 4.07e-04
|
| 95 |
+
2022-08-10 02:50:17,053 INFO [train.py:737] (2/3) Epoch 12, batch 300, loss[loss=0.3239, over 14578.00 frames; utt_duration=1578 frames, utt_pad_proportion=0.005807, over 37.00 utterances.], tot_loss[loss=0.3627, over 2241132.89 frames; utt_duration=1312 frames, utt_pad_proportion=0.04153, over 6838.98 utterances.], batch size: 37, lr: 4.07e-04
|
| 96 |
+
2022-08-10 02:59:35,401 INFO [train.py:737] (2/3) Epoch 12, batch 400, loss[loss=0.3524, over 14566.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.00605, over 40.00 utterances.], tot_loss[loss=0.3619, over 2494163.67 frames; utt_duration=1324 frames, utt_pad_proportion=0.0398, over 7544.07 utterances.], batch size: 40, lr: 4.07e-04
|
| 97 |
+
2022-08-10 03:10:03,635 INFO [train.py:737] (2/3) Epoch 12, batch 500, loss[loss=0.3425, over 14527.00 frames; utt_duration=1212 frames, utt_pad_proportion=0.01628, over 48.00 utterances.], tot_loss[loss=0.3616, over 2651572.41 frames; utt_duration=1342 frames, utt_pad_proportion=0.03654, over 7914.48 utterances.], batch size: 48, lr: 4.07e-04
|
| 98 |
+
2022-08-10 03:22:54,034 INFO [train.py:737] (2/3) Epoch 12, batch 600, loss[loss=0.3772, over 14883.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.007065, over 44.00 utterances.], tot_loss[loss=0.3621, over 2743254.13 frames; utt_duration=1338 frames, utt_pad_proportion=0.03718, over 8210.28 utterances.], batch size: 44, lr: 4.06e-04
|
| 99 |
+
2022-08-10 03:34:44,752 INFO [train.py:737] (2/3) Epoch 12, batch 700, loss[loss=0.348, over 13720.00 frames; utt_duration=964.4 frames, utt_pad_proportion=0.07178, over 57.00 utterances.], tot_loss[loss=0.3633, over 2796236.03 frames; utt_duration=1334 frames, utt_pad_proportion=0.03773, over 8396.51 utterances.], batch size: 57, lr: 4.06e-04
|
| 100 |
+
2022-08-10 03:46:11,196 INFO [train.py:737] (2/3) Epoch 12, batch 800, loss[loss=0.3651, over 14734.00 frames; utt_duration=1283 frames, utt_pad_proportion=0.008702, over 46.00 utterances.], tot_loss[loss=0.3633, over 2829566.36 frames; utt_duration=1332 frames, utt_pad_proportion=0.03837, over 8505.78 utterances.], batch size: 46, lr: 4.06e-04
|
| 101 |
+
2022-08-10 03:56:27,270 INFO [train.py:737] (2/3) Epoch 12, batch 900, loss[loss=0.4111, over 14741.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.00658, over 43.00 utterances.], tot_loss[loss=0.3658, over 2852384.74 frames; utt_duration=1333 frames, utt_pad_proportion=0.03708, over 8565.98 utterances.], batch size: 43, lr: 4.06e-04
|
| 102 |
+
2022-08-10 04:06:59,879 INFO [train.py:737] (2/3) Epoch 12, batch 1000, loss[loss=0.3791, over 14603.00 frames; utt_duration=1624 frames, utt_pad_proportion=0.007352, over 36.00 utterances.], tot_loss[loss=0.3639, over 2866703.78 frames; utt_duration=1342 frames, utt_pad_proportion=0.03504, over 8553.16 utterances.], batch size: 36, lr: 4.06e-04
|
| 103 |
+
2022-08-10 04:16:28,353 INFO [train.py:737] (2/3) Epoch 12, batch 1100, loss[loss=0.3368, over 14598.00 frames; utt_duration=1426 frames, utt_pad_proportion=0.004987, over 41.00 utterances.], tot_loss[loss=0.3628, over 2866573.37 frames; utt_duration=1333 frames, utt_pad_proportion=0.0401, over 8609.48 utterances.], batch size: 41, lr: 4.06e-04
|
| 104 |
+
2022-08-10 04:27:14,727 INFO [train.py:737] (2/3) Epoch 12, batch 1200, loss[loss=0.3212, over 14465.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01025, over 33.00 utterances.], tot_loss[loss=0.3626, over 2867659.06 frames; utt_duration=1322 frames, utt_pad_proportion=0.04292, over 8685.09 utterances.], batch size: 33, lr: 4.05e-04
|
| 105 |
+
2022-08-10 04:37:22,611 INFO [train.py:737] (2/3) Epoch 12, batch 1300, loss[loss=0.3652, over 14753.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.006058, over 43.00 utterances.], tot_loss[loss=0.3649, over 2876555.54 frames; utt_duration=1328 frames, utt_pad_proportion=0.04081, over 8676.31 utterances.], batch size: 43, lr: 4.05e-04
|
| 106 |
+
2022-08-10 04:46:39,489 INFO [train.py:737] (2/3) Epoch 12, batch 1400, loss[loss=0.4434, over 14591.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.005024, over 42.00 utterances.], tot_loss[loss=0.3657, over 2887682.12 frames; utt_duration=1342 frames, utt_pad_proportion=0.03367, over 8617.31 utterances.], batch size: 42, lr: 4.05e-04
|
| 107 |
+
2022-08-10 04:57:14,245 INFO [train.py:737] (2/3) Epoch 12, batch 1500, loss[loss=0.3832, over 14690.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.00968, over 45.00 utterances.], tot_loss[loss=0.3675, over 2876843.83 frames; utt_duration=1316 frames, utt_pad_proportion=0.0407, over 8756.65 utterances.], batch size: 45, lr: 4.05e-04
|
| 108 |
+
2022-08-10 05:09:29,813 INFO [train.py:737] (2/3) Epoch 12, batch 1600, loss[loss=0.3461, over 14769.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006351, over 39.00 utterances.], tot_loss[loss=0.3699, over 2862302.16 frames; utt_duration=1279 frames, utt_pad_proportion=0.05075, over 8964.15 utterances.], batch size: 39, lr: 4.05e-04
|
| 109 |
+
2022-08-10 05:20:38,271 INFO [train.py:737] (2/3) Epoch 12, batch 1700, loss[loss=0.4051, over 13727.00 frames; utt_duration=964.7 frames, utt_pad_proportion=0.06974, over 57.00 utterances.], tot_loss[loss=0.3678, over 2872525.42 frames; utt_duration=1298 frames, utt_pad_proportion=0.04251, over 8863.10 utterances.], batch size: 57, lr: 4.05e-04
|
| 110 |
+
2022-08-10 05:31:51,600 INFO [train.py:737] (2/3) Epoch 12, batch 1800, loss[loss=0.372, over 14438.00 frames; utt_duration=1651 frames, utt_pad_proportion=0.009341, over 35.00 utterances.], tot_loss[loss=0.3686, over 2864999.55 frames; utt_duration=1288 frames, utt_pad_proportion=0.0479, over 8909.45 utterances.], batch size: 35, lr: 4.05e-04
|
| 111 |
+
2022-08-10 05:42:58,983 INFO [train.py:737] (2/3) Epoch 12, batch 1900, loss[loss=0.3338, over 14781.00 frames; utt_duration=1600 frames, utt_pad_proportion=0.00708, over 37.00 utterances.], tot_loss[loss=0.3688, over 2868264.61 frames; utt_duration=1293 frames, utt_pad_proportion=0.04539, over 8880.71 utterances.], batch size: 37, lr: 4.04e-04
|
| 112 |
+
2022-08-10 05:52:47,053 INFO [train.py:737] (2/3) Epoch 12, batch 2000, loss[loss=0.3641, over 14751.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.005925, over 41.00 utterances.], tot_loss[loss=0.3672, over 2863652.10 frames; utt_duration=1289 frames, utt_pad_proportion=0.04993, over 8897.81 utterances.], batch size: 41, lr: 4.04e-04
|
| 113 |
+
2022-08-10 06:03:23,205 INFO [train.py:737] (2/3) Epoch 12, batch 2100, loss[loss=0.3484, over 14040.00 frames; utt_duration=1813 frames, utt_pad_proportion=0.03915, over 31.00 utterances.], tot_loss[loss=0.3677, over 2869189.03 frames; utt_duration=1302 frames, utt_pad_proportion=0.04645, over 8823.90 utterances.], batch size: 31, lr: 4.04e-04
|
| 114 |
+
2022-08-10 06:13:20,124 INFO [train.py:737] (2/3) Epoch 12, batch 2200, loss[loss=0.3324, over 14477.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.01004, over 33.00 utterances.], tot_loss[loss=0.3679, over 2867118.29 frames; utt_duration=1289 frames, utt_pad_proportion=0.04902, over 8909.76 utterances.], batch size: 33, lr: 4.04e-04
|
| 115 |
+
2022-08-10 06:24:03,211 INFO [train.py:737] (2/3) Epoch 12, batch 2300, loss[loss=0.3878, over 14765.00 frames; utt_duration=1478 frames, utt_pad_proportion=0.006052, over 40.00 utterances.], tot_loss[loss=0.3678, over 2862473.89 frames; utt_duration=1276 frames, utt_pad_proportion=0.05128, over 8981.52 utterances.], batch size: 40, lr: 4.04e-04
|
| 116 |
+
2022-08-10 06:36:03,154 INFO [train.py:737] (2/3) Epoch 12, batch 2400, loss[loss=0.3994, over 13753.00 frames; utt_duration=966.7 frames, utt_pad_proportion=0.06955, over 57.00 utterances.], tot_loss[loss=0.3705, over 2860380.92 frames; utt_duration=1272 frames, utt_pad_proportion=0.05129, over 9006.20 utterances.], batch size: 57, lr: 4.04e-04
|
| 117 |
+
2022-08-10 06:48:05,002 INFO [train.py:737] (2/3) Epoch 12, batch 2500, loss[loss=0.4012, over 14591.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.005395, over 42.00 utterances.], tot_loss[loss=0.3709, over 2856153.44 frames; utt_duration=1267 frames, utt_pad_proportion=0.05338, over 9028.94 utterances.], batch size: 42, lr: 4.03e-04
|
| 118 |
+
2022-08-10 07:00:47,850 INFO [train.py:737] (2/3) Epoch 12, batch 2600, loss[loss=0.3716, over 14598.00 frames; utt_duration=1392 frames, utt_pad_proportion=0.004615, over 42.00 utterances.], tot_loss[loss=0.3687, over 2865463.06 frames; utt_duration=1293 frames, utt_pad_proportion=0.04822, over 8877.23 utterances.], batch size: 42, lr: 4.03e-04
|
| 119 |
+
2022-08-10 07:12:58,141 INFO [train.py:737] (2/3) Epoch 12, batch 2700, loss[loss=0.4059, over 14582.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.005196, over 39.00 utterances.], tot_loss[loss=0.3666, over 2865982.43 frames; utt_duration=1294 frames, utt_pad_proportion=0.0482, over 8869.98 utterances.], batch size: 39, lr: 4.03e-04
|
| 120 |
+
2022-08-10 07:21:23,112 INFO [train.py:737] (2/3) Epoch 12, batch 2800, loss[loss=0.334, over 14572.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.00559, over 40.00 utterances.], tot_loss[loss=0.3666, over 2874610.58 frames; utt_duration=1307 frames, utt_pad_proportion=0.04123, over 8805.96 utterances.], batch size: 40, lr: 4.03e-04
|
| 121 |
+
2022-08-10 07:30:52,943 INFO [train.py:737] (2/3) Epoch 12, batch 2900, loss[loss=0.3832, over 14123.00 frames; utt_duration=1824 frames, utt_pad_proportion=0.04877, over 31.00 utterances.], tot_loss[loss=0.3844, over 2867415.09 frames; utt_duration=1286 frames, utt_pad_proportion=0.04571, over 8929.85 utterances.], batch size: 31, lr: 4.03e-04
|
| 122 |
+
2022-08-10 07:40:15,530 INFO [train.py:737] (2/3) Epoch 12, batch 3000, loss[loss=0.4214, over 14489.00 frames; utt_duration=1161 frames, utt_pad_proportion=0.0211, over 50.00 utterances.], tot_loss[loss=0.4031, over 2852506.78 frames; utt_duration=1248 frames, utt_pad_proportion=0.05516, over 9152.44 utterances.], batch size: 50, lr: 4.03e-04
|
| 123 |
+
2022-08-10 07:40:15,530 INFO [train.py:761] (2/3) Computing validation loss
|
| 124 |
+
2022-08-10 07:43:16,494 INFO [train.py:770] (2/3) Epoch 12, validation: loss=0.297, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 125 |
+
2022-08-10 07:53:07,548 INFO [train.py:737] (2/3) Epoch 12, batch 3100, loss[loss=0.4291, over 14773.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.007118, over 37.00 utterances.], tot_loss[loss=0.4168, over 2852973.01 frames; utt_duration=1251 frames, utt_pad_proportion=0.05528, over 9130.28 utterances.], batch size: 37, lr: 4.03e-04
|
| 126 |
+
2022-08-10 08:03:53,496 INFO [train.py:737] (2/3) Epoch 12, batch 3200, loss[loss=0.484, over 14687.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009798, over 45.00 utterances.], tot_loss[loss=0.4264, over 2859172.75 frames; utt_duration=1268 frames, utt_pad_proportion=0.0528, over 9026.61 utterances.], batch size: 45, lr: 4.02e-04
|
| 127 |
+
2022-08-10 08:15:52,840 INFO [train.py:737] (2/3) Epoch 12, batch 3300, loss[loss=0.4624, over 14360.00 frames; utt_duration=1085 frames, utt_pad_proportion=0.03544, over 53.00 utterances.], tot_loss[loss=0.435, over 2844949.66 frames; utt_duration=1232 frames, utt_pad_proportion=0.06218, over 9244.64 utterances.], batch size: 53, lr: 4.02e-04
|
| 128 |
+
2022-08-10 08:24:14,595 INFO [train.py:737] (2/3) Epoch 12, batch 3400, loss[loss=0.5151, over 14632.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.00836, over 44.00 utterances.], tot_loss[loss=0.4391, over 2852091.70 frames; utt_duration=1248 frames, utt_pad_proportion=0.05814, over 9155.63 utterances.], batch size: 44, lr: 4.02e-04
|
| 129 |
+
2022-08-10 08:32:25,020 INFO [train.py:737] (2/3) Epoch 12, batch 3500, loss[loss=0.4132, over 14581.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.006753, over 38.00 utterances.], tot_loss[loss=0.4429, over 2845559.99 frames; utt_duration=1234 frames, utt_pad_proportion=0.06112, over 9233.71 utterances.], batch size: 38, lr: 4.02e-04
|
| 130 |
+
2022-08-10 08:43:20,507 INFO [train.py:737] (2/3) Epoch 12, batch 3600, loss[loss=0.5518, over 14575.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.005961, over 42.00 utterances.], tot_loss[loss=0.4445, over 2846321.35 frames; utt_duration=1240 frames, utt_pad_proportion=0.06208, over 9194.77 utterances.], batch size: 42, lr: 4.02e-04
|
| 131 |
+
2022-08-10 08:54:44,670 INFO [train.py:737] (2/3) Epoch 12, batch 3700, loss[loss=0.3533, over 14714.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.0105, over 35.00 utterances.], tot_loss[loss=0.4458, over 2860059.82 frames; utt_duration=1260 frames, utt_pad_proportion=0.05389, over 9091.75 utterances.], batch size: 35, lr: 4.02e-04
|
| 132 |
+
2022-08-10 09:03:23,106 INFO [train.py:737] (2/3) Epoch 12, batch 3800, loss[loss=0.4394, over 14699.00 frames; utt_duration=1253 frames, utt_pad_proportion=0.01064, over 47.00 utterances.], tot_loss[loss=0.4468, over 2840384.96 frames; utt_duration=1225 frames, utt_pad_proportion=0.06294, over 9283.25 utterances.], batch size: 47, lr: 4.01e-04
|
| 133 |
+
2022-08-10 09:12:39,489 INFO [train.py:737] (2/3) Epoch 12, batch 3900, loss[loss=0.428, over 14742.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.005793, over 43.00 utterances.], tot_loss[loss=0.4451, over 2826440.43 frames; utt_duration=1208 frames, utt_pad_proportion=0.07209, over 9369.18 utterances.], batch size: 43, lr: 4.01e-04
|
| 134 |
+
2022-08-10 09:22:00,880 INFO [train.py:737] (2/3) Epoch 12, batch 4000, loss[loss=0.5451, over 9462.00 frames; utt_duration=404.2 frames, utt_pad_proportion=0.3625, over 94.00 utterances.], tot_loss[loss=0.4455, over 2828026.73 frames; utt_duration=1207 frames, utt_pad_proportion=0.06975, over 9380.63 utterances.], batch size: 94, lr: 4.01e-04
|
| 135 |
+
2022-08-10 09:32:08,728 INFO [train.py:737] (2/3) Epoch 12, batch 4100, loss[loss=0.4116, over 14755.00 frames; utt_duration=1596 frames, utt_pad_proportion=0.007895, over 37.00 utterances.], tot_loss[loss=0.4447, over 2821105.74 frames; utt_duration=1193 frames, utt_pad_proportion=0.07419, over 9472.25 utterances.], batch size: 37, lr: 4.01e-04
|
| 136 |
+
2022-08-10 09:41:30,768 INFO [train.py:737] (2/3) Epoch 12, batch 4200, loss[loss=0.3819, over 14459.00 frames; utt_duration=1754 frames, utt_pad_proportion=0.01004, over 33.00 utterances.], tot_loss[loss=0.4456, over 2821308.51 frames; utt_duration=1192 frames, utt_pad_proportion=0.07409, over 9475.61 utterances.], batch size: 33, lr: 4.01e-04
|
| 137 |
+
2022-08-10 09:51:28,129 INFO [train.py:737] (2/3) Epoch 12, batch 4300, loss[loss=0.456, over 14867.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.008081, over 44.00 utterances.], tot_loss[loss=0.4466, over 2818842.35 frames; utt_duration=1188 frames, utt_pad_proportion=0.07643, over 9505.59 utterances.], batch size: 44, lr: 4.01e-04
|
| 138 |
+
2022-08-10 10:02:08,399 INFO [train.py:737] (2/3) Epoch 12, batch 4400, loss[loss=0.4058, over 14584.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.008608, over 36.00 utterances.], tot_loss[loss=0.4471, over 2843347.16 frames; utt_duration=1243 frames, utt_pad_proportion=0.0621, over 9162.13 utterances.], batch size: 36, lr: 4.01e-04
|
| 139 |
+
2022-08-10 10:12:01,455 INFO [train.py:737] (2/3) Epoch 12, batch 4500, loss[loss=0.4609, over 14897.00 frames; utt_duration=1356 frames, utt_pad_proportion=0.006215, over 44.00 utterances.], tot_loss[loss=0.446, over 2831690.74 frames; utt_duration=1219 frames, utt_pad_proportion=0.07031, over 9303.28 utterances.], batch size: 44, lr: 4.00e-04
|
| 140 |
+
2022-08-10 10:22:19,559 INFO [train.py:737] (2/3) Epoch 12, batch 4600, loss[loss=0.3784, over 14761.00 frames; utt_duration=1597 frames, utt_pad_proportion=0.007357, over 37.00 utterances.], tot_loss[loss=0.4469, over 2829345.86 frames; utt_duration=1215 frames, utt_pad_proportion=0.07198, over 9329.85 utterances.], batch size: 37, lr: 4.00e-04
|
| 141 |
+
2022-08-10 10:30:47,464 INFO [train.py:737] (2/3) Epoch 12, batch 4700, loss[loss=0.523, over 14666.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.01223, over 47.00 utterances.], tot_loss[loss=0.4457, over 2817487.65 frames; utt_duration=1200 frames, utt_pad_proportion=0.07763, over 9407.30 utterances.], batch size: 47, lr: 4.00e-04
|
| 142 |
+
2022-08-10 10:43:44,062 INFO [train.py:737] (2/3) Epoch 12, batch 4800, loss[loss=0.4293, over 14645.00 frames; utt_duration=1248 frames, utt_pad_proportion=0.01459, over 47.00 utterances.], tot_loss[loss=0.4451, over 2825041.11 frames; utt_duration=1218 frames, utt_pad_proportion=0.0725, over 9292.89 utterances.], batch size: 47, lr: 4.00e-04
|
| 143 |
+
2022-08-10 10:56:42,547 INFO [train.py:737] (2/3) Epoch 12, batch 4900, loss[loss=0.422, over 14742.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.006346, over 41.00 utterances.], tot_loss[loss=0.4452, over 2829507.77 frames; utt_duration=1223 frames, utt_pad_proportion=0.07051, over 9267.58 utterances.], batch size: 41, lr: 4.00e-04
|
| 144 |
+
2022-08-10 11:12:32,190 INFO [train.py:737] (2/3) Epoch 12, batch 5000, loss[loss=0.4973, over 14392.00 frames; utt_duration=1088 frames, utt_pad_proportion=0.03413, over 53.00 utterances.], tot_loss[loss=0.4439, over 2829660.10 frames; utt_duration=1220 frames, utt_pad_proportion=0.06966, over 9292.29 utterances.], batch size: 53, lr: 4.00e-04
|
| 145 |
+
2022-08-10 11:47:05,510 INFO [train.py:737] (2/3) Epoch 12, batch 5100, loss[loss=0.5256, over 10157.00 frames; utt_duration=448 frames, utt_pad_proportion=0.314, over 91.00 utterances.], tot_loss[loss=0.4442, over 2812998.00 frames; utt_duration=1197 frames, utt_pad_proportion=0.07864, over 9413.93 utterances.], batch size: 91, lr: 4.00e-04
|
| 146 |
+
2022-08-10 12:13:52,158 INFO [train.py:737] (2/3) Epoch 12, batch 5200, loss[loss=0.4462, over 14696.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.008188, over 45.00 utterances.], tot_loss[loss=0.4438, over 2819298.60 frames; utt_duration=1203 frames, utt_pad_proportion=0.07675, over 9386.92 utterances.], batch size: 45, lr: 3.99e-04
|
| 147 |
+
2022-08-10 12:46:45,461 INFO [train.py:737] (2/3) Epoch 12, batch 5300, loss[loss=0.377, over 14852.00 frames; utt_duration=1652 frames, utt_pad_proportion=0.006782, over 36.00 utterances.], tot_loss[loss=0.4453, over 2822154.37 frames; utt_duration=1206 frames, utt_pad_proportion=0.07493, over 9373.01 utterances.], batch size: 36, lr: 3.99e-04
|
| 148 |
+
2022-08-10 13:18:04,046 INFO [train.py:737] (2/3) Epoch 12, batch 5400, loss[loss=0.4837, over 14730.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.007505, over 43.00 utterances.], tot_loss[loss=0.4484, over 2811772.08 frames; utt_duration=1191 frames, utt_pad_proportion=0.08025, over 9451.68 utterances.], batch size: 43, lr: 3.99e-04
|
| 149 |
+
2022-08-10 13:50:20,065 INFO [train.py:737] (2/3) Epoch 12, batch 5500, loss[loss=0.4787, over 13671.00 frames; utt_duration=961 frames, utt_pad_proportion=0.07509, over 57.00 utterances.], tot_loss[loss=0.4466, over 2812244.11 frames; utt_duration=1191 frames, utt_pad_proportion=0.07939, over 9460.22 utterances.], batch size: 57, lr: 3.99e-04
|
| 150 |
+
2022-08-10 14:17:39,808 INFO [train.py:737] (2/3) Epoch 12, batch 5600, loss[loss=0.4267, over 14604.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.01048, over 34.00 utterances.], tot_loss[loss=0.4447, over 2814656.56 frames; utt_duration=1199 frames, utt_pad_proportion=0.07755, over 9399.10 utterances.], batch size: 34, lr: 3.99e-04
|
| 151 |
+
2022-08-10 14:45:43,327 INFO [train.py:737] (2/3) Epoch 12, batch 5700, loss[loss=0.4602, over 14764.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005851, over 42.00 utterances.], tot_loss[loss=0.4418, over 2836272.85 frames; utt_duration=1236 frames, utt_pad_proportion=0.06481, over 9188.33 utterances.], batch size: 42, lr: 3.99e-04
|
| 152 |
+
2022-08-10 15:11:33,157 INFO [train.py:737] (2/3) Epoch 12, batch 5800, loss[loss=0.468, over 13783.00 frames; utt_duration=968.6 frames, utt_pad_proportion=0.06416, over 57.00 utterances.], tot_loss[loss=0.4432, over 2828083.64 frames; utt_duration=1218 frames, utt_pad_proportion=0.06974, over 9301.18 utterances.], batch size: 57, lr: 3.99e-04
|
| 153 |
+
2022-08-10 15:40:12,812 INFO [train.py:737] (2/3) Epoch 12, batch 5900, loss[loss=0.4606, over 14709.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.01108, over 46.00 utterances.], tot_loss[loss=0.4435, over 2823149.90 frames; utt_duration=1204 frames, utt_pad_proportion=0.07327, over 9389.54 utterances.], batch size: 46, lr: 3.98e-04
|
| 154 |
+
2022-08-10 16:07:46,685 INFO [train.py:737] (2/3) Epoch 12, batch 6000, loss[loss=0.4913, over 12816.00 frames; utt_duration=766.7 frames, utt_pad_proportion=0.1337, over 67.00 utterances.], tot_loss[loss=0.4439, over 2832494.68 frames; utt_duration=1217 frames, utt_pad_proportion=0.06668, over 9319.80 utterances.], batch size: 67, lr: 3.98e-04
|
| 155 |
+
2022-08-10 16:07:46,685 INFO [train.py:761] (2/3) Computing validation loss
|
| 156 |
+
2022-08-10 16:20:31,490 INFO [train.py:770] (2/3) Epoch 12, validation: loss=0.2954, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 157 |
+
2022-08-10 16:47:18,866 INFO [train.py:737] (2/3) Epoch 12, batch 6100, loss[loss=0.5327, over 10150.00 frames; utt_duration=442.9 frames, utt_pad_proportion=0.3197, over 92.00 utterances.], tot_loss[loss=0.4474, over 2778608.04 frames; utt_duration=1126 frames, utt_pad_proportion=0.09808, over 9888.05 utterances.], batch size: 92, lr: 3.98e-04
|
| 158 |
+
2022-08-10 17:14:26,431 INFO [train.py:737] (2/3) Epoch 13, batch 0, loss[loss=0.4479, over 14783.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.005193, over 38.00 utterances.], tot_loss[loss=0.4479, over 14783.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.005193, over 38.00 utterances.], batch size: 38, lr: 3.85e-04
|
exp/log/log-train-2022-08-11-08-52-44-0
ADDED
|
@@ -0,0 +1,278 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2022-08-11 08:52:44,620 INFO [train.py:805] (0/3) Training started
|
| 2 |
+
2022-08-11 08:52:44,621 INFO [train.py:806] (0/3) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'env_info': {'k2-version': '1.17', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '7dcabf85e8bf06984c4abab0400ef1322b5ff3df', 'k2-git-date': 'Tue Aug 2 21:22:39 2022', 'lhotse-version': '1.4.0.dev+git.59318ed.clean', 'torch-version': '1.11.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.3', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '6af45c3-dirty', 'icefall-git-date': 'Tue Aug 2 14:47:09 2022', 'icefall-path': '/thspfs/home/acct-hpc/thspzh-hpc1/git/icefall_dev', 'k2-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/k2-1.17.dev20220805+cuda11.3.torch1.11.0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'dgxs-01.cluster.thspzh.com', 'IP address': '192.168.200.101'}, 'world_size': 3, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 20, 'start_epoch': 13, 'start_batch': 0, 'exp_dir': PosixPath('cnn_ctc/exp_phone'), 'lang_dir': 'data/lang_phone', 'bpe_model': 'data/lang_bpe_500/bpe.model', 'initial_lr': 0.003, 'lr_batches': 5000, 'lr_epochs': 6, 'seed': 42, 'print_diagnostics': False, 'save_every_n': 8000, 'keep_last_k': 20, 'average_period': 100, 'use_fp16': False, 'first_out_channels': 8, 'conv_blocks': 3, 'rnn_dim': 0, 'full_libri': True, 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 600, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'input_strategy': 'PrecomputedFeatures'}
|
| 3 |
+
2022-08-11 08:52:44,964 INFO [lexicon.py:176] (0/3) Loading pre-compiled data/lang_phone/Linv.pt
|
| 4 |
+
2022-08-11 08:52:57,385 INFO [train.py:843] (0/3) About to create model
|
| 5 |
+
2022-08-11 08:52:57,408 INFO [train.py:855] (0/3) Number of model parameters: 1766904
|
| 6 |
+
2022-08-11 08:53:02,658 INFO [train.py:859] (0/3) ==========================================================================================
|
| 7 |
+
Layer (type:depth-idx) Output Shape Param #
|
| 8 |
+
==========================================================================================
|
| 9 |
+
MobileNetS4 [1, 25, 72] --
|
| 10 |
+
├─Sequential: 1-1 [1, 32, 25, 20] --
|
| 11 |
+
│ └─Conv2d: 2-1 [1, 8, 100, 80] 80
|
| 12 |
+
│ └─BatchNorm2d: 2-2 [1, 8, 100, 80] 16
|
| 13 |
+
│ └─ReLU6: 2-3 [1, 8, 100, 80] --
|
| 14 |
+
│ └─Conv2d: 2-4 [1, 16, 50, 40] 1,168
|
| 15 |
+
│ └─BatchNorm2d: 2-5 [1, 16, 50, 40] 32
|
| 16 |
+
│ └─ReLU6: 2-6 [1, 16, 50, 40] --
|
| 17 |
+
│ └─Conv2d: 2-7 [1, 32, 25, 20] 4,640
|
| 18 |
+
│ └─BatchNorm2d: 2-8 [1, 32, 25, 20] 64
|
| 19 |
+
│ └─ReLU6: 2-9 [1, 32, 25, 20] --
|
| 20 |
+
├─Sequential: 1-2 [1, 256, 25, 3] --
|
| 21 |
+
│ └─Bottleneck: 2-10 [1, 64, 25, 10] --
|
| 22 |
+
│ │ └─Sequential: 3-1 [1, 64, 25, 10] 14,400
|
| 23 |
+
│ └─Bottleneck: 2-11 [1, 64, 25, 10] --
|
| 24 |
+
│ │ └─Sequential: 3-2 [1, 64, 25, 10] 36,800
|
| 25 |
+
│ └─Bottleneck: 2-12 [1, 64, 25, 10] --
|
| 26 |
+
│ │ └─Sequential: 3-3 [1, 64, 25, 10] 36,800
|
| 27 |
+
│ └─Bottleneck: 2-13 [1, 128, 25, 5] --
|
| 28 |
+
│ │ └─Sequential: 3-4 [1, 128, 25, 5] 53,376
|
| 29 |
+
│ └─Bottleneck: 2-14 [1, 128, 25, 5] --
|
| 30 |
+
│ │ └─Sequential: 3-5 [1, 128, 25, 5] 139,136
|
| 31 |
+
│ └─Bottleneck: 2-15 [1, 128, 25, 5] --
|
| 32 |
+
│ │ └─Sequential: 3-6 [1, 128, 25, 5] 139,136
|
| 33 |
+
│ └─Bottleneck: 2-16 [1, 256, 25, 3] --
|
| 34 |
+
│ │ └─Sequential: 3-7 [1, 256, 25, 3] 205,056
|
| 35 |
+
│ └─Bottleneck: 2-17 [1, 256, 25, 3] --
|
| 36 |
+
│ │ └─Sequential: 3-8 [1, 256, 25, 3] 540,416
|
| 37 |
+
│ └─Bottleneck: 2-18 [1, 256, 25, 3] --
|
| 38 |
+
│ │ └─Sequential: 3-9 [1, 256, 25, 3] 540,416
|
| 39 |
+
├─Linear: 1-3 [1, 25, 72] 55,368
|
| 40 |
+
==========================================================================================
|
| 41 |
+
Total params: 1,766,904
|
| 42 |
+
Trainable params: 1,766,904
|
| 43 |
+
Non-trainable params: 0
|
| 44 |
+
Total mult-adds (M): 169.35
|
| 45 |
+
==========================================================================================
|
| 46 |
+
Input size (MB): 0.03
|
| 47 |
+
Forward/backward pass size (MB): 22.08
|
| 48 |
+
Params size (MB): 7.07
|
| 49 |
+
Estimated Total Size (MB): 29.18
|
| 50 |
+
==========================================================================================
|
| 51 |
+
2022-08-11 08:53:02,726 INFO [checkpoint.py:112] (0/3) Loading checkpoint from cnn_ctc/exp_phone/epoch-12.pt
|
| 52 |
+
2022-08-11 08:53:02,817 INFO [checkpoint.py:131] (0/3) Loading averaged model
|
| 53 |
+
2022-08-11 08:53:02,829 INFO [train.py:876] (0/3) Using DDP
|
| 54 |
+
2022-08-11 08:53:02,939 INFO [train.py:884] (0/3) Loading optimizer state dict
|
| 55 |
+
2022-08-11 08:53:02,986 INFO [train.py:892] (0/3) Loading scheduler state dict
|
| 56 |
+
2022-08-11 08:53:02,986 INFO [asr_datamodule.py:409] (0/3) About to get train-clean-100 cuts
|
| 57 |
+
2022-08-11 08:53:03,024 INFO [asr_datamodule.py:416] (0/3) About to get train-clean-360 cuts
|
| 58 |
+
2022-08-11 08:53:03,047 INFO [asr_datamodule.py:423] (0/3) About to get train-other-500 cuts
|
| 59 |
+
2022-08-11 08:53:03,063 INFO [asr_datamodule.py:225] (0/3) Enable MUSAN
|
| 60 |
+
2022-08-11 08:53:03,063 INFO [asr_datamodule.py:226] (0/3) About to get Musan cuts
|
| 61 |
+
2022-08-11 08:53:05,102 INFO [asr_datamodule.py:254] (0/3) Enable SpecAugment
|
| 62 |
+
2022-08-11 08:53:05,102 INFO [asr_datamodule.py:255] (0/3) Time warp factor: 80
|
| 63 |
+
2022-08-11 08:53:05,102 INFO [asr_datamodule.py:267] (0/3) Num frame mask: 10
|
| 64 |
+
2022-08-11 08:53:05,102 INFO [asr_datamodule.py:280] (0/3) About to create train dataset
|
| 65 |
+
2022-08-11 08:53:05,103 INFO [asr_datamodule.py:309] (0/3) Using DynamicBucketingSampler.
|
| 66 |
+
2022-08-11 08:53:07,694 INFO [asr_datamodule.py:324] (0/3) About to create train dataloader
|
| 67 |
+
2022-08-11 08:53:07,695 INFO [asr_datamodule.py:430] (0/3) About to get dev-clean cuts
|
| 68 |
+
2022-08-11 08:53:07,697 INFO [asr_datamodule.py:437] (0/3) About to get dev-other cuts
|
| 69 |
+
2022-08-11 08:53:07,699 INFO [asr_datamodule.py:355] (0/3) About to create dev dataset
|
| 70 |
+
2022-08-11 08:53:08,005 INFO [asr_datamodule.py:374] (0/3) About to create dev dataloader
|
| 71 |
+
2022-08-11 08:53:08,006 INFO [train.py:958] (0/3) Loading grad scaler state dict
|
| 72 |
+
2022-08-11 08:53:27,592 INFO [train.py:737] (0/3) Epoch 13, batch 0, loss[loss=0.4239, over 14762.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.005151, over 41.00 utterances.], tot_loss[loss=0.4239, over 14762.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.005151, over 41.00 utterances.], batch size: 41, lr: 3.85e-04
|
| 73 |
+
2022-08-11 08:53:27,640 INFO [distributed.py:948] (0/3) Reducer buckets have been rebuilt in this iteration.
|
| 74 |
+
2022-08-11 08:56:11,073 INFO [train.py:737] (0/3) Epoch 13, batch 100, loss[loss=0.3856, over 14759.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005184, over 41.00 utterances.], tot_loss[loss=0.3646, over 1144185.76 frames; utt_duration=1335 frames, utt_pad_proportion=0.04219, over 3431.03 utterances.], batch size: 41, lr: 3.85e-04
|
| 75 |
+
2022-08-11 08:56:18,083 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/checkpoint-136000.pt
|
| 76 |
+
2022-08-11 08:58:59,358 INFO [train.py:737] (0/3) Epoch 13, batch 200, loss[loss=0.3456, over 14884.00 frames; utt_duration=1355 frames, utt_pad_proportion=0.006682, over 44.00 utterances.], tot_loss[loss=0.3622, over 1824102.08 frames; utt_duration=1310 frames, utt_pad_proportion=0.04797, over 5577.08 utterances.], batch size: 44, lr: 3.84e-04
|
| 77 |
+
2022-08-11 09:01:26,617 INFO [train.py:737] (0/3) Epoch 13, batch 300, loss[loss=0.3944, over 13631.00 frames; utt_duration=957.9 frames, utt_pad_proportion=0.07625, over 57.00 utterances.], tot_loss[loss=0.3616, over 2245623.22 frames; utt_duration=1321 frames, utt_pad_proportion=0.04064, over 6806.17 utterances.], batch size: 57, lr: 3.84e-04
|
| 78 |
+
2022-08-11 09:03:10,459 INFO [train.py:737] (0/3) Epoch 13, batch 400, loss[loss=0.3648, over 14569.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005743, over 40.00 utterances.], tot_loss[loss=0.3628, over 2505339.24 frames; utt_duration=1327 frames, utt_pad_proportion=0.03415, over 7562.60 utterances.], batch size: 40, lr: 3.84e-04
|
| 79 |
+
2022-08-11 09:04:16,682 INFO [train.py:737] (0/3) Epoch 13, batch 500, loss[loss=0.3625, over 14720.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.009694, over 46.00 utterances.], tot_loss[loss=0.3619, over 2639727.23 frames; utt_duration=1289 frames, utt_pad_proportion=0.04549, over 8198.05 utterances.], batch size: 46, lr: 3.84e-04
|
| 80 |
+
2022-08-11 09:05:12,192 INFO [train.py:737] (0/3) Epoch 13, batch 600, loss[loss=0.3513, over 14588.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.005855, over 42.00 utterances.], tot_loss[loss=0.3588, over 2743005.03 frames; utt_duration=1332 frames, utt_pad_proportion=0.03808, over 8246.08 utterances.], batch size: 42, lr: 3.84e-04
|
| 81 |
+
2022-08-11 09:06:24,544 INFO [train.py:737] (0/3) Epoch 13, batch 700, loss[loss=0.3988, over 14653.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.007158, over 44.00 utterances.], tot_loss[loss=0.36, over 2791915.30 frames; utt_duration=1322 frames, utt_pad_proportion=0.0415, over 8454.72 utterances.], batch size: 44, lr: 3.84e-04
|
| 82 |
+
2022-08-11 09:09:10,555 INFO [train.py:737] (0/3) Epoch 13, batch 800, loss[loss=0.3452, over 14757.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.00659, over 40.00 utterances.], tot_loss[loss=0.3629, over 2819865.09 frames; utt_duration=1298 frames, utt_pad_proportion=0.04399, over 8702.34 utterances.], batch size: 40, lr: 3.84e-04
|
| 83 |
+
2022-08-11 09:11:56,854 INFO [train.py:737] (0/3) Epoch 13, batch 900, loss[loss=0.3866, over 14758.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005538, over 41.00 utterances.], tot_loss[loss=0.3617, over 2845884.66 frames; utt_duration=1321 frames, utt_pad_proportion=0.03967, over 8626.54 utterances.], batch size: 41, lr: 3.83e-04
|
| 84 |
+
2022-08-11 09:14:45,135 INFO [train.py:737] (0/3) Epoch 13, batch 1000, loss[loss=0.3692, over 14576.00 frames; utt_duration=1496 frames, utt_pad_proportion=0.006521, over 39.00 utterances.], tot_loss[loss=0.3644, over 2855691.27 frames; utt_duration=1317 frames, utt_pad_proportion=0.04254, over 8685.03 utterances.], batch size: 39, lr: 3.83e-04
|
| 85 |
+
2022-08-11 09:17:11,267 INFO [train.py:737] (0/3) Epoch 13, batch 1100, loss[loss=0.4359, over 14594.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.006511, over 38.00 utterances.], tot_loss[loss=0.3655, over 2872184.86 frames; utt_duration=1332 frames, utt_pad_proportion=0.03615, over 8636.61 utterances.], batch size: 38, lr: 3.83e-04
|
| 86 |
+
2022-08-11 09:19:38,557 INFO [train.py:737] (0/3) Epoch 13, batch 1200, loss[loss=0.4399, over 14585.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.005314, over 41.00 utterances.], tot_loss[loss=0.3649, over 2864465.85 frames; utt_duration=1304 frames, utt_pad_proportion=0.04377, over 8795.18 utterances.], batch size: 41, lr: 3.83e-04
|
| 87 |
+
2022-08-11 09:22:05,179 INFO [train.py:737] (0/3) Epoch 13, batch 1300, loss[loss=0.3236, over 14590.00 frames; utt_duration=1718 frames, utt_pad_proportion=0.01142, over 34.00 utterances.], tot_loss[loss=0.3657, over 2868756.38 frames; utt_duration=1298 frames, utt_pad_proportion=0.04271, over 8853.34 utterances.], batch size: 34, lr: 3.83e-04
|
| 88 |
+
2022-08-11 09:24:12,087 INFO [train.py:737] (0/3) Epoch 13, batch 1400, loss[loss=0.3103, over 14776.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.005948, over 39.00 utterances.], tot_loss[loss=0.368, over 2838883.29 frames; utt_duration=1234 frames, utt_pad_proportion=0.06379, over 9216.74 utterances.], batch size: 39, lr: 3.83e-04
|
| 89 |
+
2022-08-11 09:26:58,346 INFO [train.py:737] (0/3) Epoch 13, batch 1500, loss[loss=0.3025, over 14039.00 frames; utt_duration=1813 frames, utt_pad_proportion=0.05808, over 31.00 utterances.], tot_loss[loss=0.3667, over 2855015.90 frames; utt_duration=1270 frames, utt_pad_proportion=0.05465, over 9005.87 utterances.], batch size: 31, lr: 3.83e-04
|
| 90 |
+
2022-08-11 09:29:51,305 INFO [train.py:737] (0/3) Epoch 13, batch 1600, loss[loss=0.334, over 14559.00 frames; utt_duration=1575 frames, utt_pad_proportion=0.007306, over 37.00 utterances.], tot_loss[loss=0.3647, over 2853987.24 frames; utt_duration=1277 frames, utt_pad_proportion=0.05542, over 8949.91 utterances.], batch size: 37, lr: 3.82e-04
|
| 91 |
+
2022-08-11 09:32:41,586 INFO [train.py:737] (0/3) Epoch 13, batch 1700, loss[loss=0.3751, over 13004.00 frames; utt_duration=766.5 frames, utt_pad_proportion=0.1299, over 68.00 utterances.], tot_loss[loss=0.3656, over 2861673.59 frames; utt_duration=1291 frames, utt_pad_proportion=0.0498, over 8877.25 utterances.], batch size: 68, lr: 3.82e-04
|
| 92 |
+
2022-08-11 09:34:42,370 INFO [train.py:737] (0/3) Epoch 13, batch 1800, loss[loss=0.4043, over 14496.00 frames; utt_duration=1161 frames, utt_pad_proportion=0.02254, over 50.00 utterances.], tot_loss[loss=0.3657, over 2870306.89 frames; utt_duration=1306 frames, utt_pad_proportion=0.04456, over 8803.14 utterances.], batch size: 50, lr: 3.82e-04
|
| 93 |
+
2022-08-11 09:37:23,736 INFO [train.py:737] (0/3) Epoch 13, batch 1900, loss[loss=0.3474, over 14456.00 frames; utt_duration=1754 frames, utt_pad_proportion=0.0108, over 33.00 utterances.], tot_loss[loss=0.3667, over 2858408.75 frames; utt_duration=1269 frames, utt_pad_proportion=0.051, over 9021.48 utterances.], batch size: 33, lr: 3.82e-04
|
| 94 |
+
2022-08-11 09:39:48,330 INFO [train.py:737] (0/3) Epoch 13, batch 2000, loss[loss=0.3995, over 13891.00 frames; utt_duration=976.1 frames, utt_pad_proportion=0.05877, over 57.00 utterances.], tot_loss[loss=0.3653, over 2849599.72 frames; utt_duration=1252 frames, utt_pad_proportion=0.05618, over 9112.30 utterances.], batch size: 57, lr: 3.82e-04
|
| 95 |
+
2022-08-11 09:42:14,729 INFO [train.py:737] (0/3) Epoch 13, batch 2100, loss[loss=0.3129, over 14481.00 frames; utt_duration=1757 frames, utt_pad_proportion=0.009651, over 33.00 utterances.], tot_loss[loss=0.3673, over 2845466.60 frames; utt_duration=1238 frames, utt_pad_proportion=0.06127, over 9203.52 utterances.], batch size: 33, lr: 3.82e-04
|
| 96 |
+
2022-08-11 09:44:38,435 INFO [train.py:737] (0/3) Epoch 13, batch 2200, loss[loss=0.3852, over 14447.00 frames; utt_duration=1157 frames, utt_pad_proportion=0.02511, over 50.00 utterances.], tot_loss[loss=0.364, over 2859050.41 frames; utt_duration=1272 frames, utt_pad_proportion=0.05222, over 8997.82 utterances.], batch size: 50, lr: 3.82e-04
|
| 97 |
+
2022-08-11 09:47:02,615 INFO [train.py:737] (0/3) Epoch 13, batch 2300, loss[loss=0.36, over 12982.00 frames; utt_duration=776.4 frames, utt_pad_proportion=0.1237, over 67.00 utterances.], tot_loss[loss=0.3631, over 2863896.86 frames; utt_duration=1286 frames, utt_pad_proportion=0.0485, over 8920.97 utterances.], batch size: 67, lr: 3.81e-04
|
| 98 |
+
2022-08-11 09:49:33,835 INFO [train.py:737] (0/3) Epoch 13, batch 2400, loss[loss=0.3538, over 14591.00 frames; utt_duration=1718 frames, utt_pad_proportion=0.01079, over 34.00 utterances.], tot_loss[loss=0.3612, over 2867056.77 frames; utt_duration=1297 frames, utt_pad_proportion=0.04523, over 8852.37 utterances.], batch size: 34, lr: 3.81e-04
|
| 99 |
+
2022-08-11 09:52:02,611 INFO [train.py:737] (0/3) Epoch 13, batch 2500, loss[loss=0.3458, over 14770.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007504, over 37.00 utterances.], tot_loss[loss=0.3638, over 2863797.51 frames; utt_duration=1291 frames, utt_pad_proportion=0.04731, over 8883.58 utterances.], batch size: 37, lr: 3.81e-04
|
| 100 |
+
2022-08-11 09:54:20,859 INFO [train.py:737] (0/3) Epoch 13, batch 2600, loss[loss=0.3896, over 13941.00 frames; utt_duration=979.9 frames, utt_pad_proportion=0.05685, over 57.00 utterances.], tot_loss[loss=0.3664, over 2873779.97 frames; utt_duration=1306 frames, utt_pad_proportion=0.04063, over 8812.95 utterances.], batch size: 57, lr: 3.81e-04
|
| 101 |
+
2022-08-11 09:56:50,910 INFO [train.py:737] (0/3) Epoch 13, batch 2700, loss[loss=0.4127, over 14485.00 frames; utt_duration=1160 frames, utt_pad_proportion=0.02347, over 50.00 utterances.], tot_loss[loss=0.3636, over 2868626.22 frames; utt_duration=1303 frames, utt_pad_proportion=0.0447, over 8817.54 utterances.], batch size: 50, lr: 3.81e-04
|
| 102 |
+
2022-08-11 09:59:17,872 INFO [train.py:737] (0/3) Epoch 13, batch 2800, loss[loss=0.4066, over 14523.00 frames; utt_duration=1212 frames, utt_pad_proportion=0.01637, over 48.00 utterances.], tot_loss[loss=0.3673, over 2862946.37 frames; utt_duration=1293 frames, utt_pad_proportion=0.04939, over 8869.59 utterances.], batch size: 48, lr: 3.81e-04
|
| 103 |
+
2022-08-11 10:01:53,829 INFO [train.py:737] (0/3) Epoch 13, batch 2900, loss[loss=0.4804, over 14874.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.007565, over 44.00 utterances.], tot_loss[loss=0.3822, over 2855945.84 frames; utt_duration=1275 frames, utt_pad_proportion=0.05603, over 8967.04 utterances.], batch size: 44, lr: 3.81e-04
|
| 104 |
+
2022-08-11 10:04:16,978 INFO [train.py:737] (0/3) Epoch 13, batch 3000, loss[loss=0.4188, over 14379.00 frames; utt_duration=1799 frames, utt_pad_proportion=0.03551, over 32.00 utterances.], tot_loss[loss=0.396, over 2861384.64 frames; utt_duration=1300 frames, utt_pad_proportion=0.0521, over 8817.52 utterances.], batch size: 32, lr: 3.81e-04
|
| 105 |
+
2022-08-11 10:04:16,979 INFO [train.py:761] (0/3) Computing validation loss
|
| 106 |
+
2022-08-11 10:04:50,562 INFO [train.py:770] (0/3) Epoch 13, validation: loss=0.3727, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 107 |
+
2022-08-11 10:06:51,671 INFO [train.py:737] (0/3) Epoch 13, batch 3100, loss[loss=0.4163, over 14558.00 frames; utt_duration=1575 frames, utt_pad_proportion=0.007357, over 37.00 utterances.], tot_loss[loss=0.4117, over 2835675.87 frames; utt_duration=1236 frames, utt_pad_proportion=0.06722, over 9186.64 utterances.], batch size: 37, lr: 3.80e-04
|
| 108 |
+
2022-08-11 10:09:22,794 INFO [train.py:737] (0/3) Epoch 13, batch 3200, loss[loss=0.4788, over 9686.00 frames; utt_duration=422.5 frames, utt_pad_proportion=0.3491, over 92.00 utterances.], tot_loss[loss=0.4214, over 2824094.26 frames; utt_duration=1210 frames, utt_pad_proportion=0.07294, over 9350.43 utterances.], batch size: 92, lr: 3.80e-04
|
| 109 |
+
2022-08-11 10:11:51,394 INFO [train.py:737] (0/3) Epoch 13, batch 3300, loss[loss=0.3953, over 14573.00 frames; utt_duration=1496 frames, utt_pad_proportion=0.006014, over 39.00 utterances.], tot_loss[loss=0.4288, over 2834786.02 frames; utt_duration=1223 frames, utt_pad_proportion=0.06485, over 9284.72 utterances.], batch size: 39, lr: 3.80e-04
|
| 110 |
+
2022-08-11 10:14:06,316 INFO [train.py:737] (0/3) Epoch 13, batch 3400, loss[loss=0.4349, over 14594.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.005835, over 38.00 utterances.], tot_loss[loss=0.4381, over 2839056.18 frames; utt_duration=1225 frames, utt_pad_proportion=0.06377, over 9280.29 utterances.], batch size: 38, lr: 3.80e-04
|
| 111 |
+
2022-08-11 10:16:12,568 INFO [train.py:737] (0/3) Epoch 13, batch 3500, loss[loss=0.4501, over 14430.00 frames; utt_duration=1651 frames, utt_pad_proportion=0.009735, over 35.00 utterances.], tot_loss[loss=0.4433, over 2834292.49 frames; utt_duration=1218 frames, utt_pad_proportion=0.06523, over 9321.01 utterances.], batch size: 35, lr: 3.80e-04
|
| 112 |
+
2022-08-11 10:18:49,064 INFO [train.py:737] (0/3) Epoch 13, batch 3600, loss[loss=0.4546, over 14766.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005751, over 42.00 utterances.], tot_loss[loss=0.4458, over 2828646.20 frames; utt_duration=1204 frames, utt_pad_proportion=0.06878, over 9407.36 utterances.], batch size: 42, lr: 3.80e-04
|
| 113 |
+
2022-08-11 10:21:10,814 INFO [train.py:737] (0/3) Epoch 13, batch 3700, loss[loss=0.4351, over 14873.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.007765, over 44.00 utterances.], tot_loss[loss=0.4448, over 2830501.71 frames; utt_duration=1213 frames, utt_pad_proportion=0.06898, over 9345.73 utterances.], batch size: 44, lr: 3.80e-04
|
| 114 |
+
2022-08-11 10:23:39,229 INFO [train.py:737] (0/3) Epoch 13, batch 3800, loss[loss=0.4108, over 14583.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.005435, over 39.00 utterances.], tot_loss[loss=0.4442, over 2826904.65 frames; utt_duration=1207 frames, utt_pad_proportion=0.07074, over 9383.53 utterances.], batch size: 39, lr: 3.79e-04
|
| 115 |
+
2022-08-11 10:26:00,933 INFO [train.py:737] (0/3) Epoch 13, batch 3900, loss[loss=0.423, over 14414.00 frames; utt_duration=1649 frames, utt_pad_proportion=0.01073, over 35.00 utterances.], tot_loss[loss=0.4456, over 2822135.31 frames; utt_duration=1202 frames, utt_pad_proportion=0.07357, over 9402.51 utterances.], batch size: 35, lr: 3.79e-04
|
| 116 |
+
2022-08-11 10:28:33,729 INFO [train.py:737] (0/3) Epoch 13, batch 4000, loss[loss=0.394, over 14761.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.005016, over 41.00 utterances.], tot_loss[loss=0.4488, over 2821655.83 frames; utt_duration=1196 frames, utt_pad_proportion=0.07414, over 9445.15 utterances.], batch size: 41, lr: 3.79e-04
|
| 117 |
+
2022-08-11 10:30:27,525 INFO [train.py:737] (0/3) Epoch 13, batch 4100, loss[loss=0.5414, over 9584.00 frames; utt_duration=418.2 frames, utt_pad_proportion=0.3566, over 92.00 utterances.], tot_loss[loss=0.4468, over 2819566.06 frames; utt_duration=1195 frames, utt_pad_proportion=0.07627, over 9448.84 utterances.], batch size: 92, lr: 3.79e-04
|
| 118 |
+
2022-08-11 10:32:29,521 INFO [train.py:737] (0/3) Epoch 13, batch 4200, loss[loss=0.4164, over 14781.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.005445, over 38.00 utterances.], tot_loss[loss=0.445, over 2826404.78 frames; utt_duration=1203 frames, utt_pad_proportion=0.07163, over 9411.00 utterances.], batch size: 38, lr: 3.79e-04
|
| 119 |
+
2022-08-11 10:34:32,612 INFO [train.py:737] (0/3) Epoch 13, batch 4300, loss[loss=0.5155, over 9760.00 frames; utt_duration=421.5 frames, utt_pad_proportion=0.3446, over 93.00 utterances.], tot_loss[loss=0.4439, over 2827099.58 frames; utt_duration=1212 frames, utt_pad_proportion=0.07229, over 9345.34 utterances.], batch size: 93, lr: 3.79e-04
|
| 120 |
+
2022-08-11 10:36:59,752 INFO [train.py:737] (0/3) Epoch 13, batch 4400, loss[loss=0.5376, over 14681.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009367, over 45.00 utterances.], tot_loss[loss=0.4459, over 2829279.56 frames; utt_duration=1212 frames, utt_pad_proportion=0.07119, over 9347.53 utterances.], batch size: 45, lr: 3.79e-04
|
| 121 |
+
2022-08-11 10:40:20,192 INFO [train.py:737] (0/3) Epoch 13, batch 4500, loss[loss=0.3438, over 14736.00 frames; utt_duration=1685 frames, utt_pad_proportion=0.009753, over 35.00 utterances.], tot_loss[loss=0.4473, over 2824630.00 frames; utt_duration=1200 frames, utt_pad_proportion=0.07261, over 9431.07 utterances.], batch size: 35, lr: 3.78e-04
|
| 122 |
+
2022-08-11 10:44:36,547 INFO [train.py:737] (0/3) Epoch 13, batch 4600, loss[loss=0.3602, over 14760.00 frames; utt_duration=1515 frames, utt_pad_proportion=0.006973, over 39.00 utterances.], tot_loss[loss=0.4456, over 2812207.41 frames; utt_duration=1183 frames, utt_pad_proportion=0.08126, over 9523.96 utterances.], batch size: 39, lr: 3.78e-04
|
| 123 |
+
2022-08-11 10:47:17,028 INFO [train.py:737] (0/3) Epoch 13, batch 4700, loss[loss=0.4367, over 14773.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.0061, over 38.00 utterances.], tot_loss[loss=0.4455, over 2808211.61 frames; utt_duration=1186 frames, utt_pad_proportion=0.08325, over 9482.79 utterances.], batch size: 38, lr: 3.78e-04
|
| 124 |
+
2022-08-11 10:50:52,813 INFO [train.py:737] (0/3) Epoch 13, batch 4800, loss[loss=0.4648, over 12952.00 frames; utt_duration=774.7 frames, utt_pad_proportion=0.1247, over 67.00 utterances.], tot_loss[loss=0.444, over 2823614.94 frames; utt_duration=1203 frames, utt_pad_proportion=0.07332, over 9397.07 utterances.], batch size: 67, lr: 3.78e-04
|
| 125 |
+
2022-08-11 10:54:02,825 INFO [train.py:737] (0/3) Epoch 13, batch 4900, loss[loss=0.5373, over 14686.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009916, over 45.00 utterances.], tot_loss[loss=0.4437, over 2835037.49 frames; utt_duration=1227 frames, utt_pad_proportion=0.06728, over 9252.72 utterances.], batch size: 45, lr: 3.78e-04
|
| 126 |
+
2022-08-11 10:59:10,776 INFO [train.py:737] (0/3) Epoch 13, batch 5000, loss[loss=0.3545, over 14768.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007218, over 37.00 utterances.], tot_loss[loss=0.4447, over 2813786.78 frames; utt_duration=1187 frames, utt_pad_proportion=0.07966, over 9497.04 utterances.], batch size: 37, lr: 3.78e-04
|
| 127 |
+
2022-08-11 11:02:20,135 INFO [train.py:737] (0/3) Epoch 13, batch 5100, loss[loss=0.4496, over 14575.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.005842, over 42.00 utterances.], tot_loss[loss=0.4443, over 2821741.57 frames; utt_duration=1205 frames, utt_pad_proportion=0.07559, over 9378.53 utterances.], batch size: 42, lr: 3.78e-04
|
| 128 |
+
2022-08-11 11:05:44,513 INFO [train.py:737] (0/3) Epoch 13, batch 5200, loss[loss=0.4202, over 14724.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.00991, over 35.00 utterances.], tot_loss[loss=0.4398, over 2828108.47 frames; utt_duration=1220 frames, utt_pad_proportion=0.07161, over 9284.88 utterances.], batch size: 35, lr: 3.78e-04
|
| 129 |
+
2022-08-11 11:09:00,837 INFO [train.py:737] (0/3) Epoch 13, batch 5300, loss[loss=0.5405, over 9611.00 frames; utt_duration=424.1 frames, utt_pad_proportion=0.3506, over 91.00 utterances.], tot_loss[loss=0.4426, over 2833817.44 frames; utt_duration=1232 frames, utt_pad_proportion=0.06832, over 9213.08 utterances.], batch size: 91, lr: 3.77e-04
|
| 130 |
+
2022-08-11 11:14:26,398 INFO [train.py:737] (0/3) Epoch 13, batch 5400, loss[loss=0.4662, over 12955.00 frames; utt_duration=774.8 frames, utt_pad_proportion=0.1235, over 67.00 utterances.], tot_loss[loss=0.4446, over 2836739.43 frames; utt_duration=1238 frames, utt_pad_proportion=0.06561, over 9173.49 utterances.], batch size: 67, lr: 3.77e-04
|
| 131 |
+
2022-08-11 11:18:27,397 INFO [train.py:737] (0/3) Epoch 13, batch 5500, loss[loss=0.4048, over 14639.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.008241, over 44.00 utterances.], tot_loss[loss=0.4466, over 2824028.81 frames; utt_duration=1198 frames, utt_pad_proportion=0.07301, over 9438.54 utterances.], batch size: 44, lr: 3.77e-04
|
| 132 |
+
2022-08-11 11:21:56,162 INFO [train.py:737] (0/3) Epoch 13, batch 5600, loss[loss=0.425, over 14578.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.006231, over 39.00 utterances.], tot_loss[loss=0.4457, over 2838138.10 frames; utt_duration=1225 frames, utt_pad_proportion=0.06479, over 9276.23 utterances.], batch size: 39, lr: 3.77e-04
|
| 133 |
+
2022-08-11 11:24:38,524 INFO [train.py:737] (0/3) Epoch 13, batch 5700, loss[loss=0.4532, over 14562.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006846, over 40.00 utterances.], tot_loss[loss=0.443, over 2828709.55 frames; utt_duration=1214 frames, utt_pad_proportion=0.0694, over 9332.04 utterances.], batch size: 40, lr: 3.77e-04
|
| 134 |
+
2022-08-11 11:30:07,785 INFO [train.py:737] (0/3) Epoch 13, batch 5800, loss[loss=0.4407, over 14574.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.006267, over 40.00 utterances.], tot_loss[loss=0.445, over 2823526.11 frames; utt_duration=1208 frames, utt_pad_proportion=0.07375, over 9359.35 utterances.], batch size: 40, lr: 3.77e-04
|
| 135 |
+
2022-08-11 11:33:55,336 INFO [train.py:737] (0/3) Epoch 13, batch 5900, loss[loss=0.4929, over 14884.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.006303, over 44.00 utterances.], tot_loss[loss=0.4415, over 2831909.87 frames; utt_duration=1229 frames, utt_pad_proportion=0.06963, over 9230.24 utterances.], batch size: 44, lr: 3.77e-04
|
| 136 |
+
2022-08-11 11:38:37,936 INFO [train.py:737] (0/3) Epoch 13, batch 6000, loss[loss=0.3814, over 14446.00 frames; utt_duration=1653 frames, utt_pad_proportion=0.008655, over 35.00 utterances.], tot_loss[loss=0.4429, over 2842749.98 frames; utt_duration=1240 frames, utt_pad_proportion=0.0621, over 9178.55 utterances.], batch size: 35, lr: 3.76e-04
|
| 137 |
+
2022-08-11 11:38:37,937 INFO [train.py:761] (0/3) Computing validation loss
|
| 138 |
+
2022-08-11 11:39:50,679 INFO [train.py:770] (0/3) Epoch 13, validation: loss=0.2858, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 139 |
+
2022-08-11 11:44:40,577 INFO [train.py:737] (0/3) Epoch 13, batch 6100, loss[loss=0.55, over 10218.00 frames; utt_duration=445.9 frames, utt_pad_proportion=0.3141, over 92.00 utterances.], tot_loss[loss=0.4449, over 2779602.57 frames; utt_duration=1133 frames, utt_pad_proportion=0.09951, over 9829.88 utterances.], batch size: 92, lr: 3.76e-04
|
| 140 |
+
2022-08-11 11:45:13,938 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-13.pt
|
| 141 |
+
2022-08-11 11:46:31,263 INFO [train.py:737] (0/3) Epoch 14, batch 0, loss[loss=0.4175, over 14681.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.00902, over 45.00 utterances.], tot_loss[loss=0.4175, over 14681.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.00902, over 45.00 utterances.], batch size: 45, lr: 3.64e-04
|
| 142 |
+
2022-08-11 11:50:49,319 INFO [train.py:737] (0/3) Epoch 14, batch 100, loss[loss=0.412, over 14767.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.004797, over 41.00 utterances.], tot_loss[loss=0.3639, over 1146649.72 frames; utt_duration=1331 frames, utt_pad_proportion=0.0357, over 3449.87 utterances.], batch size: 41, lr: 3.64e-04
|
| 143 |
+
2022-08-11 11:54:33,951 INFO [train.py:737] (0/3) Epoch 14, batch 200, loss[loss=0.3602, over 14667.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.01291, over 47.00 utterances.], tot_loss[loss=0.3573, over 1813054.07 frames; utt_duration=1284 frames, utt_pad_proportion=0.05543, over 5653.86 utterances.], batch size: 47, lr: 3.64e-04
|
| 144 |
+
2022-08-11 11:57:32,568 INFO [train.py:737] (0/3) Epoch 14, batch 300, loss[loss=0.3493, over 14868.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.008031, over 44.00 utterances.], tot_loss[loss=0.3583, over 2234759.66 frames; utt_duration=1310 frames, utt_pad_proportion=0.04463, over 6829.56 utterances.], batch size: 44, lr: 3.64e-04
|
| 145 |
+
2022-08-11 12:02:50,131 INFO [train.py:737] (0/3) Epoch 14, batch 400, loss[loss=0.3658, over 14746.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.006546, over 43.00 utterances.], tot_loss[loss=0.3569, over 2489795.01 frames; utt_duration=1326 frames, utt_pad_proportion=0.04213, over 7519.48 utterances.], batch size: 43, lr: 3.64e-04
|
| 146 |
+
2022-08-11 12:04:25,713 INFO [train.py:737] (0/3) Epoch 14, batch 500, loss[loss=0.3636, over 14863.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.007453, over 44.00 utterances.], tot_loss[loss=0.3555, over 2649957.73 frames; utt_duration=1330 frames, utt_pad_proportion=0.03714, over 7978.44 utterances.], batch size: 44, lr: 3.64e-04
|
| 147 |
+
2022-08-11 12:10:29,636 INFO [train.py:737] (0/3) Epoch 14, batch 600, loss[loss=0.3437, over 14881.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.00642, over 44.00 utterances.], tot_loss[loss=0.3591, over 2742122.14 frames; utt_duration=1323 frames, utt_pad_proportion=0.03608, over 8298.24 utterances.], batch size: 44, lr: 3.64e-04
|
| 148 |
+
2022-08-11 12:15:28,028 INFO [train.py:737] (0/3) Epoch 14, batch 700, loss[loss=0.3377, over 14578.00 frames; utt_duration=1578 frames, utt_pad_proportion=0.005756, over 37.00 utterances.], tot_loss[loss=0.3595, over 2789570.39 frames; utt_duration=1320 frames, utt_pad_proportion=0.04112, over 8461.63 utterances.], batch size: 37, lr: 3.63e-04
|
| 149 |
+
2022-08-11 12:17:32,620 INFO [train.py:737] (0/3) Epoch 14, batch 800, loss[loss=0.3205, over 14449.00 frames; utt_duration=1653 frames, utt_pad_proportion=0.008553, over 35.00 utterances.], tot_loss[loss=0.3632, over 2819689.29 frames; utt_duration=1315 frames, utt_pad_proportion=0.04499, over 8587.80 utterances.], batch size: 35, lr: 3.63e-04
|
| 150 |
+
2022-08-11 12:22:44,897 INFO [train.py:737] (0/3) Epoch 14, batch 900, loss[loss=0.3554, over 14683.00 frames; utt_duration=1251 frames, utt_pad_proportion=0.01165, over 47.00 utterances.], tot_loss[loss=0.3612, over 2842521.03 frames; utt_duration=1325 frames, utt_pad_proportion=0.04409, over 8592.46 utterances.], batch size: 47, lr: 3.63e-04
|
| 151 |
+
2022-08-11 12:28:18,726 INFO [train.py:737] (0/3) Epoch 14, batch 1000, loss[loss=0.3741, over 14579.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.005603, over 42.00 utterances.], tot_loss[loss=0.362, over 2858509.13 frames; utt_duration=1328 frames, utt_pad_proportion=0.04174, over 8622.25 utterances.], batch size: 42, lr: 3.63e-04
|
| 152 |
+
2022-08-11 12:32:01,210 INFO [train.py:737] (0/3) Epoch 14, batch 1100, loss[loss=0.3386, over 14575.00 frames; utt_duration=1716 frames, utt_pad_proportion=0.01149, over 34.00 utterances.], tot_loss[loss=0.3623, over 2860907.99 frames; utt_duration=1311 frames, utt_pad_proportion=0.04423, over 8739.72 utterances.], batch size: 34, lr: 3.63e-04
|
| 153 |
+
2022-08-11 12:33:29,132 INFO [train.py:737] (0/3) Epoch 14, batch 1200, loss[loss=0.3218, over 14468.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01008, over 33.00 utterances.], tot_loss[loss=0.3628, over 2853935.71 frames; utt_duration=1290 frames, utt_pad_proportion=0.05094, over 8859.42 utterances.], batch size: 33, lr: 3.63e-04
|
| 154 |
+
2022-08-11 12:34:33,002 INFO [train.py:737] (0/3) Epoch 14, batch 1300, loss[loss=0.3273, over 14585.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.008286, over 36.00 utterances.], tot_loss[loss=0.3663, over 2853467.69 frames; utt_duration=1281 frames, utt_pad_proportion=0.05442, over 8921.84 utterances.], batch size: 36, lr: 3.63e-04
|
| 155 |
+
2022-08-11 12:39:36,348 INFO [train.py:737] (0/3) Epoch 14, batch 1400, loss[loss=0.422, over 14883.00 frames; utt_duration=1355 frames, utt_pad_proportion=0.006053, over 44.00 utterances.], tot_loss[loss=0.3684, over 2864708.57 frames; utt_duration=1294 frames, utt_pad_proportion=0.04845, over 8868.09 utterances.], batch size: 44, lr: 3.63e-04
|
| 156 |
+
2022-08-11 12:45:34,373 INFO [train.py:737] (0/3) Epoch 14, batch 1500, loss[loss=0.3728, over 14584.00 frames; utt_duration=1498 frames, utt_pad_proportion=0.005584, over 39.00 utterances.], tot_loss[loss=0.3674, over 2859823.03 frames; utt_duration=1273 frames, utt_pad_proportion=0.05154, over 8998.25 utterances.], batch size: 39, lr: 3.62e-04
|
| 157 |
+
2022-08-11 12:50:14,994 INFO [train.py:737] (0/3) Epoch 14, batch 1600, loss[loss=0.3308, over 14587.00 frames; utt_duration=1717 frames, utt_pad_proportion=0.01074, over 34.00 utterances.], tot_loss[loss=0.366, over 2859892.70 frames; utt_duration=1269 frames, utt_pad_proportion=0.05102, over 9028.13 utterances.], batch size: 34, lr: 3.62e-04
|
| 158 |
+
2022-08-11 12:54:22,870 INFO [train.py:737] (0/3) Epoch 14, batch 1700, loss[loss=0.3638, over 14755.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.00537, over 41.00 utterances.], tot_loss[loss=0.3686, over 2865306.85 frames; utt_duration=1282 frames, utt_pad_proportion=0.04872, over 8951.90 utterances.], batch size: 41, lr: 3.62e-04
|
| 159 |
+
2022-08-11 12:58:59,137 INFO [train.py:737] (0/3) Epoch 14, batch 1800, loss[loss=0.5037, over 9514.00 frames; utt_duration=415.1 frames, utt_pad_proportion=0.3593, over 92.00 utterances.], tot_loss[loss=0.3696, over 2857812.05 frames; utt_duration=1260 frames, utt_pad_proportion=0.05301, over 9084.84 utterances.], batch size: 92, lr: 3.62e-04
|
| 160 |
+
2022-08-11 13:03:45,766 INFO [train.py:737] (0/3) Epoch 14, batch 1900, loss[loss=0.2938, over 14774.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007269, over 37.00 utterances.], tot_loss[loss=0.364, over 2859323.58 frames; utt_duration=1277 frames, utt_pad_proportion=0.05211, over 8970.31 utterances.], batch size: 37, lr: 3.62e-04
|
| 161 |
+
2022-08-11 13:07:32,813 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/checkpoint-144000.pt
|
| 162 |
+
2022-08-11 13:08:02,546 INFO [train.py:737] (0/3) Epoch 14, batch 2000, loss[loss=0.3401, over 14756.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006456, over 40.00 utterances.], tot_loss[loss=0.3621, over 2866801.39 frames; utt_duration=1293 frames, utt_pad_proportion=0.04691, over 8879.21 utterances.], batch size: 40, lr: 3.62e-04
|
| 163 |
+
2022-08-11 13:11:18,011 INFO [train.py:737] (0/3) Epoch 14, batch 2100, loss[loss=0.2755, over 14439.00 frames; utt_duration=1752 frames, utt_pad_proportion=0.01242, over 33.00 utterances.], tot_loss[loss=0.362, over 2877228.41 frames; utt_duration=1305 frames, utt_pad_proportion=0.04028, over 8828.40 utterances.], batch size: 33, lr: 3.62e-04
|
| 164 |
+
2022-08-11 13:17:21,267 INFO [train.py:737] (0/3) Epoch 14, batch 2200, loss[loss=0.3702, over 14760.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005218, over 41.00 utterances.], tot_loss[loss=0.3607, over 2876598.40 frames; utt_duration=1318 frames, utt_pad_proportion=0.04181, over 8737.41 utterances.], batch size: 41, lr: 3.62e-04
|
| 165 |
+
2022-08-11 13:22:13,819 INFO [train.py:737] (0/3) Epoch 14, batch 2300, loss[loss=0.3964, over 14754.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.006591, over 42.00 utterances.], tot_loss[loss=0.3622, over 2856702.15 frames; utt_duration=1277 frames, utt_pad_proportion=0.05481, over 8955.35 utterances.], batch size: 42, lr: 3.61e-04
|
| 166 |
+
2022-08-11 13:27:39,732 INFO [train.py:737] (0/3) Epoch 14, batch 2400, loss[loss=0.4357, over 14644.00 frames; utt_duration=1247 frames, utt_pad_proportion=0.0131, over 47.00 utterances.], tot_loss[loss=0.3641, over 2858359.66 frames; utt_duration=1276 frames, utt_pad_proportion=0.05497, over 8968.01 utterances.], batch size: 47, lr: 3.61e-04
|
| 167 |
+
2022-08-11 13:32:15,984 INFO [train.py:737] (0/3) Epoch 14, batch 2500, loss[loss=0.3847, over 14861.00 frames; utt_duration=1653 frames, utt_pad_proportion=0.00792, over 36.00 utterances.], tot_loss[loss=0.3647, over 2860291.79 frames; utt_duration=1278 frames, utt_pad_proportion=0.05362, over 8963.33 utterances.], batch size: 36, lr: 3.61e-04
|
| 168 |
+
2022-08-11 13:38:21,582 INFO [train.py:737] (0/3) Epoch 14, batch 2600, loss[loss=0.4955, over 9719.00 frames; utt_duration=415 frames, utt_pad_proportion=0.3496, over 94.00 utterances.], tot_loss[loss=0.3674, over 2830332.57 frames; utt_duration=1221 frames, utt_pad_proportion=0.0739, over 9281.42 utterances.], batch size: 94, lr: 3.61e-04
|
| 169 |
+
2022-08-11 13:45:02,853 INFO [train.py:737] (0/3) Epoch 14, batch 2700, loss[loss=0.39, over 13066.00 frames; utt_duration=781.5 frames, utt_pad_proportion=0.117, over 67.00 utterances.], tot_loss[loss=0.3663, over 2838859.28 frames; utt_duration=1237 frames, utt_pad_proportion=0.06558, over 9190.03 utterances.], batch size: 67, lr: 3.61e-04
|
| 170 |
+
2022-08-11 13:50:17,638 INFO [train.py:737] (0/3) Epoch 14, batch 2800, loss[loss=0.4509, over 10204.00 frames; utt_duration=450 frames, utt_pad_proportion=0.3077, over 91.00 utterances.], tot_loss[loss=0.3683, over 2828399.06 frames; utt_duration=1214 frames, utt_pad_proportion=0.06984, over 9329.77 utterances.], batch size: 91, lr: 3.61e-04
|
| 171 |
+
2022-08-11 13:56:31,411 INFO [train.py:737] (0/3) Epoch 14, batch 2900, loss[loss=0.4074, over 14600.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01033, over 34.00 utterances.], tot_loss[loss=0.3798, over 2840265.99 frames; utt_duration=1233 frames, utt_pad_proportion=0.06357, over 9227.39 utterances.], batch size: 34, lr: 3.61e-04
|
| 172 |
+
2022-08-11 14:01:08,226 INFO [train.py:737] (0/3) Epoch 14, batch 3000, loss[loss=0.4235, over 14659.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01275, over 47.00 utterances.], tot_loss[loss=0.3951, over 2840509.09 frames; utt_duration=1240 frames, utt_pad_proportion=0.06368, over 9171.30 utterances.], batch size: 47, lr: 3.61e-04
|
| 173 |
+
2022-08-11 14:01:08,227 INFO [train.py:761] (0/3) Computing validation loss
|
| 174 |
+
2022-08-11 14:03:06,560 INFO [train.py:770] (0/3) Epoch 14, validation: loss=0.2899, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 175 |
+
2022-08-11 14:07:35,708 INFO [train.py:737] (0/3) Epoch 14, batch 3100, loss[loss=0.4169, over 14526.00 frames; utt_duration=1212 frames, utt_pad_proportion=0.0152, over 48.00 utterances.], tot_loss[loss=0.4092, over 2835397.83 frames; utt_duration=1233 frames, utt_pad_proportion=0.06694, over 9211.17 utterances.], batch size: 48, lr: 3.60e-04
|
| 176 |
+
2022-08-11 14:14:22,391 INFO [train.py:737] (0/3) Epoch 14, batch 3200, loss[loss=0.3686, over 14846.00 frames; utt_duration=1651 frames, utt_pad_proportion=0.009104, over 36.00 utterances.], tot_loss[loss=0.4192, over 2819301.67 frames; utt_duration=1205 frames, utt_pad_proportion=0.07604, over 9370.96 utterances.], batch size: 36, lr: 3.60e-04
|
| 177 |
+
2022-08-11 14:20:07,874 INFO [train.py:737] (0/3) Epoch 14, batch 3300, loss[loss=0.461, over 14589.00 frames; utt_duration=1460 frames, utt_pad_proportion=0.004448, over 40.00 utterances.], tot_loss[loss=0.4263, over 2821600.72 frames; utt_duration=1205 frames, utt_pad_proportion=0.0748, over 9377.22 utterances.], batch size: 40, lr: 3.60e-04
|
| 178 |
+
2022-08-11 14:23:56,193 INFO [train.py:737] (0/3) Epoch 14, batch 3400, loss[loss=0.4091, over 14477.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.009976, over 33.00 utterances.], tot_loss[loss=0.4342, over 2822579.03 frames; utt_duration=1208 frames, utt_pad_proportion=0.07353, over 9359.51 utterances.], batch size: 33, lr: 3.60e-04
|
| 179 |
+
2022-08-11 14:26:47,079 INFO [train.py:737] (0/3) Epoch 14, batch 3500, loss[loss=0.4762, over 14706.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01128, over 46.00 utterances.], tot_loss[loss=0.4403, over 2816807.69 frames; utt_duration=1193 frames, utt_pad_proportion=0.07817, over 9456.84 utterances.], batch size: 46, lr: 3.60e-04
|
| 180 |
+
2022-08-11 14:29:40,026 INFO [train.py:737] (0/3) Epoch 14, batch 3600, loss[loss=0.4119, over 14611.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.009685, over 34.00 utterances.], tot_loss[loss=0.4398, over 2837332.99 frames; utt_duration=1230 frames, utt_pad_proportion=0.06658, over 9239.73 utterances.], batch size: 34, lr: 3.60e-04
|
| 181 |
+
2022-08-11 14:32:26,121 INFO [train.py:737] (0/3) Epoch 14, batch 3700, loss[loss=0.4099, over 14574.00 frames; utt_duration=1496 frames, utt_pad_proportion=0.005946, over 39.00 utterances.], tot_loss[loss=0.4412, over 2831393.01 frames; utt_duration=1226 frames, utt_pad_proportion=0.06958, over 9245.72 utterances.], batch size: 39, lr: 3.60e-04
|
| 182 |
+
2022-08-11 14:35:07,281 INFO [train.py:737] (0/3) Epoch 14, batch 3800, loss[loss=0.3905, over 14708.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.01131, over 35.00 utterances.], tot_loss[loss=0.441, over 2845749.20 frames; utt_duration=1253 frames, utt_pad_proportion=0.05911, over 9094.00 utterances.], batch size: 35, lr: 3.60e-04
|
| 183 |
+
2022-08-11 14:37:05,347 INFO [train.py:737] (0/3) Epoch 14, batch 3900, loss[loss=0.3638, over 14498.00 frames; utt_duration=1814 frames, utt_pad_proportion=0.02596, over 32.00 utterances.], tot_loss[loss=0.44, over 2845279.42 frames; utt_duration=1252 frames, utt_pad_proportion=0.05956, over 9103.16 utterances.], batch size: 32, lr: 3.59e-04
|
| 184 |
+
2022-08-11 14:39:09,209 INFO [train.py:737] (0/3) Epoch 14, batch 4000, loss[loss=0.4322, over 14574.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006335, over 37.00 utterances.], tot_loss[loss=0.4416, over 2852897.14 frames; utt_duration=1265 frames, utt_pad_proportion=0.05458, over 9028.94 utterances.], batch size: 37, lr: 3.59e-04
|
| 185 |
+
2022-08-11 14:41:42,025 INFO [train.py:737] (0/3) Epoch 14, batch 4100, loss[loss=0.3478, over 14740.00 frames; utt_duration=1686 frames, utt_pad_proportion=0.009417, over 35.00 utterances.], tot_loss[loss=0.4411, over 2828020.88 frames; utt_duration=1226 frames, utt_pad_proportion=0.07249, over 9238.90 utterances.], batch size: 35, lr: 3.59e-04
|
| 186 |
+
2022-08-11 14:44:27,292 INFO [train.py:737] (0/3) Epoch 14, batch 4200, loss[loss=0.5025, over 14710.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.01021, over 46.00 utterances.], tot_loss[loss=0.4419, over 2839133.41 frames; utt_duration=1248 frames, utt_pad_proportion=0.06568, over 9113.50 utterances.], batch size: 46, lr: 3.59e-04
|
| 187 |
+
2022-08-11 14:46:59,733 INFO [train.py:737] (0/3) Epoch 14, batch 4300, loss[loss=0.3759, over 14611.00 frames; utt_duration=1625 frames, utt_pad_proportion=0.006673, over 36.00 utterances.], tot_loss[loss=0.4418, over 2833169.00 frames; utt_duration=1223 frames, utt_pad_proportion=0.06775, over 9278.34 utterances.], batch size: 36, lr: 3.59e-04
|
| 188 |
+
2022-08-11 14:49:20,109 INFO [train.py:737] (0/3) Epoch 14, batch 4400, loss[loss=0.528, over 14610.00 frames; utt_duration=1330 frames, utt_pad_proportion=0.009815, over 44.00 utterances.], tot_loss[loss=0.4446, over 2826223.05 frames; utt_duration=1205 frames, utt_pad_proportion=0.07165, over 9390.57 utterances.], batch size: 44, lr: 3.59e-04
|
| 189 |
+
2022-08-11 14:51:53,409 INFO [train.py:737] (0/3) Epoch 14, batch 4500, loss[loss=0.3685, over 14603.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01011, over 34.00 utterances.], tot_loss[loss=0.4427, over 2820003.03 frames; utt_duration=1200 frames, utt_pad_proportion=0.07498, over 9412.09 utterances.], batch size: 34, lr: 3.59e-04
|
| 190 |
+
2022-08-11 14:54:39,323 INFO [train.py:737] (0/3) Epoch 14, batch 4600, loss[loss=0.4052, over 14567.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.005794, over 40.00 utterances.], tot_loss[loss=0.4415, over 2827840.92 frames; utt_duration=1216 frames, utt_pad_proportion=0.07014, over 9310.32 utterances.], batch size: 40, lr: 3.59e-04
|
| 191 |
+
2022-08-11 14:56:04,615 INFO [train.py:737] (0/3) Epoch 14, batch 4700, loss[loss=0.5488, over 9642.00 frames; utt_duration=416.1 frames, utt_pad_proportion=0.3549, over 93.00 utterances.], tot_loss[loss=0.4393, over 2831631.30 frames; utt_duration=1217 frames, utt_pad_proportion=0.06908, over 9316.47 utterances.], batch size: 93, lr: 3.58e-04
|
| 192 |
+
2022-08-11 15:00:58,233 INFO [train.py:737] (0/3) Epoch 14, batch 4800, loss[loss=0.4486, over 14872.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.00702, over 44.00 utterances.], tot_loss[loss=0.4403, over 2838898.10 frames; utt_duration=1237 frames, utt_pad_proportion=0.06596, over 9193.38 utterances.], batch size: 44, lr: 3.58e-04
|
| 193 |
+
2022-08-11 15:05:17,215 INFO [train.py:737] (0/3) Epoch 14, batch 4900, loss[loss=0.5322, over 10210.00 frames; utt_duration=450.4 frames, utt_pad_proportion=0.3103, over 91.00 utterances.], tot_loss[loss=0.4423, over 2841025.59 frames; utt_duration=1230 frames, utt_pad_proportion=0.06256, over 9252.65 utterances.], batch size: 91, lr: 3.58e-04
|
| 194 |
+
2022-08-11 15:10:37,126 INFO [train.py:737] (0/3) Epoch 14, batch 5000, loss[loss=0.5121, over 9942.00 frames; utt_duration=438.6 frames, utt_pad_proportion=0.3273, over 91.00 utterances.], tot_loss[loss=0.4425, over 2825673.69 frames; utt_duration=1206 frames, utt_pad_proportion=0.07185, over 9386.13 utterances.], batch size: 91, lr: 3.58e-04
|
| 195 |
+
2022-08-11 15:16:39,154 INFO [train.py:737] (0/3) Epoch 14, batch 5100, loss[loss=0.3739, over 14633.00 frames; utt_duration=1723 frames, utt_pad_proportion=0.008749, over 34.00 utterances.], tot_loss[loss=0.4404, over 2810947.20 frames; utt_duration=1189 frames, utt_pad_proportion=0.08122, over 9465.99 utterances.], batch size: 34, lr: 3.58e-04
|
| 196 |
+
2022-08-11 15:19:11,609 INFO [train.py:737] (0/3) Epoch 14, batch 5200, loss[loss=0.4393, over 14777.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.00573, over 38.00 utterances.], tot_loss[loss=0.4429, over 2825812.66 frames; utt_duration=1215 frames, utt_pad_proportion=0.07251, over 9318.39 utterances.], batch size: 38, lr: 3.58e-04
|
| 197 |
+
2022-08-11 15:22:03,093 INFO [train.py:737] (0/3) Epoch 14, batch 5300, loss[loss=0.4518, over 14708.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.01116, over 46.00 utterances.], tot_loss[loss=0.4434, over 2824449.58 frames; utt_duration=1212 frames, utt_pad_proportion=0.07227, over 9333.36 utterances.], batch size: 46, lr: 3.58e-04
|
| 198 |
+
2022-08-11 15:24:31,020 INFO [train.py:737] (0/3) Epoch 14, batch 5400, loss[loss=0.4955, over 14693.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01121, over 46.00 utterances.], tot_loss[loss=0.4459, over 2823831.74 frames; utt_duration=1214 frames, utt_pad_proportion=0.07285, over 9316.42 utterances.], batch size: 46, lr: 3.58e-04
|
| 199 |
+
2022-08-11 15:26:31,331 INFO [train.py:737] (0/3) Epoch 14, batch 5500, loss[loss=0.3722, over 13595.00 frames; utt_duration=1814 frames, utt_pad_proportion=0.0862, over 30.00 utterances.], tot_loss[loss=0.4434, over 2830024.16 frames; utt_duration=1222 frames, utt_pad_proportion=0.0667, over 9273.95 utterances.], batch size: 30, lr: 3.57e-04
|
| 200 |
+
2022-08-11 15:29:25,758 INFO [train.py:737] (0/3) Epoch 14, batch 5600, loss[loss=0.5195, over 14592.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005549, over 41.00 utterances.], tot_loss[loss=0.4422, over 2821159.18 frames; utt_duration=1204 frames, utt_pad_proportion=0.07387, over 9384.17 utterances.], batch size: 41, lr: 3.57e-04
|
| 201 |
+
2022-08-11 15:32:05,015 INFO [train.py:737] (0/3) Epoch 14, batch 5700, loss[loss=0.469, over 14580.00 frames; utt_duration=1460 frames, utt_pad_proportion=0.004857, over 40.00 utterances.], tot_loss[loss=0.4447, over 2811989.99 frames; utt_duration=1187 frames, utt_pad_proportion=0.0793, over 9485.49 utterances.], batch size: 40, lr: 3.57e-04
|
| 202 |
+
2022-08-11 15:34:48,482 INFO [train.py:737] (0/3) Epoch 14, batch 5800, loss[loss=0.3936, over 14761.00 frames; utt_duration=1478 frames, utt_pad_proportion=0.006321, over 40.00 utterances.], tot_loss[loss=0.4431, over 2812850.17 frames; utt_duration=1188 frames, utt_pad_proportion=0.07798, over 9480.13 utterances.], batch size: 40, lr: 3.57e-04
|
| 203 |
+
2022-08-11 15:37:21,099 INFO [train.py:737] (0/3) Epoch 14, batch 5900, loss[loss=0.4457, over 14636.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.008292, over 44.00 utterances.], tot_loss[loss=0.441, over 2822956.12 frames; utt_duration=1209 frames, utt_pad_proportion=0.0727, over 9349.48 utterances.], batch size: 44, lr: 3.57e-04
|
| 204 |
+
2022-08-11 15:39:22,181 INFO [train.py:737] (0/3) Epoch 14, batch 6000, loss[loss=0.3863, over 14776.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.006517, over 37.00 utterances.], tot_loss[loss=0.4399, over 2819759.20 frames; utt_duration=1206 frames, utt_pad_proportion=0.07545, over 9366.67 utterances.], batch size: 37, lr: 3.57e-04
|
| 205 |
+
2022-08-11 15:39:22,181 INFO [train.py:761] (0/3) Computing validation loss
|
| 206 |
+
2022-08-11 15:40:05,204 INFO [train.py:770] (0/3) Epoch 14, validation: loss=0.301, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 207 |
+
2022-08-11 15:42:37,945 INFO [train.py:737] (0/3) Epoch 14, batch 6100, loss[loss=0.485, over 9525.00 frames; utt_duration=415.8 frames, utt_pad_proportion=0.3603, over 92.00 utterances.], tot_loss[loss=0.443, over 2770732.28 frames; utt_duration=1132 frames, utt_pad_proportion=0.1044, over 9802.44 utterances.], batch size: 92, lr: 3.57e-04
|
| 208 |
+
2022-08-11 15:42:58,522 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-14.pt
|
| 209 |
+
2022-08-11 15:44:19,636 INFO [train.py:737] (0/3) Epoch 15, batch 0, loss[loss=0.4259, over 14352.00 frames; utt_duration=1085 frames, utt_pad_proportion=0.03399, over 53.00 utterances.], tot_loss[loss=0.4259, over 14352.00 frames; utt_duration=1085 frames, utt_pad_proportion=0.03399, over 53.00 utterances.], batch size: 53, lr: 3.46e-04
|
| 210 |
+
2022-08-11 15:47:34,443 INFO [train.py:737] (0/3) Epoch 15, batch 100, loss[loss=0.3799, over 14050.00 frames; utt_duration=1815 frames, utt_pad_proportion=0.04036, over 31.00 utterances.], tot_loss[loss=0.3614, over 1149353.40 frames; utt_duration=1342 frames, utt_pad_proportion=0.03235, over 3430.31 utterances.], batch size: 31, lr: 3.46e-04
|
| 211 |
+
2022-08-11 15:50:08,433 INFO [train.py:737] (0/3) Epoch 15, batch 200, loss[loss=0.372, over 14783.00 frames; utt_duration=1480 frames, utt_pad_proportion=0.004357, over 40.00 utterances.], tot_loss[loss=0.3597, over 1829124.63 frames; utt_duration=1336 frames, utt_pad_proportion=0.0409, over 5484.43 utterances.], batch size: 40, lr: 3.46e-04
|
| 212 |
+
2022-08-11 15:53:20,627 INFO [train.py:737] (0/3) Epoch 15, batch 300, loss[loss=0.3496, over 14756.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006523, over 40.00 utterances.], tot_loss[loss=0.3594, over 2233245.74 frames; utt_duration=1289 frames, utt_pad_proportion=0.04886, over 6940.51 utterances.], batch size: 40, lr: 3.45e-04
|
| 213 |
+
2022-08-11 15:55:13,361 INFO [train.py:737] (0/3) Epoch 15, batch 400, loss[loss=0.301, over 14756.00 frames; utt_duration=1515 frames, utt_pad_proportion=0.007276, over 39.00 utterances.], tot_loss[loss=0.3566, over 2482351.96 frames; utt_duration=1293 frames, utt_pad_proportion=0.04853, over 7688.58 utterances.], batch size: 39, lr: 3.45e-04
|
| 214 |
+
2022-08-11 15:57:32,263 INFO [train.py:737] (0/3) Epoch 15, batch 500, loss[loss=0.3623, over 14766.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.006554, over 38.00 utterances.], tot_loss[loss=0.3573, over 2637422.94 frames; utt_duration=1290 frames, utt_pad_proportion=0.04722, over 8189.40 utterances.], batch size: 38, lr: 3.45e-04
|
| 215 |
+
2022-08-11 16:00:45,362 INFO [train.py:737] (0/3) Epoch 15, batch 600, loss[loss=0.3756, over 14693.00 frames; utt_duration=1681 frames, utt_pad_proportion=0.009647, over 35.00 utterances.], tot_loss[loss=0.3551, over 2741742.23 frames; utt_duration=1319 frames, utt_pad_proportion=0.03835, over 8324.65 utterances.], batch size: 35, lr: 3.45e-04
|
| 216 |
+
2022-08-11 16:02:53,911 INFO [train.py:737] (0/3) Epoch 15, batch 700, loss[loss=0.3063, over 14474.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.01027, over 33.00 utterances.], tot_loss[loss=0.3574, over 2784720.24 frames; utt_duration=1289 frames, utt_pad_proportion=0.04769, over 8650.83 utterances.], batch size: 33, lr: 3.45e-04
|
| 217 |
+
2022-08-11 16:05:33,899 INFO [train.py:737] (0/3) Epoch 15, batch 800, loss[loss=0.3271, over 14589.00 frames; utt_duration=1498 frames, utt_pad_proportion=0.005567, over 39.00 utterances.], tot_loss[loss=0.3644, over 2804998.48 frames; utt_duration=1246 frames, utt_pad_proportion=0.05494, over 9012.21 utterances.], batch size: 39, lr: 3.45e-04
|
| 218 |
+
2022-08-11 16:07:53,556 INFO [train.py:737] (0/3) Epoch 15, batch 900, loss[loss=0.3348, over 14770.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.006167, over 39.00 utterances.], tot_loss[loss=0.3633, over 2822940.04 frames; utt_duration=1251 frames, utt_pad_proportion=0.05758, over 9040.27 utterances.], batch size: 39, lr: 3.45e-04
|
| 219 |
+
2022-08-11 16:10:29,685 INFO [train.py:737] (0/3) Epoch 15, batch 1000, loss[loss=0.4443, over 10421.00 frames; utt_duration=454.6 frames, utt_pad_proportion=0.2995, over 92.00 utterances.], tot_loss[loss=0.3633, over 2844526.79 frames; utt_duration=1272 frames, utt_pad_proportion=0.05131, over 8955.45 utterances.], batch size: 92, lr: 3.45e-04
|
| 220 |
+
2022-08-11 16:11:56,571 INFO [train.py:737] (0/3) Epoch 15, batch 1100, loss[loss=0.3869, over 14750.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.006075, over 43.00 utterances.], tot_loss[loss=0.3657, over 2857335.68 frames; utt_duration=1283 frames, utt_pad_proportion=0.04557, over 8919.34 utterances.], batch size: 43, lr: 3.45e-04
|
| 221 |
+
2022-08-11 16:13:12,427 INFO [train.py:737] (0/3) Epoch 15, batch 1200, loss[loss=0.3245, over 14708.00 frames; utt_duration=1682 frames, utt_pad_proportion=0.00917, over 35.00 utterances.], tot_loss[loss=0.3637, over 2864267.32 frames; utt_duration=1297 frames, utt_pad_proportion=0.04418, over 8843.86 utterances.], batch size: 35, lr: 3.44e-04
|
| 222 |
+
2022-08-11 16:16:21,712 INFO [train.py:737] (0/3) Epoch 15, batch 1300, loss[loss=0.3828, over 14680.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.01025, over 45.00 utterances.], tot_loss[loss=0.3673, over 2848598.39 frames; utt_duration=1264 frames, utt_pad_proportion=0.05678, over 9022.84 utterances.], batch size: 45, lr: 3.44e-04
|
| 223 |
+
2022-08-11 16:19:25,059 INFO [train.py:737] (0/3) Epoch 15, batch 1400, loss[loss=0.3219, over 14490.00 frames; utt_duration=1758 frames, utt_pad_proportion=0.008831, over 33.00 utterances.], tot_loss[loss=0.3645, over 2851980.55 frames; utt_duration=1269 frames, utt_pad_proportion=0.05452, over 9003.17 utterances.], batch size: 33, lr: 3.44e-04
|
| 224 |
+
2022-08-11 16:22:02,438 INFO [train.py:737] (0/3) Epoch 15, batch 1500, loss[loss=0.4011, over 14617.00 frames; utt_duration=1330 frames, utt_pad_proportion=0.008874, over 44.00 utterances.], tot_loss[loss=0.3628, over 2859530.77 frames; utt_duration=1286 frames, utt_pad_proportion=0.05039, over 8904.86 utterances.], batch size: 44, lr: 3.44e-04
|
| 225 |
+
2022-08-11 16:25:07,903 INFO [train.py:737] (0/3) Epoch 15, batch 1600, loss[loss=0.4087, over 14587.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005872, over 41.00 utterances.], tot_loss[loss=0.3637, over 2859294.36 frames; utt_duration=1273 frames, utt_pad_proportion=0.05156, over 8996.25 utterances.], batch size: 41, lr: 3.44e-04
|
| 226 |
+
2022-08-11 16:27:01,396 INFO [train.py:737] (0/3) Epoch 15, batch 1700, loss[loss=0.2775, over 14772.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.006167, over 38.00 utterances.], tot_loss[loss=0.3614, over 2868621.78 frames; utt_duration=1292 frames, utt_pad_proportion=0.04515, over 8888.47 utterances.], batch size: 38, lr: 3.44e-04
|
| 227 |
+
2022-08-11 16:28:51,803 INFO [train.py:737] (0/3) Epoch 15, batch 1800, loss[loss=0.4531, over 14684.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.007863, over 45.00 utterances.], tot_loss[loss=0.3636, over 2856626.80 frames; utt_duration=1270 frames, utt_pad_proportion=0.05296, over 9011.22 utterances.], batch size: 45, lr: 3.44e-04
|
| 228 |
+
2022-08-11 16:31:57,528 INFO [train.py:737] (0/3) Epoch 15, batch 1900, loss[loss=0.3478, over 14591.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.006005, over 38.00 utterances.], tot_loss[loss=0.3632, over 2852828.87 frames; utt_duration=1257 frames, utt_pad_proportion=0.05594, over 9090.31 utterances.], batch size: 38, lr: 3.44e-04
|
| 229 |
+
2022-08-11 16:33:53,038 INFO [train.py:737] (0/3) Epoch 15, batch 2000, loss[loss=0.3027, over 14831.00 frames; utt_duration=1650 frames, utt_pad_proportion=0.009209, over 36.00 utterances.], tot_loss[loss=0.3604, over 2858706.82 frames; utt_duration=1265 frames, utt_pad_proportion=0.05263, over 9051.31 utterances.], batch size: 36, lr: 3.44e-04
|
| 230 |
+
2022-08-11 16:36:46,009 INFO [train.py:737] (0/3) Epoch 15, batch 2100, loss[loss=0.388, over 14701.00 frames; utt_duration=1682 frames, utt_pad_proportion=0.01137, over 35.00 utterances.], tot_loss[loss=0.3619, over 2836827.26 frames; utt_duration=1225 frames, utt_pad_proportion=0.06628, over 9275.37 utterances.], batch size: 35, lr: 3.43e-04
|
| 231 |
+
2022-08-11 16:39:59,183 INFO [train.py:737] (0/3) Epoch 15, batch 2200, loss[loss=0.3504, over 14687.00 frames; utt_duration=1680 frames, utt_pad_proportion=0.01003, over 35.00 utterances.], tot_loss[loss=0.3615, over 2847208.08 frames; utt_duration=1255 frames, utt_pad_proportion=0.05987, over 9086.74 utterances.], batch size: 35, lr: 3.43e-04
|
| 232 |
+
2022-08-11 16:42:56,527 INFO [train.py:737] (0/3) Epoch 15, batch 2300, loss[loss=0.391, over 14444.00 frames; utt_duration=1653 frames, utt_pad_proportion=0.00857, over 35.00 utterances.], tot_loss[loss=0.3643, over 2845707.30 frames; utt_duration=1252 frames, utt_pad_proportion=0.05932, over 9105.49 utterances.], batch size: 35, lr: 3.43e-04
|
| 233 |
+
2022-08-11 16:46:01,612 INFO [train.py:737] (0/3) Epoch 15, batch 2400, loss[loss=0.3908, over 14455.00 frames; utt_duration=1092 frames, utt_pad_proportion=0.02915, over 53.00 utterances.], tot_loss[loss=0.3632, over 2852811.02 frames; utt_duration=1267 frames, utt_pad_proportion=0.05616, over 9015.07 utterances.], batch size: 53, lr: 3.43e-04
|
| 234 |
+
2022-08-11 16:48:07,644 INFO [train.py:737] (0/3) Epoch 15, batch 2500, loss[loss=0.3949, over 14591.00 frames; utt_duration=1579 frames, utt_pad_proportion=0.004939, over 37.00 utterances.], tot_loss[loss=0.3611, over 2857823.98 frames; utt_duration=1282 frames, utt_pad_proportion=0.05361, over 8923.63 utterances.], batch size: 37, lr: 3.43e-04
|
| 235 |
+
2022-08-11 16:50:38,869 INFO [train.py:737] (0/3) Epoch 15, batch 2600, loss[loss=0.3427, over 14753.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005706, over 41.00 utterances.], tot_loss[loss=0.3595, over 2849985.04 frames; utt_duration=1265 frames, utt_pad_proportion=0.05966, over 9022.03 utterances.], batch size: 41, lr: 3.43e-04
|
| 236 |
+
2022-08-11 16:53:31,182 INFO [train.py:737] (0/3) Epoch 15, batch 2700, loss[loss=0.317, over 14469.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01047, over 33.00 utterances.], tot_loss[loss=0.3599, over 2853299.98 frames; utt_duration=1262 frames, utt_pad_proportion=0.05624, over 9057.94 utterances.], batch size: 33, lr: 3.43e-04
|
| 237 |
+
2022-08-11 16:56:22,270 INFO [train.py:737] (0/3) Epoch 15, batch 2800, loss[loss=0.3046, over 14606.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.00999, over 34.00 utterances.], tot_loss[loss=0.3622, over 2853557.94 frames; utt_duration=1276 frames, utt_pad_proportion=0.0562, over 8954.43 utterances.], batch size: 34, lr: 3.43e-04
|
| 238 |
+
2022-08-11 16:59:23,579 INFO [train.py:737] (0/3) Epoch 15, batch 2900, loss[loss=0.5053, over 12977.00 frames; utt_duration=776.4 frames, utt_pad_proportion=0.1247, over 67.00 utterances.], tot_loss[loss=0.38, over 2843427.95 frames; utt_duration=1246 frames, utt_pad_proportion=0.06165, over 9136.20 utterances.], batch size: 67, lr: 3.43e-04
|
| 239 |
+
2022-08-11 17:02:14,381 INFO [train.py:737] (0/3) Epoch 15, batch 3000, loss[loss=0.3911, over 14572.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.006199, over 40.00 utterances.], tot_loss[loss=0.396, over 2852318.64 frames; utt_duration=1259 frames, utt_pad_proportion=0.05735, over 9072.49 utterances.], batch size: 40, lr: 3.42e-04
|
| 240 |
+
2022-08-11 17:02:14,382 INFO [train.py:761] (0/3) Computing validation loss
|
| 241 |
+
2022-08-11 17:02:58,430 INFO [train.py:770] (0/3) Epoch 15, validation: loss=0.2831, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 242 |
+
2022-08-11 17:05:29,809 INFO [train.py:737] (0/3) Epoch 15, batch 3100, loss[loss=0.4218, over 14565.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.00671, over 37.00 utterances.], tot_loss[loss=0.4101, over 2822989.72 frames; utt_duration=1205 frames, utt_pad_proportion=0.0762, over 9386.31 utterances.], batch size: 37, lr: 3.42e-04
|
| 243 |
+
2022-08-11 17:08:07,075 INFO [train.py:737] (0/3) Epoch 15, batch 3200, loss[loss=0.4891, over 14588.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.005889, over 42.00 utterances.], tot_loss[loss=0.422, over 2823290.54 frames; utt_duration=1206 frames, utt_pad_proportion=0.07555, over 9379.16 utterances.], batch size: 42, lr: 3.42e-04
|
| 244 |
+
2022-08-11 17:10:37,553 INFO [train.py:737] (0/3) Epoch 15, batch 3300, loss[loss=0.4147, over 14595.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005344, over 41.00 utterances.], tot_loss[loss=0.429, over 2822433.77 frames; utt_duration=1206 frames, utt_pad_proportion=0.07509, over 9369.72 utterances.], batch size: 41, lr: 3.42e-04
|
| 245 |
+
2022-08-11 17:12:19,930 INFO [train.py:737] (0/3) Epoch 15, batch 3400, loss[loss=0.4237, over 14762.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.005519, over 43.00 utterances.], tot_loss[loss=0.4315, over 2834752.35 frames; utt_duration=1231 frames, utt_pad_proportion=0.0675, over 9225.66 utterances.], batch size: 43, lr: 3.42e-04
|
| 246 |
+
2022-08-11 17:14:52,685 INFO [train.py:737] (0/3) Epoch 15, batch 3500, loss[loss=0.3737, over 14771.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.007684, over 37.00 utterances.], tot_loss[loss=0.4334, over 2835879.83 frames; utt_duration=1234 frames, utt_pad_proportion=0.06666, over 9200.88 utterances.], batch size: 37, lr: 3.42e-04
|
| 247 |
+
2022-08-11 17:17:37,568 INFO [train.py:737] (0/3) Epoch 15, batch 3600, loss[loss=0.5204, over 9491.00 frames; utt_duration=409.6 frames, utt_pad_proportion=0.356, over 93.00 utterances.], tot_loss[loss=0.4357, over 2843358.66 frames; utt_duration=1247 frames, utt_pad_proportion=0.06193, over 9130.81 utterances.], batch size: 93, lr: 3.42e-04
|
| 248 |
+
2022-08-11 17:20:24,454 INFO [train.py:737] (0/3) Epoch 15, batch 3700, loss[loss=0.4512, over 14452.00 frames; utt_duration=1158 frames, utt_pad_proportion=0.02373, over 50.00 utterances.], tot_loss[loss=0.4406, over 2830245.13 frames; utt_duration=1214 frames, utt_pad_proportion=0.06942, over 9340.54 utterances.], batch size: 50, lr: 3.42e-04
|
| 249 |
+
2022-08-11 17:23:11,071 INFO [train.py:737] (0/3) Epoch 15, batch 3800, loss[loss=0.4503, over 13070.00 frames; utt_duration=781.8 frames, utt_pad_proportion=0.1166, over 67.00 utterances.], tot_loss[loss=0.4438, over 2831088.44 frames; utt_duration=1212 frames, utt_pad_proportion=0.06847, over 9354.56 utterances.], batch size: 67, lr: 3.41e-04
|
| 250 |
+
2022-08-11 17:24:59,471 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/checkpoint-152000.pt
|
| 251 |
+
2022-08-11 17:25:37,106 INFO [train.py:737] (0/3) Epoch 15, batch 3900, loss[loss=0.4594, over 14643.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.007717, over 44.00 utterances.], tot_loss[loss=0.4434, over 2830416.05 frames; utt_duration=1214 frames, utt_pad_proportion=0.06919, over 9340.99 utterances.], batch size: 44, lr: 3.41e-04
|
| 252 |
+
2022-08-11 17:28:09,794 INFO [train.py:737] (0/3) Epoch 15, batch 4000, loss[loss=0.4443, over 14563.00 frames; utt_duration=1495 frames, utt_pad_proportion=0.005916, over 39.00 utterances.], tot_loss[loss=0.4451, over 2832353.94 frames; utt_duration=1223 frames, utt_pad_proportion=0.06806, over 9278.34 utterances.], batch size: 39, lr: 3.41e-04
|
| 253 |
+
2022-08-11 17:30:45,453 INFO [train.py:737] (0/3) Epoch 15, batch 4100, loss[loss=0.4138, over 14571.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005556, over 40.00 utterances.], tot_loss[loss=0.4421, over 2831696.13 frames; utt_duration=1232 frames, utt_pad_proportion=0.06816, over 9203.96 utterances.], batch size: 40, lr: 3.41e-04
|
| 254 |
+
2022-08-11 17:33:21,444 INFO [train.py:737] (0/3) Epoch 15, batch 4200, loss[loss=0.4878, over 14467.00 frames; utt_duration=1159 frames, utt_pad_proportion=0.02203, over 50.00 utterances.], tot_loss[loss=0.4404, over 2837447.35 frames; utt_duration=1237 frames, utt_pad_proportion=0.06534, over 9188.17 utterances.], batch size: 50, lr: 3.41e-04
|
| 255 |
+
2022-08-11 17:35:41,153 INFO [train.py:737] (0/3) Epoch 15, batch 4300, loss[loss=0.4624, over 13299.00 frames; utt_duration=795.2 frames, utt_pad_proportion=0.09939, over 67.00 utterances.], tot_loss[loss=0.4428, over 2832784.60 frames; utt_duration=1223 frames, utt_pad_proportion=0.06761, over 9279.00 utterances.], batch size: 67, lr: 3.41e-04
|
| 256 |
+
2022-08-11 17:38:29,486 INFO [train.py:737] (0/3) Epoch 15, batch 4400, loss[loss=0.4743, over 14696.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01117, over 46.00 utterances.], tot_loss[loss=0.4445, over 2831392.59 frames; utt_duration=1214 frames, utt_pad_proportion=0.06774, over 9341.24 utterances.], batch size: 46, lr: 3.41e-04
|
| 257 |
+
2022-08-11 17:41:15,835 INFO [train.py:737] (0/3) Epoch 15, batch 4500, loss[loss=0.4222, over 14640.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.007886, over 44.00 utterances.], tot_loss[loss=0.4419, over 2836032.79 frames; utt_duration=1220 frames, utt_pad_proportion=0.06567, over 9308.70 utterances.], batch size: 44, lr: 3.41e-04
|
| 258 |
+
2022-08-11 17:43:38,600 INFO [train.py:737] (0/3) Epoch 15, batch 4600, loss[loss=0.4314, over 14776.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005381, over 42.00 utterances.], tot_loss[loss=0.4445, over 2816938.85 frames; utt_duration=1184 frames, utt_pad_proportion=0.07884, over 9530.92 utterances.], batch size: 42, lr: 3.41e-04
|
| 259 |
+
2022-08-11 17:46:05,260 INFO [train.py:737] (0/3) Epoch 15, batch 4700, loss[loss=0.3277, over 14610.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.01015, over 34.00 utterances.], tot_loss[loss=0.44, over 2824048.96 frames; utt_duration=1212 frames, utt_pad_proportion=0.07631, over 9333.14 utterances.], batch size: 34, lr: 3.40e-04
|
| 260 |
+
2022-08-11 17:48:30,594 INFO [train.py:737] (0/3) Epoch 15, batch 4800, loss[loss=0.3862, over 14605.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.009973, over 34.00 utterances.], tot_loss[loss=0.4434, over 2812811.21 frames; utt_duration=1184 frames, utt_pad_proportion=0.08093, over 9511.99 utterances.], batch size: 34, lr: 3.40e-04
|
| 261 |
+
2022-08-11 17:50:07,722 INFO [train.py:737] (0/3) Epoch 15, batch 4900, loss[loss=0.4297, over 14867.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.008098, over 44.00 utterances.], tot_loss[loss=0.4447, over 2822930.89 frames; utt_duration=1202 frames, utt_pad_proportion=0.07395, over 9402.60 utterances.], batch size: 44, lr: 3.40e-04
|
| 262 |
+
2022-08-11 17:52:17,903 INFO [train.py:737] (0/3) Epoch 15, batch 5000, loss[loss=0.4134, over 14769.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006267, over 39.00 utterances.], tot_loss[loss=0.4442, over 2810134.66 frames; utt_duration=1185 frames, utt_pad_proportion=0.08143, over 9501.16 utterances.], batch size: 39, lr: 3.40e-04
|
| 263 |
+
2022-08-11 17:54:42,993 INFO [train.py:737] (0/3) Epoch 15, batch 5100, loss[loss=0.5111, over 9159.00 frames; utt_duration=399.7 frames, utt_pad_proportion=0.3823, over 92.00 utterances.], tot_loss[loss=0.4416, over 2799800.49 frames; utt_duration=1172 frames, utt_pad_proportion=0.08892, over 9566.32 utterances.], batch size: 92, lr: 3.40e-04
|
| 264 |
+
2022-08-11 17:57:27,535 INFO [train.py:737] (0/3) Epoch 15, batch 5200, loss[loss=0.4453, over 14634.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.008157, over 44.00 utterances.], tot_loss[loss=0.4405, over 2806602.58 frames; utt_duration=1187 frames, utt_pad_proportion=0.0866, over 9467.06 utterances.], batch size: 44, lr: 3.40e-04
|
| 265 |
+
2022-08-11 18:00:09,222 INFO [train.py:737] (0/3) Epoch 15, batch 5300, loss[loss=0.4387, over 13735.00 frames; utt_duration=965.4 frames, utt_pad_proportion=0.06992, over 57.00 utterances.], tot_loss[loss=0.4392, over 2820328.20 frames; utt_duration=1211 frames, utt_pad_proportion=0.07794, over 9326.50 utterances.], batch size: 57, lr: 3.40e-04
|
| 266 |
+
2022-08-11 18:02:48,925 INFO [train.py:737] (0/3) Epoch 15, batch 5400, loss[loss=0.4459, over 12880.00 frames; utt_duration=770.5 frames, utt_pad_proportion=0.1274, over 67.00 utterances.], tot_loss[loss=0.4431, over 2817421.54 frames; utt_duration=1198 frames, utt_pad_proportion=0.07886, over 9415.13 utterances.], batch size: 67, lr: 3.40e-04
|
| 267 |
+
2022-08-11 18:05:10,739 INFO [train.py:737] (0/3) Epoch 15, batch 5500, loss[loss=0.4707, over 13723.00 frames; utt_duration=964.8 frames, utt_pad_proportion=0.07053, over 57.00 utterances.], tot_loss[loss=0.4452, over 2821075.90 frames; utt_duration=1202 frames, utt_pad_proportion=0.07561, over 9402.71 utterances.], batch size: 57, lr: 3.40e-04
|
| 268 |
+
2022-08-11 18:06:37,415 INFO [train.py:737] (0/3) Epoch 15, batch 5600, loss[loss=0.3873, over 14768.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006284, over 39.00 utterances.], tot_loss[loss=0.4406, over 2826747.53 frames; utt_duration=1219 frames, utt_pad_proportion=0.07227, over 9286.32 utterances.], batch size: 39, lr: 3.39e-04
|
| 269 |
+
2022-08-11 18:09:12,335 INFO [train.py:737] (0/3) Epoch 15, batch 5700, loss[loss=0.406, over 14767.00 frames; utt_duration=1478 frames, utt_pad_proportion=0.005299, over 40.00 utterances.], tot_loss[loss=0.4409, over 2833716.92 frames; utt_duration=1235 frames, utt_pad_proportion=0.06847, over 9190.26 utterances.], batch size: 40, lr: 3.39e-04
|
| 270 |
+
2022-08-11 18:12:03,389 INFO [train.py:737] (0/3) Epoch 15, batch 5800, loss[loss=0.362, over 13641.00 frames; utt_duration=1820 frames, utt_pad_proportion=0.07028, over 30.00 utterances.], tot_loss[loss=0.4406, over 2832952.70 frames; utt_duration=1235 frames, utt_pad_proportion=0.06938, over 9187.06 utterances.], batch size: 30, lr: 3.39e-04
|
| 271 |
+
2022-08-11 18:14:35,534 INFO [train.py:737] (0/3) Epoch 15, batch 5900, loss[loss=0.4036, over 14565.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.006186, over 37.00 utterances.], tot_loss[loss=0.4402, over 2819992.62 frames; utt_duration=1206 frames, utt_pad_proportion=0.07689, over 9367.56 utterances.], batch size: 37, lr: 3.39e-04
|
| 272 |
+
2022-08-11 18:17:10,948 INFO [train.py:737] (0/3) Epoch 15, batch 6000, loss[loss=0.4122, over 14433.00 frames; utt_duration=1751 frames, utt_pad_proportion=0.01006, over 33.00 utterances.], tot_loss[loss=0.4432, over 2817141.59 frames; utt_duration=1194 frames, utt_pad_proportion=0.07766, over 9451.91 utterances.], batch size: 33, lr: 3.39e-04
|
| 273 |
+
2022-08-11 18:17:10,949 INFO [train.py:761] (0/3) Computing validation loss
|
| 274 |
+
2022-08-11 18:18:08,642 INFO [train.py:770] (0/3) Epoch 15, validation: loss=0.3006, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 275 |
+
2022-08-11 18:20:56,946 INFO [train.py:737] (0/3) Epoch 15, batch 6100, loss[loss=0.4841, over 9573.00 frames; utt_duration=422.3 frames, utt_pad_proportion=0.3523, over 91.00 utterances.], tot_loss[loss=0.4474, over 2752839.78 frames; utt_duration=1092 frames, utt_pad_proportion=0.114, over 10093.28 utterances.], batch size: 91, lr: 3.39e-04
|
| 276 |
+
2022-08-11 18:21:22,971 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-15.pt
|
| 277 |
+
2022-08-11 18:22:06,668 INFO [train.py:737] (0/3) Epoch 16, batch 0, loss[loss=0.4228, over 14659.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01277, over 47.00 utterances.], tot_loss[loss=0.4228, over 14659.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01277, over 47.00 utterances.], batch size: 47, lr: 3.29e-04
|
| 278 |
+
2022-08-11 18:25:01,424 INFO [train.py:737] (0/3) Epoch 16, batch 100, loss[loss=0.3833, over 14585.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.00611, over 41.00 utterances.], tot_loss[loss=0.3679, over 1149046.77 frames; utt_duration=1349 frames, utt_pad_proportion=0.03297, over 3409.69 utterances.], batch size: 41, lr: 3.29e-04
|
exp/log/log-train-2022-08-11-08-52-44-1
ADDED
|
@@ -0,0 +1,225 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2022-08-11 08:52:44,618 INFO [train.py:805] (1/3) Training started
|
| 2 |
+
2022-08-11 08:52:44,618 INFO [train.py:806] (1/3) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'env_info': {'k2-version': '1.17', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '7dcabf85e8bf06984c4abab0400ef1322b5ff3df', 'k2-git-date': 'Tue Aug 2 21:22:39 2022', 'lhotse-version': '1.4.0.dev+git.59318ed.clean', 'torch-version': '1.11.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.3', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '6af45c3-dirty', 'icefall-git-date': 'Tue Aug 2 14:47:09 2022', 'icefall-path': '/thspfs/home/acct-hpc/thspzh-hpc1/git/icefall_dev', 'k2-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/k2-1.17.dev20220805+cuda11.3.torch1.11.0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'dgxs-01.cluster.thspzh.com', 'IP address': '192.168.200.101'}, 'world_size': 3, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 20, 'start_epoch': 13, 'start_batch': 0, 'exp_dir': PosixPath('cnn_ctc/exp_phone'), 'lang_dir': 'data/lang_phone', 'bpe_model': 'data/lang_bpe_500/bpe.model', 'initial_lr': 0.003, 'lr_batches': 5000, 'lr_epochs': 6, 'seed': 42, 'print_diagnostics': False, 'save_every_n': 8000, 'keep_last_k': 20, 'average_period': 100, 'use_fp16': False, 'first_out_channels': 8, 'conv_blocks': 3, 'rnn_dim': 0, 'full_libri': True, 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 600, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'input_strategy': 'PrecomputedFeatures'}
|
| 3 |
+
2022-08-11 08:52:44,969 INFO [lexicon.py:176] (1/3) Loading pre-compiled data/lang_phone/Linv.pt
|
| 4 |
+
2022-08-11 08:52:57,153 INFO [train.py:843] (1/3) About to create model
|
| 5 |
+
2022-08-11 08:52:57,177 INFO [checkpoint.py:112] (1/3) Loading checkpoint from cnn_ctc/exp_phone/epoch-12.pt
|
| 6 |
+
2022-08-11 08:52:57,919 INFO [train.py:876] (1/3) Using DDP
|
| 7 |
+
2022-08-11 08:53:02,939 INFO [train.py:884] (1/3) Loading optimizer state dict
|
| 8 |
+
2022-08-11 08:53:02,984 INFO [train.py:892] (1/3) Loading scheduler state dict
|
| 9 |
+
2022-08-11 08:53:02,984 INFO [asr_datamodule.py:409] (1/3) About to get train-clean-100 cuts
|
| 10 |
+
2022-08-11 08:53:03,025 INFO [asr_datamodule.py:416] (1/3) About to get train-clean-360 cuts
|
| 11 |
+
2022-08-11 08:53:03,046 INFO [asr_datamodule.py:423] (1/3) About to get train-other-500 cuts
|
| 12 |
+
2022-08-11 08:53:03,063 INFO [asr_datamodule.py:225] (1/3) Enable MUSAN
|
| 13 |
+
2022-08-11 08:53:03,063 INFO [asr_datamodule.py:226] (1/3) About to get Musan cuts
|
| 14 |
+
2022-08-11 08:53:05,068 INFO [asr_datamodule.py:254] (1/3) Enable SpecAugment
|
| 15 |
+
2022-08-11 08:53:05,069 INFO [asr_datamodule.py:255] (1/3) Time warp factor: 80
|
| 16 |
+
2022-08-11 08:53:05,069 INFO [asr_datamodule.py:267] (1/3) Num frame mask: 10
|
| 17 |
+
2022-08-11 08:53:05,069 INFO [asr_datamodule.py:280] (1/3) About to create train dataset
|
| 18 |
+
2022-08-11 08:53:05,069 INFO [asr_datamodule.py:309] (1/3) Using DynamicBucketingSampler.
|
| 19 |
+
2022-08-11 08:53:07,411 INFO [asr_datamodule.py:324] (1/3) About to create train dataloader
|
| 20 |
+
2022-08-11 08:53:07,412 INFO [asr_datamodule.py:430] (1/3) About to get dev-clean cuts
|
| 21 |
+
2022-08-11 08:53:07,413 INFO [asr_datamodule.py:437] (1/3) About to get dev-other cuts
|
| 22 |
+
2022-08-11 08:53:07,415 INFO [asr_datamodule.py:355] (1/3) About to create dev dataset
|
| 23 |
+
2022-08-11 08:53:07,720 INFO [asr_datamodule.py:374] (1/3) About to create dev dataloader
|
| 24 |
+
2022-08-11 08:53:07,720 INFO [train.py:958] (1/3) Loading grad scaler state dict
|
| 25 |
+
2022-08-11 08:53:27,570 INFO [train.py:737] (1/3) Epoch 13, batch 0, loss[loss=0.4348, over 14678.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.009671, over 45.00 utterances.], tot_loss[loss=0.4348, over 14678.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.009671, over 45.00 utterances.], batch size: 45, lr: 3.85e-04
|
| 26 |
+
2022-08-11 08:53:27,640 INFO [distributed.py:948] (1/3) Reducer buckets have been rebuilt in this iteration.
|
| 27 |
+
2022-08-11 08:56:11,073 INFO [train.py:737] (1/3) Epoch 13, batch 100, loss[loss=0.355, over 14763.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006671, over 39.00 utterances.], tot_loss[loss=0.3668, over 1137714.09 frames; utt_duration=1290 frames, utt_pad_proportion=0.0453, over 3533.19 utterances.], batch size: 39, lr: 3.85e-04
|
| 28 |
+
2022-08-11 08:58:59,356 INFO [train.py:737] (1/3) Epoch 13, batch 200, loss[loss=0.3987, over 14743.00 frames; utt_duration=1406 frames, utt_pad_proportion=0.006546, over 42.00 utterances.], tot_loss[loss=0.3661, over 1813658.93 frames; utt_duration=1287 frames, utt_pad_proportion=0.05415, over 5642.03 utterances.], batch size: 42, lr: 3.84e-04
|
| 29 |
+
2022-08-11 09:01:26,616 INFO [train.py:737] (1/3) Epoch 13, batch 300, loss[loss=0.3045, over 14601.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.009691, over 34.00 utterances.], tot_loss[loss=0.3628, over 2234681.77 frames; utt_duration=1301 frames, utt_pad_proportion=0.04682, over 6876.45 utterances.], batch size: 34, lr: 3.84e-04
|
| 30 |
+
2022-08-11 09:03:10,458 INFO [train.py:737] (1/3) Epoch 13, batch 400, loss[loss=0.3585, over 14579.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005858, over 40.00 utterances.], tot_loss[loss=0.361, over 2478732.56 frames; utt_duration=1290 frames, utt_pad_proportion=0.05115, over 7695.88 utterances.], batch size: 40, lr: 3.84e-04
|
| 31 |
+
2022-08-11 09:04:16,681 INFO [train.py:737] (1/3) Epoch 13, batch 500, loss[loss=0.297, over 14582.00 frames; utt_duration=1717 frames, utt_pad_proportion=0.01044, over 34.00 utterances.], tot_loss[loss=0.3626, over 2628168.28 frames; utt_duration=1275 frames, utt_pad_proportion=0.05356, over 8257.38 utterances.], batch size: 34, lr: 3.84e-04
|
| 32 |
+
2022-08-11 09:05:12,191 INFO [train.py:737] (1/3) Epoch 13, batch 600, loss[loss=0.3456, over 14755.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005588, over 41.00 utterances.], tot_loss[loss=0.3585, over 2728721.07 frames; utt_duration=1303 frames, utt_pad_proportion=0.04691, over 8384.08 utterances.], batch size: 41, lr: 3.84e-04
|
| 33 |
+
2022-08-11 09:06:24,543 INFO [train.py:737] (1/3) Epoch 13, batch 700, loss[loss=0.362, over 14751.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.006077, over 41.00 utterances.], tot_loss[loss=0.3636, over 2775947.08 frames; utt_duration=1280 frames, utt_pad_proportion=0.05264, over 8687.67 utterances.], batch size: 41, lr: 3.84e-04
|
| 34 |
+
2022-08-11 09:09:10,554 INFO [train.py:737] (1/3) Epoch 13, batch 800, loss[loss=0.3466, over 13996.00 frames; utt_duration=1807 frames, utt_pad_proportion=0.03965, over 31.00 utterances.], tot_loss[loss=0.362, over 2815425.32 frames; utt_duration=1301 frames, utt_pad_proportion=0.04945, over 8663.68 utterances.], batch size: 31, lr: 3.84e-04
|
| 35 |
+
2022-08-11 09:11:56,853 INFO [train.py:737] (1/3) Epoch 13, batch 900, loss[loss=0.3244, over 14060.00 frames; utt_duration=1816 frames, utt_pad_proportion=0.05589, over 31.00 utterances.], tot_loss[loss=0.3668, over 2841541.62 frames; utt_duration=1304 frames, utt_pad_proportion=0.04512, over 8726.23 utterances.], batch size: 31, lr: 3.83e-04
|
| 36 |
+
2022-08-11 09:14:45,133 INFO [train.py:737] (1/3) Epoch 13, batch 1000, loss[loss=0.358, over 14774.00 frames; utt_duration=1479 frames, utt_pad_proportion=0.005531, over 40.00 utterances.], tot_loss[loss=0.3669, over 2855577.12 frames; utt_duration=1309 frames, utt_pad_proportion=0.04328, over 8738.98 utterances.], batch size: 40, lr: 3.83e-04
|
| 37 |
+
2022-08-11 09:17:11,267 INFO [train.py:737] (1/3) Epoch 13, batch 1100, loss[loss=0.3284, over 14569.00 frames; utt_duration=1496 frames, utt_pad_proportion=0.006133, over 39.00 utterances.], tot_loss[loss=0.3676, over 2863839.17 frames; utt_duration=1301 frames, utt_pad_proportion=0.04217, over 8817.18 utterances.], batch size: 39, lr: 3.83e-04
|
| 38 |
+
2022-08-11 09:19:38,556 INFO [train.py:737] (1/3) Epoch 13, batch 1200, loss[loss=0.3965, over 14509.00 frames; utt_duration=1162 frames, utt_pad_proportion=0.02098, over 50.00 utterances.], tot_loss[loss=0.3663, over 2867836.93 frames; utt_duration=1302 frames, utt_pad_proportion=0.042, over 8818.39 utterances.], batch size: 50, lr: 3.83e-04
|
| 39 |
+
2022-08-11 09:22:05,179 INFO [train.py:737] (1/3) Epoch 13, batch 1300, loss[loss=0.4931, over 14575.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006569, over 42.00 utterances.], tot_loss[loss=0.3653, over 2872115.08 frames; utt_duration=1316 frames, utt_pad_proportion=0.0413, over 8741.28 utterances.], batch size: 42, lr: 3.83e-04
|
| 40 |
+
2022-08-11 09:24:12,080 INFO [train.py:737] (1/3) Epoch 13, batch 1400, loss[loss=0.3946, over 14722.00 frames; utt_duration=1282 frames, utt_pad_proportion=0.01014, over 46.00 utterances.], tot_loss[loss=0.3649, over 2854985.41 frames; utt_duration=1280 frames, utt_pad_proportion=0.05345, over 8931.58 utterances.], batch size: 46, lr: 3.83e-04
|
| 41 |
+
2022-08-11 09:26:58,344 INFO [train.py:737] (1/3) Epoch 13, batch 1500, loss[loss=0.3598, over 14761.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.006205, over 42.00 utterances.], tot_loss[loss=0.3692, over 2859457.66 frames; utt_duration=1277 frames, utt_pad_proportion=0.05137, over 8967.77 utterances.], batch size: 42, lr: 3.83e-04
|
| 42 |
+
2022-08-11 09:29:51,304 INFO [train.py:737] (1/3) Epoch 13, batch 1600, loss[loss=0.3575, over 14405.00 frames; utt_duration=1154 frames, utt_pad_proportion=0.02867, over 50.00 utterances.], tot_loss[loss=0.368, over 2859921.75 frames; utt_duration=1269 frames, utt_pad_proportion=0.05291, over 9021.88 utterances.], batch size: 50, lr: 3.82e-04
|
| 43 |
+
2022-08-11 09:32:41,585 INFO [train.py:737] (1/3) Epoch 13, batch 1700, loss[loss=0.3331, over 14751.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.007028, over 40.00 utterances.], tot_loss[loss=0.366, over 2859021.69 frames; utt_duration=1262 frames, utt_pad_proportion=0.0539, over 9071.36 utterances.], batch size: 40, lr: 3.82e-04
|
| 44 |
+
2022-08-11 09:34:42,368 INFO [train.py:737] (1/3) Epoch 13, batch 1800, loss[loss=0.3785, over 14624.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.0088, over 44.00 utterances.], tot_loss[loss=0.3674, over 2849077.38 frames; utt_duration=1235 frames, utt_pad_proportion=0.05888, over 9238.65 utterances.], batch size: 44, lr: 3.82e-04
|
| 45 |
+
2022-08-11 09:37:23,735 INFO [train.py:737] (1/3) Epoch 13, batch 1900, loss[loss=0.3992, over 14737.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.006197, over 43.00 utterances.], tot_loss[loss=0.3663, over 2852004.77 frames; utt_duration=1253 frames, utt_pad_proportion=0.05789, over 9115.20 utterances.], batch size: 43, lr: 3.82e-04
|
| 46 |
+
2022-08-11 09:39:48,329 INFO [train.py:737] (1/3) Epoch 13, batch 2000, loss[loss=0.3334, over 14478.00 frames; utt_duration=1757 frames, utt_pad_proportion=0.009702, over 33.00 utterances.], tot_loss[loss=0.3642, over 2845819.67 frames; utt_duration=1244 frames, utt_pad_proportion=0.06125, over 9161.69 utterances.], batch size: 33, lr: 3.82e-04
|
| 47 |
+
2022-08-11 09:42:14,728 INFO [train.py:737] (1/3) Epoch 13, batch 2100, loss[loss=0.3805, over 14755.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005504, over 41.00 utterances.], tot_loss[loss=0.3597, over 2863897.97 frames; utt_duration=1299 frames, utt_pad_proportion=0.04998, over 8830.31 utterances.], batch size: 41, lr: 3.82e-04
|
| 48 |
+
2022-08-11 09:44:38,435 INFO [train.py:737] (1/3) Epoch 13, batch 2200, loss[loss=0.329, over 14483.00 frames; utt_duration=1757 frames, utt_pad_proportion=0.006852, over 33.00 utterances.], tot_loss[loss=0.3652, over 2871739.99 frames; utt_duration=1304 frames, utt_pad_proportion=0.04404, over 8815.92 utterances.], batch size: 33, lr: 3.82e-04
|
| 49 |
+
2022-08-11 09:47:02,613 INFO [train.py:737] (1/3) Epoch 13, batch 2300, loss[loss=0.4436, over 14651.00 frames; utt_duration=1248 frames, utt_pad_proportion=0.01151, over 47.00 utterances.], tot_loss[loss=0.3641, over 2867059.75 frames; utt_duration=1289 frames, utt_pad_proportion=0.04679, over 8908.43 utterances.], batch size: 47, lr: 3.81e-04
|
| 50 |
+
2022-08-11 09:49:33,833 INFO [train.py:737] (1/3) Epoch 13, batch 2400, loss[loss=0.3776, over 14647.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.007446, over 44.00 utterances.], tot_loss[loss=0.3628, over 2863180.37 frames; utt_duration=1287 frames, utt_pad_proportion=0.05094, over 8911.08 utterances.], batch size: 44, lr: 3.81e-04
|
| 51 |
+
2022-08-11 09:52:02,610 INFO [train.py:737] (1/3) Epoch 13, batch 2500, loss[loss=0.3481, over 12946.00 frames; utt_duration=774.2 frames, utt_pad_proportion=0.1252, over 67.00 utterances.], tot_loss[loss=0.3627, over 2851152.32 frames; utt_duration=1263 frames, utt_pad_proportion=0.05662, over 9037.52 utterances.], batch size: 67, lr: 3.81e-04
|
| 52 |
+
2022-08-11 09:54:20,857 INFO [train.py:737] (1/3) Epoch 13, batch 2600, loss[loss=0.3594, over 14778.00 frames; utt_duration=1409 frames, utt_pad_proportion=0.005179, over 42.00 utterances.], tot_loss[loss=0.3621, over 2853824.32 frames; utt_duration=1269 frames, utt_pad_proportion=0.05664, over 9005.99 utterances.], batch size: 42, lr: 3.81e-04
|
| 53 |
+
2022-08-11 09:56:50,909 INFO [train.py:737] (1/3) Epoch 13, batch 2700, loss[loss=0.3418, over 14762.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007739, over 37.00 utterances.], tot_loss[loss=0.3628, over 2849690.69 frames; utt_duration=1259 frames, utt_pad_proportion=0.05913, over 9063.61 utterances.], batch size: 37, lr: 3.81e-04
|
| 54 |
+
2022-08-11 09:59:17,871 INFO [train.py:737] (1/3) Epoch 13, batch 2800, loss[loss=0.4041, over 14573.00 frames; utt_duration=1423 frames, utt_pad_proportion=0.006944, over 41.00 utterances.], tot_loss[loss=0.3671, over 2848133.04 frames; utt_duration=1245 frames, utt_pad_proportion=0.05834, over 9163.25 utterances.], batch size: 41, lr: 3.81e-04
|
| 55 |
+
2022-08-11 10:01:53,829 INFO [train.py:737] (1/3) Epoch 13, batch 2900, loss[loss=0.4476, over 14788.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.005008, over 38.00 utterances.], tot_loss[loss=0.3816, over 2855255.73 frames; utt_duration=1252 frames, utt_pad_proportion=0.05436, over 9133.97 utterances.], batch size: 38, lr: 3.81e-04
|
| 56 |
+
2022-08-11 10:04:16,977 INFO [train.py:737] (1/3) Epoch 13, batch 3000, loss[loss=0.4159, over 10287.00 frames; utt_duration=453.6 frames, utt_pad_proportion=0.3064, over 91.00 utterances.], tot_loss[loss=0.4009, over 2853255.32 frames; utt_duration=1249 frames, utt_pad_proportion=0.05415, over 9146.92 utterances.], batch size: 91, lr: 3.81e-04
|
| 57 |
+
2022-08-11 10:04:16,978 INFO [train.py:761] (1/3) Computing validation loss
|
| 58 |
+
2022-08-11 10:04:50,565 INFO [train.py:770] (1/3) Epoch 13, validation: loss=0.3727, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 59 |
+
2022-08-11 10:06:51,669 INFO [train.py:737] (1/3) Epoch 13, batch 3100, loss[loss=0.4649, over 14492.00 frames; utt_duration=1209 frames, utt_pad_proportion=0.01767, over 48.00 utterances.], tot_loss[loss=0.4138, over 2859013.35 frames; utt_duration=1269 frames, utt_pad_proportion=0.05148, over 9021.84 utterances.], batch size: 48, lr: 3.80e-04
|
| 60 |
+
2022-08-11 10:09:22,792 INFO [train.py:737] (1/3) Epoch 13, batch 3200, loss[loss=0.3933, over 14775.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.006933, over 37.00 utterances.], tot_loss[loss=0.424, over 2844624.73 frames; utt_duration=1236 frames, utt_pad_proportion=0.06178, over 9214.41 utterances.], batch size: 37, lr: 3.80e-04
|
| 61 |
+
2022-08-11 10:11:51,393 INFO [train.py:737] (1/3) Epoch 13, batch 3300, loss[loss=0.4459, over 9640.00 frames; utt_duration=416.2 frames, utt_pad_proportion=0.3547, over 93.00 utterances.], tot_loss[loss=0.4331, over 2824545.08 frames; utt_duration=1200 frames, utt_pad_proportion=0.073, over 9426.22 utterances.], batch size: 93, lr: 3.80e-04
|
| 62 |
+
2022-08-11 10:14:06,314 INFO [train.py:737] (1/3) Epoch 13, batch 3400, loss[loss=0.5013, over 14458.00 frames; utt_duration=1158 frames, utt_pad_proportion=0.02487, over 50.00 utterances.], tot_loss[loss=0.436, over 2829414.56 frames; utt_duration=1216 frames, utt_pad_proportion=0.07118, over 9320.58 utterances.], batch size: 50, lr: 3.80e-04
|
| 63 |
+
2022-08-11 10:16:12,566 INFO [train.py:737] (1/3) Epoch 13, batch 3500, loss[loss=0.4705, over 14752.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006742, over 40.00 utterances.], tot_loss[loss=0.4375, over 2821759.66 frames; utt_duration=1202 frames, utt_pad_proportion=0.07526, over 9399.40 utterances.], batch size: 40, lr: 3.80e-04
|
| 64 |
+
2022-08-11 10:18:49,063 INFO [train.py:737] (1/3) Epoch 13, batch 3600, loss[loss=0.4687, over 14583.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.006129, over 39.00 utterances.], tot_loss[loss=0.4395, over 2822634.39 frames; utt_duration=1212 frames, utt_pad_proportion=0.07496, over 9328.04 utterances.], batch size: 39, lr: 3.80e-04
|
| 65 |
+
2022-08-11 10:21:10,813 INFO [train.py:737] (1/3) Epoch 13, batch 3700, loss[loss=0.4346, over 14604.00 frames; utt_duration=1624 frames, utt_pad_proportion=0.006677, over 36.00 utterances.], tot_loss[loss=0.4422, over 2809980.07 frames; utt_duration=1186 frames, utt_pad_proportion=0.08141, over 9492.91 utterances.], batch size: 36, lr: 3.80e-04
|
| 66 |
+
2022-08-11 10:23:39,228 INFO [train.py:737] (1/3) Epoch 13, batch 3800, loss[loss=0.4071, over 14758.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.005821, over 40.00 utterances.], tot_loss[loss=0.4425, over 2832861.92 frames; utt_duration=1229 frames, utt_pad_proportion=0.06831, over 9230.02 utterances.], batch size: 40, lr: 3.79e-04
|
| 67 |
+
2022-08-11 10:26:00,932 INFO [train.py:737] (1/3) Epoch 13, batch 3900, loss[loss=0.431, over 14773.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.00765, over 37.00 utterances.], tot_loss[loss=0.4419, over 2824625.04 frames; utt_duration=1217 frames, utt_pad_proportion=0.07217, over 9296.53 utterances.], batch size: 37, lr: 3.79e-04
|
| 68 |
+
2022-08-11 10:28:33,728 INFO [train.py:737] (1/3) Epoch 13, batch 4000, loss[loss=0.4633, over 14570.00 frames; utt_duration=1496 frames, utt_pad_proportion=0.006082, over 39.00 utterances.], tot_loss[loss=0.4423, over 2829857.11 frames; utt_duration=1226 frames, utt_pad_proportion=0.06861, over 9246.82 utterances.], batch size: 39, lr: 3.79e-04
|
| 69 |
+
2022-08-11 10:30:27,524 INFO [train.py:737] (1/3) Epoch 13, batch 4100, loss[loss=0.4587, over 14522.00 frames; utt_duration=1212 frames, utt_pad_proportion=0.01664, over 48.00 utterances.], tot_loss[loss=0.4411, over 2826164.25 frames; utt_duration=1219 frames, utt_pad_proportion=0.07133, over 9284.11 utterances.], batch size: 48, lr: 3.79e-04
|
| 70 |
+
2022-08-11 10:32:29,520 INFO [train.py:737] (1/3) Epoch 13, batch 4200, loss[loss=0.3957, over 14772.00 frames; utt_duration=1479 frames, utt_pad_proportion=0.005548, over 40.00 utterances.], tot_loss[loss=0.4397, over 2832160.45 frames; utt_duration=1232 frames, utt_pad_proportion=0.06818, over 9203.21 utterances.], batch size: 40, lr: 3.79e-04
|
| 71 |
+
2022-08-11 10:34:32,605 INFO [train.py:737] (1/3) Epoch 13, batch 4300, loss[loss=0.455, over 14554.00 frames; utt_duration=1457 frames, utt_pad_proportion=0.006207, over 40.00 utterances.], tot_loss[loss=0.4397, over 2829155.70 frames; utt_duration=1222 frames, utt_pad_proportion=0.06914, over 9270.50 utterances.], batch size: 40, lr: 3.79e-04
|
| 72 |
+
2022-08-11 10:36:59,751 INFO [train.py:737] (1/3) Epoch 13, batch 4400, loss[loss=0.4068, over 14634.00 frames; utt_duration=1723 frames, utt_pad_proportion=0.008664, over 34.00 utterances.], tot_loss[loss=0.4431, over 2815736.55 frames; utt_duration=1199 frames, utt_pad_proportion=0.07628, over 9403.89 utterances.], batch size: 34, lr: 3.79e-04
|
| 73 |
+
2022-08-11 10:40:20,191 INFO [train.py:737] (1/3) Epoch 13, batch 4500, loss[loss=0.5125, over 14442.00 frames; utt_duration=1157 frames, utt_pad_proportion=0.0246, over 50.00 utterances.], tot_loss[loss=0.4444, over 2817589.83 frames; utt_duration=1203 frames, utt_pad_proportion=0.07712, over 9380.62 utterances.], batch size: 50, lr: 3.78e-04
|
| 74 |
+
2022-08-11 10:44:36,546 INFO [train.py:737] (1/3) Epoch 13, batch 4600, loss[loss=0.494, over 14567.00 frames; utt_duration=1423 frames, utt_pad_proportion=0.006574, over 41.00 utterances.], tot_loss[loss=0.4417, over 2838722.08 frames; utt_duration=1241 frames, utt_pad_proportion=0.06312, over 9160.84 utterances.], batch size: 41, lr: 3.78e-04
|
| 75 |
+
2022-08-11 10:47:17,026 INFO [train.py:737] (1/3) Epoch 13, batch 4700, loss[loss=0.5018, over 10003.00 frames; utt_duration=436.2 frames, utt_pad_proportion=0.3289, over 92.00 utterances.], tot_loss[loss=0.4445, over 2827215.24 frames; utt_duration=1209 frames, utt_pad_proportion=0.0709, over 9366.07 utterances.], batch size: 92, lr: 3.78e-04
|
| 76 |
+
2022-08-11 10:50:52,812 INFO [train.py:737] (1/3) Epoch 13, batch 4800, loss[loss=0.4575, over 14764.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005919, over 42.00 utterances.], tot_loss[loss=0.4435, over 2830552.23 frames; utt_duration=1216 frames, utt_pad_proportion=0.06854, over 9325.91 utterances.], batch size: 42, lr: 3.78e-04
|
| 77 |
+
2022-08-11 10:54:02,824 INFO [train.py:737] (1/3) Epoch 13, batch 4900, loss[loss=0.4299, over 13168.00 frames; utt_duration=776.2 frames, utt_pad_proportion=0.1129, over 68.00 utterances.], tot_loss[loss=0.4404, over 2819491.35 frames; utt_duration=1196 frames, utt_pad_proportion=0.07501, over 9442.17 utterances.], batch size: 68, lr: 3.78e-04
|
| 78 |
+
2022-08-11 10:59:10,774 INFO [train.py:737] (1/3) Epoch 13, batch 5000, loss[loss=0.4499, over 14878.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.006636, over 44.00 utterances.], tot_loss[loss=0.4413, over 2815005.85 frames; utt_duration=1192 frames, utt_pad_proportion=0.07868, over 9457.05 utterances.], batch size: 44, lr: 3.78e-04
|
| 79 |
+
2022-08-11 11:02:20,134 INFO [train.py:737] (1/3) Epoch 13, batch 5100, loss[loss=0.3795, over 14588.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.008388, over 36.00 utterances.], tot_loss[loss=0.4434, over 2816049.93 frames; utt_duration=1194 frames, utt_pad_proportion=0.07839, over 9448.88 utterances.], batch size: 36, lr: 3.78e-04
|
| 80 |
+
2022-08-11 11:05:44,511 INFO [train.py:737] (1/3) Epoch 13, batch 5200, loss[loss=0.5078, over 14115.00 frames; utt_duration=975.3 frames, utt_pad_proportion=0.05583, over 58.00 utterances.], tot_loss[loss=0.4476, over 2812214.03 frames; utt_duration=1174 frames, utt_pad_proportion=0.07939, over 9594.05 utterances.], batch size: 58, lr: 3.78e-04
|
| 81 |
+
2022-08-11 11:09:00,836 INFO [train.py:737] (1/3) Epoch 13, batch 5300, loss[loss=0.4465, over 14629.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.008648, over 44.00 utterances.], tot_loss[loss=0.4457, over 2822727.51 frames; utt_duration=1197 frames, utt_pad_proportion=0.07372, over 9447.37 utterances.], batch size: 44, lr: 3.77e-04
|
| 82 |
+
2022-08-11 11:14:26,396 INFO [train.py:737] (1/3) Epoch 13, batch 5400, loss[loss=0.4229, over 14776.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005465, over 42.00 utterances.], tot_loss[loss=0.4457, over 2818335.57 frames; utt_duration=1190 frames, utt_pad_proportion=0.07619, over 9483.42 utterances.], batch size: 42, lr: 3.77e-04
|
| 83 |
+
2022-08-11 11:18:27,396 INFO [train.py:737] (1/3) Epoch 13, batch 5500, loss[loss=0.4658, over 14548.00 frames; utt_duration=1456 frames, utt_pad_proportion=0.007243, over 40.00 utterances.], tot_loss[loss=0.4454, over 2808841.69 frames; utt_duration=1186 frames, utt_pad_proportion=0.08288, over 9485.09 utterances.], batch size: 40, lr: 3.77e-04
|
| 84 |
+
2022-08-11 11:21:56,160 INFO [train.py:737] (1/3) Epoch 13, batch 5600, loss[loss=0.4975, over 14640.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.007852, over 44.00 utterances.], tot_loss[loss=0.4421, over 2840197.89 frames; utt_duration=1244 frames, utt_pad_proportion=0.06426, over 9140.26 utterances.], batch size: 44, lr: 3.77e-04
|
| 85 |
+
2022-08-11 11:24:38,524 INFO [train.py:737] (1/3) Epoch 13, batch 5700, loss[loss=0.3948, over 14607.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.01036, over 34.00 utterances.], tot_loss[loss=0.446, over 2824173.04 frames; utt_duration=1205 frames, utt_pad_proportion=0.07361, over 9385.90 utterances.], batch size: 34, lr: 3.77e-04
|
| 86 |
+
2022-08-11 11:30:07,783 INFO [train.py:737] (1/3) Epoch 13, batch 5800, loss[loss=0.4347, over 14644.00 frames; utt_duration=1248 frames, utt_pad_proportion=0.01357, over 47.00 utterances.], tot_loss[loss=0.4451, over 2837509.40 frames; utt_duration=1226 frames, utt_pad_proportion=0.06517, over 9268.82 utterances.], batch size: 47, lr: 3.77e-04
|
| 87 |
+
2022-08-11 11:33:55,335 INFO [train.py:737] (1/3) Epoch 13, batch 5900, loss[loss=0.4244, over 14586.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.006515, over 38.00 utterances.], tot_loss[loss=0.4444, over 2823132.55 frames; utt_duration=1199 frames, utt_pad_proportion=0.0738, over 9429.85 utterances.], batch size: 38, lr: 3.77e-04
|
| 88 |
+
2022-08-11 11:38:37,936 INFO [train.py:737] (1/3) Epoch 13, batch 6000, loss[loss=0.4294, over 14330.00 frames; utt_duration=1083 frames, utt_pad_proportion=0.03752, over 53.00 utterances.], tot_loss[loss=0.4449, over 2834181.20 frames; utt_duration=1213 frames, utt_pad_proportion=0.0665, over 9355.61 utterances.], batch size: 53, lr: 3.76e-04
|
| 89 |
+
2022-08-11 11:38:37,936 INFO [train.py:761] (1/3) Computing validation loss
|
| 90 |
+
2022-08-11 11:39:50,684 INFO [train.py:770] (1/3) Epoch 13, validation: loss=0.2858, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 91 |
+
2022-08-11 11:44:40,576 INFO [train.py:737] (1/3) Epoch 13, batch 6100, loss[loss=0.53, over 9477.00 frames; utt_duration=413.5 frames, utt_pad_proportion=0.3658, over 92.00 utterances.], tot_loss[loss=0.4496, over 2755682.22 frames; utt_duration=1089 frames, utt_pad_proportion=0.1117, over 10139.60 utterances.], batch size: 92, lr: 3.76e-04
|
| 92 |
+
2022-08-11 11:46:31,259 INFO [train.py:737] (1/3) Epoch 14, batch 0, loss[loss=0.3934, over 14640.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.007717, over 44.00 utterances.], tot_loss[loss=0.3934, over 14640.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.007717, over 44.00 utterances.], batch size: 44, lr: 3.64e-04
|
| 93 |
+
2022-08-11 11:50:49,319 INFO [train.py:737] (1/3) Epoch 14, batch 100, loss[loss=0.328, over 14760.00 frames; utt_duration=1515 frames, utt_pad_proportion=0.00704, over 39.00 utterances.], tot_loss[loss=0.3658, over 1149860.55 frames; utt_duration=1348 frames, utt_pad_proportion=0.03033, over 3414.79 utterances.], batch size: 39, lr: 3.64e-04
|
| 94 |
+
2022-08-11 11:54:33,951 INFO [train.py:737] (1/3) Epoch 14, batch 200, loss[loss=0.2943, over 14772.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.007717, over 37.00 utterances.], tot_loss[loss=0.3593, over 1841472.69 frames; utt_duration=1347 frames, utt_pad_proportion=0.02668, over 5473.01 utterances.], batch size: 37, lr: 3.64e-04
|
| 95 |
+
2022-08-11 11:57:32,568 INFO [train.py:737] (1/3) Epoch 14, batch 300, loss[loss=0.39, over 14655.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01207, over 47.00 utterances.], tot_loss[loss=0.3597, over 2238845.24 frames; utt_duration=1303 frames, utt_pad_proportion=0.04295, over 6878.38 utterances.], batch size: 47, lr: 3.64e-04
|
| 96 |
+
2022-08-11 12:02:50,130 INFO [train.py:737] (1/3) Epoch 14, batch 400, loss[loss=0.4084, over 14735.00 frames; utt_duration=1283 frames, utt_pad_proportion=0.00877, over 46.00 utterances.], tot_loss[loss=0.3612, over 2492996.80 frames; utt_duration=1304 frames, utt_pad_proportion=0.04215, over 7653.42 utterances.], batch size: 46, lr: 3.64e-04
|
| 97 |
+
2022-08-11 12:04:25,712 INFO [train.py:737] (1/3) Epoch 14, batch 500, loss[loss=0.3883, over 14752.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006944, over 40.00 utterances.], tot_loss[loss=0.3612, over 2626654.90 frames; utt_duration=1263 frames, utt_pad_proportion=0.05376, over 8328.24 utterances.], batch size: 40, lr: 3.64e-04
|
| 98 |
+
2022-08-11 12:10:29,635 INFO [train.py:737] (1/3) Epoch 14, batch 600, loss[loss=0.2883, over 13984.00 frames; utt_duration=1806 frames, utt_pad_proportion=0.0415, over 31.00 utterances.], tot_loss[loss=0.3613, over 2727602.04 frames; utt_duration=1289 frames, utt_pad_proportion=0.0478, over 8471.24 utterances.], batch size: 31, lr: 3.64e-04
|
| 99 |
+
2022-08-11 12:15:28,027 INFO [train.py:737] (1/3) Epoch 14, batch 700, loss[loss=0.373, over 14866.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.006675, over 44.00 utterances.], tot_loss[loss=0.3646, over 2784737.11 frames; utt_duration=1279 frames, utt_pad_proportion=0.047, over 8716.00 utterances.], batch size: 44, lr: 3.63e-04
|
| 100 |
+
2022-08-11 12:17:32,619 INFO [train.py:737] (1/3) Epoch 14, batch 800, loss[loss=0.3568, over 14665.00 frames; utt_duration=1305 frames, utt_pad_proportion=0.009577, over 45.00 utterances.], tot_loss[loss=0.3634, over 2826799.84 frames; utt_duration=1299 frames, utt_pad_proportion=0.04142, over 8715.91 utterances.], batch size: 45, lr: 3.63e-04
|
| 101 |
+
2022-08-11 12:22:44,896 INFO [train.py:737] (1/3) Epoch 14, batch 900, loss[loss=0.3071, over 14569.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006574, over 37.00 utterances.], tot_loss[loss=0.3629, over 2841372.68 frames; utt_duration=1294 frames, utt_pad_proportion=0.04544, over 8795.08 utterances.], batch size: 37, lr: 3.63e-04
|
| 102 |
+
2022-08-11 12:28:18,725 INFO [train.py:737] (1/3) Epoch 14, batch 1000, loss[loss=0.3647, over 14772.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007885, over 37.00 utterances.], tot_loss[loss=0.3603, over 2854474.18 frames; utt_duration=1310 frames, utt_pad_proportion=0.04352, over 8727.75 utterances.], batch size: 37, lr: 3.63e-04
|
| 103 |
+
2022-08-11 12:32:01,209 INFO [train.py:737] (1/3) Epoch 14, batch 1100, loss[loss=0.3297, over 14724.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.01041, over 35.00 utterances.], tot_loss[loss=0.3615, over 2868442.17 frames; utt_duration=1328 frames, utt_pad_proportion=0.03971, over 8648.07 utterances.], batch size: 35, lr: 3.63e-04
|
| 104 |
+
2022-08-11 12:33:29,130 INFO [train.py:737] (1/3) Epoch 14, batch 1200, loss[loss=0.4059, over 14640.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.007937, over 44.00 utterances.], tot_loss[loss=0.3618, over 2869590.12 frames; utt_duration=1310 frames, utt_pad_proportion=0.04211, over 8771.74 utterances.], batch size: 44, lr: 3.63e-04
|
| 105 |
+
2022-08-11 12:34:33,003 INFO [train.py:737] (1/3) Epoch 14, batch 1300, loss[loss=0.4013, over 14706.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.008855, over 45.00 utterances.], tot_loss[loss=0.361, over 2857794.41 frames; utt_duration=1287 frames, utt_pad_proportion=0.05181, over 8893.00 utterances.], batch size: 45, lr: 3.63e-04
|
| 106 |
+
2022-08-11 12:39:36,347 INFO [train.py:737] (1/3) Epoch 14, batch 1400, loss[loss=0.3089, over 14730.00 frames; utt_duration=1685 frames, utt_pad_proportion=0.01007, over 35.00 utterances.], tot_loss[loss=0.3654, over 2832126.41 frames; utt_duration=1237 frames, utt_pad_proportion=0.06818, over 9167.23 utterances.], batch size: 35, lr: 3.63e-04
|
| 107 |
+
2022-08-11 12:45:34,372 INFO [train.py:737] (1/3) Epoch 14, batch 1500, loss[loss=0.4109, over 14581.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006229, over 42.00 utterances.], tot_loss[loss=0.3685, over 2852961.44 frames; utt_duration=1270 frames, utt_pad_proportion=0.05502, over 8996.90 utterances.], batch size: 42, lr: 3.62e-04
|
| 108 |
+
2022-08-11 12:50:14,994 INFO [train.py:737] (1/3) Epoch 14, batch 1600, loss[loss=0.2891, over 14467.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01019, over 33.00 utterances.], tot_loss[loss=0.367, over 2865894.40 frames; utt_duration=1294 frames, utt_pad_proportion=0.04828, over 8865.94 utterances.], batch size: 33, lr: 3.62e-04
|
| 109 |
+
2022-08-11 12:54:22,869 INFO [train.py:737] (1/3) Epoch 14, batch 1700, loss[loss=0.3284, over 14768.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006519, over 39.00 utterances.], tot_loss[loss=0.3676, over 2855068.93 frames; utt_duration=1269 frames, utt_pad_proportion=0.05528, over 9010.88 utterances.], batch size: 39, lr: 3.62e-04
|
| 110 |
+
2022-08-11 12:58:59,135 INFO [train.py:737] (1/3) Epoch 14, batch 1800, loss[loss=0.3789, over 14715.00 frames; utt_duration=1309 frames, utt_pad_proportion=0.006508, over 45.00 utterances.], tot_loss[loss=0.3665, over 2854573.69 frames; utt_duration=1266 frames, utt_pad_proportion=0.05471, over 9027.48 utterances.], batch size: 45, lr: 3.62e-04
|
| 111 |
+
2022-08-11 13:03:45,765 INFO [train.py:737] (1/3) Epoch 14, batch 1900, loss[loss=0.3438, over 14785.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.00531, over 38.00 utterances.], tot_loss[loss=0.3656, over 2848448.30 frames; utt_duration=1259 frames, utt_pad_proportion=0.05825, over 9061.89 utterances.], batch size: 38, lr: 3.62e-04
|
| 112 |
+
2022-08-11 13:08:02,545 INFO [train.py:737] (1/3) Epoch 14, batch 2000, loss[loss=0.3388, over 14772.00 frames; utt_duration=1479 frames, utt_pad_proportion=0.005632, over 40.00 utterances.], tot_loss[loss=0.3645, over 2852166.01 frames; utt_duration=1260 frames, utt_pad_proportion=0.05656, over 9068.24 utterances.], batch size: 40, lr: 3.62e-04
|
| 113 |
+
2022-08-11 13:11:18,009 INFO [train.py:737] (1/3) Epoch 14, batch 2100, loss[loss=0.4105, over 14530.00 frames; utt_duration=1164 frames, utt_pad_proportion=0.01912, over 50.00 utterances.], tot_loss[loss=0.3635, over 2870774.91 frames; utt_duration=1299 frames, utt_pad_proportion=0.04586, over 8849.91 utterances.], batch size: 50, lr: 3.62e-04
|
| 114 |
+
2022-08-11 13:17:21,266 INFO [train.py:737] (1/3) Epoch 14, batch 2200, loss[loss=0.4031, over 14474.00 frames; utt_duration=1160 frames, utt_pad_proportion=0.02308, over 50.00 utterances.], tot_loss[loss=0.364, over 2849363.49 frames; utt_duration=1260 frames, utt_pad_proportion=0.0585, over 9056.23 utterances.], batch size: 50, lr: 3.62e-04
|
| 115 |
+
2022-08-11 13:22:13,817 INFO [train.py:737] (1/3) Epoch 14, batch 2300, loss[loss=0.3301, over 14865.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.00747, over 44.00 utterances.], tot_loss[loss=0.364, over 2853178.80 frames; utt_duration=1259 frames, utt_pad_proportion=0.0565, over 9076.26 utterances.], batch size: 44, lr: 3.61e-04
|
| 116 |
+
2022-08-11 13:27:39,731 INFO [train.py:737] (1/3) Epoch 14, batch 2400, loss[loss=0.4335, over 9836.00 frames; utt_duration=429.3 frames, utt_pad_proportion=0.3406, over 92.00 utterances.], tot_loss[loss=0.3631, over 2862116.47 frames; utt_duration=1274 frames, utt_pad_proportion=0.05013, over 8994.80 utterances.], batch size: 92, lr: 3.61e-04
|
| 117 |
+
2022-08-11 13:32:15,983 INFO [train.py:737] (1/3) Epoch 14, batch 2500, loss[loss=0.4227, over 14694.00 frames; utt_duration=1279 frames, utt_pad_proportion=0.01066, over 46.00 utterances.], tot_loss[loss=0.3613, over 2858488.67 frames; utt_duration=1276 frames, utt_pad_proportion=0.05447, over 8974.26 utterances.], batch size: 46, lr: 3.61e-04
|
| 118 |
+
2022-08-11 13:38:21,581 INFO [train.py:737] (1/3) Epoch 14, batch 2600, loss[loss=0.3767, over 14575.00 frames; utt_duration=1496 frames, utt_pad_proportion=0.005132, over 39.00 utterances.], tot_loss[loss=0.3613, over 2862713.51 frames; utt_duration=1285 frames, utt_pad_proportion=0.05012, over 8922.83 utterances.], batch size: 39, lr: 3.61e-04
|
| 119 |
+
2022-08-11 13:45:02,852 INFO [train.py:737] (1/3) Epoch 14, batch 2700, loss[loss=0.3574, over 14852.00 frames; utt_duration=1652 frames, utt_pad_proportion=0.008653, over 36.00 utterances.], tot_loss[loss=0.3606, over 2882670.40 frames; utt_duration=1325 frames, utt_pad_proportion=0.03816, over 8712.80 utterances.], batch size: 36, lr: 3.61e-04
|
| 120 |
+
2022-08-11 13:50:17,637 INFO [train.py:737] (1/3) Epoch 14, batch 2800, loss[loss=0.3252, over 14633.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.00836, over 44.00 utterances.], tot_loss[loss=0.3616, over 2888249.36 frames; utt_duration=1327 frames, utt_pad_proportion=0.03479, over 8713.91 utterances.], batch size: 44, lr: 3.61e-04
|
| 121 |
+
2022-08-11 13:56:31,409 INFO [train.py:737] (1/3) Epoch 14, batch 2900, loss[loss=0.4091, over 14747.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.005726, over 43.00 utterances.], tot_loss[loss=0.3763, over 2884377.49 frames; utt_duration=1325 frames, utt_pad_proportion=0.03648, over 8715.79 utterances.], batch size: 43, lr: 3.61e-04
|
| 122 |
+
2022-08-11 14:01:08,226 INFO [train.py:737] (1/3) Epoch 14, batch 3000, loss[loss=0.4121, over 14536.00 frames; utt_duration=1213 frames, utt_pad_proportion=0.01547, over 48.00 utterances.], tot_loss[loss=0.3986, over 2872206.04 frames; utt_duration=1290 frames, utt_pad_proportion=0.04435, over 8914.81 utterances.], batch size: 48, lr: 3.61e-04
|
| 123 |
+
2022-08-11 14:01:08,226 INFO [train.py:761] (1/3) Computing validation loss
|
| 124 |
+
2022-08-11 14:03:06,563 INFO [train.py:770] (1/3) Epoch 14, validation: loss=0.2899, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 125 |
+
2022-08-11 14:07:35,707 INFO [train.py:737] (1/3) Epoch 14, batch 3100, loss[loss=0.3758, over 14613.00 frames; utt_duration=1721 frames, utt_pad_proportion=0.009381, over 34.00 utterances.], tot_loss[loss=0.4131, over 2857262.84 frames; utt_duration=1265 frames, utt_pad_proportion=0.05305, over 9044.98 utterances.], batch size: 34, lr: 3.60e-04
|
| 126 |
+
2022-08-11 14:14:22,389 INFO [train.py:737] (1/3) Epoch 14, batch 3200, loss[loss=0.5724, over 14444.00 frames; utt_duration=1157 frames, utt_pad_proportion=0.02532, over 50.00 utterances.], tot_loss[loss=0.4226, over 2863975.50 frames; utt_duration=1271 frames, utt_pad_proportion=0.04905, over 9026.07 utterances.], batch size: 50, lr: 3.60e-04
|
| 127 |
+
2022-08-11 14:20:07,873 INFO [train.py:737] (1/3) Epoch 14, batch 3300, loss[loss=0.4575, over 14737.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.007185, over 43.00 utterances.], tot_loss[loss=0.4286, over 2844455.45 frames; utt_duration=1250 frames, utt_pad_proportion=0.06011, over 9114.01 utterances.], batch size: 43, lr: 3.60e-04
|
| 128 |
+
2022-08-11 14:23:56,193 INFO [train.py:737] (1/3) Epoch 14, batch 3400, loss[loss=0.4174, over 14598.00 frames; utt_duration=1624 frames, utt_pad_proportion=0.006881, over 36.00 utterances.], tot_loss[loss=0.436, over 2834345.19 frames; utt_duration=1223 frames, utt_pad_proportion=0.0681, over 9278.02 utterances.], batch size: 36, lr: 3.60e-04
|
| 129 |
+
2022-08-11 14:26:47,078 INFO [train.py:737] (1/3) Epoch 14, batch 3500, loss[loss=0.452, over 14850.00 frames; utt_duration=1351 frames, utt_pad_proportion=0.009231, over 44.00 utterances.], tot_loss[loss=0.4363, over 2843288.06 frames; utt_duration=1234 frames, utt_pad_proportion=0.06169, over 9224.63 utterances.], batch size: 44, lr: 3.60e-04
|
| 130 |
+
2022-08-11 14:29:40,025 INFO [train.py:737] (1/3) Epoch 14, batch 3600, loss[loss=0.3826, over 14774.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.006032, over 39.00 utterances.], tot_loss[loss=0.4378, over 2841928.28 frames; utt_duration=1251 frames, utt_pad_proportion=0.06258, over 9094.55 utterances.], batch size: 39, lr: 3.60e-04
|
| 131 |
+
2022-08-11 14:32:26,120 INFO [train.py:737] (1/3) Epoch 14, batch 3700, loss[loss=0.445, over 14613.00 frames; utt_duration=1330 frames, utt_pad_proportion=0.008338, over 44.00 utterances.], tot_loss[loss=0.4428, over 2817229.26 frames; utt_duration=1199 frames, utt_pad_proportion=0.07889, over 9414.26 utterances.], batch size: 44, lr: 3.60e-04
|
| 132 |
+
2022-08-11 14:35:07,280 INFO [train.py:737] (1/3) Epoch 14, batch 3800, loss[loss=0.4191, over 14588.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.008269, over 36.00 utterances.], tot_loss[loss=0.4442, over 2818413.61 frames; utt_duration=1201 frames, utt_pad_proportion=0.07842, over 9400.01 utterances.], batch size: 36, lr: 3.60e-04
|
| 133 |
+
2022-08-11 14:37:05,346 INFO [train.py:737] (1/3) Epoch 14, batch 3900, loss[loss=0.4182, over 14561.00 frames; utt_duration=1457 frames, utt_pad_proportion=0.00717, over 40.00 utterances.], tot_loss[loss=0.4456, over 2817506.99 frames; utt_duration=1193 frames, utt_pad_proportion=0.07951, over 9460.16 utterances.], batch size: 40, lr: 3.59e-04
|
| 134 |
+
2022-08-11 14:39:09,208 INFO [train.py:737] (1/3) Epoch 14, batch 4000, loss[loss=0.4584, over 14713.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.01084, over 46.00 utterances.], tot_loss[loss=0.445, over 2812430.48 frames; utt_duration=1187 frames, utt_pad_proportion=0.0803, over 9489.37 utterances.], batch size: 46, lr: 3.59e-04
|
| 135 |
+
2022-08-11 14:41:42,024 INFO [train.py:737] (1/3) Epoch 14, batch 4100, loss[loss=0.5519, over 9694.00 frames; utt_duration=427.5 frames, utt_pad_proportion=0.3453, over 91.00 utterances.], tot_loss[loss=0.446, over 2802590.89 frames; utt_duration=1163 frames, utt_pad_proportion=0.0835, over 9651.50 utterances.], batch size: 91, lr: 3.59e-04
|
| 136 |
+
2022-08-11 14:44:27,291 INFO [train.py:737] (1/3) Epoch 14, batch 4200, loss[loss=0.3621, over 14005.00 frames; utt_duration=1808 frames, utt_pad_proportion=0.05034, over 31.00 utterances.], tot_loss[loss=0.4452, over 2802448.40 frames; utt_duration=1164 frames, utt_pad_proportion=0.08457, over 9641.85 utterances.], batch size: 31, lr: 3.59e-04
|
| 137 |
+
2022-08-11 14:46:59,732 INFO [train.py:737] (1/3) Epoch 14, batch 4300, loss[loss=0.4626, over 14766.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.006688, over 38.00 utterances.], tot_loss[loss=0.445, over 2816178.74 frames; utt_duration=1193 frames, utt_pad_proportion=0.07782, over 9452.66 utterances.], batch size: 38, lr: 3.59e-04
|
| 138 |
+
2022-08-11 14:49:20,107 INFO [train.py:737] (1/3) Epoch 14, batch 4400, loss[loss=0.373, over 14718.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.01083, over 35.00 utterances.], tot_loss[loss=0.4453, over 2812430.13 frames; utt_duration=1193 frames, utt_pad_proportion=0.08109, over 9444.76 utterances.], batch size: 35, lr: 3.59e-04
|
| 139 |
+
2022-08-11 14:51:53,409 INFO [train.py:737] (1/3) Epoch 14, batch 4500, loss[loss=0.5022, over 14447.00 frames; utt_duration=1157 frames, utt_pad_proportion=0.02537, over 50.00 utterances.], tot_loss[loss=0.4441, over 2820464.56 frames; utt_duration=1203 frames, utt_pad_proportion=0.07608, over 9387.50 utterances.], batch size: 50, lr: 3.59e-04
|
| 140 |
+
2022-08-11 14:54:39,323 INFO [train.py:737] (1/3) Epoch 14, batch 4600, loss[loss=0.542, over 9701.00 frames; utt_duration=423.3 frames, utt_pad_proportion=0.3488, over 92.00 utterances.], tot_loss[loss=0.4428, over 2824728.71 frames; utt_duration=1209 frames, utt_pad_proportion=0.07381, over 9360.12 utterances.], batch size: 92, lr: 3.59e-04
|
| 141 |
+
2022-08-11 14:56:04,614 INFO [train.py:737] (1/3) Epoch 14, batch 4700, loss[loss=0.4791, over 14479.00 frames; utt_duration=1208 frames, utt_pad_proportion=0.01953, over 48.00 utterances.], tot_loss[loss=0.4442, over 2822568.08 frames; utt_duration=1210 frames, utt_pad_proportion=0.07395, over 9340.20 utterances.], batch size: 48, lr: 3.58e-04
|
| 142 |
+
2022-08-11 15:00:58,233 INFO [train.py:737] (1/3) Epoch 14, batch 4800, loss[loss=0.3865, over 14584.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.0079, over 36.00 utterances.], tot_loss[loss=0.4441, over 2829895.02 frames; utt_duration=1221 frames, utt_pad_proportion=0.06923, over 9285.36 utterances.], batch size: 36, lr: 3.58e-04
|
| 143 |
+
2022-08-11 15:05:17,214 INFO [train.py:737] (1/3) Epoch 14, batch 4900, loss[loss=0.4613, over 14687.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009064, over 45.00 utterances.], tot_loss[loss=0.4402, over 2821060.84 frames; utt_duration=1212 frames, utt_pad_proportion=0.07564, over 9321.51 utterances.], batch size: 45, lr: 3.58e-04
|
| 144 |
+
2022-08-11 15:10:37,125 INFO [train.py:737] (1/3) Epoch 14, batch 5000, loss[loss=0.4328, over 14753.00 frames; utt_duration=1406 frames, utt_pad_proportion=0.006793, over 42.00 utterances.], tot_loss[loss=0.4407, over 2823514.80 frames; utt_duration=1212 frames, utt_pad_proportion=0.07493, over 9333.25 utterances.], batch size: 42, lr: 3.58e-04
|
| 145 |
+
2022-08-11 15:16:39,152 INFO [train.py:737] (1/3) Epoch 14, batch 5100, loss[loss=0.4432, over 14453.00 frames; utt_duration=1158 frames, utt_pad_proportion=0.02465, over 50.00 utterances.], tot_loss[loss=0.4398, over 2834774.81 frames; utt_duration=1229 frames, utt_pad_proportion=0.067, over 9239.78 utterances.], batch size: 50, lr: 3.58e-04
|
| 146 |
+
2022-08-11 15:19:11,608 INFO [train.py:737] (1/3) Epoch 14, batch 5200, loss[loss=0.4896, over 13944.00 frames; utt_duration=980 frames, utt_pad_proportion=0.05584, over 57.00 utterances.], tot_loss[loss=0.44, over 2826809.27 frames; utt_duration=1211 frames, utt_pad_proportion=0.07287, over 9347.50 utterances.], batch size: 57, lr: 3.58e-04
|
| 147 |
+
2022-08-11 15:22:03,092 INFO [train.py:737] (1/3) Epoch 14, batch 5300, loss[loss=0.5343, over 9897.00 frames; utt_duration=436.5 frames, utt_pad_proportion=0.3316, over 91.00 utterances.], tot_loss[loss=0.4405, over 2823373.64 frames; utt_duration=1200 frames, utt_pad_proportion=0.07536, over 9422.23 utterances.], batch size: 91, lr: 3.58e-04
|
| 148 |
+
2022-08-11 15:24:31,019 INFO [train.py:737] (1/3) Epoch 14, batch 5400, loss[loss=0.4437, over 14369.00 frames; utt_duration=1086 frames, utt_pad_proportion=0.03554, over 53.00 utterances.], tot_loss[loss=0.4418, over 2827185.72 frames; utt_duration=1206 frames, utt_pad_proportion=0.07355, over 9386.86 utterances.], batch size: 53, lr: 3.58e-04
|
| 149 |
+
2022-08-11 15:26:31,330 INFO [train.py:737] (1/3) Epoch 14, batch 5500, loss[loss=0.4902, over 14623.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.00766, over 44.00 utterances.], tot_loss[loss=0.4404, over 2830964.17 frames; utt_duration=1220 frames, utt_pad_proportion=0.07134, over 9294.16 utterances.], batch size: 44, lr: 3.57e-04
|
| 150 |
+
2022-08-11 15:29:25,756 INFO [train.py:737] (1/3) Epoch 14, batch 5600, loss[loss=0.4268, over 14579.00 frames; utt_duration=1578 frames, utt_pad_proportion=0.005978, over 37.00 utterances.], tot_loss[loss=0.4413, over 2825730.12 frames; utt_duration=1211 frames, utt_pad_proportion=0.07305, over 9348.58 utterances.], batch size: 37, lr: 3.57e-04
|
| 151 |
+
2022-08-11 15:32:05,014 INFO [train.py:737] (1/3) Epoch 14, batch 5700, loss[loss=0.4554, over 12692.00 frames; utt_duration=759.3 frames, utt_pad_proportion=0.1421, over 67.00 utterances.], tot_loss[loss=0.4419, over 2815675.55 frames; utt_duration=1191 frames, utt_pad_proportion=0.07807, over 9466.27 utterances.], batch size: 67, lr: 3.57e-04
|
| 152 |
+
2022-08-11 15:34:48,482 INFO [train.py:737] (1/3) Epoch 14, batch 5800, loss[loss=0.3579, over 14589.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.006851, over 38.00 utterances.], tot_loss[loss=0.4453, over 2812923.25 frames; utt_duration=1187 frames, utt_pad_proportion=0.08018, over 9490.27 utterances.], batch size: 38, lr: 3.57e-04
|
| 153 |
+
2022-08-11 15:37:21,098 INFO [train.py:737] (1/3) Epoch 14, batch 5900, loss[loss=0.5392, over 14764.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.004898, over 41.00 utterances.], tot_loss[loss=0.4434, over 2807954.28 frames; utt_duration=1179 frames, utt_pad_proportion=0.08279, over 9535.25 utterances.], batch size: 41, lr: 3.57e-04
|
| 154 |
+
2022-08-11 15:39:22,179 INFO [train.py:737] (1/3) Epoch 14, batch 6000, loss[loss=0.4592, over 14767.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006284, over 39.00 utterances.], tot_loss[loss=0.4429, over 2815410.35 frames; utt_duration=1190 frames, utt_pad_proportion=0.07801, over 9476.91 utterances.], batch size: 39, lr: 3.57e-04
|
| 155 |
+
2022-08-11 15:39:22,180 INFO [train.py:761] (1/3) Computing validation loss
|
| 156 |
+
2022-08-11 15:40:05,207 INFO [train.py:770] (1/3) Epoch 14, validation: loss=0.301, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 157 |
+
2022-08-11 15:42:37,944 INFO [train.py:737] (1/3) Epoch 14, batch 6100, loss[loss=0.5632, over 10201.00 frames; utt_duration=435.4 frames, utt_pad_proportion=0.3175, over 94.00 utterances.], tot_loss[loss=0.4486, over 2752661.56 frames; utt_duration=1089 frames, utt_pad_proportion=0.1138, over 10125.92 utterances.], batch size: 94, lr: 3.57e-04
|
| 158 |
+
2022-08-11 15:44:19,634 INFO [train.py:737] (1/3) Epoch 15, batch 0, loss[loss=0.3726, over 14752.00 frames; utt_duration=1515 frames, utt_pad_proportion=0.007376, over 39.00 utterances.], tot_loss[loss=0.3726, over 14752.00 frames; utt_duration=1515 frames, utt_pad_proportion=0.007376, over 39.00 utterances.], batch size: 39, lr: 3.46e-04
|
| 159 |
+
2022-08-11 15:47:34,443 INFO [train.py:737] (1/3) Epoch 15, batch 100, loss[loss=0.3221, over 14588.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.006209, over 38.00 utterances.], tot_loss[loss=0.3612, over 1146718.41 frames; utt_duration=1331 frames, utt_pad_proportion=0.03124, over 3449.45 utterances.], batch size: 38, lr: 3.46e-04
|
| 160 |
+
2022-08-11 15:50:08,433 INFO [train.py:737] (1/3) Epoch 15, batch 200, loss[loss=0.3599, over 14587.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.007798, over 36.00 utterances.], tot_loss[loss=0.3567, over 1835085.70 frames; utt_duration=1336 frames, utt_pad_proportion=0.02914, over 5499.55 utterances.], batch size: 36, lr: 3.46e-04
|
| 161 |
+
2022-08-11 15:53:20,627 INFO [train.py:737] (1/3) Epoch 15, batch 300, loss[loss=0.3579, over 14506.00 frames; utt_duration=1211 frames, utt_pad_proportion=0.01662, over 48.00 utterances.], tot_loss[loss=0.3589, over 2239507.53 frames; utt_duration=1311 frames, utt_pad_proportion=0.04217, over 6842.97 utterances.], batch size: 48, lr: 3.45e-04
|
| 162 |
+
2022-08-11 15:55:13,360 INFO [train.py:737] (1/3) Epoch 15, batch 400, loss[loss=0.312, over 14605.00 frames; utt_duration=1539 frames, utt_pad_proportion=0.005205, over 38.00 utterances.], tot_loss[loss=0.358, over 2497396.15 frames; utt_duration=1325 frames, utt_pad_proportion=0.03774, over 7549.11 utterances.], batch size: 38, lr: 3.45e-04
|
| 163 |
+
2022-08-11 15:57:32,262 INFO [train.py:737] (1/3) Epoch 15, batch 500, loss[loss=0.3434, over 14582.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.00611, over 42.00 utterances.], tot_loss[loss=0.3553, over 2640867.45 frames; utt_duration=1305 frames, utt_pad_proportion=0.04258, over 8103.05 utterances.], batch size: 42, lr: 3.45e-04
|
| 164 |
+
2022-08-11 16:00:45,360 INFO [train.py:737] (1/3) Epoch 15, batch 600, loss[loss=0.3072, over 14781.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.006614, over 37.00 utterances.], tot_loss[loss=0.3569, over 2746801.66 frames; utt_duration=1347 frames, utt_pad_proportion=0.03283, over 8166.44 utterances.], batch size: 37, lr: 3.45e-04
|
| 165 |
+
2022-08-11 16:02:53,911 INFO [train.py:737] (1/3) Epoch 15, batch 700, loss[loss=0.3541, over 14783.00 frames; utt_duration=1600 frames, utt_pad_proportion=0.005762, over 37.00 utterances.], tot_loss[loss=0.3565, over 2791621.29 frames; utt_duration=1326 frames, utt_pad_proportion=0.04168, over 8431.92 utterances.], batch size: 37, lr: 3.45e-04
|
| 166 |
+
2022-08-11 16:05:33,898 INFO [train.py:737] (1/3) Epoch 15, batch 800, loss[loss=0.399, over 14781.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.00603, over 37.00 utterances.], tot_loss[loss=0.358, over 2832160.32 frames; utt_duration=1337 frames, utt_pad_proportion=0.03833, over 8484.60 utterances.], batch size: 37, lr: 3.45e-04
|
| 167 |
+
2022-08-11 16:07:53,555 INFO [train.py:737] (1/3) Epoch 15, batch 900, loss[loss=0.352, over 14781.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.005478, over 38.00 utterances.], tot_loss[loss=0.3597, over 2854547.82 frames; utt_duration=1331 frames, utt_pad_proportion=0.03661, over 8589.37 utterances.], batch size: 38, lr: 3.45e-04
|
| 168 |
+
2022-08-11 16:10:29,685 INFO [train.py:737] (1/3) Epoch 15, batch 1000, loss[loss=0.3854, over 14707.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.0112, over 46.00 utterances.], tot_loss[loss=0.3589, over 2856761.70 frames; utt_duration=1312 frames, utt_pad_proportion=0.04212, over 8721.27 utterances.], batch size: 46, lr: 3.45e-04
|
| 169 |
+
2022-08-11 16:11:56,570 INFO [train.py:737] (1/3) Epoch 15, batch 1100, loss[loss=0.3775, over 14596.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.006579, over 38.00 utterances.], tot_loss[loss=0.3622, over 2869754.08 frames; utt_duration=1328 frames, utt_pad_proportion=0.03915, over 8652.58 utterances.], batch size: 38, lr: 3.45e-04
|
| 170 |
+
2022-08-11 16:13:12,427 INFO [train.py:737] (1/3) Epoch 15, batch 1200, loss[loss=0.3372, over 14646.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.007395, over 44.00 utterances.], tot_loss[loss=0.3599, over 2882730.75 frames; utt_duration=1355 frames, utt_pad_proportion=0.03444, over 8520.68 utterances.], batch size: 44, lr: 3.44e-04
|
| 171 |
+
2022-08-11 16:16:21,711 INFO [train.py:737] (1/3) Epoch 15, batch 1300, loss[loss=0.3651, over 14776.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.007583, over 37.00 utterances.], tot_loss[loss=0.3634, over 2875595.65 frames; utt_duration=1325 frames, utt_pad_proportion=0.04047, over 8688.35 utterances.], batch size: 37, lr: 3.44e-04
|
| 172 |
+
2022-08-11 16:19:25,058 INFO [train.py:737] (1/3) Epoch 15, batch 1400, loss[loss=0.3295, over 14732.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.007337, over 43.00 utterances.], tot_loss[loss=0.3609, over 2884919.67 frames; utt_duration=1337 frames, utt_pad_proportion=0.03422, over 8643.32 utterances.], batch size: 43, lr: 3.44e-04
|
| 173 |
+
2022-08-11 16:22:02,436 INFO [train.py:737] (1/3) Epoch 15, batch 1500, loss[loss=0.4316, over 14575.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005368, over 40.00 utterances.], tot_loss[loss=0.3611, over 2876293.79 frames; utt_duration=1310 frames, utt_pad_proportion=0.04196, over 8794.68 utterances.], batch size: 40, lr: 3.44e-04
|
| 174 |
+
2022-08-11 16:25:07,902 INFO [train.py:737] (1/3) Epoch 15, batch 1600, loss[loss=0.3629, over 14583.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.00528, over 42.00 utterances.], tot_loss[loss=0.3611, over 2874644.79 frames; utt_duration=1306 frames, utt_pad_proportion=0.04161, over 8813.74 utterances.], batch size: 42, lr: 3.44e-04
|
| 175 |
+
2022-08-11 16:27:01,394 INFO [train.py:737] (1/3) Epoch 15, batch 1700, loss[loss=0.3928, over 14604.00 frames; utt_duration=1426 frames, utt_pad_proportion=0.0048, over 41.00 utterances.], tot_loss[loss=0.3635, over 2858133.33 frames; utt_duration=1262 frames, utt_pad_proportion=0.05249, over 9072.24 utterances.], batch size: 41, lr: 3.44e-04
|
| 176 |
+
2022-08-11 16:28:51,800 INFO [train.py:737] (1/3) Epoch 15, batch 1800, loss[loss=0.3308, over 14765.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006137, over 39.00 utterances.], tot_loss[loss=0.3643, over 2861110.19 frames; utt_duration=1260 frames, utt_pad_proportion=0.05021, over 9090.86 utterances.], batch size: 39, lr: 3.44e-04
|
| 177 |
+
2022-08-11 16:31:57,526 INFO [train.py:737] (1/3) Epoch 15, batch 1900, loss[loss=0.3328, over 14577.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006297, over 42.00 utterances.], tot_loss[loss=0.3606, over 2869319.46 frames; utt_duration=1286 frames, utt_pad_proportion=0.04573, over 8935.02 utterances.], batch size: 42, lr: 3.44e-04
|
| 178 |
+
2022-08-11 16:33:53,035 INFO [train.py:737] (1/3) Epoch 15, batch 2000, loss[loss=0.3216, over 14469.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.01042, over 33.00 utterances.], tot_loss[loss=0.3604, over 2867521.77 frames; utt_duration=1287 frames, utt_pad_proportion=0.04627, over 8922.54 utterances.], batch size: 33, lr: 3.44e-04
|
| 179 |
+
2022-08-11 16:36:46,008 INFO [train.py:737] (1/3) Epoch 15, batch 2100, loss[loss=0.4279, over 14655.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01368, over 47.00 utterances.], tot_loss[loss=0.3592, over 2864646.42 frames; utt_duration=1290 frames, utt_pad_proportion=0.04832, over 8895.85 utterances.], batch size: 47, lr: 3.43e-04
|
| 180 |
+
2022-08-11 16:39:59,181 INFO [train.py:737] (1/3) Epoch 15, batch 2200, loss[loss=0.3778, over 14749.00 frames; utt_duration=1476 frames, utt_pad_proportion=0.007078, over 40.00 utterances.], tot_loss[loss=0.3629, over 2848172.36 frames; utt_duration=1247 frames, utt_pad_proportion=0.05978, over 9150.03 utterances.], batch size: 40, lr: 3.43e-04
|
| 181 |
+
2022-08-11 16:42:56,526 INFO [train.py:737] (1/3) Epoch 15, batch 2300, loss[loss=0.3326, over 14779.00 frames; utt_duration=1409 frames, utt_pad_proportion=0.005246, over 42.00 utterances.], tot_loss[loss=0.3624, over 2854209.21 frames; utt_duration=1262 frames, utt_pad_proportion=0.05679, over 9055.01 utterances.], batch size: 42, lr: 3.43e-04
|
| 182 |
+
2022-08-11 16:46:01,611 INFO [train.py:737] (1/3) Epoch 15, batch 2400, loss[loss=0.3545, over 14765.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.006688, over 38.00 utterances.], tot_loss[loss=0.3633, over 2844286.13 frames; utt_duration=1248 frames, utt_pad_proportion=0.0622, over 9129.72 utterances.], batch size: 38, lr: 3.43e-04
|
| 183 |
+
2022-08-11 16:48:07,642 INFO [train.py:737] (1/3) Epoch 15, batch 2500, loss[loss=0.288, over 14746.00 frames; utt_duration=1687 frames, utt_pad_proportion=0.008415, over 35.00 utterances.], tot_loss[loss=0.3634, over 2838646.23 frames; utt_duration=1233 frames, utt_pad_proportion=0.06601, over 9218.38 utterances.], batch size: 35, lr: 3.43e-04
|
| 184 |
+
2022-08-11 16:50:38,868 INFO [train.py:737] (1/3) Epoch 15, batch 2600, loss[loss=0.3862, over 14779.00 frames; utt_duration=1409 frames, utt_pad_proportion=0.005196, over 42.00 utterances.], tot_loss[loss=0.3636, over 2836966.62 frames; utt_duration=1234 frames, utt_pad_proportion=0.06688, over 9209.63 utterances.], batch size: 42, lr: 3.43e-04
|
| 185 |
+
2022-08-11 16:53:31,181 INFO [train.py:737] (1/3) Epoch 15, batch 2700, loss[loss=0.3309, over 14792.00 frames; utt_duration=1519 frames, utt_pad_proportion=0.004789, over 39.00 utterances.], tot_loss[loss=0.3623, over 2845846.41 frames; utt_duration=1253 frames, utt_pad_proportion=0.06067, over 9092.62 utterances.], batch size: 39, lr: 3.43e-04
|
| 186 |
+
2022-08-11 16:56:22,269 INFO [train.py:737] (1/3) Epoch 15, batch 2800, loss[loss=0.3733, over 14760.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.005502, over 42.00 utterances.], tot_loss[loss=0.3667, over 2838756.99 frames; utt_duration=1234 frames, utt_pad_proportion=0.06473, over 9209.59 utterances.], batch size: 42, lr: 3.43e-04
|
| 187 |
+
2022-08-11 16:59:23,578 INFO [train.py:737] (1/3) Epoch 15, batch 2900, loss[loss=0.3833, over 14626.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.008749, over 44.00 utterances.], tot_loss[loss=0.3786, over 2849317.09 frames; utt_duration=1260 frames, utt_pad_proportion=0.05948, over 9053.61 utterances.], batch size: 44, lr: 3.43e-04
|
| 188 |
+
2022-08-11 17:02:14,380 INFO [train.py:737] (1/3) Epoch 15, batch 3000, loss[loss=0.4674, over 13102.00 frames; utt_duration=772.3 frames, utt_pad_proportion=0.1244, over 68.00 utterances.], tot_loss[loss=0.3976, over 2837655.76 frames; utt_duration=1236 frames, utt_pad_proportion=0.06496, over 9193.96 utterances.], batch size: 68, lr: 3.42e-04
|
| 189 |
+
2022-08-11 17:02:14,380 INFO [train.py:761] (1/3) Computing validation loss
|
| 190 |
+
2022-08-11 17:02:58,433 INFO [train.py:770] (1/3) Epoch 15, validation: loss=0.2831, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 191 |
+
2022-08-11 17:05:29,808 INFO [train.py:737] (1/3) Epoch 15, batch 3100, loss[loss=0.4863, over 13201.00 frames; utt_duration=778.1 frames, utt_pad_proportion=0.1178, over 68.00 utterances.], tot_loss[loss=0.413, over 2833156.42 frames; utt_duration=1220 frames, utt_pad_proportion=0.0674, over 9300.70 utterances.], batch size: 68, lr: 3.42e-04
|
| 192 |
+
2022-08-11 17:08:07,074 INFO [train.py:737] (1/3) Epoch 15, batch 3200, loss[loss=0.3766, over 14753.00 frames; utt_duration=1688 frames, utt_pad_proportion=0.008511, over 35.00 utterances.], tot_loss[loss=0.4231, over 2823262.71 frames; utt_duration=1199 frames, utt_pad_proportion=0.07364, over 9430.50 utterances.], batch size: 35, lr: 3.42e-04
|
| 193 |
+
2022-08-11 17:10:37,552 INFO [train.py:737] (1/3) Epoch 15, batch 3300, loss[loss=0.3896, over 14768.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006452, over 39.00 utterances.], tot_loss[loss=0.4293, over 2844799.89 frames; utt_duration=1243 frames, utt_pad_proportion=0.06092, over 9167.99 utterances.], batch size: 39, lr: 3.42e-04
|
| 194 |
+
2022-08-11 17:12:19,929 INFO [train.py:737] (1/3) Epoch 15, batch 3400, loss[loss=0.4067, over 14763.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005216, over 42.00 utterances.], tot_loss[loss=0.435, over 2842575.64 frames; utt_duration=1240 frames, utt_pad_proportion=0.06249, over 9184.39 utterances.], batch size: 42, lr: 3.42e-04
|
| 195 |
+
2022-08-11 17:14:52,684 INFO [train.py:737] (1/3) Epoch 15, batch 3500, loss[loss=0.4131, over 14668.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.01218, over 47.00 utterances.], tot_loss[loss=0.437, over 2835321.49 frames; utt_duration=1229 frames, utt_pad_proportion=0.0654, over 9239.87 utterances.], batch size: 47, lr: 3.42e-04
|
| 196 |
+
2022-08-11 17:17:37,567 INFO [train.py:737] (1/3) Epoch 15, batch 3600, loss[loss=0.4867, over 14585.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.005143, over 42.00 utterances.], tot_loss[loss=0.4406, over 2834906.41 frames; utt_duration=1226 frames, utt_pad_proportion=0.06611, over 9261.07 utterances.], batch size: 42, lr: 3.42e-04
|
| 197 |
+
2022-08-11 17:20:24,453 INFO [train.py:737] (1/3) Epoch 15, batch 3700, loss[loss=0.3726, over 14729.00 frames; utt_duration=1685 frames, utt_pad_proportion=0.01011, over 35.00 utterances.], tot_loss[loss=0.4393, over 2831860.66 frames; utt_duration=1222 frames, utt_pad_proportion=0.06792, over 9282.64 utterances.], batch size: 35, lr: 3.42e-04
|
| 198 |
+
2022-08-11 17:23:11,070 INFO [train.py:737] (1/3) Epoch 15, batch 3800, loss[loss=0.4545, over 14596.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.004599, over 41.00 utterances.], tot_loss[loss=0.4402, over 2824270.80 frames; utt_duration=1217 frames, utt_pad_proportion=0.07366, over 9291.05 utterances.], batch size: 41, lr: 3.41e-04
|
| 199 |
+
2022-08-11 17:25:37,104 INFO [train.py:737] (1/3) Epoch 15, batch 3900, loss[loss=0.4474, over 14778.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.005882, over 38.00 utterances.], tot_loss[loss=0.4416, over 2813518.38 frames; utt_duration=1191 frames, utt_pad_proportion=0.07973, over 9457.83 utterances.], batch size: 38, lr: 3.41e-04
|
| 200 |
+
2022-08-11 17:28:09,794 INFO [train.py:737] (1/3) Epoch 15, batch 4000, loss[loss=0.4699, over 14786.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.005361, over 38.00 utterances.], tot_loss[loss=0.4407, over 2820384.33 frames; utt_duration=1200 frames, utt_pad_proportion=0.07597, over 9411.24 utterances.], batch size: 38, lr: 3.41e-04
|
| 201 |
+
2022-08-11 17:30:45,452 INFO [train.py:737] (1/3) Epoch 15, batch 4100, loss[loss=0.5007, over 14449.00 frames; utt_duration=1157 frames, utt_pad_proportion=0.02606, over 50.00 utterances.], tot_loss[loss=0.4412, over 2833619.95 frames; utt_duration=1222 frames, utt_pad_proportion=0.06798, over 9283.75 utterances.], batch size: 50, lr: 3.41e-04
|
| 202 |
+
2022-08-11 17:33:21,443 INFO [train.py:737] (1/3) Epoch 15, batch 4200, loss[loss=0.4757, over 14578.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.0063, over 39.00 utterances.], tot_loss[loss=0.4425, over 2820785.10 frames; utt_duration=1200 frames, utt_pad_proportion=0.0752, over 9415.82 utterances.], batch size: 39, lr: 3.41e-04
|
| 203 |
+
2022-08-11 17:35:41,151 INFO [train.py:737] (1/3) Epoch 15, batch 4300, loss[loss=0.4614, over 13786.00 frames; utt_duration=968.7 frames, utt_pad_proportion=0.06764, over 57.00 utterances.], tot_loss[loss=0.4403, over 2832814.09 frames; utt_duration=1229 frames, utt_pad_proportion=0.06956, over 9231.93 utterances.], batch size: 57, lr: 3.41e-04
|
| 204 |
+
2022-08-11 17:38:29,485 INFO [train.py:737] (1/3) Epoch 15, batch 4400, loss[loss=0.4995, over 14751.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005757, over 41.00 utterances.], tot_loss[loss=0.4398, over 2826720.17 frames; utt_duration=1227 frames, utt_pad_proportion=0.07374, over 9222.83 utterances.], batch size: 41, lr: 3.41e-04
|
| 205 |
+
2022-08-11 17:41:15,836 INFO [train.py:737] (1/3) Epoch 15, batch 4500, loss[loss=0.4027, over 14764.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.00654, over 40.00 utterances.], tot_loss[loss=0.4403, over 2821243.31 frames; utt_duration=1212 frames, utt_pad_proportion=0.07593, over 9324.85 utterances.], batch size: 40, lr: 3.41e-04
|
| 206 |
+
2022-08-11 17:43:38,600 INFO [train.py:737] (1/3) Epoch 15, batch 4600, loss[loss=0.4346, over 14676.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.01013, over 45.00 utterances.], tot_loss[loss=0.4375, over 2818830.98 frames; utt_duration=1214 frames, utt_pad_proportion=0.07641, over 9298.19 utterances.], batch size: 45, lr: 3.41e-04
|
| 207 |
+
2022-08-11 17:46:05,259 INFO [train.py:737] (1/3) Epoch 15, batch 4700, loss[loss=0.4436, over 14609.00 frames; utt_duration=1539 frames, utt_pad_proportion=0.004899, over 38.00 utterances.], tot_loss[loss=0.4402, over 2824938.96 frames; utt_duration=1210 frames, utt_pad_proportion=0.07196, over 9352.52 utterances.], batch size: 38, lr: 3.40e-04
|
| 208 |
+
2022-08-11 17:48:30,593 INFO [train.py:737] (1/3) Epoch 15, batch 4800, loss[loss=0.4436, over 14768.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.006352, over 38.00 utterances.], tot_loss[loss=0.439, over 2837287.74 frames; utt_duration=1236 frames, utt_pad_proportion=0.06501, over 9190.52 utterances.], batch size: 38, lr: 3.40e-04
|
| 209 |
+
2022-08-11 17:50:07,720 INFO [train.py:737] (1/3) Epoch 15, batch 4900, loss[loss=0.4975, over 9694.00 frames; utt_duration=418.6 frames, utt_pad_proportion=0.3429, over 93.00 utterances.], tot_loss[loss=0.4393, over 2815833.35 frames; utt_duration=1199 frames, utt_pad_proportion=0.07904, over 9409.36 utterances.], batch size: 93, lr: 3.40e-04
|
| 210 |
+
2022-08-11 17:52:17,902 INFO [train.py:737] (1/3) Epoch 15, batch 5000, loss[loss=0.429, over 13812.00 frames; utt_duration=970.7 frames, utt_pad_proportion=0.06487, over 57.00 utterances.], tot_loss[loss=0.4403, over 2820293.85 frames; utt_duration=1194 frames, utt_pad_proportion=0.07547, over 9459.88 utterances.], batch size: 57, lr: 3.40e-04
|
| 211 |
+
2022-08-11 17:54:42,991 INFO [train.py:737] (1/3) Epoch 15, batch 5100, loss[loss=0.4343, over 14476.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.01004, over 33.00 utterances.], tot_loss[loss=0.4414, over 2836514.20 frames; utt_duration=1216 frames, utt_pad_proportion=0.06556, over 9343.34 utterances.], batch size: 33, lr: 3.40e-04
|
| 212 |
+
2022-08-11 17:57:27,534 INFO [train.py:737] (1/3) Epoch 15, batch 5200, loss[loss=0.4564, over 14509.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.0175, over 48.00 utterances.], tot_loss[loss=0.4437, over 2835963.66 frames; utt_duration=1212 frames, utt_pad_proportion=0.06546, over 9368.42 utterances.], batch size: 48, lr: 3.40e-04
|
| 213 |
+
2022-08-11 18:00:09,220 INFO [train.py:737] (1/3) Epoch 15, batch 5300, loss[loss=0.373, over 14563.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.005746, over 37.00 utterances.], tot_loss[loss=0.4414, over 2836095.37 frames; utt_duration=1219 frames, utt_pad_proportion=0.06667, over 9317.10 utterances.], batch size: 37, lr: 3.40e-04
|
| 214 |
+
2022-08-11 18:02:48,923 INFO [train.py:737] (1/3) Epoch 15, batch 5400, loss[loss=0.3904, over 14588.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.007747, over 36.00 utterances.], tot_loss[loss=0.4418, over 2820958.24 frames; utt_duration=1195 frames, utt_pad_proportion=0.07535, over 9452.90 utterances.], batch size: 36, lr: 3.40e-04
|
| 215 |
+
2022-08-11 18:05:10,737 INFO [train.py:737] (1/3) Epoch 15, batch 5500, loss[loss=0.4082, over 14575.00 frames; utt_duration=1496 frames, utt_pad_proportion=0.006402, over 39.00 utterances.], tot_loss[loss=0.4417, over 2823566.34 frames; utt_duration=1206 frames, utt_pad_proportion=0.07376, over 9373.03 utterances.], batch size: 39, lr: 3.40e-04
|
| 216 |
+
2022-08-11 18:06:37,414 INFO [train.py:737] (1/3) Epoch 15, batch 5600, loss[loss=0.5341, over 10067.00 frames; utt_duration=439.2 frames, utt_pad_proportion=0.3254, over 92.00 utterances.], tot_loss[loss=0.4414, over 2817932.88 frames; utt_duration=1193 frames, utt_pad_proportion=0.07654, over 9463.64 utterances.], batch size: 92, lr: 3.39e-04
|
| 217 |
+
2022-08-11 18:09:12,334 INFO [train.py:737] (1/3) Epoch 15, batch 5700, loss[loss=0.5071, over 9572.00 frames; utt_duration=417.6 frames, utt_pad_proportion=0.3545, over 92.00 utterances.], tot_loss[loss=0.4418, over 2809023.16 frames; utt_duration=1173 frames, utt_pad_proportion=0.08103, over 9589.84 utterances.], batch size: 92, lr: 3.39e-04
|
| 218 |
+
2022-08-11 18:12:03,387 INFO [train.py:737] (1/3) Epoch 15, batch 5800, loss[loss=0.4552, over 13796.00 frames; utt_duration=969.8 frames, utt_pad_proportion=0.06661, over 57.00 utterances.], tot_loss[loss=0.4393, over 2828567.89 frames; utt_duration=1215 frames, utt_pad_proportion=0.07006, over 9321.63 utterances.], batch size: 57, lr: 3.39e-04
|
| 219 |
+
2022-08-11 18:14:35,533 INFO [train.py:737] (1/3) Epoch 15, batch 5900, loss[loss=0.427, over 14589.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.00626, over 38.00 utterances.], tot_loss[loss=0.4394, over 2831019.68 frames; utt_duration=1216 frames, utt_pad_proportion=0.06836, over 9321.37 utterances.], batch size: 38, lr: 3.39e-04
|
| 220 |
+
2022-08-11 18:17:10,947 INFO [train.py:737] (1/3) Epoch 15, batch 6000, loss[loss=0.4934, over 13005.00 frames; utt_duration=777.8 frames, utt_pad_proportion=0.1211, over 67.00 utterances.], tot_loss[loss=0.4398, over 2836962.93 frames; utt_duration=1234 frames, utt_pad_proportion=0.0664, over 9209.50 utterances.], batch size: 67, lr: 3.39e-04
|
| 221 |
+
2022-08-11 18:17:10,948 INFO [train.py:761] (1/3) Computing validation loss
|
| 222 |
+
2022-08-11 18:18:08,645 INFO [train.py:770] (1/3) Epoch 15, validation: loss=0.3006, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 223 |
+
2022-08-11 18:20:56,945 INFO [train.py:737] (1/3) Epoch 15, batch 6100, loss[loss=0.5431, over 9511.00 frames; utt_duration=415.2 frames, utt_pad_proportion=0.3622, over 92.00 utterances.], tot_loss[loss=0.4458, over 2763590.17 frames; utt_duration=1111 frames, utt_pad_proportion=0.1091, over 9962.48 utterances.], batch size: 92, lr: 3.39e-04
|
| 224 |
+
2022-08-11 18:22:06,668 INFO [train.py:737] (1/3) Epoch 16, batch 0, loss[loss=0.438, over 9397.00 frames; utt_duration=410.2 frames, utt_pad_proportion=0.368, over 92.00 utterances.], tot_loss[loss=0.438, over 9397.00 frames; utt_duration=410.2 frames, utt_pad_proportion=0.368, over 92.00 utterances.], batch size: 92, lr: 3.29e-04
|
| 225 |
+
2022-08-11 18:25:01,423 INFO [train.py:737] (1/3) Epoch 16, batch 100, loss[loss=0.432, over 10034.00 frames; utt_duration=442.6 frames, utt_pad_proportion=0.3221, over 91.00 utterances.], tot_loss[loss=0.3662, over 1126386.17 frames; utt_duration=1256 frames, utt_pad_proportion=0.06815, over 3592.53 utterances.], batch size: 91, lr: 3.29e-04
|
exp/log/log-train-2022-08-11-08-52-44-2
ADDED
|
@@ -0,0 +1,225 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2022-08-11 08:52:44,616 INFO [train.py:805] (2/3) Training started
|
| 2 |
+
2022-08-11 08:52:44,616 INFO [train.py:806] (2/3) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'env_info': {'k2-version': '1.17', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '7dcabf85e8bf06984c4abab0400ef1322b5ff3df', 'k2-git-date': 'Tue Aug 2 21:22:39 2022', 'lhotse-version': '1.4.0.dev+git.59318ed.clean', 'torch-version': '1.11.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.3', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '6af45c3-dirty', 'icefall-git-date': 'Tue Aug 2 14:47:09 2022', 'icefall-path': '/thspfs/home/acct-hpc/thspzh-hpc1/git/icefall_dev', 'k2-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/k2-1.17.dev20220805+cuda11.3.torch1.11.0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'dgxs-01.cluster.thspzh.com', 'IP address': '192.168.200.101'}, 'world_size': 3, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 20, 'start_epoch': 13, 'start_batch': 0, 'exp_dir': PosixPath('cnn_ctc/exp_phone'), 'lang_dir': 'data/lang_phone', 'bpe_model': 'data/lang_bpe_500/bpe.model', 'initial_lr': 0.003, 'lr_batches': 5000, 'lr_epochs': 6, 'seed': 42, 'print_diagnostics': False, 'save_every_n': 8000, 'keep_last_k': 20, 'average_period': 100, 'use_fp16': False, 'first_out_channels': 8, 'conv_blocks': 3, 'rnn_dim': 0, 'full_libri': True, 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 600, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'input_strategy': 'PrecomputedFeatures'}
|
| 3 |
+
2022-08-11 08:52:44,965 INFO [lexicon.py:176] (2/3) Loading pre-compiled data/lang_phone/Linv.pt
|
| 4 |
+
2022-08-11 08:52:57,153 INFO [train.py:843] (2/3) About to create model
|
| 5 |
+
2022-08-11 08:52:57,176 INFO [checkpoint.py:112] (2/3) Loading checkpoint from cnn_ctc/exp_phone/epoch-12.pt
|
| 6 |
+
2022-08-11 08:52:57,918 INFO [train.py:876] (2/3) Using DDP
|
| 7 |
+
2022-08-11 08:53:02,939 INFO [train.py:884] (2/3) Loading optimizer state dict
|
| 8 |
+
2022-08-11 08:53:02,988 INFO [train.py:892] (2/3) Loading scheduler state dict
|
| 9 |
+
2022-08-11 08:53:02,988 INFO [asr_datamodule.py:409] (2/3) About to get train-clean-100 cuts
|
| 10 |
+
2022-08-11 08:53:03,024 INFO [asr_datamodule.py:416] (2/3) About to get train-clean-360 cuts
|
| 11 |
+
2022-08-11 08:53:03,046 INFO [asr_datamodule.py:423] (2/3) About to get train-other-500 cuts
|
| 12 |
+
2022-08-11 08:53:03,064 INFO [asr_datamodule.py:225] (2/3) Enable MUSAN
|
| 13 |
+
2022-08-11 08:53:03,064 INFO [asr_datamodule.py:226] (2/3) About to get Musan cuts
|
| 14 |
+
2022-08-11 08:53:05,058 INFO [asr_datamodule.py:254] (2/3) Enable SpecAugment
|
| 15 |
+
2022-08-11 08:53:05,058 INFO [asr_datamodule.py:255] (2/3) Time warp factor: 80
|
| 16 |
+
2022-08-11 08:53:05,058 INFO [asr_datamodule.py:267] (2/3) Num frame mask: 10
|
| 17 |
+
2022-08-11 08:53:05,058 INFO [asr_datamodule.py:280] (2/3) About to create train dataset
|
| 18 |
+
2022-08-11 08:53:05,058 INFO [asr_datamodule.py:309] (2/3) Using DynamicBucketingSampler.
|
| 19 |
+
2022-08-11 08:53:07,346 INFO [asr_datamodule.py:324] (2/3) About to create train dataloader
|
| 20 |
+
2022-08-11 08:53:07,358 INFO [asr_datamodule.py:430] (2/3) About to get dev-clean cuts
|
| 21 |
+
2022-08-11 08:53:07,368 INFO [asr_datamodule.py:437] (2/3) About to get dev-other cuts
|
| 22 |
+
2022-08-11 08:53:07,382 INFO [asr_datamodule.py:355] (2/3) About to create dev dataset
|
| 23 |
+
2022-08-11 08:53:07,684 INFO [asr_datamodule.py:374] (2/3) About to create dev dataloader
|
| 24 |
+
2022-08-11 08:53:07,684 INFO [train.py:958] (2/3) Loading grad scaler state dict
|
| 25 |
+
2022-08-11 08:53:27,592 INFO [train.py:737] (2/3) Epoch 13, batch 0, loss[loss=0.4479, over 14783.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.005193, over 38.00 utterances.], tot_loss[loss=0.4479, over 14783.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.005193, over 38.00 utterances.], batch size: 38, lr: 3.85e-04
|
| 26 |
+
2022-08-11 08:53:27,642 INFO [distributed.py:948] (2/3) Reducer buckets have been rebuilt in this iteration.
|
| 27 |
+
2022-08-11 08:56:11,072 INFO [train.py:737] (2/3) Epoch 13, batch 100, loss[loss=0.3395, over 14772.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.006049, over 39.00 utterances.], tot_loss[loss=0.368, over 1130545.74 frames; utt_duration=1253 frames, utt_pad_proportion=0.06418, over 3612.74 utterances.], batch size: 39, lr: 3.85e-04
|
| 28 |
+
2022-08-11 08:58:59,357 INFO [train.py:737] (2/3) Epoch 13, batch 200, loss[loss=0.3228, over 14599.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.006154, over 38.00 utterances.], tot_loss[loss=0.3588, over 1822205.23 frames; utt_duration=1326 frames, utt_pad_proportion=0.04429, over 5503.16 utterances.], batch size: 38, lr: 3.84e-04
|
| 29 |
+
2022-08-11 09:01:26,615 INFO [train.py:737] (2/3) Epoch 13, batch 300, loss[loss=0.4178, over 14591.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.008286, over 36.00 utterances.], tot_loss[loss=0.3584, over 2249512.83 frames; utt_duration=1359 frames, utt_pad_proportion=0.03164, over 6629.46 utterances.], batch size: 36, lr: 3.84e-04
|
| 30 |
+
2022-08-11 09:03:10,458 INFO [train.py:737] (2/3) Epoch 13, batch 400, loss[loss=0.3563, over 14746.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.005692, over 43.00 utterances.], tot_loss[loss=0.3574, over 2491454.32 frames; utt_duration=1317 frames, utt_pad_proportion=0.03879, over 7575.36 utterances.], batch size: 43, lr: 3.84e-04
|
| 31 |
+
2022-08-11 09:04:16,681 INFO [train.py:737] (2/3) Epoch 13, batch 500, loss[loss=0.3532, over 14755.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006876, over 40.00 utterances.], tot_loss[loss=0.3562, over 2641470.29 frames; utt_duration=1311 frames, utt_pad_proportion=0.04072, over 8066.57 utterances.], batch size: 40, lr: 3.84e-04
|
| 32 |
+
2022-08-11 09:05:12,191 INFO [train.py:737] (2/3) Epoch 13, batch 600, loss[loss=0.3661, over 14693.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009646, over 45.00 utterances.], tot_loss[loss=0.3579, over 2730920.59 frames; utt_duration=1312 frames, utt_pad_proportion=0.04346, over 8332.98 utterances.], batch size: 45, lr: 3.84e-04
|
| 33 |
+
2022-08-11 09:06:24,543 INFO [train.py:737] (2/3) Epoch 13, batch 700, loss[loss=0.3561, over 13064.00 frames; utt_duration=781.5 frames, utt_pad_proportion=0.117, over 67.00 utterances.], tot_loss[loss=0.3602, over 2785220.17 frames; utt_duration=1302 frames, utt_pad_proportion=0.04515, over 8565.16 utterances.], batch size: 67, lr: 3.84e-04
|
| 34 |
+
2022-08-11 09:09:10,554 INFO [train.py:737] (2/3) Epoch 13, batch 800, loss[loss=0.4344, over 14568.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006574, over 40.00 utterances.], tot_loss[loss=0.3635, over 2827488.27 frames; utt_duration=1310 frames, utt_pad_proportion=0.03985, over 8642.46 utterances.], batch size: 40, lr: 3.84e-04
|
| 35 |
+
2022-08-11 09:11:56,854 INFO [train.py:737] (2/3) Epoch 13, batch 900, loss[loss=0.4191, over 12874.00 frames; utt_duration=770.2 frames, utt_pad_proportion=0.1297, over 67.00 utterances.], tot_loss[loss=0.3663, over 2839400.39 frames; utt_duration=1284 frames, utt_pad_proportion=0.04374, over 8855.47 utterances.], batch size: 67, lr: 3.83e-04
|
| 36 |
+
2022-08-11 09:14:45,133 INFO [train.py:737] (2/3) Epoch 13, batch 1000, loss[loss=0.3751, over 14551.00 frames; utt_duration=1214 frames, utt_pad_proportion=0.01464, over 48.00 utterances.], tot_loss[loss=0.3635, over 2849183.77 frames; utt_duration=1281 frames, utt_pad_proportion=0.04688, over 8905.19 utterances.], batch size: 48, lr: 3.83e-04
|
| 37 |
+
2022-08-11 09:17:11,265 INFO [train.py:737] (2/3) Epoch 13, batch 1100, loss[loss=0.4212, over 14630.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.008394, over 44.00 utterances.], tot_loss[loss=0.3653, over 2841233.64 frames; utt_duration=1254 frames, utt_pad_proportion=0.05574, over 9071.76 utterances.], batch size: 44, lr: 3.83e-04
|
| 38 |
+
2022-08-11 09:19:38,557 INFO [train.py:737] (2/3) Epoch 13, batch 1200, loss[loss=0.3598, over 14868.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.006591, over 44.00 utterances.], tot_loss[loss=0.3658, over 2853120.25 frames; utt_duration=1272 frames, utt_pad_proportion=0.05263, over 8980.21 utterances.], batch size: 44, lr: 3.83e-04
|
| 39 |
+
2022-08-11 09:22:05,178 INFO [train.py:737] (2/3) Epoch 13, batch 1300, loss[loss=0.3679, over 14429.00 frames; utt_duration=1650 frames, utt_pad_proportion=0.01015, over 35.00 utterances.], tot_loss[loss=0.3656, over 2854392.18 frames; utt_duration=1279 frames, utt_pad_proportion=0.05354, over 8938.87 utterances.], batch size: 35, lr: 3.83e-04
|
| 40 |
+
2022-08-11 09:24:12,080 INFO [train.py:737] (2/3) Epoch 13, batch 1400, loss[loss=0.3407, over 14608.00 frames; utt_duration=1539 frames, utt_pad_proportion=0.005746, over 38.00 utterances.], tot_loss[loss=0.3668, over 2856373.62 frames; utt_duration=1284 frames, utt_pad_proportion=0.05219, over 8911.49 utterances.], batch size: 38, lr: 3.83e-04
|
| 41 |
+
2022-08-11 09:26:58,344 INFO [train.py:737] (2/3) Epoch 13, batch 1500, loss[loss=0.2732, over 14436.00 frames; utt_duration=1651 frames, utt_pad_proportion=0.009461, over 35.00 utterances.], tot_loss[loss=0.3661, over 2865712.78 frames; utt_duration=1293 frames, utt_pad_proportion=0.04681, over 8876.47 utterances.], batch size: 35, lr: 3.83e-04
|
| 42 |
+
2022-08-11 09:29:51,304 INFO [train.py:737] (2/3) Epoch 13, batch 1600, loss[loss=0.4342, over 10512.00 frames; utt_duration=458.4 frames, utt_pad_proportion=0.2947, over 92.00 utterances.], tot_loss[loss=0.3655, over 2860570.74 frames; utt_duration=1278 frames, utt_pad_proportion=0.0518, over 8964.46 utterances.], batch size: 92, lr: 3.82e-04
|
| 43 |
+
2022-08-11 09:32:41,585 INFO [train.py:737] (2/3) Epoch 13, batch 1700, loss[loss=0.3463, over 13739.00 frames; utt_duration=965.5 frames, utt_pad_proportion=0.06899, over 57.00 utterances.], tot_loss[loss=0.3647, over 2861084.53 frames; utt_duration=1287 frames, utt_pad_proportion=0.05082, over 8902.71 utterances.], batch size: 57, lr: 3.82e-04
|
| 44 |
+
2022-08-11 09:34:42,369 INFO [train.py:737] (2/3) Epoch 13, batch 1800, loss[loss=0.4181, over 14562.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006152, over 40.00 utterances.], tot_loss[loss=0.3649, over 2853504.21 frames; utt_duration=1281 frames, utt_pad_proportion=0.05648, over 8923.20 utterances.], batch size: 40, lr: 3.82e-04
|
| 45 |
+
2022-08-11 09:37:23,734 INFO [train.py:737] (2/3) Epoch 13, batch 1900, loss[loss=0.3677, over 14792.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.004941, over 38.00 utterances.], tot_loss[loss=0.365, over 2859859.39 frames; utt_duration=1288 frames, utt_pad_proportion=0.05167, over 8891.88 utterances.], batch size: 38, lr: 3.82e-04
|
| 46 |
+
2022-08-11 09:39:48,329 INFO [train.py:737] (2/3) Epoch 13, batch 2000, loss[loss=0.4014, over 14585.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.00577, over 41.00 utterances.], tot_loss[loss=0.3648, over 2875356.90 frames; utt_duration=1316 frames, utt_pad_proportion=0.04364, over 8752.09 utterances.], batch size: 41, lr: 3.82e-04
|
| 47 |
+
2022-08-11 09:42:14,728 INFO [train.py:737] (2/3) Epoch 13, batch 2100, loss[loss=0.3358, over 14346.00 frames; utt_duration=1084 frames, utt_pad_proportion=0.03697, over 53.00 utterances.], tot_loss[loss=0.3638, over 2854092.49 frames; utt_duration=1263 frames, utt_pad_proportion=0.05618, over 9047.39 utterances.], batch size: 53, lr: 3.82e-04
|
| 48 |
+
2022-08-11 09:44:38,434 INFO [train.py:737] (2/3) Epoch 13, batch 2200, loss[loss=0.4013, over 14361.00 frames; utt_duration=1085 frames, utt_pad_proportion=0.03618, over 53.00 utterances.], tot_loss[loss=0.3644, over 2862226.52 frames; utt_duration=1283 frames, utt_pad_proportion=0.05156, over 8932.63 utterances.], batch size: 53, lr: 3.82e-04
|
| 49 |
+
2022-08-11 09:47:02,613 INFO [train.py:737] (2/3) Epoch 13, batch 2300, loss[loss=0.3997, over 14443.00 frames; utt_duration=1157 frames, utt_pad_proportion=0.02596, over 50.00 utterances.], tot_loss[loss=0.3644, over 2875102.98 frames; utt_duration=1307 frames, utt_pad_proportion=0.04305, over 8809.57 utterances.], batch size: 50, lr: 3.81e-04
|
| 50 |
+
2022-08-11 09:49:33,833 INFO [train.py:737] (2/3) Epoch 13, batch 2400, loss[loss=0.3254, over 14585.00 frames; utt_duration=1578 frames, utt_pad_proportion=0.005484, over 37.00 utterances.], tot_loss[loss=0.3653, over 2856376.52 frames; utt_duration=1268 frames, utt_pad_proportion=0.05274, over 9019.29 utterances.], batch size: 37, lr: 3.81e-04
|
| 51 |
+
2022-08-11 09:52:02,610 INFO [train.py:737] (2/3) Epoch 13, batch 2500, loss[loss=0.3753, over 14718.00 frames; utt_duration=1282 frames, utt_pad_proportion=0.01041, over 46.00 utterances.], tot_loss[loss=0.3642, over 2857978.32 frames; utt_duration=1270 frames, utt_pad_proportion=0.05376, over 9009.54 utterances.], batch size: 46, lr: 3.81e-04
|
| 52 |
+
2022-08-11 09:54:20,857 INFO [train.py:737] (2/3) Epoch 13, batch 2600, loss[loss=0.3019, over 14587.00 frames; utt_duration=1718 frames, utt_pad_proportion=0.01111, over 34.00 utterances.], tot_loss[loss=0.3645, over 2847593.82 frames; utt_duration=1248 frames, utt_pad_proportion=0.06061, over 9141.13 utterances.], batch size: 34, lr: 3.81e-04
|
| 53 |
+
2022-08-11 09:56:50,909 INFO [train.py:737] (2/3) Epoch 13, batch 2700, loss[loss=0.3306, over 14748.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.006428, over 43.00 utterances.], tot_loss[loss=0.3646, over 2861872.25 frames; utt_duration=1267 frames, utt_pad_proportion=0.05232, over 9047.67 utterances.], batch size: 43, lr: 3.81e-04
|
| 54 |
+
2022-08-11 09:59:17,870 INFO [train.py:737] (2/3) Epoch 13, batch 2800, loss[loss=0.4039, over 14886.00 frames; utt_duration=1355 frames, utt_pad_proportion=0.005869, over 44.00 utterances.], tot_loss[loss=0.3649, over 2859140.06 frames; utt_duration=1271 frames, utt_pad_proportion=0.05459, over 9009.68 utterances.], batch size: 44, lr: 3.81e-04
|
| 55 |
+
2022-08-11 10:01:53,828 INFO [train.py:737] (2/3) Epoch 13, batch 2900, loss[loss=0.3855, over 14589.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.007072, over 38.00 utterances.], tot_loss[loss=0.3793, over 2853610.80 frames; utt_duration=1262 frames, utt_pad_proportion=0.05551, over 9053.79 utterances.], batch size: 38, lr: 3.81e-04
|
| 56 |
+
2022-08-11 10:04:16,976 INFO [train.py:737] (2/3) Epoch 13, batch 3000, loss[loss=0.4065, over 14571.00 frames; utt_duration=1535 frames, utt_pad_proportion=0.007519, over 38.00 utterances.], tot_loss[loss=0.3984, over 2850093.51 frames; utt_duration=1246 frames, utt_pad_proportion=0.05841, over 9160.99 utterances.], batch size: 38, lr: 3.81e-04
|
| 57 |
+
2022-08-11 10:04:16,977 INFO [train.py:761] (2/3) Computing validation loss
|
| 58 |
+
2022-08-11 10:04:50,565 INFO [train.py:770] (2/3) Epoch 13, validation: loss=0.3727, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 59 |
+
2022-08-11 10:06:51,669 INFO [train.py:737] (2/3) Epoch 13, batch 3100, loss[loss=0.4317, over 14642.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.007062, over 44.00 utterances.], tot_loss[loss=0.414, over 2838041.86 frames; utt_duration=1226 frames, utt_pad_proportion=0.06653, over 9269.91 utterances.], batch size: 44, lr: 3.80e-04
|
| 60 |
+
2022-08-11 10:09:22,791 INFO [train.py:737] (2/3) Epoch 13, batch 3200, loss[loss=0.4894, over 14682.00 frames; utt_duration=1251 frames, utt_pad_proportion=0.01027, over 47.00 utterances.], tot_loss[loss=0.4242, over 2835455.19 frames; utt_duration=1227 frames, utt_pad_proportion=0.06754, over 9257.86 utterances.], batch size: 47, lr: 3.80e-04
|
| 61 |
+
2022-08-11 10:11:51,393 INFO [train.py:737] (2/3) Epoch 13, batch 3300, loss[loss=0.5024, over 13682.00 frames; utt_duration=961.5 frames, utt_pad_proportion=0.07097, over 57.00 utterances.], tot_loss[loss=0.4316, over 2835011.11 frames; utt_duration=1230 frames, utt_pad_proportion=0.0674, over 9231.13 utterances.], batch size: 57, lr: 3.80e-04
|
| 62 |
+
2022-08-11 10:14:06,314 INFO [train.py:737] (2/3) Epoch 13, batch 3400, loss[loss=0.4343, over 14501.00 frames; utt_duration=1162 frames, utt_pad_proportion=0.02059, over 50.00 utterances.], tot_loss[loss=0.4362, over 2828080.08 frames; utt_duration=1214 frames, utt_pad_proportion=0.07154, over 9328.22 utterances.], batch size: 50, lr: 3.80e-04
|
| 63 |
+
2022-08-11 10:16:12,565 INFO [train.py:737] (2/3) Epoch 13, batch 3500, loss[loss=0.5002, over 14748.00 frames; utt_duration=1406 frames, utt_pad_proportion=0.005708, over 42.00 utterances.], tot_loss[loss=0.4369, over 2830336.78 frames; utt_duration=1222 frames, utt_pad_proportion=0.07084, over 9275.70 utterances.], batch size: 42, lr: 3.80e-04
|
| 64 |
+
2022-08-11 10:18:49,063 INFO [train.py:737] (2/3) Epoch 13, batch 3600, loss[loss=0.3988, over 14765.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.004881, over 41.00 utterances.], tot_loss[loss=0.4409, over 2822437.27 frames; utt_duration=1209 frames, utt_pad_proportion=0.0758, over 9353.48 utterances.], batch size: 41, lr: 3.80e-04
|
| 65 |
+
2022-08-11 10:21:10,812 INFO [train.py:737] (2/3) Epoch 13, batch 3700, loss[loss=0.4307, over 14615.00 frames; utt_duration=1721 frames, utt_pad_proportion=0.009798, over 34.00 utterances.], tot_loss[loss=0.4455, over 2836110.18 frames; utt_duration=1227 frames, utt_pad_proportion=0.06762, over 9260.66 utterances.], batch size: 34, lr: 3.80e-04
|
| 66 |
+
2022-08-11 10:23:39,228 INFO [train.py:737] (2/3) Epoch 13, batch 3800, loss[loss=0.532, over 9706.00 frames; utt_duration=423.3 frames, utt_pad_proportion=0.3507, over 92.00 utterances.], tot_loss[loss=0.4465, over 2812009.49 frames; utt_duration=1185 frames, utt_pad_proportion=0.08116, over 9507.71 utterances.], batch size: 92, lr: 3.79e-04
|
| 67 |
+
2022-08-11 10:26:00,931 INFO [train.py:737] (2/3) Epoch 13, batch 3900, loss[loss=0.4396, over 14763.00 frames; utt_duration=1478 frames, utt_pad_proportion=0.005968, over 40.00 utterances.], tot_loss[loss=0.446, over 2814381.94 frames; utt_duration=1177 frames, utt_pad_proportion=0.08039, over 9577.40 utterances.], batch size: 40, lr: 3.79e-04
|
| 68 |
+
2022-08-11 10:28:33,727 INFO [train.py:737] (2/3) Epoch 13, batch 4000, loss[loss=0.4383, over 14589.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005838, over 41.00 utterances.], tot_loss[loss=0.447, over 2833366.92 frames; utt_duration=1223 frames, utt_pad_proportion=0.06987, over 9281.10 utterances.], batch size: 41, lr: 3.79e-04
|
| 69 |
+
2022-08-11 10:30:27,524 INFO [train.py:737] (2/3) Epoch 13, batch 4100, loss[loss=0.4849, over 14678.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.009738, over 45.00 utterances.], tot_loss[loss=0.4431, over 2838846.18 frames; utt_duration=1235 frames, utt_pad_proportion=0.06556, over 9208.05 utterances.], batch size: 45, lr: 3.79e-04
|
| 70 |
+
2022-08-11 10:32:29,520 INFO [train.py:737] (2/3) Epoch 13, batch 4200, loss[loss=0.4139, over 14577.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005978, over 40.00 utterances.], tot_loss[loss=0.4436, over 2833659.43 frames; utt_duration=1226 frames, utt_pad_proportion=0.0683, over 9254.01 utterances.], batch size: 40, lr: 3.79e-04
|
| 71 |
+
2022-08-11 10:34:32,604 INFO [train.py:737] (2/3) Epoch 13, batch 4300, loss[loss=0.5115, over 14692.00 frames; utt_duration=1279 frames, utt_pad_proportion=0.01164, over 46.00 utterances.], tot_loss[loss=0.4455, over 2826086.17 frames; utt_duration=1208 frames, utt_pad_proportion=0.07264, over 9366.65 utterances.], batch size: 46, lr: 3.79e-04
|
| 72 |
+
2022-08-11 10:36:59,752 INFO [train.py:737] (2/3) Epoch 13, batch 4400, loss[loss=0.3665, over 14705.00 frames; utt_duration=1682 frames, utt_pad_proportion=0.01102, over 35.00 utterances.], tot_loss[loss=0.4458, over 2828418.77 frames; utt_duration=1215 frames, utt_pad_proportion=0.07208, over 9320.04 utterances.], batch size: 35, lr: 3.79e-04
|
| 73 |
+
2022-08-11 10:40:20,190 INFO [train.py:737] (2/3) Epoch 13, batch 4500, loss[loss=0.4412, over 14770.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007235, over 37.00 utterances.], tot_loss[loss=0.4484, over 2806016.64 frames; utt_duration=1178 frames, utt_pad_proportion=0.08578, over 9539.00 utterances.], batch size: 37, lr: 3.78e-04
|
| 74 |
+
2022-08-11 10:44:36,545 INFO [train.py:737] (2/3) Epoch 13, batch 4600, loss[loss=0.403, over 14495.00 frames; utt_duration=1161 frames, utt_pad_proportion=0.02269, over 50.00 utterances.], tot_loss[loss=0.4467, over 2831874.36 frames; utt_duration=1225 frames, utt_pad_proportion=0.06961, over 9259.12 utterances.], batch size: 50, lr: 3.78e-04
|
| 75 |
+
2022-08-11 10:47:17,026 INFO [train.py:737] (2/3) Epoch 13, batch 4700, loss[loss=0.4325, over 14582.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.00594, over 41.00 utterances.], tot_loss[loss=0.444, over 2834421.05 frames; utt_duration=1230 frames, utt_pad_proportion=0.06843, over 9232.04 utterances.], batch size: 41, lr: 3.78e-04
|
| 76 |
+
2022-08-11 10:50:52,812 INFO [train.py:737] (2/3) Epoch 13, batch 4800, loss[loss=0.3935, over 14626.00 frames; utt_duration=1722 frames, utt_pad_proportion=0.007929, over 34.00 utterances.], tot_loss[loss=0.4446, over 2839488.06 frames; utt_duration=1247 frames, utt_pad_proportion=0.06529, over 9116.01 utterances.], batch size: 34, lr: 3.78e-04
|
| 77 |
+
2022-08-11 10:54:02,823 INFO [train.py:737] (2/3) Epoch 13, batch 4900, loss[loss=0.4734, over 14765.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005233, over 42.00 utterances.], tot_loss[loss=0.4438, over 2837057.69 frames; utt_duration=1236 frames, utt_pad_proportion=0.06705, over 9193.66 utterances.], batch size: 42, lr: 3.78e-04
|
| 78 |
+
2022-08-11 10:59:10,774 INFO [train.py:737] (2/3) Epoch 13, batch 5000, loss[loss=0.5726, over 14660.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01178, over 47.00 utterances.], tot_loss[loss=0.4412, over 2849130.21 frames; utt_duration=1256 frames, utt_pad_proportion=0.05817, over 9083.62 utterances.], batch size: 47, lr: 3.78e-04
|
| 79 |
+
2022-08-11 11:02:20,134 INFO [train.py:737] (2/3) Epoch 13, batch 5100, loss[loss=0.4396, over 14483.00 frames; utt_duration=1757 frames, utt_pad_proportion=0.00838, over 33.00 utterances.], tot_loss[loss=0.4447, over 2834928.43 frames; utt_duration=1229 frames, utt_pad_proportion=0.06662, over 9239.50 utterances.], batch size: 33, lr: 3.78e-04
|
| 80 |
+
2022-08-11 11:05:44,511 INFO [train.py:737] (2/3) Epoch 13, batch 5200, loss[loss=0.5038, over 12884.00 frames; utt_duration=770.4 frames, utt_pad_proportion=0.1305, over 67.00 utterances.], tot_loss[loss=0.4429, over 2827622.43 frames; utt_duration=1215 frames, utt_pad_proportion=0.07107, over 9319.08 utterances.], batch size: 67, lr: 3.78e-04
|
| 81 |
+
2022-08-11 11:09:00,835 INFO [train.py:737] (2/3) Epoch 13, batch 5300, loss[loss=0.4994, over 14769.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006519, over 39.00 utterances.], tot_loss[loss=0.4417, over 2821725.23 frames; utt_duration=1203 frames, utt_pad_proportion=0.07518, over 9397.01 utterances.], batch size: 39, lr: 3.77e-04
|
| 82 |
+
2022-08-11 11:14:26,396 INFO [train.py:737] (2/3) Epoch 13, batch 5400, loss[loss=0.4212, over 14013.00 frames; utt_duration=1810 frames, utt_pad_proportion=0.0405, over 31.00 utterances.], tot_loss[loss=0.4441, over 2823572.54 frames; utt_duration=1215 frames, utt_pad_proportion=0.07493, over 9306.46 utterances.], batch size: 31, lr: 3.77e-04
|
| 83 |
+
2022-08-11 11:18:27,396 INFO [train.py:737] (2/3) Epoch 13, batch 5500, loss[loss=0.3908, over 14714.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.01045, over 35.00 utterances.], tot_loss[loss=0.4428, over 2830291.42 frames; utt_duration=1225 frames, utt_pad_proportion=0.06946, over 9250.31 utterances.], batch size: 35, lr: 3.77e-04
|
| 84 |
+
2022-08-11 11:21:56,160 INFO [train.py:737] (2/3) Epoch 13, batch 5600, loss[loss=0.4064, over 14705.00 frames; utt_duration=1682 frames, utt_pad_proportion=0.01102, over 35.00 utterances.], tot_loss[loss=0.4439, over 2815614.06 frames; utt_duration=1197 frames, utt_pad_proportion=0.07728, over 9424.59 utterances.], batch size: 35, lr: 3.77e-04
|
| 85 |
+
2022-08-11 11:24:38,522 INFO [train.py:737] (2/3) Epoch 13, batch 5700, loss[loss=0.3973, over 14769.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006284, over 39.00 utterances.], tot_loss[loss=0.4429, over 2819959.84 frames; utt_duration=1208 frames, utt_pad_proportion=0.07537, over 9346.72 utterances.], batch size: 39, lr: 3.77e-04
|
| 86 |
+
2022-08-11 11:30:07,783 INFO [train.py:737] (2/3) Epoch 13, batch 5800, loss[loss=0.3963, over 14636.00 frames; utt_duration=1723 frames, utt_pad_proportion=0.00784, over 34.00 utterances.], tot_loss[loss=0.4395, over 2815585.06 frames; utt_duration=1204 frames, utt_pad_proportion=0.07831, over 9365.03 utterances.], batch size: 34, lr: 3.77e-04
|
| 87 |
+
2022-08-11 11:33:55,334 INFO [train.py:737] (2/3) Epoch 13, batch 5900, loss[loss=0.4607, over 14686.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.01002, over 45.00 utterances.], tot_loss[loss=0.441, over 2815536.52 frames; utt_duration=1199 frames, utt_pad_proportion=0.07742, over 9408.44 utterances.], batch size: 45, lr: 3.77e-04
|
| 88 |
+
2022-08-11 11:38:37,935 INFO [train.py:737] (2/3) Epoch 13, batch 6000, loss[loss=0.3886, over 14600.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01035, over 34.00 utterances.], tot_loss[loss=0.4412, over 2820119.79 frames; utt_duration=1211 frames, utt_pad_proportion=0.07668, over 9325.05 utterances.], batch size: 34, lr: 3.76e-04
|
| 89 |
+
2022-08-11 11:38:37,935 INFO [train.py:761] (2/3) Computing validation loss
|
| 90 |
+
2022-08-11 11:39:50,682 INFO [train.py:770] (2/3) Epoch 13, validation: loss=0.2858, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 91 |
+
2022-08-11 11:44:40,576 INFO [train.py:737] (2/3) Epoch 13, batch 6100, loss[loss=0.4839, over 9636.00 frames; utt_duration=420.6 frames, utt_pad_proportion=0.353, over 92.00 utterances.], tot_loss[loss=0.4459, over 2750764.14 frames; utt_duration=1095 frames, utt_pad_proportion=0.1149, over 10066.21 utterances.], batch size: 92, lr: 3.76e-04
|
| 92 |
+
2022-08-11 11:46:31,259 INFO [train.py:737] (2/3) Epoch 14, batch 0, loss[loss=0.3735, over 14787.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.005075, over 38.00 utterances.], tot_loss[loss=0.3735, over 14787.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.005075, over 38.00 utterances.], batch size: 38, lr: 3.64e-04
|
| 93 |
+
2022-08-11 11:50:49,318 INFO [train.py:737] (2/3) Epoch 14, batch 100, loss[loss=0.2888, over 14076.00 frames; utt_duration=1818 frames, utt_pad_proportion=0.03519, over 31.00 utterances.], tot_loss[loss=0.3717, over 1148225.94 frames; utt_duration=1314 frames, utt_pad_proportion=0.03375, over 3500.25 utterances.], batch size: 31, lr: 3.64e-04
|
| 94 |
+
2022-08-11 11:54:33,950 INFO [train.py:737] (2/3) Epoch 14, batch 200, loss[loss=0.4329, over 13917.00 frames; utt_duration=978.2 frames, utt_pad_proportion=0.05854, over 57.00 utterances.], tot_loss[loss=0.3676, over 1806125.82 frames; utt_duration=1248 frames, utt_pad_proportion=0.0622, over 5797.30 utterances.], batch size: 57, lr: 3.64e-04
|
| 95 |
+
2022-08-11 11:57:32,567 INFO [train.py:737] (2/3) Epoch 14, batch 300, loss[loss=0.3462, over 14780.00 frames; utt_duration=1600 frames, utt_pad_proportion=0.007013, over 37.00 utterances.], tot_loss[loss=0.3646, over 2218632.14 frames; utt_duration=1251 frames, utt_pad_proportion=0.05764, over 7102.58 utterances.], batch size: 37, lr: 3.64e-04
|
| 96 |
+
2022-08-11 12:02:50,129 INFO [train.py:737] (2/3) Epoch 14, batch 400, loss[loss=0.3266, over 14506.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.01757, over 48.00 utterances.], tot_loss[loss=0.3621, over 2451247.99 frames; utt_duration=1218 frames, utt_pad_proportion=0.06853, over 8060.24 utterances.], batch size: 48, lr: 3.64e-04
|
| 97 |
+
2022-08-11 12:04:25,712 INFO [train.py:737] (2/3) Epoch 14, batch 500, loss[loss=0.3258, over 14579.00 frames; utt_duration=1460 frames, utt_pad_proportion=0.005739, over 40.00 utterances.], tot_loss[loss=0.3589, over 2618064.40 frames; utt_duration=1260 frames, utt_pad_proportion=0.05811, over 8321.03 utterances.], batch size: 40, lr: 3.64e-04
|
| 98 |
+
2022-08-11 12:10:29,635 INFO [train.py:737] (2/3) Epoch 14, batch 600, loss[loss=0.4081, over 14656.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.0129, over 47.00 utterances.], tot_loss[loss=0.362, over 2721513.83 frames; utt_duration=1279 frames, utt_pad_proportion=0.04992, over 8521.20 utterances.], batch size: 47, lr: 3.64e-04
|
| 99 |
+
2022-08-11 12:15:28,027 INFO [train.py:737] (2/3) Epoch 14, batch 700, loss[loss=0.3919, over 14730.00 frames; utt_duration=1371 frames, utt_pad_proportion=0.007606, over 43.00 utterances.], tot_loss[loss=0.3623, over 2790536.27 frames; utt_duration=1309 frames, utt_pad_proportion=0.04099, over 8534.80 utterances.], batch size: 43, lr: 3.63e-04
|
| 100 |
+
2022-08-11 12:17:32,619 INFO [train.py:737] (2/3) Epoch 14, batch 800, loss[loss=0.3715, over 14780.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.006081, over 37.00 utterances.], tot_loss[loss=0.3647, over 2832239.83 frames; utt_duration=1325 frames, utt_pad_proportion=0.03648, over 8558.59 utterances.], batch size: 37, lr: 3.63e-04
|
| 101 |
+
2022-08-11 12:22:44,896 INFO [train.py:737] (2/3) Epoch 14, batch 900, loss[loss=0.3119, over 14616.00 frames; utt_duration=1625 frames, utt_pad_proportion=0.006605, over 36.00 utterances.], tot_loss[loss=0.3636, over 2850037.58 frames; utt_duration=1320 frames, utt_pad_proportion=0.0379, over 8649.05 utterances.], batch size: 36, lr: 3.63e-04
|
| 102 |
+
2022-08-11 12:28:18,724 INFO [train.py:737] (2/3) Epoch 14, batch 1000, loss[loss=0.2756, over 14605.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.01053, over 34.00 utterances.], tot_loss[loss=0.3626, over 2862725.29 frames; utt_duration=1321 frames, utt_pad_proportion=0.03852, over 8681.28 utterances.], batch size: 34, lr: 3.63e-04
|
| 103 |
+
2022-08-11 12:32:01,208 INFO [train.py:737] (2/3) Epoch 14, batch 1100, loss[loss=0.3499, over 14723.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.009361, over 35.00 utterances.], tot_loss[loss=0.3631, over 2873629.97 frames; utt_duration=1329 frames, utt_pad_proportion=0.03558, over 8661.97 utterances.], batch size: 35, lr: 3.63e-04
|
| 104 |
+
2022-08-11 12:33:29,130 INFO [train.py:737] (2/3) Epoch 14, batch 1200, loss[loss=0.3663, over 14762.00 frames; utt_duration=1375 frames, utt_pad_proportion=0.005385, over 43.00 utterances.], tot_loss[loss=0.3641, over 2866131.56 frames; utt_duration=1295 frames, utt_pad_proportion=0.04409, over 8860.14 utterances.], batch size: 43, lr: 3.63e-04
|
| 105 |
+
2022-08-11 12:34:33,001 INFO [train.py:737] (2/3) Epoch 14, batch 1300, loss[loss=0.3583, over 14587.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.005331, over 42.00 utterances.], tot_loss[loss=0.3649, over 2862748.40 frames; utt_duration=1285 frames, utt_pad_proportion=0.04632, over 8924.68 utterances.], batch size: 42, lr: 3.63e-04
|
| 106 |
+
2022-08-11 12:39:36,346 INFO [train.py:737] (2/3) Epoch 14, batch 1400, loss[loss=0.3266, over 14598.00 frames; utt_duration=1623 frames, utt_pad_proportion=0.007895, over 36.00 utterances.], tot_loss[loss=0.3643, over 2859970.27 frames; utt_duration=1272 frames, utt_pad_proportion=0.04935, over 9006.07 utterances.], batch size: 36, lr: 3.63e-04
|
| 107 |
+
2022-08-11 12:45:34,371 INFO [train.py:737] (2/3) Epoch 14, batch 1500, loss[loss=0.3984, over 14605.00 frames; utt_duration=1329 frames, utt_pad_proportion=0.01032, over 44.00 utterances.], tot_loss[loss=0.3629, over 2864609.64 frames; utt_duration=1291 frames, utt_pad_proportion=0.04769, over 8883.80 utterances.], batch size: 44, lr: 3.62e-04
|
| 108 |
+
2022-08-11 12:50:14,993 INFO [train.py:737] (2/3) Epoch 14, batch 1600, loss[loss=0.3725, over 14604.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.009505, over 34.00 utterances.], tot_loss[loss=0.3615, over 2868041.40 frames; utt_duration=1308 frames, utt_pad_proportion=0.04517, over 8783.14 utterances.], batch size: 34, lr: 3.62e-04
|
| 109 |
+
2022-08-11 12:54:22,868 INFO [train.py:737] (2/3) Epoch 14, batch 1700, loss[loss=0.3604, over 14769.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006284, over 39.00 utterances.], tot_loss[loss=0.3621, over 2881167.59 frames; utt_duration=1327 frames, utt_pad_proportion=0.03715, over 8696.79 utterances.], batch size: 39, lr: 3.62e-04
|
| 110 |
+
2022-08-11 12:58:59,135 INFO [train.py:737] (2/3) Epoch 14, batch 1800, loss[loss=0.4112, over 14701.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01143, over 46.00 utterances.], tot_loss[loss=0.3649, over 2855095.92 frames; utt_duration=1275 frames, utt_pad_proportion=0.0561, over 8970.31 utterances.], batch size: 46, lr: 3.62e-04
|
| 111 |
+
2022-08-11 13:03:45,764 INFO [train.py:737] (2/3) Epoch 14, batch 1900, loss[loss=0.3309, over 14740.00 frames; utt_duration=1686 frames, utt_pad_proportion=0.009434, over 35.00 utterances.], tot_loss[loss=0.3647, over 2850175.33 frames; utt_duration=1252 frames, utt_pad_proportion=0.05923, over 9120.02 utterances.], batch size: 35, lr: 3.62e-04
|
| 112 |
+
2022-08-11 13:08:02,545 INFO [train.py:737] (2/3) Epoch 14, batch 2000, loss[loss=0.3744, over 12841.00 frames; utt_duration=768.3 frames, utt_pad_proportion=0.1309, over 67.00 utterances.], tot_loss[loss=0.3659, over 2849122.04 frames; utt_duration=1248 frames, utt_pad_proportion=0.0596, over 9145.73 utterances.], batch size: 67, lr: 3.62e-04
|
| 113 |
+
2022-08-11 13:11:18,009 INFO [train.py:737] (2/3) Epoch 14, batch 2100, loss[loss=0.3565, over 13996.00 frames; utt_duration=1807 frames, utt_pad_proportion=0.03862, over 31.00 utterances.], tot_loss[loss=0.3652, over 2855991.81 frames; utt_duration=1263 frames, utt_pad_proportion=0.0541, over 9054.27 utterances.], batch size: 31, lr: 3.62e-04
|
| 114 |
+
2022-08-11 13:17:21,265 INFO [train.py:737] (2/3) Epoch 14, batch 2200, loss[loss=0.3358, over 14586.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.004569, over 39.00 utterances.], tot_loss[loss=0.3676, over 2848302.70 frames; utt_duration=1245 frames, utt_pad_proportion=0.05904, over 9163.69 utterances.], batch size: 39, lr: 3.62e-04
|
| 115 |
+
2022-08-11 13:22:13,817 INFO [train.py:737] (2/3) Epoch 14, batch 2300, loss[loss=0.3684, over 14740.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.006765, over 43.00 utterances.], tot_loss[loss=0.3663, over 2845362.96 frames; utt_duration=1249 frames, utt_pad_proportion=0.05978, over 9122.70 utterances.], batch size: 43, lr: 3.61e-04
|
| 116 |
+
2022-08-11 13:27:39,730 INFO [train.py:737] (2/3) Epoch 14, batch 2400, loss[loss=0.3458, over 14755.00 frames; utt_duration=1515 frames, utt_pad_proportion=0.007208, over 39.00 utterances.], tot_loss[loss=0.3656, over 2860510.72 frames; utt_duration=1278 frames, utt_pad_proportion=0.04963, over 8964.86 utterances.], batch size: 39, lr: 3.61e-04
|
| 117 |
+
2022-08-11 13:32:15,983 INFO [train.py:737] (2/3) Epoch 14, batch 2500, loss[loss=0.4086, over 10046.00 frames; utt_duration=438 frames, utt_pad_proportion=0.3272, over 92.00 utterances.], tot_loss[loss=0.3644, over 2859639.39 frames; utt_duration=1272 frames, utt_pad_proportion=0.05064, over 9006.00 utterances.], batch size: 92, lr: 3.61e-04
|
| 118 |
+
2022-08-11 13:38:21,581 INFO [train.py:737] (2/3) Epoch 14, batch 2600, loss[loss=0.2845, over 14504.00 frames; utt_duration=1759 frames, utt_pad_proportion=0.008216, over 33.00 utterances.], tot_loss[loss=0.361, over 2852532.27 frames; utt_duration=1269 frames, utt_pad_proportion=0.0552, over 9002.19 utterances.], batch size: 33, lr: 3.61e-04
|
| 119 |
+
2022-08-11 13:45:02,851 INFO [train.py:737] (2/3) Epoch 14, batch 2700, loss[loss=0.3605, over 14586.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.00582, over 42.00 utterances.], tot_loss[loss=0.3596, over 2860799.28 frames; utt_duration=1287 frames, utt_pad_proportion=0.0508, over 8900.28 utterances.], batch size: 42, lr: 3.61e-04
|
| 120 |
+
2022-08-11 13:50:17,637 INFO [train.py:737] (2/3) Epoch 14, batch 2800, loss[loss=0.4018, over 14710.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.01033, over 46.00 utterances.], tot_loss[loss=0.3603, over 2863286.19 frames; utt_duration=1296 frames, utt_pad_proportion=0.04925, over 8846.82 utterances.], batch size: 46, lr: 3.61e-04
|
| 121 |
+
2022-08-11 13:56:31,409 INFO [train.py:737] (2/3) Epoch 14, batch 2900, loss[loss=0.4254, over 14681.00 frames; utt_duration=1278 frames, utt_pad_proportion=0.01228, over 46.00 utterances.], tot_loss[loss=0.3765, over 2854742.87 frames; utt_duration=1273 frames, utt_pad_proportion=0.05493, over 8980.78 utterances.], batch size: 46, lr: 3.61e-04
|
| 122 |
+
2022-08-11 14:01:08,225 INFO [train.py:737] (2/3) Epoch 14, batch 3000, loss[loss=0.3678, over 14801.00 frames; utt_duration=1601 frames, utt_pad_proportion=0.005305, over 37.00 utterances.], tot_loss[loss=0.3955, over 2862651.75 frames; utt_duration=1272 frames, utt_pad_proportion=0.04947, over 9011.86 utterances.], batch size: 37, lr: 3.61e-04
|
| 123 |
+
2022-08-11 14:01:08,225 INFO [train.py:761] (2/3) Computing validation loss
|
| 124 |
+
2022-08-11 14:03:06,562 INFO [train.py:770] (2/3) Epoch 14, validation: loss=0.2899, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 125 |
+
2022-08-11 14:07:35,706 INFO [train.py:737] (2/3) Epoch 14, batch 3100, loss[loss=0.3733, over 14438.00 frames; utt_duration=1752 frames, utt_pad_proportion=0.01092, over 33.00 utterances.], tot_loss[loss=0.4116, over 2854391.15 frames; utt_duration=1260 frames, utt_pad_proportion=0.05615, over 9073.17 utterances.], batch size: 33, lr: 3.60e-04
|
| 126 |
+
2022-08-11 14:14:22,388 INFO [train.py:737] (2/3) Epoch 14, batch 3200, loss[loss=0.4351, over 14862.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.00747, over 44.00 utterances.], tot_loss[loss=0.4228, over 2835469.74 frames; utt_duration=1216 frames, utt_pad_proportion=0.06754, over 9338.03 utterances.], batch size: 44, lr: 3.60e-04
|
| 127 |
+
2022-08-11 14:20:07,872 INFO [train.py:737] (2/3) Epoch 14, batch 3300, loss[loss=0.4173, over 14468.00 frames; utt_duration=1159 frames, utt_pad_proportion=0.02448, over 50.00 utterances.], tot_loss[loss=0.43, over 2837013.48 frames; utt_duration=1222 frames, utt_pad_proportion=0.06728, over 9298.49 utterances.], batch size: 50, lr: 3.60e-04
|
| 128 |
+
2022-08-11 14:23:56,192 INFO [train.py:737] (2/3) Epoch 14, batch 3400, loss[loss=0.4329, over 14707.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.01142, over 35.00 utterances.], tot_loss[loss=0.4357, over 2831733.04 frames; utt_duration=1218 frames, utt_pad_proportion=0.07017, over 9309.85 utterances.], batch size: 35, lr: 3.60e-04
|
| 129 |
+
2022-08-11 14:26:47,077 INFO [train.py:737] (2/3) Epoch 14, batch 3500, loss[loss=0.4501, over 14447.00 frames; utt_duration=1157 frames, utt_pad_proportion=0.02581, over 50.00 utterances.], tot_loss[loss=0.4366, over 2840762.51 frames; utt_duration=1235 frames, utt_pad_proportion=0.06339, over 9214.10 utterances.], batch size: 50, lr: 3.60e-04
|
| 130 |
+
2022-08-11 14:29:40,025 INFO [train.py:737] (2/3) Epoch 14, batch 3600, loss[loss=0.4312, over 14691.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.008946, over 45.00 utterances.], tot_loss[loss=0.4454, over 2806258.73 frames; utt_duration=1163 frames, utt_pad_proportion=0.08255, over 9666.35 utterances.], batch size: 45, lr: 3.60e-04
|
| 131 |
+
2022-08-11 14:32:26,119 INFO [train.py:737] (2/3) Epoch 14, batch 3700, loss[loss=0.385, over 14095.00 frames; utt_duration=1820 frames, utt_pad_proportion=0.03284, over 31.00 utterances.], tot_loss[loss=0.4443, over 2824385.79 frames; utt_duration=1198 frames, utt_pad_proportion=0.0716, over 9439.58 utterances.], batch size: 31, lr: 3.60e-04
|
| 132 |
+
2022-08-11 14:35:07,280 INFO [train.py:737] (2/3) Epoch 14, batch 3800, loss[loss=0.4115, over 14452.00 frames; utt_duration=1753 frames, utt_pad_proportion=0.00845, over 33.00 utterances.], tot_loss[loss=0.4463, over 2820713.99 frames; utt_duration=1196 frames, utt_pad_proportion=0.07446, over 9447.36 utterances.], batch size: 33, lr: 3.60e-04
|
| 133 |
+
2022-08-11 14:37:05,346 INFO [train.py:737] (2/3) Epoch 14, batch 3900, loss[loss=0.4792, over 14588.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.005769, over 42.00 utterances.], tot_loss[loss=0.4456, over 2808842.56 frames; utt_duration=1174 frames, utt_pad_proportion=0.08154, over 9580.54 utterances.], batch size: 42, lr: 3.59e-04
|
| 134 |
+
2022-08-11 14:39:09,208 INFO [train.py:737] (2/3) Epoch 14, batch 4000, loss[loss=0.5277, over 14500.00 frames; utt_duration=1162 frames, utt_pad_proportion=0.02231, over 50.00 utterances.], tot_loss[loss=0.4431, over 2827938.46 frames; utt_duration=1207 frames, utt_pad_proportion=0.07152, over 9380.14 utterances.], batch size: 50, lr: 3.59e-04
|
| 135 |
+
2022-08-11 14:41:42,024 INFO [train.py:737] (2/3) Epoch 14, batch 4100, loss[loss=0.486, over 12742.00 frames; utt_duration=762.3 frames, utt_pad_proportion=0.1406, over 67.00 utterances.], tot_loss[loss=0.4437, over 2832376.55 frames; utt_duration=1223 frames, utt_pad_proportion=0.06926, over 9274.67 utterances.], batch size: 67, lr: 3.59e-04
|
| 136 |
+
2022-08-11 14:44:27,290 INFO [train.py:737] (2/3) Epoch 14, batch 4200, loss[loss=0.4346, over 14314.00 frames; utt_duration=1082 frames, utt_pad_proportion=0.03831, over 53.00 utterances.], tot_loss[loss=0.4432, over 2835220.12 frames; utt_duration=1225 frames, utt_pad_proportion=0.06728, over 9269.49 utterances.], batch size: 53, lr: 3.59e-04
|
| 137 |
+
2022-08-11 14:46:59,732 INFO [train.py:737] (2/3) Epoch 14, batch 4300, loss[loss=0.3785, over 14605.00 frames; utt_duration=1539 frames, utt_pad_proportion=0.005018, over 38.00 utterances.], tot_loss[loss=0.4401, over 2840204.27 frames; utt_duration=1242 frames, utt_pad_proportion=0.06424, over 9157.52 utterances.], batch size: 38, lr: 3.59e-04
|
| 138 |
+
2022-08-11 14:49:20,107 INFO [train.py:737] (2/3) Epoch 14, batch 4400, loss[loss=0.4356, over 14778.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.005646, over 38.00 utterances.], tot_loss[loss=0.4399, over 2850584.95 frames; utt_duration=1261 frames, utt_pad_proportion=0.05715, over 9056.56 utterances.], batch size: 38, lr: 3.59e-04
|
| 139 |
+
2022-08-11 14:51:53,408 INFO [train.py:737] (2/3) Epoch 14, batch 4500, loss[loss=0.4358, over 14595.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.005582, over 42.00 utterances.], tot_loss[loss=0.4442, over 2831017.75 frames; utt_duration=1223 frames, utt_pad_proportion=0.06956, over 9270.22 utterances.], batch size: 42, lr: 3.59e-04
|
| 140 |
+
2022-08-11 14:54:39,322 INFO [train.py:737] (2/3) Epoch 14, batch 4600, loss[loss=0.4219, over 14875.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.00687, over 44.00 utterances.], tot_loss[loss=0.4466, over 2833825.94 frames; utt_duration=1220 frames, utt_pad_proportion=0.06809, over 9302.14 utterances.], batch size: 44, lr: 3.59e-04
|
| 141 |
+
2022-08-11 14:56:04,614 INFO [train.py:737] (2/3) Epoch 14, batch 4700, loss[loss=0.4535, over 14758.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.005688, over 43.00 utterances.], tot_loss[loss=0.4434, over 2800521.47 frames; utt_duration=1168 frames, utt_pad_proportion=0.08739, over 9601.67 utterances.], batch size: 43, lr: 3.58e-04
|
| 142 |
+
2022-08-11 15:00:58,231 INFO [train.py:737] (2/3) Epoch 14, batch 4800, loss[loss=0.3449, over 14476.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.01008, over 33.00 utterances.], tot_loss[loss=0.4419, over 2807507.35 frames; utt_duration=1183 frames, utt_pad_proportion=0.08255, over 9504.04 utterances.], batch size: 33, lr: 3.58e-04
|
| 143 |
+
2022-08-11 15:05:17,213 INFO [train.py:737] (2/3) Epoch 14, batch 4900, loss[loss=0.4007, over 14591.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.008337, over 36.00 utterances.], tot_loss[loss=0.4427, over 2827852.69 frames; utt_duration=1211 frames, utt_pad_proportion=0.07126, over 9351.77 utterances.], batch size: 36, lr: 3.58e-04
|
| 144 |
+
2022-08-11 15:10:37,123 INFO [train.py:737] (2/3) Epoch 14, batch 5000, loss[loss=0.4389, over 14568.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.00593, over 37.00 utterances.], tot_loss[loss=0.4432, over 2837307.42 frames; utt_duration=1223 frames, utt_pad_proportion=0.06503, over 9294.76 utterances.], batch size: 37, lr: 3.58e-04
|
| 145 |
+
2022-08-11 15:16:39,153 INFO [train.py:737] (2/3) Epoch 14, batch 5100, loss[loss=0.4299, over 14578.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.006163, over 39.00 utterances.], tot_loss[loss=0.4444, over 2819189.94 frames; utt_duration=1193 frames, utt_pad_proportion=0.07611, over 9461.50 utterances.], batch size: 39, lr: 3.58e-04
|
| 146 |
+
2022-08-11 15:19:11,608 INFO [train.py:737] (2/3) Epoch 14, batch 5200, loss[loss=0.3703, over 14610.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.008561, over 34.00 utterances.], tot_loss[loss=0.4434, over 2816322.17 frames; utt_duration=1195 frames, utt_pad_proportion=0.07796, over 9436.09 utterances.], batch size: 34, lr: 3.58e-04
|
| 147 |
+
2022-08-11 15:22:03,091 INFO [train.py:737] (2/3) Epoch 14, batch 5300, loss[loss=0.5026, over 14320.00 frames; utt_duration=1082 frames, utt_pad_proportion=0.03874, over 53.00 utterances.], tot_loss[loss=0.4414, over 2827685.61 frames; utt_duration=1216 frames, utt_pad_proportion=0.0714, over 9311.49 utterances.], batch size: 53, lr: 3.58e-04
|
| 148 |
+
2022-08-11 15:24:31,018 INFO [train.py:737] (2/3) Epoch 14, batch 5400, loss[loss=0.4804, over 9885.00 frames; utt_duration=431.3 frames, utt_pad_proportion=0.3375, over 92.00 utterances.], tot_loss[loss=0.4451, over 2824167.04 frames; utt_duration=1203 frames, utt_pad_proportion=0.07345, over 9399.89 utterances.], batch size: 92, lr: 3.58e-04
|
| 149 |
+
2022-08-11 15:26:31,329 INFO [train.py:737] (2/3) Epoch 14, batch 5500, loss[loss=0.4509, over 13566.00 frames; utt_duration=953.8 frames, utt_pad_proportion=0.08201, over 57.00 utterances.], tot_loss[loss=0.4458, over 2827325.59 frames; utt_duration=1211 frames, utt_pad_proportion=0.07174, over 9353.55 utterances.], batch size: 57, lr: 3.57e-04
|
| 150 |
+
2022-08-11 15:29:25,757 INFO [train.py:737] (2/3) Epoch 14, batch 5600, loss[loss=0.463, over 14746.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.005978, over 43.00 utterances.], tot_loss[loss=0.442, over 2838706.66 frames; utt_duration=1234 frames, utt_pad_proportion=0.06526, over 9214.29 utterances.], batch size: 43, lr: 3.57e-04
|
| 151 |
+
2022-08-11 15:32:05,014 INFO [train.py:737] (2/3) Epoch 14, batch 5700, loss[loss=0.4328, over 14670.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.01014, over 45.00 utterances.], tot_loss[loss=0.4413, over 2847082.07 frames; utt_duration=1249 frames, utt_pad_proportion=0.06044, over 9129.62 utterances.], batch size: 45, lr: 3.57e-04
|
| 152 |
+
2022-08-11 15:34:48,481 INFO [train.py:737] (2/3) Epoch 14, batch 5800, loss[loss=0.4394, over 14623.00 frames; utt_duration=1246 frames, utt_pad_proportion=0.01499, over 47.00 utterances.], tot_loss[loss=0.44, over 2847285.35 frames; utt_duration=1250 frames, utt_pad_proportion=0.06083, over 9119.85 utterances.], batch size: 47, lr: 3.57e-04
|
| 153 |
+
2022-08-11 15:37:21,099 INFO [train.py:737] (2/3) Epoch 14, batch 5900, loss[loss=0.4175, over 14585.00 frames; utt_duration=1460 frames, utt_pad_proportion=0.004635, over 40.00 utterances.], tot_loss[loss=0.4369, over 2840067.21 frames; utt_duration=1236 frames, utt_pad_proportion=0.06523, over 9203.58 utterances.], batch size: 40, lr: 3.57e-04
|
| 154 |
+
2022-08-11 15:39:22,179 INFO [train.py:737] (2/3) Epoch 14, batch 6000, loss[loss=0.4347, over 14885.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.006998, over 44.00 utterances.], tot_loss[loss=0.4414, over 2840431.57 frames; utt_duration=1232 frames, utt_pad_proportion=0.06428, over 9229.98 utterances.], batch size: 44, lr: 3.57e-04
|
| 155 |
+
2022-08-11 15:39:22,180 INFO [train.py:761] (2/3) Computing validation loss
|
| 156 |
+
2022-08-11 15:40:05,206 INFO [train.py:770] (2/3) Epoch 14, validation: loss=0.301, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 157 |
+
2022-08-11 15:42:37,944 INFO [train.py:737] (2/3) Epoch 14, batch 6100, loss[loss=0.5063, over 9320.00 frames; utt_duration=406.8 frames, utt_pad_proportion=0.3751, over 92.00 utterances.], tot_loss[loss=0.4469, over 2762251.71 frames; utt_duration=1099 frames, utt_pad_proportion=0.1076, over 10071.10 utterances.], batch size: 92, lr: 3.57e-04
|
| 158 |
+
2022-08-11 15:44:19,634 INFO [train.py:737] (2/3) Epoch 15, batch 0, loss[loss=0.3618, over 14552.00 frames; utt_duration=1575 frames, utt_pad_proportion=0.007834, over 37.00 utterances.], tot_loss[loss=0.3618, over 14552.00 frames; utt_duration=1575 frames, utt_pad_proportion=0.007834, over 37.00 utterances.], batch size: 37, lr: 3.46e-04
|
| 159 |
+
2022-08-11 15:47:34,442 INFO [train.py:737] (2/3) Epoch 15, batch 100, loss[loss=0.3721, over 14484.00 frames; utt_duration=1209 frames, utt_pad_proportion=0.01826, over 48.00 utterances.], tot_loss[loss=0.3718, over 1133022.53 frames; utt_duration=1264 frames, utt_pad_proportion=0.05886, over 3590.34 utterances.], batch size: 48, lr: 3.46e-04
|
| 160 |
+
2022-08-11 15:50:08,432 INFO [train.py:737] (2/3) Epoch 15, batch 200, loss[loss=0.3321, over 14711.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.01062, over 35.00 utterances.], tot_loss[loss=0.3612, over 1815205.07 frames; utt_duration=1285 frames, utt_pad_proportion=0.05474, over 5654.91 utterances.], batch size: 35, lr: 3.46e-04
|
| 161 |
+
2022-08-11 15:53:20,627 INFO [train.py:737] (2/3) Epoch 15, batch 300, loss[loss=0.3209, over 14746.00 frames; utt_duration=1687 frames, utt_pad_proportion=0.008432, over 35.00 utterances.], tot_loss[loss=0.3604, over 2228648.20 frames; utt_duration=1281 frames, utt_pad_proportion=0.05477, over 6966.23 utterances.], batch size: 35, lr: 3.45e-04
|
| 162 |
+
2022-08-11 15:55:13,360 INFO [train.py:737] (2/3) Epoch 15, batch 400, loss[loss=0.3143, over 14456.00 frames; utt_duration=1654 frames, utt_pad_proportion=0.007764, over 35.00 utterances.], tot_loss[loss=0.3584, over 2495282.79 frames; utt_duration=1333 frames, utt_pad_proportion=0.04012, over 7494.78 utterances.], batch size: 35, lr: 3.45e-04
|
| 163 |
+
2022-08-11 15:57:32,262 INFO [train.py:737] (2/3) Epoch 15, batch 500, loss[loss=0.4116, over 14738.00 frames; utt_duration=1405 frames, utt_pad_proportion=0.006832, over 42.00 utterances.], tot_loss[loss=0.3578, over 2626642.43 frames; utt_duration=1280 frames, utt_pad_proportion=0.05436, over 8220.23 utterances.], batch size: 42, lr: 3.45e-04
|
| 164 |
+
2022-08-11 16:00:45,359 INFO [train.py:737] (2/3) Epoch 15, batch 600, loss[loss=0.3598, over 14077.00 frames; utt_duration=1817 frames, utt_pad_proportion=0.03123, over 31.00 utterances.], tot_loss[loss=0.3595, over 2731338.07 frames; utt_duration=1304 frames, utt_pad_proportion=0.04458, over 8387.54 utterances.], batch size: 31, lr: 3.45e-04
|
| 165 |
+
2022-08-11 16:02:53,910 INFO [train.py:737] (2/3) Epoch 15, batch 700, loss[loss=0.2835, over 14717.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.01036, over 35.00 utterances.], tot_loss[loss=0.3602, over 2781855.96 frames; utt_duration=1297 frames, utt_pad_proportion=0.04609, over 8592.13 utterances.], batch size: 35, lr: 3.45e-04
|
| 166 |
+
2022-08-11 16:05:33,898 INFO [train.py:737] (2/3) Epoch 15, batch 800, loss[loss=0.3614, over 14624.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.008952, over 44.00 utterances.], tot_loss[loss=0.3621, over 2821921.29 frames; utt_duration=1306 frames, utt_pad_proportion=0.04192, over 8649.69 utterances.], batch size: 44, lr: 3.45e-04
|
| 167 |
+
2022-08-11 16:07:53,555 INFO [train.py:737] (2/3) Epoch 15, batch 900, loss[loss=0.3668, over 14496.00 frames; utt_duration=1161 frames, utt_pad_proportion=0.02184, over 50.00 utterances.], tot_loss[loss=0.3634, over 2845810.30 frames; utt_duration=1314 frames, utt_pad_proportion=0.03912, over 8672.17 utterances.], batch size: 50, lr: 3.45e-04
|
| 168 |
+
2022-08-11 16:10:29,684 INFO [train.py:737] (2/3) Epoch 15, batch 1000, loss[loss=0.355, over 14776.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.007549, over 37.00 utterances.], tot_loss[loss=0.3605, over 2850710.72 frames; utt_duration=1305 frames, utt_pad_proportion=0.04329, over 8744.96 utterances.], batch size: 37, lr: 3.45e-04
|
| 169 |
+
2022-08-11 16:11:56,570 INFO [train.py:737] (2/3) Epoch 15, batch 1100, loss[loss=0.3141, over 14471.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.01039, over 33.00 utterances.], tot_loss[loss=0.3647, over 2842477.56 frames; utt_duration=1268 frames, utt_pad_proportion=0.05435, over 8975.48 utterances.], batch size: 33, lr: 3.45e-04
|
| 170 |
+
2022-08-11 16:13:12,426 INFO [train.py:737] (2/3) Epoch 15, batch 1200, loss[loss=0.3732, over 14751.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.005372, over 43.00 utterances.], tot_loss[loss=0.3654, over 2844804.03 frames; utt_duration=1263 frames, utt_pad_proportion=0.05628, over 9018.76 utterances.], batch size: 43, lr: 3.44e-04
|
| 171 |
+
2022-08-11 16:16:21,711 INFO [train.py:737] (2/3) Epoch 15, batch 1300, loss[loss=0.4338, over 14745.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.006127, over 41.00 utterances.], tot_loss[loss=0.3668, over 2846713.93 frames; utt_duration=1256 frames, utt_pad_proportion=0.0567, over 9079.35 utterances.], batch size: 41, lr: 3.44e-04
|
| 172 |
+
2022-08-11 16:19:25,057 INFO [train.py:737] (2/3) Epoch 15, batch 1400, loss[loss=0.3435, over 14742.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.006664, over 43.00 utterances.], tot_loss[loss=0.3609, over 2873632.59 frames; utt_duration=1324 frames, utt_pad_proportion=0.04161, over 8689.05 utterances.], batch size: 43, lr: 3.44e-04
|
| 173 |
+
2022-08-11 16:22:02,436 INFO [train.py:737] (2/3) Epoch 15, batch 1500, loss[loss=0.3589, over 14481.00 frames; utt_duration=1160 frames, utt_pad_proportion=0.02354, over 50.00 utterances.], tot_loss[loss=0.3609, over 2873374.15 frames; utt_duration=1318 frames, utt_pad_proportion=0.04188, over 8730.76 utterances.], batch size: 50, lr: 3.44e-04
|
| 174 |
+
2022-08-11 16:25:07,902 INFO [train.py:737] (2/3) Epoch 15, batch 1600, loss[loss=0.3475, over 14564.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006033, over 40.00 utterances.], tot_loss[loss=0.363, over 2876414.65 frames; utt_duration=1318 frames, utt_pad_proportion=0.03945, over 8741.46 utterances.], batch size: 40, lr: 3.44e-04
|
| 175 |
+
2022-08-11 16:27:01,396 INFO [train.py:737] (2/3) Epoch 15, batch 1700, loss[loss=0.4145, over 13084.00 frames; utt_duration=782.7 frames, utt_pad_proportion=0.1146, over 67.00 utterances.], tot_loss[loss=0.3602, over 2871789.15 frames; utt_duration=1302 frames, utt_pad_proportion=0.04431, over 8835.05 utterances.], batch size: 67, lr: 3.44e-04
|
| 176 |
+
2022-08-11 16:28:51,801 INFO [train.py:737] (2/3) Epoch 15, batch 1800, loss[loss=0.336, over 14582.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.00628, over 42.00 utterances.], tot_loss[loss=0.359, over 2885627.36 frames; utt_duration=1342 frames, utt_pad_proportion=0.0351, over 8613.55 utterances.], batch size: 42, lr: 3.44e-04
|
| 177 |
+
2022-08-11 16:31:57,526 INFO [train.py:737] (2/3) Epoch 15, batch 1900, loss[loss=0.3828, over 13724.00 frames; utt_duration=964.7 frames, utt_pad_proportion=0.07154, over 57.00 utterances.], tot_loss[loss=0.3594, over 2884288.00 frames; utt_duration=1338 frames, utt_pad_proportion=0.03687, over 8633.18 utterances.], batch size: 57, lr: 3.44e-04
|
| 178 |
+
2022-08-11 16:33:53,035 INFO [train.py:737] (2/3) Epoch 15, batch 2000, loss[loss=0.3158, over 14710.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.008317, over 35.00 utterances.], tot_loss[loss=0.3596, over 2867643.23 frames; utt_duration=1307 frames, utt_pad_proportion=0.04726, over 8784.68 utterances.], batch size: 35, lr: 3.44e-04
|
| 179 |
+
2022-08-11 16:36:46,008 INFO [train.py:737] (2/3) Epoch 15, batch 2100, loss[loss=0.3332, over 14557.00 frames; utt_duration=1457 frames, utt_pad_proportion=0.007238, over 40.00 utterances.], tot_loss[loss=0.361, over 2862375.47 frames; utt_duration=1287 frames, utt_pad_proportion=0.05159, over 8908.80 utterances.], batch size: 40, lr: 3.43e-04
|
| 180 |
+
2022-08-11 16:39:59,181 INFO [train.py:737] (2/3) Epoch 15, batch 2200, loss[loss=0.345, over 14774.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.0062, over 39.00 utterances.], tot_loss[loss=0.3598, over 2873405.60 frames; utt_duration=1312 frames, utt_pad_proportion=0.04403, over 8769.03 utterances.], batch size: 39, lr: 3.43e-04
|
| 181 |
+
2022-08-11 16:42:56,526 INFO [train.py:737] (2/3) Epoch 15, batch 2300, loss[loss=0.3834, over 14683.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.0103, over 45.00 utterances.], tot_loss[loss=0.3603, over 2868742.80 frames; utt_duration=1301 frames, utt_pad_proportion=0.04663, over 8832.74 utterances.], batch size: 45, lr: 3.43e-04
|
| 182 |
+
2022-08-11 16:46:01,610 INFO [train.py:737] (2/3) Epoch 15, batch 2400, loss[loss=0.3413, over 14581.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.00601, over 39.00 utterances.], tot_loss[loss=0.3608, over 2867897.00 frames; utt_duration=1291 frames, utt_pad_proportion=0.04707, over 8894.40 utterances.], batch size: 39, lr: 3.43e-04
|
| 183 |
+
2022-08-11 16:48:07,641 INFO [train.py:737] (2/3) Epoch 15, batch 2500, loss[loss=0.316, over 14759.00 frames; utt_duration=1515 frames, utt_pad_proportion=0.007091, over 39.00 utterances.], tot_loss[loss=0.3591, over 2868532.91 frames; utt_duration=1286 frames, utt_pad_proportion=0.04647, over 8935.25 utterances.], batch size: 39, lr: 3.43e-04
|
| 184 |
+
2022-08-11 16:50:38,868 INFO [train.py:737] (2/3) Epoch 15, batch 2600, loss[loss=0.3316, over 14864.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.00762, over 44.00 utterances.], tot_loss[loss=0.36, over 2866378.46 frames; utt_duration=1291 frames, utt_pad_proportion=0.04818, over 8894.27 utterances.], batch size: 44, lr: 3.43e-04
|
| 185 |
+
2022-08-11 16:53:31,180 INFO [train.py:737] (2/3) Epoch 15, batch 2700, loss[loss=0.2792, over 14429.00 frames; utt_duration=1751 frames, utt_pad_proportion=0.01322, over 33.00 utterances.], tot_loss[loss=0.3613, over 2874019.14 frames; utt_duration=1304 frames, utt_pad_proportion=0.04458, over 8828.77 utterances.], batch size: 33, lr: 3.43e-04
|
| 186 |
+
2022-08-11 16:56:22,279 INFO [train.py:737] (2/3) Epoch 15, batch 2800, loss[loss=0.3974, over 14683.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.01005, over 45.00 utterances.], tot_loss[loss=0.3626, over 2875394.97 frames; utt_duration=1307 frames, utt_pad_proportion=0.04304, over 8812.84 utterances.], batch size: 45, lr: 3.43e-04
|
| 187 |
+
2022-08-11 16:59:23,577 INFO [train.py:737] (2/3) Epoch 15, batch 2900, loss[loss=0.3791, over 14776.00 frames; utt_duration=1409 frames, utt_pad_proportion=0.005112, over 42.00 utterances.], tot_loss[loss=0.3777, over 2875359.12 frames; utt_duration=1301 frames, utt_pad_proportion=0.04248, over 8851.06 utterances.], batch size: 42, lr: 3.43e-04
|
| 188 |
+
2022-08-11 17:02:14,380 INFO [train.py:737] (2/3) Epoch 15, batch 3000, loss[loss=0.4443, over 13785.00 frames; utt_duration=968.8 frames, utt_pad_proportion=0.06758, over 57.00 utterances.], tot_loss[loss=0.3941, over 2868963.75 frames; utt_duration=1284 frames, utt_pad_proportion=0.0459, over 8951.40 utterances.], batch size: 57, lr: 3.42e-04
|
| 189 |
+
2022-08-11 17:02:14,380 INFO [train.py:761] (2/3) Computing validation loss
|
| 190 |
+
2022-08-11 17:02:58,433 INFO [train.py:770] (2/3) Epoch 15, validation: loss=0.2831, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 191 |
+
2022-08-11 17:05:29,808 INFO [train.py:737] (2/3) Epoch 15, batch 3100, loss[loss=0.3612, over 14604.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.007654, over 34.00 utterances.], tot_loss[loss=0.4075, over 2860067.71 frames; utt_duration=1280 frames, utt_pad_proportion=0.05096, over 8949.03 utterances.], batch size: 34, lr: 3.42e-04
|
| 192 |
+
2022-08-11 17:08:07,073 INFO [train.py:737] (2/3) Epoch 15, batch 3200, loss[loss=0.4748, over 14594.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005378, over 41.00 utterances.], tot_loss[loss=0.4163, over 2856649.24 frames; utt_duration=1271 frames, utt_pad_proportion=0.05306, over 9000.19 utterances.], batch size: 41, lr: 3.42e-04
|
| 193 |
+
2022-08-11 17:10:37,551 INFO [train.py:737] (2/3) Epoch 15, batch 3300, loss[loss=0.4735, over 13073.00 frames; utt_duration=782 frames, utt_pad_proportion=0.1154, over 67.00 utterances.], tot_loss[loss=0.4258, over 2844370.12 frames; utt_duration=1240 frames, utt_pad_proportion=0.0599, over 9187.15 utterances.], batch size: 67, lr: 3.42e-04
|
| 194 |
+
2022-08-11 17:12:19,929 INFO [train.py:737] (2/3) Epoch 15, batch 3400, loss[loss=0.4514, over 14776.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.005847, over 39.00 utterances.], tot_loss[loss=0.4308, over 2832343.25 frames; utt_duration=1217 frames, utt_pad_proportion=0.06766, over 9318.17 utterances.], batch size: 39, lr: 3.42e-04
|
| 195 |
+
2022-08-11 17:14:52,684 INFO [train.py:737] (2/3) Epoch 15, batch 3500, loss[loss=0.4295, over 14715.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.01062, over 35.00 utterances.], tot_loss[loss=0.4381, over 2819531.77 frames; utt_duration=1191 frames, utt_pad_proportion=0.07577, over 9479.13 utterances.], batch size: 35, lr: 3.42e-04
|
| 196 |
+
2022-08-11 17:17:37,566 INFO [train.py:737] (2/3) Epoch 15, batch 3600, loss[loss=0.4811, over 14752.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006792, over 40.00 utterances.], tot_loss[loss=0.4413, over 2803434.77 frames; utt_duration=1166 frames, utt_pad_proportion=0.08684, over 9626.22 utterances.], batch size: 40, lr: 3.42e-04
|
| 197 |
+
2022-08-11 17:20:24,452 INFO [train.py:737] (2/3) Epoch 15, batch 3700, loss[loss=0.4339, over 13646.00 frames; utt_duration=1821 frames, utt_pad_proportion=0.07741, over 30.00 utterances.], tot_loss[loss=0.4438, over 2819789.14 frames; utt_duration=1202 frames, utt_pad_proportion=0.07757, over 9396.05 utterances.], batch size: 30, lr: 3.42e-04
|
| 198 |
+
2022-08-11 17:23:11,070 INFO [train.py:737] (2/3) Epoch 15, batch 3800, loss[loss=0.4121, over 14564.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.007033, over 37.00 utterances.], tot_loss[loss=0.4484, over 2819964.88 frames; utt_duration=1191 frames, utt_pad_proportion=0.07653, over 9486.49 utterances.], batch size: 37, lr: 3.41e-04
|
| 199 |
+
2022-08-11 17:25:37,104 INFO [train.py:737] (2/3) Epoch 15, batch 3900, loss[loss=0.4427, over 14865.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.00727, over 44.00 utterances.], tot_loss[loss=0.4435, over 2835266.63 frames; utt_duration=1224 frames, utt_pad_proportion=0.06774, over 9278.17 utterances.], batch size: 44, lr: 3.41e-04
|
| 200 |
+
2022-08-11 17:28:09,792 INFO [train.py:737] (2/3) Epoch 15, batch 4000, loss[loss=0.4057, over 14760.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005269, over 41.00 utterances.], tot_loss[loss=0.4419, over 2825938.42 frames; utt_duration=1212 frames, utt_pad_proportion=0.07202, over 9338.97 utterances.], batch size: 41, lr: 3.41e-04
|
| 201 |
+
2022-08-11 17:30:45,452 INFO [train.py:737] (2/3) Epoch 15, batch 4100, loss[loss=0.4093, over 14572.00 frames; utt_duration=1716 frames, utt_pad_proportion=0.01229, over 34.00 utterances.], tot_loss[loss=0.4429, over 2818142.55 frames; utt_duration=1197 frames, utt_pad_proportion=0.07634, over 9425.24 utterances.], batch size: 34, lr: 3.41e-04
|
| 202 |
+
2022-08-11 17:33:21,442 INFO [train.py:737] (2/3) Epoch 15, batch 4200, loss[loss=0.3661, over 14481.00 frames; utt_duration=1757 frames, utt_pad_proportion=0.009788, over 33.00 utterances.], tot_loss[loss=0.4433, over 2811693.75 frames; utt_duration=1189 frames, utt_pad_proportion=0.08141, over 9471.96 utterances.], batch size: 33, lr: 3.41e-04
|
| 203 |
+
2022-08-11 17:35:41,151 INFO [train.py:737] (2/3) Epoch 15, batch 4300, loss[loss=0.4333, over 14700.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01168, over 46.00 utterances.], tot_loss[loss=0.4432, over 2810090.37 frames; utt_duration=1180 frames, utt_pad_proportion=0.08125, over 9534.15 utterances.], batch size: 46, lr: 3.41e-04
|
| 204 |
+
2022-08-11 17:38:29,485 INFO [train.py:737] (2/3) Epoch 15, batch 4400, loss[loss=0.4174, over 14597.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.004752, over 41.00 utterances.], tot_loss[loss=0.4441, over 2826194.26 frames; utt_duration=1206 frames, utt_pad_proportion=0.07142, over 9386.09 utterances.], batch size: 41, lr: 3.41e-04
|
| 205 |
+
2022-08-11 17:41:15,834 INFO [train.py:737] (2/3) Epoch 15, batch 4500, loss[loss=0.4284, over 14452.00 frames; utt_duration=1653 frames, utt_pad_proportion=0.008467, over 35.00 utterances.], tot_loss[loss=0.4428, over 2827542.57 frames; utt_duration=1213 frames, utt_pad_proportion=0.07167, over 9335.89 utterances.], batch size: 35, lr: 3.41e-04
|
| 206 |
+
2022-08-11 17:43:38,599 INFO [train.py:737] (2/3) Epoch 15, batch 4600, loss[loss=0.4723, over 13054.00 frames; utt_duration=769.2 frames, utt_pad_proportion=0.1279, over 68.00 utterances.], tot_loss[loss=0.4419, over 2824912.32 frames; utt_duration=1213 frames, utt_pad_proportion=0.07267, over 9328.64 utterances.], batch size: 68, lr: 3.41e-04
|
| 207 |
+
2022-08-11 17:46:05,258 INFO [train.py:737] (2/3) Epoch 15, batch 4700, loss[loss=0.4302, over 14682.00 frames; utt_duration=1251 frames, utt_pad_proportion=0.0112, over 47.00 utterances.], tot_loss[loss=0.4392, over 2833436.26 frames; utt_duration=1223 frames, utt_pad_proportion=0.06726, over 9278.12 utterances.], batch size: 47, lr: 3.40e-04
|
| 208 |
+
2022-08-11 17:48:30,593 INFO [train.py:737] (2/3) Epoch 15, batch 4800, loss[loss=0.4956, over 14663.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.01224, over 47.00 utterances.], tot_loss[loss=0.4392, over 2837297.24 frames; utt_duration=1230 frames, utt_pad_proportion=0.06488, over 9236.44 utterances.], batch size: 47, lr: 3.40e-04
|
| 209 |
+
2022-08-11 17:50:07,720 INFO [train.py:737] (2/3) Epoch 15, batch 4900, loss[loss=0.5362, over 9735.00 frames; utt_duration=429.4 frames, utt_pad_proportion=0.3414, over 91.00 utterances.], tot_loss[loss=0.4375, over 2846772.63 frames; utt_duration=1245 frames, utt_pad_proportion=0.05972, over 9153.86 utterances.], batch size: 91, lr: 3.40e-04
|
| 210 |
+
2022-08-11 17:52:17,902 INFO [train.py:737] (2/3) Epoch 15, batch 5000, loss[loss=0.3926, over 14734.00 frames; utt_duration=1685 frames, utt_pad_proportion=0.009703, over 35.00 utterances.], tot_loss[loss=0.437, over 2843056.58 frames; utt_duration=1246 frames, utt_pad_proportion=0.0628, over 9137.83 utterances.], batch size: 35, lr: 3.40e-04
|
| 211 |
+
2022-08-11 17:54:42,991 INFO [train.py:737] (2/3) Epoch 15, batch 5100, loss[loss=0.505, over 14580.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.006485, over 41.00 utterances.], tot_loss[loss=0.4355, over 2851085.35 frames; utt_duration=1260 frames, utt_pad_proportion=0.05652, over 9059.37 utterances.], batch size: 41, lr: 3.40e-04
|
| 212 |
+
2022-08-11 17:57:27,534 INFO [train.py:737] (2/3) Epoch 15, batch 5200, loss[loss=0.4136, over 14762.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006788, over 39.00 utterances.], tot_loss[loss=0.4355, over 2841254.17 frames; utt_duration=1242 frames, utt_pad_proportion=0.06076, over 9159.72 utterances.], batch size: 39, lr: 3.40e-04
|
| 213 |
+
2022-08-11 18:00:09,220 INFO [train.py:737] (2/3) Epoch 15, batch 5300, loss[loss=0.4539, over 14660.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01338, over 47.00 utterances.], tot_loss[loss=0.4369, over 2827572.11 frames; utt_duration=1212 frames, utt_pad_proportion=0.0694, over 9347.17 utterances.], batch size: 47, lr: 3.40e-04
|
| 214 |
+
2022-08-11 18:02:48,924 INFO [train.py:737] (2/3) Epoch 15, batch 5400, loss[loss=0.4391, over 14874.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.007598, over 44.00 utterances.], tot_loss[loss=0.4367, over 2841033.84 frames; utt_duration=1241 frames, utt_pad_proportion=0.06292, over 9170.92 utterances.], batch size: 44, lr: 3.40e-04
|
| 215 |
+
2022-08-11 18:05:10,737 INFO [train.py:737] (2/3) Epoch 15, batch 5500, loss[loss=0.5011, over 14575.00 frames; utt_duration=1578 frames, utt_pad_proportion=0.005944, over 37.00 utterances.], tot_loss[loss=0.4395, over 2828804.42 frames; utt_duration=1219 frames, utt_pad_proportion=0.06976, over 9296.40 utterances.], batch size: 37, lr: 3.40e-04
|
| 216 |
+
2022-08-11 18:06:37,413 INFO [train.py:737] (2/3) Epoch 15, batch 5600, loss[loss=0.4423, over 14579.00 frames; utt_duration=1460 frames, utt_pad_proportion=0.00562, over 40.00 utterances.], tot_loss[loss=0.4385, over 2835818.97 frames; utt_duration=1233 frames, utt_pad_proportion=0.06601, over 9210.33 utterances.], batch size: 40, lr: 3.39e-04
|
| 217 |
+
2022-08-11 18:09:12,334 INFO [train.py:737] (2/3) Epoch 15, batch 5700, loss[loss=0.4645, over 14701.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.008036, over 45.00 utterances.], tot_loss[loss=0.4385, over 2826980.78 frames; utt_duration=1216 frames, utt_pad_proportion=0.07278, over 9311.35 utterances.], batch size: 45, lr: 3.39e-04
|
| 218 |
+
2022-08-11 18:12:03,387 INFO [train.py:737] (2/3) Epoch 15, batch 5800, loss[loss=0.4285, over 14424.00 frames; utt_duration=1090 frames, utt_pad_proportion=0.0274, over 53.00 utterances.], tot_loss[loss=0.4411, over 2813644.85 frames; utt_duration=1183 frames, utt_pad_proportion=0.0788, over 9525.35 utterances.], batch size: 53, lr: 3.39e-04
|
| 219 |
+
2022-08-11 18:14:35,533 INFO [train.py:737] (2/3) Epoch 15, batch 5900, loss[loss=0.431, over 14717.00 frames; utt_duration=1309 frames, utt_pad_proportion=0.008013, over 45.00 utterances.], tot_loss[loss=0.4412, over 2840417.74 frames; utt_duration=1230 frames, utt_pad_proportion=0.06265, over 9250.15 utterances.], batch size: 45, lr: 3.39e-04
|
| 220 |
+
2022-08-11 18:17:10,948 INFO [train.py:737] (2/3) Epoch 15, batch 6000, loss[loss=0.3729, over 14613.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.009618, over 34.00 utterances.], tot_loss[loss=0.4405, over 2840169.49 frames; utt_duration=1237 frames, utt_pad_proportion=0.06235, over 9191.69 utterances.], batch size: 34, lr: 3.39e-04
|
| 221 |
+
2022-08-11 18:17:10,948 INFO [train.py:761] (2/3) Computing validation loss
|
| 222 |
+
2022-08-11 18:18:08,644 INFO [train.py:770] (2/3) Epoch 15, validation: loss=0.3006, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 223 |
+
2022-08-11 18:20:56,945 INFO [train.py:737] (2/3) Epoch 15, batch 6100, loss[loss=0.4987, over 9992.00 frames; utt_duration=435.8 frames, utt_pad_proportion=0.3253, over 92.00 utterances.], tot_loss[loss=0.4477, over 2758996.03 frames; utt_duration=1099 frames, utt_pad_proportion=0.11, over 10053.71 utterances.], batch size: 92, lr: 3.39e-04
|
| 224 |
+
2022-08-11 18:22:06,667 INFO [train.py:737] (2/3) Epoch 16, batch 0, loss[loss=0.3967, over 14557.00 frames; utt_duration=1457 frames, utt_pad_proportion=0.006003, over 40.00 utterances.], tot_loss[loss=0.3967, over 14557.00 frames; utt_duration=1457 frames, utt_pad_proportion=0.006003, over 40.00 utterances.], batch size: 40, lr: 3.29e-04
|
| 225 |
+
2022-08-11 18:25:01,422 INFO [train.py:737] (2/3) Epoch 16, batch 100, loss[loss=0.3833, over 14701.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.008889, over 45.00 utterances.], tot_loss[loss=0.3635, over 1147951.84 frames; utt_duration=1362 frames, utt_pad_proportion=0.03543, over 3374.94 utterances.], batch size: 45, lr: 3.29e-04
|
exp/log/log-train-2022-08-11-18-53-52-0
ADDED
|
@@ -0,0 +1,344 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2022-08-11 18:53:52,374 INFO [train.py:805] (0/3) Training started
|
| 2 |
+
2022-08-11 18:53:52,375 INFO [train.py:806] (0/3) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'env_info': {'k2-version': '1.17', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '7dcabf85e8bf06984c4abab0400ef1322b5ff3df', 'k2-git-date': 'Tue Aug 2 21:22:39 2022', 'lhotse-version': '1.4.0.dev+git.59318ed.clean', 'torch-version': '1.11.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.3', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '6af45c3-dirty', 'icefall-git-date': 'Tue Aug 2 14:47:09 2022', 'icefall-path': '/thspfs/home/acct-hpc/thspzh-hpc1/git/icefall_dev', 'k2-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/k2-1.17.dev20220805+cuda11.3.torch1.11.0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'dgxs-01.cluster.thspzh.com', 'IP address': '192.168.200.101'}, 'world_size': 3, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 25, 'start_epoch': 16, 'start_batch': 0, 'exp_dir': PosixPath('cnn_ctc/exp_phone'), 'lang_dir': 'data/lang_phone', 'bpe_model': 'data/lang_bpe_500/bpe.model', 'initial_lr': 0.003, 'lr_batches': 5000, 'lr_epochs': 6, 'seed': 42, 'print_diagnostics': False, 'save_every_n': 8000, 'keep_last_k': 20, 'average_period': 100, 'use_fp16': False, 'first_out_channels': 8, 'conv_blocks': 3, 'rnn_dim': 0, 'full_libri': True, 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 600, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'input_strategy': 'PrecomputedFeatures'}
|
| 3 |
+
2022-08-11 18:53:52,750 INFO [lexicon.py:176] (0/3) Loading pre-compiled data/lang_phone/Linv.pt
|
| 4 |
+
2022-08-11 18:54:04,308 INFO [train.py:843] (0/3) About to create model
|
| 5 |
+
2022-08-11 18:54:04,332 INFO [train.py:855] (0/3) Number of model parameters: 1766904
|
| 6 |
+
2022-08-11 18:54:11,385 INFO [train.py:859] (0/3) ==========================================================================================
|
| 7 |
+
Layer (type:depth-idx) Output Shape Param #
|
| 8 |
+
==========================================================================================
|
| 9 |
+
MobileNetS4 [1, 25, 72] --
|
| 10 |
+
├─Sequential: 1-1 [1, 32, 25, 20] --
|
| 11 |
+
│ └─Conv2d: 2-1 [1, 8, 100, 80] 80
|
| 12 |
+
│ └─BatchNorm2d: 2-2 [1, 8, 100, 80] 16
|
| 13 |
+
│ └─ReLU6: 2-3 [1, 8, 100, 80] --
|
| 14 |
+
│ └─Conv2d: 2-4 [1, 16, 50, 40] 1,168
|
| 15 |
+
│ └─BatchNorm2d: 2-5 [1, 16, 50, 40] 32
|
| 16 |
+
│ └─ReLU6: 2-6 [1, 16, 50, 40] --
|
| 17 |
+
│ └─Conv2d: 2-7 [1, 32, 25, 20] 4,640
|
| 18 |
+
│ └─BatchNorm2d: 2-8 [1, 32, 25, 20] 64
|
| 19 |
+
│ └─ReLU6: 2-9 [1, 32, 25, 20] --
|
| 20 |
+
├─Sequential: 1-2 [1, 256, 25, 3] --
|
| 21 |
+
│ └─Bottleneck: 2-10 [1, 64, 25, 10] --
|
| 22 |
+
│ │ └─Sequential: 3-1 [1, 64, 25, 10] 14,400
|
| 23 |
+
│ └─Bottleneck: 2-11 [1, 64, 25, 10] --
|
| 24 |
+
│ │ └─Sequential: 3-2 [1, 64, 25, 10] 36,800
|
| 25 |
+
│ └─Bottleneck: 2-12 [1, 64, 25, 10] --
|
| 26 |
+
│ │ └─Sequential: 3-3 [1, 64, 25, 10] 36,800
|
| 27 |
+
│ └─Bottleneck: 2-13 [1, 128, 25, 5] --
|
| 28 |
+
│ │ └─Sequential: 3-4 [1, 128, 25, 5] 53,376
|
| 29 |
+
│ └─Bottleneck: 2-14 [1, 128, 25, 5] --
|
| 30 |
+
│ │ └─Sequential: 3-5 [1, 128, 25, 5] 139,136
|
| 31 |
+
│ └─Bottleneck: 2-15 [1, 128, 25, 5] --
|
| 32 |
+
│ │ └─Sequential: 3-6 [1, 128, 25, 5] 139,136
|
| 33 |
+
│ └─Bottleneck: 2-16 [1, 256, 25, 3] --
|
| 34 |
+
│ │ └─Sequential: 3-7 [1, 256, 25, 3] 205,056
|
| 35 |
+
│ └─Bottleneck: 2-17 [1, 256, 25, 3] --
|
| 36 |
+
│ │ └─Sequential: 3-8 [1, 256, 25, 3] 540,416
|
| 37 |
+
│ └─Bottleneck: 2-18 [1, 256, 25, 3] --
|
| 38 |
+
│ │ └─Sequential: 3-9 [1, 256, 25, 3] 540,416
|
| 39 |
+
├─Linear: 1-3 [1, 25, 72] 55,368
|
| 40 |
+
==========================================================================================
|
| 41 |
+
Total params: 1,766,904
|
| 42 |
+
Trainable params: 1,766,904
|
| 43 |
+
Non-trainable params: 0
|
| 44 |
+
Total mult-adds (M): 169.35
|
| 45 |
+
==========================================================================================
|
| 46 |
+
Input size (MB): 0.03
|
| 47 |
+
Forward/backward pass size (MB): 22.08
|
| 48 |
+
Params size (MB): 7.07
|
| 49 |
+
Estimated Total Size (MB): 29.18
|
| 50 |
+
==========================================================================================
|
| 51 |
+
2022-08-11 18:54:11,470 INFO [checkpoint.py:112] (0/3) Loading checkpoint from cnn_ctc/exp_phone/epoch-15.pt
|
| 52 |
+
2022-08-11 18:54:11,558 INFO [checkpoint.py:131] (0/3) Loading averaged model
|
| 53 |
+
2022-08-11 18:54:11,570 INFO [train.py:876] (0/3) Using DDP
|
| 54 |
+
2022-08-11 18:54:11,786 INFO [train.py:884] (0/3) Loading optimizer state dict
|
| 55 |
+
2022-08-11 18:54:11,830 INFO [train.py:892] (0/3) Loading scheduler state dict
|
| 56 |
+
2022-08-11 18:54:11,830 INFO [asr_datamodule.py:409] (0/3) About to get train-clean-100 cuts
|
| 57 |
+
2022-08-11 18:54:11,883 INFO [asr_datamodule.py:416] (0/3) About to get train-clean-360 cuts
|
| 58 |
+
2022-08-11 18:54:11,896 INFO [asr_datamodule.py:423] (0/3) About to get train-other-500 cuts
|
| 59 |
+
2022-08-11 18:54:11,915 INFO [asr_datamodule.py:225] (0/3) Enable MUSAN
|
| 60 |
+
2022-08-11 18:54:11,915 INFO [asr_datamodule.py:226] (0/3) About to get Musan cuts
|
| 61 |
+
2022-08-11 18:54:13,888 INFO [asr_datamodule.py:254] (0/3) Enable SpecAugment
|
| 62 |
+
2022-08-11 18:54:13,888 INFO [asr_datamodule.py:255] (0/3) Time warp factor: 80
|
| 63 |
+
2022-08-11 18:54:13,888 INFO [asr_datamodule.py:267] (0/3) Num frame mask: 10
|
| 64 |
+
2022-08-11 18:54:13,888 INFO [asr_datamodule.py:280] (0/3) About to create train dataset
|
| 65 |
+
2022-08-11 18:54:13,889 INFO [asr_datamodule.py:309] (0/3) Using DynamicBucketingSampler.
|
| 66 |
+
2022-08-11 18:54:16,428 INFO [asr_datamodule.py:324] (0/3) About to create train dataloader
|
| 67 |
+
2022-08-11 18:54:16,447 INFO [asr_datamodule.py:430] (0/3) About to get dev-clean cuts
|
| 68 |
+
2022-08-11 18:54:16,460 INFO [asr_datamodule.py:437] (0/3) About to get dev-other cuts
|
| 69 |
+
2022-08-11 18:54:16,478 INFO [asr_datamodule.py:355] (0/3) About to create dev dataset
|
| 70 |
+
2022-08-11 18:54:16,785 INFO [asr_datamodule.py:374] (0/3) About to create dev dataloader
|
| 71 |
+
2022-08-11 18:54:16,785 INFO [train.py:959] (0/3) Loading grad scaler state dict
|
| 72 |
+
2022-08-11 18:54:39,535 INFO [train.py:737] (0/3) Epoch 16, batch 0, loss[loss=0.4228, over 14659.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01277, over 47.00 utterances.], tot_loss[loss=0.4228, over 14659.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01277, over 47.00 utterances.], batch size: 47, lr: 3.29e-04
|
| 73 |
+
2022-08-11 18:54:39,567 INFO [distributed.py:948] (0/3) Reducer buckets have been rebuilt in this iteration.
|
| 74 |
+
2022-08-11 19:00:36,847 INFO [train.py:737] (0/3) Epoch 16, batch 100, loss[loss=0.3838, over 14585.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.00611, over 41.00 utterances.], tot_loss[loss=0.3676, over 1149046.77 frames; utt_duration=1349 frames, utt_pad_proportion=0.03297, over 3409.69 utterances.], batch size: 41, lr: 3.29e-04
|
| 75 |
+
2022-08-11 19:06:33,772 INFO [train.py:737] (0/3) Epoch 16, batch 200, loss[loss=0.3273, over 14766.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.004982, over 41.00 utterances.], tot_loss[loss=0.3619, over 1838817.69 frames; utt_duration=1353 frames, utt_pad_proportion=0.02953, over 5443.21 utterances.], batch size: 41, lr: 3.29e-04
|
| 76 |
+
2022-08-11 19:12:27,340 INFO [train.py:737] (0/3) Epoch 16, batch 300, loss[loss=0.351, over 13962.00 frames; utt_duration=964.5 frames, utt_pad_proportion=0.06359, over 58.00 utterances.], tot_loss[loss=0.3606, over 2239358.21 frames; utt_duration=1307 frames, utt_pad_proportion=0.04248, over 6861.51 utterances.], batch size: 58, lr: 3.29e-04
|
| 77 |
+
2022-08-11 19:18:24,971 INFO [train.py:737] (0/3) Epoch 16, batch 400, loss[loss=0.3931, over 14593.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.005378, over 42.00 utterances.], tot_loss[loss=0.3604, over 2486436.67 frames; utt_duration=1287 frames, utt_pad_proportion=0.04332, over 7739.67 utterances.], batch size: 42, lr: 3.29e-04
|
| 78 |
+
2022-08-11 19:24:20,559 INFO [train.py:737] (0/3) Epoch 16, batch 500, loss[loss=0.3424, over 14592.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005412, over 41.00 utterances.], tot_loss[loss=0.3569, over 2632995.61 frames; utt_duration=1282 frames, utt_pad_proportion=0.04713, over 8226.82 utterances.], batch size: 41, lr: 3.29e-04
|
| 79 |
+
2022-08-11 19:30:14,318 INFO [train.py:737] (0/3) Epoch 16, batch 600, loss[loss=0.3805, over 14321.00 frames; utt_duration=1082 frames, utt_pad_proportion=0.03723, over 53.00 utterances.], tot_loss[loss=0.3546, over 2736082.41 frames; utt_duration=1319 frames, utt_pad_proportion=0.04045, over 8309.29 utterances.], batch size: 53, lr: 3.28e-04
|
| 80 |
+
2022-08-11 19:36:07,464 INFO [train.py:737] (0/3) Epoch 16, batch 700, loss[loss=0.3861, over 14791.00 frames; utt_duration=1559 frames, utt_pad_proportion=0.004756, over 38.00 utterances.], tot_loss[loss=0.3568, over 2796290.00 frames; utt_duration=1324 frames, utt_pad_proportion=0.03802, over 8455.27 utterances.], batch size: 38, lr: 3.28e-04
|
| 81 |
+
2022-08-11 19:42:04,438 INFO [train.py:737] (0/3) Epoch 16, batch 800, loss[loss=0.3534, over 14763.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005969, over 42.00 utterances.], tot_loss[loss=0.3593, over 2821925.60 frames; utt_duration=1315 frames, utt_pad_proportion=0.04475, over 8596.84 utterances.], batch size: 42, lr: 3.28e-04
|
| 82 |
+
2022-08-11 19:48:01,464 INFO [train.py:737] (0/3) Epoch 16, batch 900, loss[loss=0.3762, over 14633.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.008478, over 44.00 utterances.], tot_loss[loss=0.3612, over 2850986.03 frames; utt_duration=1325 frames, utt_pad_proportion=0.03864, over 8617.88 utterances.], batch size: 44, lr: 3.28e-04
|
| 83 |
+
2022-08-11 19:54:00,643 INFO [train.py:737] (0/3) Epoch 16, batch 1000, loss[loss=0.2909, over 14477.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.009924, over 33.00 utterances.], tot_loss[loss=0.3636, over 2865027.22 frames; utt_duration=1324 frames, utt_pad_proportion=0.0366, over 8662.90 utterances.], batch size: 33, lr: 3.28e-04
|
| 84 |
+
2022-08-11 19:59:54,048 INFO [train.py:737] (0/3) Epoch 16, batch 1100, loss[loss=0.3519, over 14743.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.005441, over 41.00 utterances.], tot_loss[loss=0.3634, over 2855992.15 frames; utt_duration=1298 frames, utt_pad_proportion=0.04799, over 8812.39 utterances.], batch size: 41, lr: 3.28e-04
|
| 85 |
+
2022-08-11 20:05:56,450 INFO [train.py:737] (0/3) Epoch 16, batch 1200, loss[loss=0.424, over 14443.00 frames; utt_duration=1652 frames, utt_pad_proportion=0.008913, over 35.00 utterances.], tot_loss[loss=0.3632, over 2862269.58 frames; utt_duration=1296 frames, utt_pad_proportion=0.04613, over 8846.30 utterances.], batch size: 35, lr: 3.28e-04
|
| 86 |
+
2022-08-11 20:11:53,132 INFO [train.py:737] (0/3) Epoch 16, batch 1300, loss[loss=0.3692, over 14753.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.005288, over 43.00 utterances.], tot_loss[loss=0.3649, over 2867000.11 frames; utt_duration=1285 frames, utt_pad_proportion=0.04395, over 8938.09 utterances.], batch size: 43, lr: 3.28e-04
|
| 87 |
+
2022-08-11 20:17:51,405 INFO [train.py:737] (0/3) Epoch 16, batch 1400, loss[loss=0.3212, over 14748.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.005608, over 43.00 utterances.], tot_loss[loss=0.3624, over 2873153.91 frames; utt_duration=1309 frames, utt_pad_proportion=0.04243, over 8786.55 utterances.], batch size: 43, lr: 3.28e-04
|
| 88 |
+
2022-08-11 20:23:48,377 INFO [train.py:737] (0/3) Epoch 16, batch 1500, loss[loss=0.3593, over 14116.00 frames; utt_duration=1822 frames, utt_pad_proportion=0.0398, over 31.00 utterances.], tot_loss[loss=0.3633, over 2870627.14 frames; utt_duration=1302 frames, utt_pad_proportion=0.04396, over 8828.57 utterances.], batch size: 31, lr: 3.27e-04
|
| 89 |
+
2022-08-11 20:29:43,720 INFO [train.py:737] (0/3) Epoch 16, batch 1600, loss[loss=0.2752, over 14629.00 frames; utt_duration=1722 frames, utt_pad_proportion=0.008415, over 34.00 utterances.], tot_loss[loss=0.3653, over 2868232.08 frames; utt_duration=1293 frames, utt_pad_proportion=0.04654, over 8881.99 utterances.], batch size: 34, lr: 3.27e-04
|
| 90 |
+
2022-08-11 20:35:44,437 INFO [train.py:737] (0/3) Epoch 16, batch 1700, loss[loss=0.3415, over 14773.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007403, over 37.00 utterances.], tot_loss[loss=0.3657, over 2869135.45 frames; utt_duration=1285 frames, utt_pad_proportion=0.04594, over 8940.81 utterances.], batch size: 37, lr: 3.27e-04
|
| 91 |
+
2022-08-11 20:41:36,432 INFO [train.py:737] (0/3) Epoch 16, batch 1800, loss[loss=0.362, over 14720.00 frames; utt_duration=1310 frames, utt_pad_proportion=0.006655, over 45.00 utterances.], tot_loss[loss=0.3637, over 2869375.99 frames; utt_duration=1285 frames, utt_pad_proportion=0.04671, over 8938.88 utterances.], batch size: 45, lr: 3.27e-04
|
| 92 |
+
2022-08-11 20:47:35,923 INFO [train.py:737] (0/3) Epoch 16, batch 1900, loss[loss=0.3634, over 14761.00 frames; utt_duration=1555 frames, utt_pad_proportion=0.006974, over 38.00 utterances.], tot_loss[loss=0.3626, over 2865610.41 frames; utt_duration=1280 frames, utt_pad_proportion=0.04687, over 8968.35 utterances.], batch size: 38, lr: 3.27e-04
|
| 93 |
+
2022-08-11 20:53:36,130 INFO [train.py:737] (0/3) Epoch 16, batch 2000, loss[loss=0.3751, over 14576.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006063, over 37.00 utterances.], tot_loss[loss=0.3603, over 2867209.71 frames; utt_duration=1289 frames, utt_pad_proportion=0.04748, over 8905.63 utterances.], batch size: 37, lr: 3.27e-04
|
| 94 |
+
2022-08-11 20:59:32,537 INFO [train.py:737] (0/3) Epoch 16, batch 2100, loss[loss=0.4078, over 14768.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.004747, over 41.00 utterances.], tot_loss[loss=0.3613, over 2880115.51 frames; utt_duration=1319 frames, utt_pad_proportion=0.03937, over 8741.42 utterances.], batch size: 41, lr: 3.27e-04
|
| 95 |
+
2022-08-11 21:05:29,582 INFO [train.py:737] (0/3) Epoch 16, batch 2200, loss[loss=0.4265, over 10178.00 frames; utt_duration=444.1 frames, utt_pad_proportion=0.3147, over 92.00 utterances.], tot_loss[loss=0.3625, over 2854422.81 frames; utt_duration=1276 frames, utt_pad_proportion=0.05722, over 8957.21 utterances.], batch size: 92, lr: 3.27e-04
|
| 96 |
+
2022-08-11 21:11:25,896 INFO [train.py:737] (0/3) Epoch 16, batch 2300, loss[loss=0.283, over 14781.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.005495, over 38.00 utterances.], tot_loss[loss=0.3589, over 2863054.84 frames; utt_duration=1294 frames, utt_pad_proportion=0.05103, over 8860.08 utterances.], batch size: 38, lr: 3.27e-04
|
| 97 |
+
2022-08-11 21:17:21,664 INFO [train.py:737] (0/3) Epoch 16, batch 2400, loss[loss=0.306, over 14731.00 frames; utt_duration=1685 frames, utt_pad_proportion=0.00944, over 35.00 utterances.], tot_loss[loss=0.3593, over 2856940.44 frames; utt_duration=1279 frames, utt_pad_proportion=0.05538, over 8948.15 utterances.], batch size: 35, lr: 3.27e-04
|
| 98 |
+
2022-08-11 21:23:16,090 INFO [train.py:737] (0/3) Epoch 16, batch 2500, loss[loss=0.3387, over 14571.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005368, over 40.00 utterances.], tot_loss[loss=0.3614, over 2847760.30 frames; utt_duration=1261 frames, utt_pad_proportion=0.06025, over 9041.80 utterances.], batch size: 40, lr: 3.26e-04
|
| 99 |
+
2022-08-11 21:29:14,049 INFO [train.py:737] (0/3) Epoch 16, batch 2600, loss[loss=0.3823, over 14724.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.01054, over 35.00 utterances.], tot_loss[loss=0.3621, over 2834933.46 frames; utt_duration=1228 frames, utt_pad_proportion=0.06825, over 9243.86 utterances.], batch size: 35, lr: 3.26e-04
|
| 100 |
+
2022-08-11 21:35:11,490 INFO [train.py:737] (0/3) Epoch 16, batch 2700, loss[loss=0.3063, over 14576.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.005571, over 39.00 utterances.], tot_loss[loss=0.3599, over 2845416.17 frames; utt_duration=1254 frames, utt_pad_proportion=0.06054, over 9089.11 utterances.], batch size: 39, lr: 3.26e-04
|
| 101 |
+
2022-08-11 21:41:05,997 INFO [train.py:737] (0/3) Epoch 16, batch 2800, loss[loss=0.4384, over 14072.00 frames; utt_duration=1817 frames, utt_pad_proportion=0.03293, over 31.00 utterances.], tot_loss[loss=0.3626, over 2843816.01 frames; utt_duration=1244 frames, utt_pad_proportion=0.06153, over 9152.37 utterances.], batch size: 31, lr: 3.26e-04
|
| 102 |
+
2022-08-11 21:46:58,170 INFO [train.py:737] (0/3) Epoch 16, batch 2900, loss[loss=0.4315, over 9782.00 frames; utt_duration=422.2 frames, utt_pad_proportion=0.3394, over 93.00 utterances.], tot_loss[loss=0.3784, over 2829836.27 frames; utt_duration=1218 frames, utt_pad_proportion=0.06978, over 9301.25 utterances.], batch size: 93, lr: 3.26e-04
|
| 103 |
+
2022-08-11 21:52:53,676 INFO [train.py:737] (0/3) Epoch 16, batch 3000, loss[loss=0.3671, over 14610.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.01012, over 34.00 utterances.], tot_loss[loss=0.3956, over 2834831.94 frames; utt_duration=1227 frames, utt_pad_proportion=0.06644, over 9255.76 utterances.], batch size: 34, lr: 3.26e-04
|
| 104 |
+
2022-08-11 21:52:53,677 INFO [train.py:761] (0/3) Computing validation loss
|
| 105 |
+
2022-08-11 21:54:35,582 INFO [train.py:770] (0/3) Epoch 16, validation: loss=0.2879, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 106 |
+
2022-08-11 22:00:26,546 INFO [train.py:737] (0/3) Epoch 16, batch 3100, loss[loss=0.4527, over 14585.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.006498, over 38.00 utterances.], tot_loss[loss=0.4121, over 2831283.58 frames; utt_duration=1214 frames, utt_pad_proportion=0.06776, over 9338.14 utterances.], batch size: 38, lr: 3.26e-04
|
| 107 |
+
2022-08-11 22:06:13,235 INFO [train.py:737] (0/3) Epoch 16, batch 3200, loss[loss=0.3984, over 14453.00 frames; utt_duration=1753 frames, utt_pad_proportion=0.0117, over 33.00 utterances.], tot_loss[loss=0.4198, over 2835501.99 frames; utt_duration=1233 frames, utt_pad_proportion=0.06751, over 9209.06 utterances.], batch size: 33, lr: 3.26e-04
|
| 108 |
+
2022-08-11 22:12:06,478 INFO [train.py:737] (0/3) Epoch 16, batch 3300, loss[loss=0.4094, over 13723.00 frames; utt_duration=964.2 frames, utt_pad_proportion=0.06926, over 57.00 utterances.], tot_loss[loss=0.4273, over 2835506.37 frames; utt_duration=1220 frames, utt_pad_proportion=0.06681, over 9304.43 utterances.], batch size: 57, lr: 3.26e-04
|
| 109 |
+
2022-08-11 22:17:57,672 INFO [train.py:737] (0/3) Epoch 16, batch 3400, loss[loss=0.5208, over 14717.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.009129, over 46.00 utterances.], tot_loss[loss=0.4351, over 2835258.14 frames; utt_duration=1225 frames, utt_pad_proportion=0.06714, over 9271.76 utterances.], batch size: 46, lr: 3.26e-04
|
| 110 |
+
2022-08-11 22:23:49,963 INFO [train.py:737] (0/3) Epoch 16, batch 3500, loss[loss=0.4335, over 14623.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.008817, over 44.00 utterances.], tot_loss[loss=0.4389, over 2836370.59 frames; utt_duration=1227 frames, utt_pad_proportion=0.06724, over 9259.08 utterances.], batch size: 44, lr: 3.25e-04
|
| 111 |
+
2022-08-11 22:29:40,561 INFO [train.py:737] (0/3) Epoch 16, batch 3600, loss[loss=0.4285, over 14748.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.006092, over 43.00 utterances.], tot_loss[loss=0.4403, over 2834071.06 frames; utt_duration=1224 frames, utt_pad_proportion=0.06841, over 9276.57 utterances.], batch size: 43, lr: 3.25e-04
|
| 112 |
+
2022-08-11 22:35:33,287 INFO [train.py:737] (0/3) Epoch 16, batch 3700, loss[loss=0.4431, over 14576.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.005588, over 39.00 utterances.], tot_loss[loss=0.4419, over 2812688.06 frames; utt_duration=1189 frames, utt_pad_proportion=0.08139, over 9473.87 utterances.], batch size: 39, lr: 3.25e-04
|
| 113 |
+
2022-08-11 22:41:20,225 INFO [train.py:737] (0/3) Epoch 16, batch 3800, loss[loss=0.3946, over 14566.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.006778, over 37.00 utterances.], tot_loss[loss=0.4416, over 2824141.72 frames; utt_duration=1212 frames, utt_pad_proportion=0.07407, over 9335.91 utterances.], batch size: 37, lr: 3.25e-04
|
| 114 |
+
2022-08-11 22:47:10,485 INFO [train.py:737] (0/3) Epoch 16, batch 3900, loss[loss=0.3831, over 14780.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.005596, over 38.00 utterances.], tot_loss[loss=0.4417, over 2827025.73 frames; utt_duration=1220 frames, utt_pad_proportion=0.07271, over 9277.10 utterances.], batch size: 38, lr: 3.25e-04
|
| 115 |
+
2022-08-11 22:53:00,685 INFO [train.py:737] (0/3) Epoch 16, batch 4000, loss[loss=0.5266, over 14690.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009014, over 45.00 utterances.], tot_loss[loss=0.4435, over 2829735.12 frames; utt_duration=1218 frames, utt_pad_proportion=0.07013, over 9301.57 utterances.], batch size: 45, lr: 3.25e-04
|
| 116 |
+
2022-08-11 22:58:51,024 INFO [train.py:737] (0/3) Epoch 16, batch 4100, loss[loss=0.415, over 14796.00 frames; utt_duration=1601 frames, utt_pad_proportion=0.00564, over 37.00 utterances.], tot_loss[loss=0.4456, over 2829565.43 frames; utt_duration=1218 frames, utt_pad_proportion=0.07024, over 9302.78 utterances.], batch size: 37, lr: 3.25e-04
|
| 117 |
+
2022-08-11 23:04:44,506 INFO [train.py:737] (0/3) Epoch 16, batch 4200, loss[loss=0.4502, over 13021.00 frames; utt_duration=778.9 frames, utt_pad_proportion=0.1189, over 67.00 utterances.], tot_loss[loss=0.4453, over 2822338.46 frames; utt_duration=1197 frames, utt_pad_proportion=0.0745, over 9442.56 utterances.], batch size: 67, lr: 3.25e-04
|
| 118 |
+
2022-08-11 23:10:34,628 INFO [train.py:737] (0/3) Epoch 16, batch 4300, loss[loss=0.4394, over 14377.00 frames; utt_duration=1086 frames, utt_pad_proportion=0.03087, over 53.00 utterances.], tot_loss[loss=0.4435, over 2822963.18 frames; utt_duration=1198 frames, utt_pad_proportion=0.07501, over 9433.97 utterances.], batch size: 53, lr: 3.25e-04
|
| 119 |
+
2022-08-11 23:16:26,640 INFO [train.py:737] (0/3) Epoch 16, batch 4400, loss[loss=0.4002, over 14010.00 frames; utt_duration=1809 frames, utt_pad_proportion=0.04978, over 31.00 utterances.], tot_loss[loss=0.4437, over 2818617.48 frames; utt_duration=1192 frames, utt_pad_proportion=0.07765, over 9473.23 utterances.], batch size: 31, lr: 3.24e-04
|
| 120 |
+
2022-08-11 23:22:14,315 INFO [train.py:737] (0/3) Epoch 16, batch 4500, loss[loss=0.4353, over 14504.00 frames; utt_duration=1760 frames, utt_pad_proportion=0.008097, over 33.00 utterances.], tot_loss[loss=0.4421, over 2818790.94 frames; utt_duration=1195 frames, utt_pad_proportion=0.07722, over 9444.25 utterances.], batch size: 33, lr: 3.24e-04
|
| 121 |
+
2022-08-11 23:28:03,895 INFO [train.py:737] (0/3) Epoch 16, batch 4600, loss[loss=0.3652, over 14452.00 frames; utt_duration=1653 frames, utt_pad_proportion=0.008141, over 35.00 utterances.], tot_loss[loss=0.4393, over 2831335.30 frames; utt_duration=1226 frames, utt_pad_proportion=0.07028, over 9249.64 utterances.], batch size: 35, lr: 3.24e-04
|
| 122 |
+
2022-08-11 23:33:56,639 INFO [train.py:737] (0/3) Epoch 16, batch 4700, loss[loss=0.4549, over 12595.00 frames; utt_duration=753.4 frames, utt_pad_proportion=0.1506, over 67.00 utterances.], tot_loss[loss=0.444, over 2814255.06 frames; utt_duration=1186 frames, utt_pad_proportion=0.08082, over 9501.75 utterances.], batch size: 67, lr: 3.24e-04
|
| 123 |
+
2022-08-11 23:39:49,292 INFO [train.py:737] (0/3) Epoch 16, batch 4800, loss[loss=0.5191, over 10003.00 frames; utt_duration=436.2 frames, utt_pad_proportion=0.3288, over 92.00 utterances.], tot_loss[loss=0.4454, over 2802547.58 frames; utt_duration=1165 frames, utt_pad_proportion=0.08578, over 9631.49 utterances.], batch size: 92, lr: 3.24e-04
|
| 124 |
+
2022-08-11 23:45:40,797 INFO [train.py:737] (0/3) Epoch 16, batch 4900, loss[loss=0.5578, over 14631.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.008563, over 44.00 utterances.], tot_loss[loss=0.4417, over 2813131.90 frames; utt_duration=1186 frames, utt_pad_proportion=0.08061, over 9495.72 utterances.], batch size: 44, lr: 3.24e-04
|
| 125 |
+
2022-08-11 23:51:33,919 INFO [train.py:737] (0/3) Epoch 16, batch 5000, loss[loss=0.4718, over 14891.00 frames; utt_duration=1355 frames, utt_pad_proportion=0.006415, over 44.00 utterances.], tot_loss[loss=0.4435, over 2805211.24 frames; utt_duration=1167 frames, utt_pad_proportion=0.08498, over 9625.33 utterances.], batch size: 44, lr: 3.24e-04
|
| 126 |
+
2022-08-11 23:57:27,494 INFO [train.py:737] (0/3) Epoch 16, batch 5100, loss[loss=0.4076, over 14572.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005624, over 40.00 utterances.], tot_loss[loss=0.4429, over 2820774.73 frames; utt_duration=1192 frames, utt_pad_proportion=0.07376, over 9476.55 utterances.], batch size: 40, lr: 3.24e-04
|
| 127 |
+
2022-08-12 00:03:19,188 INFO [train.py:737] (0/3) Epoch 16, batch 5200, loss[loss=0.4404, over 14739.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.00618, over 43.00 utterances.], tot_loss[loss=0.4408, over 2817888.23 frames; utt_duration=1198 frames, utt_pad_proportion=0.07679, over 9423.95 utterances.], batch size: 43, lr: 3.24e-04
|
| 128 |
+
2022-08-12 00:09:07,287 INFO [train.py:737] (0/3) Epoch 16, batch 5300, loss[loss=0.5776, over 9873.00 frames; utt_duration=430.9 frames, utt_pad_proportion=0.3341, over 92.00 utterances.], tot_loss[loss=0.4395, over 2837463.30 frames; utt_duration=1232 frames, utt_pad_proportion=0.06611, over 9223.92 utterances.], batch size: 92, lr: 3.24e-04
|
| 129 |
+
2022-08-12 00:15:01,425 INFO [train.py:737] (0/3) Epoch 16, batch 5400, loss[loss=0.4593, over 14692.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009596, over 45.00 utterances.], tot_loss[loss=0.443, over 2841433.54 frames; utt_duration=1228 frames, utt_pad_proportion=0.06288, over 9265.03 utterances.], batch size: 45, lr: 3.23e-04
|
| 130 |
+
2022-08-12 00:20:52,397 INFO [train.py:737] (0/3) Epoch 16, batch 5500, loss[loss=0.4861, over 14673.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.01076, over 45.00 utterances.], tot_loss[loss=0.4429, over 2826697.43 frames; utt_duration=1203 frames, utt_pad_proportion=0.0729, over 9409.90 utterances.], batch size: 45, lr: 3.23e-04
|
| 131 |
+
2022-08-12 00:26:40,779 INFO [train.py:737] (0/3) Epoch 16, batch 5600, loss[loss=0.412, over 14568.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006625, over 40.00 utterances.], tot_loss[loss=0.4393, over 2823558.03 frames; utt_duration=1200 frames, utt_pad_proportion=0.07383, over 9421.60 utterances.], batch size: 40, lr: 3.23e-04
|
| 132 |
+
2022-08-12 00:32:31,457 INFO [train.py:737] (0/3) Epoch 16, batch 5700, loss[loss=0.386, over 14752.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006809, over 40.00 utterances.], tot_loss[loss=0.4374, over 2824430.62 frames; utt_duration=1209 frames, utt_pad_proportion=0.07397, over 9357.21 utterances.], batch size: 40, lr: 3.23e-04
|
| 133 |
+
2022-08-12 00:35:42,076 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/checkpoint-160000.pt
|
| 134 |
+
2022-08-12 00:38:21,054 INFO [train.py:737] (0/3) Epoch 16, batch 5800, loss[loss=0.5221, over 14424.00 frames; utt_duration=1155 frames, utt_pad_proportion=0.02427, over 50.00 utterances.], tot_loss[loss=0.4377, over 2823598.28 frames; utt_duration=1213 frames, utt_pad_proportion=0.07327, over 9321.41 utterances.], batch size: 50, lr: 3.23e-04
|
| 135 |
+
2022-08-12 00:44:13,574 INFO [train.py:737] (0/3) Epoch 16, batch 5900, loss[loss=0.4084, over 14763.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.004898, over 41.00 utterances.], tot_loss[loss=0.4404, over 2816783.08 frames; utt_duration=1194 frames, utt_pad_proportion=0.07754, over 9446.68 utterances.], batch size: 41, lr: 3.23e-04
|
| 136 |
+
2022-08-12 00:50:02,626 INFO [train.py:737] (0/3) Epoch 16, batch 6000, loss[loss=0.3877, over 14448.00 frames; utt_duration=1753 frames, utt_pad_proportion=0.01182, over 33.00 utterances.], tot_loss[loss=0.4391, over 2835532.53 frames; utt_duration=1229 frames, utt_pad_proportion=0.0677, over 9237.57 utterances.], batch size: 33, lr: 3.23e-04
|
| 137 |
+
2022-08-12 00:50:02,626 INFO [train.py:761] (0/3) Computing validation loss
|
| 138 |
+
2022-08-12 00:51:42,066 INFO [train.py:770] (0/3) Epoch 16, validation: loss=0.2945, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 139 |
+
2022-08-12 00:57:06,528 INFO [train.py:737] (0/3) Epoch 16, batch 6100, loss[loss=0.5061, over 9911.00 frames; utt_duration=423.3 frames, utt_pad_proportion=0.3366, over 94.00 utterances.], tot_loss[loss=0.4417, over 2773291.61 frames; utt_duration=1132 frames, utt_pad_proportion=0.1047, over 9810.62 utterances.], batch size: 94, lr: 3.23e-04
|
| 140 |
+
2022-08-12 00:57:37,397 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-16.pt
|
| 141 |
+
2022-08-12 00:58:41,879 INFO [train.py:737] (0/3) Epoch 17, batch 0, loss[loss=0.3757, over 14754.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.005822, over 43.00 utterances.], tot_loss[loss=0.3757, over 14754.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.005822, over 43.00 utterances.], batch size: 43, lr: 3.14e-04
|
| 142 |
+
2022-08-12 01:04:37,182 INFO [train.py:737] (0/3) Epoch 17, batch 100, loss[loss=0.3886, over 14656.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01355, over 47.00 utterances.], tot_loss[loss=0.3605, over 1131654.56 frames; utt_duration=1248 frames, utt_pad_proportion=0.05962, over 3630.25 utterances.], batch size: 47, lr: 3.14e-04
|
| 143 |
+
2022-08-12 01:10:31,073 INFO [train.py:737] (0/3) Epoch 17, batch 200, loss[loss=0.3429, over 14586.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.006093, over 41.00 utterances.], tot_loss[loss=0.3544, over 1812548.85 frames; utt_duration=1277 frames, utt_pad_proportion=0.05525, over 5686.30 utterances.], batch size: 41, lr: 3.14e-04
|
| 144 |
+
2022-08-12 01:16:27,827 INFO [train.py:737] (0/3) Epoch 17, batch 300, loss[loss=0.3645, over 14640.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.007564, over 44.00 utterances.], tot_loss[loss=0.3582, over 2230129.46 frames; utt_duration=1276 frames, utt_pad_proportion=0.05194, over 7001.29 utterances.], batch size: 44, lr: 3.14e-04
|
| 145 |
+
2022-08-12 01:22:20,486 INFO [train.py:737] (0/3) Epoch 17, batch 400, loss[loss=0.3311, over 14766.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007688, over 37.00 utterances.], tot_loss[loss=0.3568, over 2481621.52 frames; utt_duration=1283 frames, utt_pad_proportion=0.04997, over 7745.14 utterances.], batch size: 37, lr: 3.13e-04
|
| 146 |
+
2022-08-12 01:28:17,404 INFO [train.py:737] (0/3) Epoch 17, batch 500, loss[loss=0.35, over 14395.00 frames; utt_duration=1088 frames, utt_pad_proportion=0.03376, over 53.00 utterances.], tot_loss[loss=0.3574, over 2633382.86 frames; utt_duration=1291 frames, utt_pad_proportion=0.0496, over 8166.02 utterances.], batch size: 53, lr: 3.13e-04
|
| 147 |
+
2022-08-12 01:34:10,596 INFO [train.py:737] (0/3) Epoch 17, batch 600, loss[loss=0.4143, over 14748.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.006294, over 43.00 utterances.], tot_loss[loss=0.3602, over 2713799.96 frames; utt_duration=1263 frames, utt_pad_proportion=0.05563, over 8601.89 utterances.], batch size: 43, lr: 3.13e-04
|
| 148 |
+
2022-08-12 01:40:07,110 INFO [train.py:737] (0/3) Epoch 17, batch 700, loss[loss=0.3422, over 14874.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.006358, over 44.00 utterances.], tot_loss[loss=0.3594, over 2778608.15 frames; utt_duration=1281 frames, utt_pad_proportion=0.04929, over 8687.76 utterances.], batch size: 44, lr: 3.13e-04
|
| 149 |
+
2022-08-12 01:46:04,163 INFO [train.py:737] (0/3) Epoch 17, batch 800, loss[loss=0.3507, over 14778.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.006664, over 37.00 utterances.], tot_loss[loss=0.3598, over 2807875.54 frames; utt_duration=1284 frames, utt_pad_proportion=0.05154, over 8756.49 utterances.], batch size: 37, lr: 3.13e-04
|
| 150 |
+
2022-08-12 01:52:03,017 INFO [train.py:737] (0/3) Epoch 17, batch 900, loss[loss=0.3632, over 14488.00 frames; utt_duration=1209 frames, utt_pad_proportion=0.01809, over 48.00 utterances.], tot_loss[loss=0.3663, over 2830567.03 frames; utt_duration=1287 frames, utt_pad_proportion=0.05034, over 8805.49 utterances.], batch size: 48, lr: 3.13e-04
|
| 151 |
+
2022-08-12 01:57:59,863 INFO [train.py:737] (0/3) Epoch 17, batch 1000, loss[loss=0.314, over 13556.00 frames; utt_duration=1809 frames, utt_pad_proportion=0.07987, over 30.00 utterances.], tot_loss[loss=0.364, over 2840892.70 frames; utt_duration=1280 frames, utt_pad_proportion=0.05065, over 8887.11 utterances.], batch size: 30, lr: 3.13e-04
|
| 152 |
+
2022-08-12 02:03:56,400 INFO [train.py:737] (0/3) Epoch 17, batch 1100, loss[loss=0.3524, over 14452.00 frames; utt_duration=1158 frames, utt_pad_proportion=0.02316, over 50.00 utterances.], tot_loss[loss=0.3629, over 2852393.79 frames; utt_duration=1279 frames, utt_pad_proportion=0.0491, over 8928.53 utterances.], batch size: 50, lr: 3.13e-04
|
| 153 |
+
2022-08-12 02:09:56,094 INFO [train.py:737] (0/3) Epoch 17, batch 1200, loss[loss=0.3851, over 14766.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005936, over 42.00 utterances.], tot_loss[loss=0.3618, over 2867231.22 frames; utt_duration=1307 frames, utt_pad_proportion=0.04272, over 8785.12 utterances.], batch size: 42, lr: 3.13e-04
|
| 154 |
+
2022-08-12 02:15:53,093 INFO [train.py:737] (0/3) Epoch 17, batch 1300, loss[loss=0.3091, over 14029.00 frames; utt_duration=1812 frames, utt_pad_proportion=0.04536, over 31.00 utterances.], tot_loss[loss=0.3634, over 2869037.56 frames; utt_duration=1299 frames, utt_pad_proportion=0.04327, over 8843.62 utterances.], batch size: 31, lr: 3.13e-04
|
| 155 |
+
2022-08-12 02:21:48,042 INFO [train.py:737] (0/3) Epoch 17, batch 1400, loss[loss=0.378, over 13237.00 frames; utt_duration=791.6 frames, utt_pad_proportion=0.1085, over 67.00 utterances.], tot_loss[loss=0.3616, over 2876714.34 frames; utt_duration=1317 frames, utt_pad_proportion=0.03908, over 8745.44 utterances.], batch size: 67, lr: 3.12e-04
|
| 156 |
+
2022-08-12 02:27:45,480 INFO [train.py:737] (0/3) Epoch 17, batch 1500, loss[loss=0.408, over 14476.00 frames; utt_duration=1208 frames, utt_pad_proportion=0.01873, over 48.00 utterances.], tot_loss[loss=0.3613, over 2866228.05 frames; utt_duration=1296 frames, utt_pad_proportion=0.04687, over 8854.70 utterances.], batch size: 48, lr: 3.12e-04
|
| 157 |
+
2022-08-12 02:33:41,558 INFO [train.py:737] (0/3) Epoch 17, batch 1600, loss[loss=0.4874, over 9304.00 frames; utt_duration=406 frames, utt_pad_proportion=0.3744, over 92.00 utterances.], tot_loss[loss=0.3623, over 2863825.51 frames; utt_duration=1295 frames, utt_pad_proportion=0.05019, over 8856.12 utterances.], batch size: 92, lr: 3.12e-04
|
| 158 |
+
2022-08-12 02:39:40,258 INFO [train.py:737] (0/3) Epoch 17, batch 1700, loss[loss=0.3205, over 14881.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.006503, over 44.00 utterances.], tot_loss[loss=0.3629, over 2855892.01 frames; utt_duration=1267 frames, utt_pad_proportion=0.05426, over 9024.42 utterances.], batch size: 44, lr: 3.12e-04
|
| 159 |
+
2022-08-12 02:45:36,012 INFO [train.py:737] (0/3) Epoch 17, batch 1800, loss[loss=0.3716, over 14562.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006186, over 40.00 utterances.], tot_loss[loss=0.3614, over 2859021.24 frames; utt_duration=1270 frames, utt_pad_proportion=0.05214, over 9012.32 utterances.], batch size: 40, lr: 3.12e-04
|
| 160 |
+
2022-08-12 02:51:32,198 INFO [train.py:737] (0/3) Epoch 17, batch 1900, loss[loss=0.3229, over 14424.00 frames; utt_duration=1650 frames, utt_pad_proportion=0.01004, over 35.00 utterances.], tot_loss[loss=0.3601, over 2863668.87 frames; utt_duration=1289 frames, utt_pad_proportion=0.04959, over 8894.18 utterances.], batch size: 35, lr: 3.12e-04
|
| 161 |
+
2022-08-12 02:57:29,847 INFO [train.py:737] (0/3) Epoch 17, batch 2000, loss[loss=0.3218, over 14615.00 frames; utt_duration=1625 frames, utt_pad_proportion=0.005895, over 36.00 utterances.], tot_loss[loss=0.3612, over 2857032.23 frames; utt_duration=1276 frames, utt_pad_proportion=0.05357, over 8967.55 utterances.], batch size: 36, lr: 3.12e-04
|
| 162 |
+
2022-08-12 03:03:23,355 INFO [train.py:737] (0/3) Epoch 17, batch 2100, loss[loss=0.3537, over 14746.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.005072, over 43.00 utterances.], tot_loss[loss=0.3613, over 2855343.58 frames; utt_duration=1266 frames, utt_pad_proportion=0.05448, over 9032.25 utterances.], batch size: 43, lr: 3.12e-04
|
| 163 |
+
2022-08-12 03:09:19,728 INFO [train.py:737] (0/3) Epoch 17, batch 2200, loss[loss=0.3483, over 14844.00 frames; utt_duration=1651 frames, utt_pad_proportion=0.009137, over 36.00 utterances.], tot_loss[loss=0.3626, over 2844198.75 frames; utt_duration=1248 frames, utt_pad_proportion=0.06061, over 9128.16 utterances.], batch size: 36, lr: 3.12e-04
|
| 164 |
+
2022-08-12 03:15:11,264 INFO [train.py:737] (0/3) Epoch 17, batch 2300, loss[loss=0.4434, over 9576.00 frames; utt_duration=422.3 frames, utt_pad_proportion=0.3543, over 91.00 utterances.], tot_loss[loss=0.3623, over 2839451.85 frames; utt_duration=1249 frames, utt_pad_proportion=0.06435, over 9101.68 utterances.], batch size: 91, lr: 3.12e-04
|
| 165 |
+
2022-08-12 03:21:06,343 INFO [train.py:737] (0/3) Epoch 17, batch 2400, loss[loss=0.316, over 14763.00 frames; utt_duration=1555 frames, utt_pad_proportion=0.006755, over 38.00 utterances.], tot_loss[loss=0.3612, over 2847430.22 frames; utt_duration=1266 frames, utt_pad_proportion=0.05993, over 9004.91 utterances.], batch size: 38, lr: 3.12e-04
|
| 166 |
+
2022-08-12 03:27:05,203 INFO [train.py:737] (0/3) Epoch 17, batch 2500, loss[loss=0.3851, over 13691.00 frames; utt_duration=962 frames, utt_pad_proportion=0.07139, over 57.00 utterances.], tot_loss[loss=0.3645, over 2829423.49 frames; utt_duration=1229 frames, utt_pad_proportion=0.07009, over 9219.52 utterances.], batch size: 57, lr: 3.11e-04
|
| 167 |
+
2022-08-12 03:33:01,497 INFO [train.py:737] (0/3) Epoch 17, batch 2600, loss[loss=0.3581, over 14743.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.006262, over 41.00 utterances.], tot_loss[loss=0.3633, over 2833494.55 frames; utt_duration=1227 frames, utt_pad_proportion=0.06844, over 9250.56 utterances.], batch size: 41, lr: 3.11e-04
|
| 168 |
+
2022-08-12 03:38:55,047 INFO [train.py:737] (0/3) Epoch 17, batch 2700, loss[loss=0.4544, over 14585.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.006093, over 41.00 utterances.], tot_loss[loss=0.3596, over 2830646.40 frames; utt_duration=1235 frames, utt_pad_proportion=0.07, over 9178.51 utterances.], batch size: 41, lr: 3.11e-04
|
| 169 |
+
2022-08-12 03:44:52,667 INFO [train.py:737] (0/3) Epoch 17, batch 2800, loss[loss=0.4035, over 14670.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.01019, over 45.00 utterances.], tot_loss[loss=0.3641, over 2839713.62 frames; utt_duration=1236 frames, utt_pad_proportion=0.06431, over 9204.76 utterances.], batch size: 45, lr: 3.11e-04
|
| 170 |
+
2022-08-12 03:50:46,292 INFO [train.py:737] (0/3) Epoch 17, batch 2900, loss[loss=0.4411, over 14390.00 frames; utt_duration=1088 frames, utt_pad_proportion=0.03408, over 53.00 utterances.], tot_loss[loss=0.3806, over 2830645.90 frames; utt_duration=1210 frames, utt_pad_proportion=0.06994, over 9372.51 utterances.], batch size: 53, lr: 3.11e-04
|
| 171 |
+
2022-08-12 03:56:37,635 INFO [train.py:737] (0/3) Epoch 17, batch 3000, loss[loss=0.4084, over 14635.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.00836, over 44.00 utterances.], tot_loss[loss=0.3972, over 2851494.47 frames; utt_duration=1256 frames, utt_pad_proportion=0.05702, over 9095.52 utterances.], batch size: 44, lr: 3.11e-04
|
| 172 |
+
2022-08-12 03:56:37,635 INFO [train.py:761] (0/3) Computing validation loss
|
| 173 |
+
2022-08-12 03:58:13,084 INFO [train.py:770] (0/3) Epoch 17, validation: loss=0.2805, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 174 |
+
2022-08-12 04:04:03,703 INFO [train.py:737] (0/3) Epoch 17, batch 3100, loss[loss=0.4263, over 14573.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005556, over 40.00 utterances.], tot_loss[loss=0.4114, over 2836363.75 frames; utt_duration=1224 frames, utt_pad_proportion=0.06682, over 9278.77 utterances.], batch size: 40, lr: 3.11e-04
|
| 175 |
+
2022-08-12 04:09:52,794 INFO [train.py:737] (0/3) Epoch 17, batch 3200, loss[loss=0.406, over 14587.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.005263, over 41.00 utterances.], tot_loss[loss=0.4199, over 2827553.05 frames; utt_duration=1212 frames, utt_pad_proportion=0.07077, over 9341.96 utterances.], batch size: 41, lr: 3.11e-04
|
| 176 |
+
2022-08-12 04:15:46,142 INFO [train.py:737] (0/3) Epoch 17, batch 3300, loss[loss=0.4601, over 14702.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.008154, over 45.00 utterances.], tot_loss[loss=0.4271, over 2828223.03 frames; utt_duration=1213 frames, utt_pad_proportion=0.07045, over 9340.55 utterances.], batch size: 45, lr: 3.11e-04
|
| 177 |
+
2022-08-12 04:21:38,716 INFO [train.py:737] (0/3) Epoch 17, batch 3400, loss[loss=0.4556, over 14750.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005774, over 41.00 utterances.], tot_loss[loss=0.4335, over 2830022.58 frames; utt_duration=1219 frames, utt_pad_proportion=0.07059, over 9300.31 utterances.], batch size: 41, lr: 3.11e-04
|
| 178 |
+
2022-08-12 04:27:31,852 INFO [train.py:737] (0/3) Epoch 17, batch 3500, loss[loss=0.4674, over 14634.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.008292, over 44.00 utterances.], tot_loss[loss=0.4365, over 2837314.95 frames; utt_duration=1228 frames, utt_pad_proportion=0.06592, over 9255.57 utterances.], batch size: 44, lr: 3.10e-04
|
| 179 |
+
2022-08-12 04:33:19,370 INFO [train.py:737] (0/3) Epoch 17, batch 3600, loss[loss=0.361, over 14456.00 frames; utt_duration=1754 frames, utt_pad_proportion=0.01012, over 33.00 utterances.], tot_loss[loss=0.4376, over 2829919.15 frames; utt_duration=1221 frames, utt_pad_proportion=0.07075, over 9279.23 utterances.], batch size: 33, lr: 3.10e-04
|
| 180 |
+
2022-08-12 04:39:12,740 INFO [train.py:737] (0/3) Epoch 17, batch 3700, loss[loss=0.4071, over 14862.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.007504, over 44.00 utterances.], tot_loss[loss=0.4359, over 2831740.56 frames; utt_duration=1220 frames, utt_pad_proportion=0.07048, over 9292.21 utterances.], batch size: 44, lr: 3.10e-04
|
| 181 |
+
2022-08-12 04:45:00,417 INFO [train.py:737] (0/3) Epoch 17, batch 3800, loss[loss=0.3872, over 14583.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.005247, over 39.00 utterances.], tot_loss[loss=0.4394, over 2850350.57 frames; utt_duration=1255 frames, utt_pad_proportion=0.05774, over 9092.29 utterances.], batch size: 39, lr: 3.10e-04
|
| 182 |
+
2022-08-12 04:50:52,731 INFO [train.py:737] (0/3) Epoch 17, batch 3900, loss[loss=0.5186, over 14631.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.00858, over 44.00 utterances.], tot_loss[loss=0.44, over 2832918.47 frames; utt_duration=1234 frames, utt_pad_proportion=0.06878, over 9192.88 utterances.], batch size: 44, lr: 3.10e-04
|
| 183 |
+
2022-08-12 04:56:42,374 INFO [train.py:737] (0/3) Epoch 17, batch 4000, loss[loss=0.4207, over 14776.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.005881, over 39.00 utterances.], tot_loss[loss=0.4412, over 2826162.25 frames; utt_duration=1221 frames, utt_pad_proportion=0.07254, over 9267.76 utterances.], batch size: 39, lr: 3.10e-04
|
| 184 |
+
2022-08-12 05:02:33,027 INFO [train.py:737] (0/3) Epoch 17, batch 4100, loss[loss=0.5044, over 12958.00 frames; utt_duration=775.1 frames, utt_pad_proportion=0.1222, over 67.00 utterances.], tot_loss[loss=0.439, over 2837608.78 frames; utt_duration=1234 frames, utt_pad_proportion=0.06525, over 9212.20 utterances.], batch size: 67, lr: 3.10e-04
|
| 185 |
+
2022-08-12 05:08:23,748 INFO [train.py:737] (0/3) Epoch 17, batch 4200, loss[loss=0.473, over 13203.00 frames; utt_duration=777.9 frames, utt_pad_proportion=0.1171, over 68.00 utterances.], tot_loss[loss=0.4399, over 2827746.46 frames; utt_duration=1212 frames, utt_pad_proportion=0.07142, over 9340.83 utterances.], batch size: 68, lr: 3.10e-04
|
| 186 |
+
2022-08-12 05:14:14,534 INFO [train.py:737] (0/3) Epoch 17, batch 4300, loss[loss=0.4793, over 14520.00 frames; utt_duration=1211 frames, utt_pad_proportion=0.01689, over 48.00 utterances.], tot_loss[loss=0.4415, over 2812068.07 frames; utt_duration=1186 frames, utt_pad_proportion=0.08103, over 9493.99 utterances.], batch size: 48, lr: 3.10e-04
|
| 187 |
+
2022-08-12 05:20:08,613 INFO [train.py:737] (0/3) Epoch 17, batch 4400, loss[loss=0.3524, over 14692.00 frames; utt_duration=1680 frames, utt_pad_proportion=0.01153, over 35.00 utterances.], tot_loss[loss=0.4414, over 2833949.72 frames; utt_duration=1222 frames, utt_pad_proportion=0.06696, over 9284.91 utterances.], batch size: 35, lr: 3.10e-04
|
| 188 |
+
2022-08-12 05:26:02,686 INFO [train.py:737] (0/3) Epoch 17, batch 4500, loss[loss=0.4692, over 13047.00 frames; utt_duration=769 frames, utt_pad_proportion=0.1241, over 68.00 utterances.], tot_loss[loss=0.4403, over 2810696.90 frames; utt_duration=1184 frames, utt_pad_proportion=0.07881, over 9509.02 utterances.], batch size: 68, lr: 3.10e-04
|
| 189 |
+
2022-08-12 05:31:53,044 INFO [train.py:737] (0/3) Epoch 17, batch 4600, loss[loss=0.4659, over 14781.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.005629, over 39.00 utterances.], tot_loss[loss=0.4397, over 2828529.06 frames; utt_duration=1223 frames, utt_pad_proportion=0.07037, over 9260.00 utterances.], batch size: 39, lr: 3.09e-04
|
| 190 |
+
2022-08-12 05:37:40,347 INFO [train.py:737] (0/3) Epoch 17, batch 4700, loss[loss=0.4607, over 14589.00 frames; utt_duration=1579 frames, utt_pad_proportion=0.00533, over 37.00 utterances.], tot_loss[loss=0.4395, over 2830279.22 frames; utt_duration=1227 frames, utt_pad_proportion=0.06939, over 9239.10 utterances.], batch size: 37, lr: 3.09e-04
|
| 191 |
+
2022-08-12 05:43:30,040 INFO [train.py:737] (0/3) Epoch 17, batch 4800, loss[loss=0.4548, over 14728.00 frames; utt_duration=1282 frames, utt_pad_proportion=0.009921, over 46.00 utterances.], tot_loss[loss=0.4389, over 2822911.03 frames; utt_duration=1214 frames, utt_pad_proportion=0.07414, over 9313.46 utterances.], batch size: 46, lr: 3.09e-04
|
| 192 |
+
2022-08-12 05:49:21,886 INFO [train.py:737] (0/3) Epoch 17, batch 4900, loss[loss=0.4528, over 14764.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007722, over 37.00 utterances.], tot_loss[loss=0.4415, over 2824302.59 frames; utt_duration=1208 frames, utt_pad_proportion=0.07354, over 9360.13 utterances.], batch size: 37, lr: 3.09e-04
|
| 193 |
+
2022-08-12 05:55:09,869 INFO [train.py:737] (0/3) Epoch 17, batch 5000, loss[loss=0.3848, over 14464.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.009679, over 33.00 utterances.], tot_loss[loss=0.4425, over 2819399.52 frames; utt_duration=1199 frames, utt_pad_proportion=0.076, over 9418.92 utterances.], batch size: 33, lr: 3.09e-04
|
| 194 |
+
2022-08-12 06:00:59,658 INFO [train.py:737] (0/3) Epoch 17, batch 5100, loss[loss=0.4454, over 14499.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.01816, over 48.00 utterances.], tot_loss[loss=0.4411, over 2829821.53 frames; utt_duration=1212 frames, utt_pad_proportion=0.0692, over 9349.74 utterances.], batch size: 48, lr: 3.09e-04
|
| 195 |
+
2022-08-12 06:06:51,513 INFO [train.py:737] (0/3) Epoch 17, batch 5200, loss[loss=0.4141, over 14590.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.004992, over 39.00 utterances.], tot_loss[loss=0.4378, over 2835942.44 frames; utt_duration=1227 frames, utt_pad_proportion=0.06482, over 9254.93 utterances.], batch size: 39, lr: 3.09e-04
|
| 196 |
+
2022-08-12 06:12:42,098 INFO [train.py:737] (0/3) Epoch 17, batch 5300, loss[loss=0.5217, over 9580.00 frames; utt_duration=417.8 frames, utt_pad_proportion=0.3572, over 92.00 utterances.], tot_loss[loss=0.44, over 2816158.21 frames; utt_duration=1198 frames, utt_pad_proportion=0.07757, over 9416.26 utterances.], batch size: 92, lr: 3.09e-04
|
| 197 |
+
2022-08-12 06:18:31,725 INFO [train.py:737] (0/3) Epoch 17, batch 5400, loss[loss=0.4922, over 14632.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.007756, over 44.00 utterances.], tot_loss[loss=0.4388, over 2833806.34 frames; utt_duration=1235 frames, utt_pad_proportion=0.06681, over 9188.79 utterances.], batch size: 44, lr: 3.09e-04
|
| 198 |
+
2022-08-12 06:24:22,452 INFO [train.py:737] (0/3) Epoch 17, batch 5500, loss[loss=0.3514, over 14471.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01047, over 33.00 utterances.], tot_loss[loss=0.4385, over 2821627.24 frames; utt_duration=1211 frames, utt_pad_proportion=0.07519, over 9333.54 utterances.], batch size: 33, lr: 3.09e-04
|
| 199 |
+
2022-08-12 06:30:10,217 INFO [train.py:737] (0/3) Epoch 17, batch 5600, loss[loss=0.5475, over 9036.00 frames; utt_duration=394.5 frames, utt_pad_proportion=0.3903, over 92.00 utterances.], tot_loss[loss=0.4389, over 2818627.94 frames; utt_duration=1206 frames, utt_pad_proportion=0.07749, over 9359.83 utterances.], batch size: 92, lr: 3.08e-04
|
| 200 |
+
2022-08-12 06:35:59,296 INFO [train.py:737] (0/3) Epoch 17, batch 5700, loss[loss=0.4081, over 14569.00 frames; utt_duration=1495 frames, utt_pad_proportion=0.005694, over 39.00 utterances.], tot_loss[loss=0.4382, over 2817223.09 frames; utt_duration=1204 frames, utt_pad_proportion=0.07883, over 9369.67 utterances.], batch size: 39, lr: 3.08e-04
|
| 201 |
+
2022-08-12 06:41:51,109 INFO [train.py:737] (0/3) Epoch 17, batch 5800, loss[loss=0.3665, over 14877.00 frames; utt_duration=1654 frames, utt_pad_proportion=0.00717, over 36.00 utterances.], tot_loss[loss=0.4364, over 2825309.72 frames; utt_duration=1217 frames, utt_pad_proportion=0.07263, over 9301.18 utterances.], batch size: 36, lr: 3.08e-04
|
| 202 |
+
2022-08-12 06:47:36,188 INFO [train.py:737] (0/3) Epoch 17, batch 5900, loss[loss=0.3978, over 14592.00 frames; utt_duration=1718 frames, utt_pad_proportion=0.01074, over 34.00 utterances.], tot_loss[loss=0.4391, over 2818001.94 frames; utt_duration=1206 frames, utt_pad_proportion=0.07689, over 9357.49 utterances.], batch size: 34, lr: 3.08e-04
|
| 203 |
+
2022-08-12 06:53:24,448 INFO [train.py:737] (0/3) Epoch 17, batch 6000, loss[loss=0.4092, over 14382.00 frames; utt_duration=1087 frames, utt_pad_proportion=0.0339, over 53.00 utterances.], tot_loss[loss=0.4405, over 2818761.55 frames; utt_duration=1196 frames, utt_pad_proportion=0.07566, over 9436.77 utterances.], batch size: 53, lr: 3.08e-04
|
| 204 |
+
2022-08-12 06:53:24,449 INFO [train.py:761] (0/3) Computing validation loss
|
| 205 |
+
2022-08-12 06:55:07,773 INFO [train.py:770] (0/3) Epoch 17, validation: loss=0.3115, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 206 |
+
2022-08-12 07:00:34,845 INFO [train.py:737] (0/3) Epoch 17, batch 6100, loss[loss=0.5333, over 10449.00 frames; utt_duration=450.9 frames, utt_pad_proportion=0.301, over 93.00 utterances.], tot_loss[loss=0.4457, over 2757889.93 frames; utt_duration=1094 frames, utt_pad_proportion=0.1078, over 10100.02 utterances.], batch size: 93, lr: 3.08e-04
|
| 207 |
+
2022-08-12 07:01:05,933 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-17.pt
|
| 208 |
+
2022-08-12 07:02:15,427 INFO [train.py:737] (0/3) Epoch 18, batch 0, loss[loss=0.3749, over 14582.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005824, over 40.00 utterances.], tot_loss[loss=0.3749, over 14582.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005824, over 40.00 utterances.], batch size: 40, lr: 3.00e-04
|
| 209 |
+
2022-08-12 07:08:15,724 INFO [train.py:737] (0/3) Epoch 18, batch 100, loss[loss=0.3953, over 14358.00 frames; utt_duration=1085 frames, utt_pad_proportion=0.03466, over 53.00 utterances.], tot_loss[loss=0.3608, over 1140192.15 frames; utt_duration=1287 frames, utt_pad_proportion=0.04879, over 3548.21 utterances.], batch size: 53, lr: 3.00e-04
|
| 210 |
+
2022-08-12 07:14:10,520 INFO [train.py:737] (0/3) Epoch 18, batch 200, loss[loss=0.2566, over 14601.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01098, over 34.00 utterances.], tot_loss[loss=0.3561, over 1831859.90 frames; utt_duration=1331 frames, utt_pad_proportion=0.038, over 5510.07 utterances.], batch size: 34, lr: 3.00e-04
|
| 211 |
+
2022-08-12 07:20:03,955 INFO [train.py:737] (0/3) Epoch 18, batch 300, loss[loss=0.3382, over 14763.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.006121, over 42.00 utterances.], tot_loss[loss=0.3581, over 2244707.89 frames; utt_duration=1315 frames, utt_pad_proportion=0.04199, over 6834.64 utterances.], batch size: 42, lr: 3.00e-04
|
| 212 |
+
2022-08-12 07:25:56,867 INFO [train.py:737] (0/3) Epoch 18, batch 400, loss[loss=0.3414, over 14784.00 frames; utt_duration=1600 frames, utt_pad_proportion=0.006828, over 37.00 utterances.], tot_loss[loss=0.3528, over 2498474.11 frames; utt_duration=1337 frames, utt_pad_proportion=0.03931, over 7482.60 utterances.], batch size: 37, lr: 3.00e-04
|
| 213 |
+
2022-08-12 07:31:51,510 INFO [train.py:737] (0/3) Epoch 18, batch 500, loss[loss=0.3602, over 14760.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.004833, over 43.00 utterances.], tot_loss[loss=0.3528, over 2649820.45 frames; utt_duration=1328 frames, utt_pad_proportion=0.03975, over 7993.24 utterances.], batch size: 43, lr: 2.99e-04
|
| 214 |
+
2022-08-12 07:37:46,287 INFO [train.py:737] (0/3) Epoch 18, batch 600, loss[loss=0.3662, over 14454.00 frames; utt_duration=1158 frames, utt_pad_proportion=0.02436, over 50.00 utterances.], tot_loss[loss=0.354, over 2747822.89 frames; utt_duration=1342 frames, utt_pad_proportion=0.03429, over 8198.41 utterances.], batch size: 50, lr: 2.99e-04
|
| 215 |
+
2022-08-12 07:43:44,662 INFO [train.py:737] (0/3) Epoch 18, batch 700, loss[loss=0.3673, over 14563.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.00688, over 40.00 utterances.], tot_loss[loss=0.3541, over 2790404.53 frames; utt_duration=1324 frames, utt_pad_proportion=0.04328, over 8438.93 utterances.], batch size: 40, lr: 2.99e-04
|
| 216 |
+
2022-08-12 07:49:44,655 INFO [train.py:737] (0/3) Epoch 18, batch 800, loss[loss=0.3864, over 14582.00 frames; utt_duration=1578 frames, utt_pad_proportion=0.00579, over 37.00 utterances.], tot_loss[loss=0.3559, over 2816206.66 frames; utt_duration=1308 frames, utt_pad_proportion=0.04798, over 8620.94 utterances.], batch size: 37, lr: 2.99e-04
|
| 217 |
+
2022-08-12 07:55:42,380 INFO [train.py:737] (0/3) Epoch 18, batch 900, loss[loss=0.2938, over 14791.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.00489, over 38.00 utterances.], tot_loss[loss=0.3562, over 2840359.14 frames; utt_duration=1312 frames, utt_pad_proportion=0.04544, over 8668.20 utterances.], batch size: 38, lr: 2.99e-04
|
| 218 |
+
2022-08-12 08:01:36,734 INFO [train.py:737] (0/3) Epoch 18, batch 1000, loss[loss=0.2672, over 14462.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01078, over 33.00 utterances.], tot_loss[loss=0.359, over 2857761.57 frames; utt_duration=1317 frames, utt_pad_proportion=0.04098, over 8692.29 utterances.], batch size: 33, lr: 2.99e-04
|
| 219 |
+
2022-08-12 08:07:31,600 INFO [train.py:737] (0/3) Epoch 18, batch 1100, loss[loss=0.4356, over 9930.00 frames; utt_duration=433.1 frames, utt_pad_proportion=0.3348, over 92.00 utterances.], tot_loss[loss=0.36, over 2861437.75 frames; utt_duration=1303 frames, utt_pad_proportion=0.044, over 8793.16 utterances.], batch size: 92, lr: 2.99e-04
|
| 220 |
+
2022-08-12 08:13:30,145 INFO [train.py:737] (0/3) Epoch 18, batch 1200, loss[loss=0.4052, over 14697.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.009226, over 45.00 utterances.], tot_loss[loss=0.3599, over 2866229.96 frames; utt_duration=1317 frames, utt_pad_proportion=0.04346, over 8714.04 utterances.], batch size: 45, lr: 2.99e-04
|
| 221 |
+
2022-08-12 08:19:25,567 INFO [train.py:737] (0/3) Epoch 18, batch 1300, loss[loss=0.3067, over 14774.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.00615, over 38.00 utterances.], tot_loss[loss=0.3599, over 2879962.96 frames; utt_duration=1341 frames, utt_pad_proportion=0.03606, over 8601.04 utterances.], batch size: 38, lr: 2.99e-04
|
| 222 |
+
2022-08-12 08:25:25,692 INFO [train.py:737] (0/3) Epoch 18, batch 1400, loss[loss=0.3504, over 14782.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.004691, over 38.00 utterances.], tot_loss[loss=0.3624, over 2864255.03 frames; utt_duration=1297 frames, utt_pad_proportion=0.04775, over 8841.68 utterances.], batch size: 38, lr: 2.99e-04
|
| 223 |
+
2022-08-12 08:31:23,719 INFO [train.py:737] (0/3) Epoch 18, batch 1500, loss[loss=0.3505, over 14747.00 frames; utt_duration=1514 frames, utt_pad_proportion=0.006679, over 39.00 utterances.], tot_loss[loss=0.3606, over 2872457.97 frames; utt_duration=1307 frames, utt_pad_proportion=0.04257, over 8802.83 utterances.], batch size: 39, lr: 2.99e-04
|
| 224 |
+
2022-08-12 08:32:45,918 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/checkpoint-168000.pt
|
| 225 |
+
2022-08-12 08:37:25,240 INFO [train.py:737] (0/3) Epoch 18, batch 1600, loss[loss=0.3173, over 14754.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.006625, over 42.00 utterances.], tot_loss[loss=0.3605, over 2880708.99 frames; utt_duration=1320 frames, utt_pad_proportion=0.03743, over 8736.16 utterances.], batch size: 42, lr: 2.98e-04
|
| 226 |
+
2022-08-12 08:43:19,481 INFO [train.py:737] (0/3) Epoch 18, batch 1700, loss[loss=0.4221, over 14771.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005566, over 42.00 utterances.], tot_loss[loss=0.3609, over 2879180.15 frames; utt_duration=1322 frames, utt_pad_proportion=0.03925, over 8719.26 utterances.], batch size: 42, lr: 2.98e-04
|
| 227 |
+
2022-08-12 08:49:17,870 INFO [train.py:737] (0/3) Epoch 18, batch 1800, loss[loss=0.3424, over 14072.00 frames; utt_duration=1817 frames, utt_pad_proportion=0.04514, over 31.00 utterances.], tot_loss[loss=0.3585, over 2867181.09 frames; utt_duration=1299 frames, utt_pad_proportion=0.04669, over 8841.08 utterances.], batch size: 31, lr: 2.98e-04
|
| 228 |
+
2022-08-12 08:55:13,601 INFO [train.py:737] (0/3) Epoch 18, batch 1900, loss[loss=0.291, over 14708.00 frames; utt_duration=1682 frames, utt_pad_proportion=0.01039, over 35.00 utterances.], tot_loss[loss=0.3596, over 2869465.07 frames; utt_duration=1305 frames, utt_pad_proportion=0.04583, over 8806.16 utterances.], batch size: 35, lr: 2.98e-04
|
| 229 |
+
2022-08-12 08:59:37,386 INFO [train.py:737] (0/3) Epoch 18, batch 2000, loss[loss=0.313, over 14019.00 frames; utt_duration=1810 frames, utt_pad_proportion=0.03711, over 31.00 utterances.], tot_loss[loss=0.3588, over 2859154.47 frames; utt_duration=1283 frames, utt_pad_proportion=0.05378, over 8926.35 utterances.], batch size: 31, lr: 2.98e-04
|
| 230 |
+
2022-08-12 09:02:23,575 INFO [train.py:737] (0/3) Epoch 18, batch 2100, loss[loss=0.3466, over 14493.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.01507, over 48.00 utterances.], tot_loss[loss=0.3571, over 2864015.53 frames; utt_duration=1286 frames, utt_pad_proportion=0.04941, over 8918.97 utterances.], batch size: 48, lr: 2.98e-04
|
| 231 |
+
2022-08-12 09:05:26,496 INFO [train.py:737] (0/3) Epoch 18, batch 2200, loss[loss=0.4556, over 9723.00 frames; utt_duration=428.7 frames, utt_pad_proportion=0.3445, over 91.00 utterances.], tot_loss[loss=0.3584, over 2865184.12 frames; utt_duration=1287 frames, utt_pad_proportion=0.04897, over 8917.90 utterances.], batch size: 91, lr: 2.98e-04
|
| 232 |
+
2022-08-12 09:07:43,370 INFO [train.py:737] (0/3) Epoch 18, batch 2300, loss[loss=0.3666, over 14769.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.00488, over 42.00 utterances.], tot_loss[loss=0.3596, over 2871126.78 frames; utt_duration=1297 frames, utt_pad_proportion=0.04577, over 8862.52 utterances.], batch size: 42, lr: 2.98e-04
|
| 233 |
+
2022-08-12 09:10:16,145 INFO [train.py:737] (0/3) Epoch 18, batch 2400, loss[loss=0.337, over 14869.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.007437, over 44.00 utterances.], tot_loss[loss=0.3605, over 2856773.38 frames; utt_duration=1270 frames, utt_pad_proportion=0.05545, over 9009.26 utterances.], batch size: 44, lr: 2.98e-04
|
| 234 |
+
2022-08-12 09:12:54,437 INFO [train.py:737] (0/3) Epoch 18, batch 2500, loss[loss=0.3698, over 14763.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.005117, over 41.00 utterances.], tot_loss[loss=0.3593, over 2851885.93 frames; utt_duration=1264 frames, utt_pad_proportion=0.05896, over 9038.52 utterances.], batch size: 41, lr: 2.98e-04
|
| 235 |
+
2022-08-12 09:16:00,772 INFO [train.py:737] (0/3) Epoch 18, batch 2600, loss[loss=0.271, over 14819.00 frames; utt_duration=1649 frames, utt_pad_proportion=0.01045, over 36.00 utterances.], tot_loss[loss=0.3593, over 2852010.41 frames; utt_duration=1261 frames, utt_pad_proportion=0.05849, over 9059.60 utterances.], batch size: 36, lr: 2.98e-04
|
| 236 |
+
2022-08-12 09:18:22,664 INFO [train.py:737] (0/3) Epoch 18, batch 2700, loss[loss=0.2898, over 14772.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007885, over 37.00 utterances.], tot_loss[loss=0.3591, over 2842002.96 frames; utt_duration=1246 frames, utt_pad_proportion=0.06484, over 9134.94 utterances.], batch size: 37, lr: 2.97e-04
|
| 237 |
+
2022-08-12 09:21:19,620 INFO [train.py:737] (0/3) Epoch 18, batch 2800, loss[loss=0.303, over 14472.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.009896, over 33.00 utterances.], tot_loss[loss=0.3632, over 2833924.15 frames; utt_duration=1225 frames, utt_pad_proportion=0.0684, over 9267.15 utterances.], batch size: 33, lr: 2.97e-04
|
| 238 |
+
2022-08-12 09:24:12,307 INFO [train.py:737] (0/3) Epoch 18, batch 2900, loss[loss=0.3993, over 14640.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.007852, over 44.00 utterances.], tot_loss[loss=0.3768, over 2842372.44 frames; utt_duration=1243 frames, utt_pad_proportion=0.06285, over 9161.52 utterances.], batch size: 44, lr: 2.97e-04
|
| 239 |
+
2022-08-12 09:26:27,702 INFO [train.py:737] (0/3) Epoch 18, batch 3000, loss[loss=0.4385, over 14590.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.006192, over 38.00 utterances.], tot_loss[loss=0.393, over 2853139.38 frames; utt_duration=1264 frames, utt_pad_proportion=0.05567, over 9042.36 utterances.], batch size: 38, lr: 2.97e-04
|
| 240 |
+
2022-08-12 09:26:27,703 INFO [train.py:761] (0/3) Computing validation loss
|
| 241 |
+
2022-08-12 09:27:08,098 INFO [train.py:770] (0/3) Epoch 18, validation: loss=0.2876, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 242 |
+
2022-08-12 09:29:25,968 INFO [train.py:737] (0/3) Epoch 18, batch 3100, loss[loss=0.4463, over 14589.00 frames; utt_duration=1391 frames, utt_pad_proportion=0.005616, over 42.00 utterances.], tot_loss[loss=0.4071, over 2844547.17 frames; utt_duration=1242 frames, utt_pad_proportion=0.06102, over 9170.94 utterances.], batch size: 42, lr: 2.97e-04
|
| 243 |
+
2022-08-12 09:32:24,406 INFO [train.py:737] (0/3) Epoch 18, batch 3200, loss[loss=0.4653, over 14774.00 frames; utt_duration=1479 frames, utt_pad_proportion=0.005414, over 40.00 utterances.], tot_loss[loss=0.4165, over 2848660.15 frames; utt_duration=1250 frames, utt_pad_proportion=0.05933, over 9123.76 utterances.], batch size: 40, lr: 2.97e-04
|
| 244 |
+
2022-08-12 09:34:49,898 INFO [train.py:737] (0/3) Epoch 18, batch 3300, loss[loss=0.4845, over 14497.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.01616, over 48.00 utterances.], tot_loss[loss=0.4237, over 2826769.56 frames; utt_duration=1208 frames, utt_pad_proportion=0.07216, over 9372.44 utterances.], batch size: 48, lr: 2.97e-04
|
| 245 |
+
2022-08-12 09:37:01,563 INFO [train.py:737] (0/3) Epoch 18, batch 3400, loss[loss=0.4697, over 14701.00 frames; utt_duration=1253 frames, utt_pad_proportion=0.01066, over 47.00 utterances.], tot_loss[loss=0.4305, over 2828893.78 frames; utt_duration=1218 frames, utt_pad_proportion=0.0701, over 9299.85 utterances.], batch size: 47, lr: 2.97e-04
|
| 246 |
+
2022-08-12 09:39:45,278 INFO [train.py:737] (0/3) Epoch 18, batch 3500, loss[loss=0.5173, over 14728.00 frames; utt_duration=1282 frames, utt_pad_proportion=0.009955, over 46.00 utterances.], tot_loss[loss=0.4314, over 2834355.78 frames; utt_duration=1236 frames, utt_pad_proportion=0.06868, over 9182.60 utterances.], batch size: 46, lr: 2.97e-04
|
| 247 |
+
2022-08-12 09:41:58,169 INFO [train.py:737] (0/3) Epoch 18, batch 3600, loss[loss=0.4194, over 14864.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.007437, over 44.00 utterances.], tot_loss[loss=0.4347, over 2832134.06 frames; utt_duration=1225 frames, utt_pad_proportion=0.06951, over 9262.82 utterances.], batch size: 44, lr: 2.97e-04
|
| 248 |
+
2022-08-12 09:44:05,887 INFO [train.py:737] (0/3) Epoch 18, batch 3700, loss[loss=0.3737, over 14774.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.005966, over 38.00 utterances.], tot_loss[loss=0.435, over 2829634.74 frames; utt_duration=1214 frames, utt_pad_proportion=0.07006, over 9334.64 utterances.], batch size: 38, lr: 2.97e-04
|
| 249 |
+
2022-08-12 09:46:59,241 INFO [train.py:737] (0/3) Epoch 18, batch 3800, loss[loss=0.408, over 14742.00 frames; utt_duration=1686 frames, utt_pad_proportion=0.008684, over 35.00 utterances.], tot_loss[loss=0.4387, over 2827494.14 frames; utt_duration=1211 frames, utt_pad_proportion=0.0713, over 9348.65 utterances.], batch size: 35, lr: 2.97e-04
|
| 250 |
+
2022-08-12 09:49:10,331 INFO [train.py:737] (0/3) Epoch 18, batch 3900, loss[loss=0.3861, over 14764.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.005333, over 39.00 utterances.], tot_loss[loss=0.4378, over 2828239.20 frames; utt_duration=1224 frames, utt_pad_proportion=0.07022, over 9256.03 utterances.], batch size: 39, lr: 2.96e-04
|
| 251 |
+
2022-08-12 09:51:49,475 INFO [train.py:737] (0/3) Epoch 18, batch 4000, loss[loss=0.3776, over 14429.00 frames; utt_duration=1650 frames, utt_pad_proportion=0.009907, over 35.00 utterances.], tot_loss[loss=0.4403, over 2822593.66 frames; utt_duration=1212 frames, utt_pad_proportion=0.07518, over 9329.54 utterances.], batch size: 35, lr: 2.96e-04
|
| 252 |
+
2022-08-12 09:54:08,256 INFO [train.py:737] (0/3) Epoch 18, batch 4100, loss[loss=0.3826, over 14443.00 frames; utt_duration=1752 frames, utt_pad_proportion=0.01172, over 33.00 utterances.], tot_loss[loss=0.4396, over 2844250.72 frames; utt_duration=1246 frames, utt_pad_proportion=0.0622, over 9141.38 utterances.], batch size: 33, lr: 2.96e-04
|
| 253 |
+
2022-08-12 09:56:28,399 INFO [train.py:737] (0/3) Epoch 18, batch 4200, loss[loss=0.4951, over 14293.00 frames; utt_duration=1080 frames, utt_pad_proportion=0.03711, over 53.00 utterances.], tot_loss[loss=0.4419, over 2826075.87 frames; utt_duration=1205 frames, utt_pad_proportion=0.07244, over 9393.22 utterances.], batch size: 53, lr: 2.96e-04
|
| 254 |
+
2022-08-12 09:58:21,327 INFO [train.py:737] (0/3) Epoch 18, batch 4300, loss[loss=0.3902, over 14701.00 frames; utt_duration=1682 frames, utt_pad_proportion=0.01142, over 35.00 utterances.], tot_loss[loss=0.4398, over 2824682.81 frames; utt_duration=1205 frames, utt_pad_proportion=0.07286, over 9384.53 utterances.], batch size: 35, lr: 2.96e-04
|
| 255 |
+
2022-08-12 10:01:09,184 INFO [train.py:737] (0/3) Epoch 18, batch 4400, loss[loss=0.5305, over 9479.00 frames; utt_duration=413.7 frames, utt_pad_proportion=0.3606, over 92.00 utterances.], tot_loss[loss=0.4415, over 2814249.34 frames; utt_duration=1187 frames, utt_pad_proportion=0.07898, over 9495.01 utterances.], batch size: 92, lr: 2.96e-04
|
| 256 |
+
2022-08-12 10:04:01,949 INFO [train.py:737] (0/3) Epoch 18, batch 4500, loss[loss=0.3861, over 14701.00 frames; utt_duration=1682 frames, utt_pad_proportion=0.01127, over 35.00 utterances.], tot_loss[loss=0.4403, over 2827913.37 frames; utt_duration=1210 frames, utt_pad_proportion=0.0714, over 9359.83 utterances.], batch size: 35, lr: 2.96e-04
|
| 257 |
+
2022-08-12 10:07:00,420 INFO [train.py:737] (0/3) Epoch 18, batch 4600, loss[loss=0.3545, over 14764.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.006638, over 38.00 utterances.], tot_loss[loss=0.4384, over 2836118.08 frames; utt_duration=1224 frames, utt_pad_proportion=0.06688, over 9276.40 utterances.], batch size: 38, lr: 2.96e-04
|
| 258 |
+
2022-08-12 10:09:39,661 INFO [train.py:737] (0/3) Epoch 18, batch 4700, loss[loss=0.4595, over 14743.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.005843, over 43.00 utterances.], tot_loss[loss=0.4376, over 2838637.33 frames; utt_duration=1241 frames, utt_pad_proportion=0.06472, over 9157.31 utterances.], batch size: 43, lr: 2.96e-04
|
| 259 |
+
2022-08-12 10:12:23,882 INFO [train.py:737] (0/3) Epoch 18, batch 4800, loss[loss=0.5125, over 12911.00 frames; utt_duration=772.4 frames, utt_pad_proportion=0.1283, over 67.00 utterances.], tot_loss[loss=0.4389, over 2831483.06 frames; utt_duration=1233 frames, utt_pad_proportion=0.06931, over 9194.12 utterances.], batch size: 67, lr: 2.96e-04
|
| 260 |
+
2022-08-12 10:15:16,742 INFO [train.py:737] (0/3) Epoch 18, batch 4900, loss[loss=0.4462, over 14573.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006437, over 37.00 utterances.], tot_loss[loss=0.4373, over 2833809.06 frames; utt_duration=1229 frames, utt_pad_proportion=0.06838, over 9231.86 utterances.], batch size: 37, lr: 2.96e-04
|
| 261 |
+
2022-08-12 10:17:43,025 INFO [train.py:737] (0/3) Epoch 18, batch 5000, loss[loss=0.4582, over 14599.00 frames; utt_duration=1623 frames, utt_pad_proportion=0.007675, over 36.00 utterances.], tot_loss[loss=0.4387, over 2832550.22 frames; utt_duration=1209 frames, utt_pad_proportion=0.06801, over 9385.30 utterances.], batch size: 36, lr: 2.95e-04
|
| 262 |
+
2022-08-12 10:19:42,978 INFO [train.py:737] (0/3) Epoch 18, batch 5100, loss[loss=0.4518, over 14711.00 frames; utt_duration=1309 frames, utt_pad_proportion=0.008434, over 45.00 utterances.], tot_loss[loss=0.439, over 2834548.59 frames; utt_duration=1214 frames, utt_pad_proportion=0.06773, over 9349.46 utterances.], batch size: 45, lr: 2.95e-04
|
| 263 |
+
2022-08-12 10:22:03,878 INFO [train.py:737] (0/3) Epoch 18, batch 5200, loss[loss=0.4724, over 14408.00 frames; utt_duration=1089 frames, utt_pad_proportion=0.03286, over 53.00 utterances.], tot_loss[loss=0.4411, over 2832252.16 frames; utt_duration=1206 frames, utt_pad_proportion=0.06803, over 9405.16 utterances.], batch size: 53, lr: 2.95e-04
|
| 264 |
+
2022-08-12 10:24:26,636 INFO [train.py:737] (0/3) Epoch 18, batch 5300, loss[loss=0.4471, over 14763.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006688, over 39.00 utterances.], tot_loss[loss=0.4402, over 2835341.11 frames; utt_duration=1220 frames, utt_pad_proportion=0.06848, over 9306.43 utterances.], batch size: 39, lr: 2.95e-04
|
| 265 |
+
2022-08-12 10:26:43,150 INFO [train.py:737] (0/3) Epoch 18, batch 5400, loss[loss=0.5003, over 14775.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.005898, over 39.00 utterances.], tot_loss[loss=0.4405, over 2823645.12 frames; utt_duration=1197 frames, utt_pad_proportion=0.07528, over 9445.03 utterances.], batch size: 39, lr: 2.95e-04
|
| 266 |
+
2022-08-12 10:28:50,808 INFO [train.py:737] (0/3) Epoch 18, batch 5500, loss[loss=0.432, over 14705.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01143, over 46.00 utterances.], tot_loss[loss=0.4396, over 2839722.43 frames; utt_duration=1236 frames, utt_pad_proportion=0.06536, over 9204.10 utterances.], batch size: 46, lr: 2.95e-04
|
| 267 |
+
2022-08-12 10:30:53,581 INFO [train.py:737] (0/3) Epoch 18, batch 5600, loss[loss=0.3983, over 14741.00 frames; utt_duration=1686 frames, utt_pad_proportion=0.008785, over 35.00 utterances.], tot_loss[loss=0.441, over 2846803.66 frames; utt_duration=1245 frames, utt_pad_proportion=0.05993, over 9158.47 utterances.], batch size: 35, lr: 2.95e-04
|
| 268 |
+
2022-08-12 10:33:04,874 INFO [train.py:737] (0/3) Epoch 18, batch 5700, loss[loss=0.3829, over 14600.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01088, over 34.00 utterances.], tot_loss[loss=0.4352, over 2845708.16 frames; utt_duration=1257 frames, utt_pad_proportion=0.06155, over 9065.67 utterances.], batch size: 34, lr: 2.95e-04
|
| 269 |
+
2022-08-12 10:35:25,141 INFO [train.py:737] (0/3) Epoch 18, batch 5800, loss[loss=0.4728, over 14738.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.006248, over 43.00 utterances.], tot_loss[loss=0.435, over 2837448.49 frames; utt_duration=1246 frames, utt_pad_proportion=0.06617, over 9117.01 utterances.], batch size: 43, lr: 2.95e-04
|
| 270 |
+
2022-08-12 10:37:04,066 INFO [train.py:737] (0/3) Epoch 18, batch 5900, loss[loss=0.5257, over 9935.00 frames; utt_duration=438.3 frames, utt_pad_proportion=0.3288, over 91.00 utterances.], tot_loss[loss=0.4376, over 2831201.56 frames; utt_duration=1224 frames, utt_pad_proportion=0.0688, over 9261.29 utterances.], batch size: 91, lr: 2.95e-04
|
| 271 |
+
2022-08-12 10:38:07,036 INFO [train.py:737] (0/3) Epoch 18, batch 6000, loss[loss=0.5095, over 14579.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.006214, over 39.00 utterances.], tot_loss[loss=0.436, over 2826749.39 frames; utt_duration=1221 frames, utt_pad_proportion=0.07115, over 9271.62 utterances.], batch size: 39, lr: 2.95e-04
|
| 272 |
+
2022-08-12 10:38:07,037 INFO [train.py:761] (0/3) Computing validation loss
|
| 273 |
+
2022-08-12 10:38:38,841 INFO [train.py:770] (0/3) Epoch 18, validation: loss=0.3055, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 274 |
+
2022-08-12 10:40:35,065 INFO [train.py:737] (0/3) Epoch 18, batch 6100, loss[loss=0.5063, over 9674.00 frames; utt_duration=422 frames, utt_pad_proportion=0.3527, over 92.00 utterances.], tot_loss[loss=0.44, over 2761807.81 frames; utt_duration=1110 frames, utt_pad_proportion=0.1087, over 9963.09 utterances.], batch size: 92, lr: 2.95e-04
|
| 275 |
+
2022-08-12 10:41:03,418 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-18.pt
|
| 276 |
+
2022-08-12 10:41:44,279 INFO [train.py:737] (0/3) Epoch 19, batch 0, loss[loss=0.4093, over 14504.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.0177, over 48.00 utterances.], tot_loss[loss=0.4093, over 14504.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.0177, over 48.00 utterances.], batch size: 48, lr: 2.87e-04
|
| 277 |
+
2022-08-12 10:44:11,839 INFO [train.py:737] (0/3) Epoch 19, batch 100, loss[loss=0.3082, over 14763.00 frames; utt_duration=1597 frames, utt_pad_proportion=0.007256, over 37.00 utterances.], tot_loss[loss=0.3588, over 1144409.55 frames; utt_duration=1354 frames, utt_pad_proportion=0.03435, over 3384.47 utterances.], batch size: 37, lr: 2.87e-04
|
| 278 |
+
2022-08-12 10:46:22,384 INFO [train.py:737] (0/3) Epoch 19, batch 200, loss[loss=0.297, over 14438.00 frames; utt_duration=1752 frames, utt_pad_proportion=0.01264, over 33.00 utterances.], tot_loss[loss=0.3576, over 1833260.98 frames; utt_duration=1352 frames, utt_pad_proportion=0.03556, over 5428.61 utterances.], batch size: 33, lr: 2.87e-04
|
| 279 |
+
2022-08-12 10:48:43,568 INFO [train.py:737] (0/3) Epoch 19, batch 300, loss[loss=0.2966, over 14493.00 frames; utt_duration=1758 frames, utt_pad_proportion=0.008848, over 33.00 utterances.], tot_loss[loss=0.3536, over 2248678.40 frames; utt_duration=1333 frames, utt_pad_proportion=0.03527, over 6757.46 utterances.], batch size: 33, lr: 2.87e-04
|
| 280 |
+
2022-08-12 10:51:23,914 INFO [train.py:737] (0/3) Epoch 19, batch 400, loss[loss=0.3659, over 13023.00 frames; utt_duration=778.9 frames, utt_pad_proportion=0.1209, over 67.00 utterances.], tot_loss[loss=0.353, over 2491838.57 frames; utt_duration=1309 frames, utt_pad_proportion=0.04074, over 7625.96 utterances.], batch size: 67, lr: 2.87e-04
|
| 281 |
+
2022-08-12 10:53:02,774 INFO [train.py:737] (0/3) Epoch 19, batch 500, loss[loss=0.3689, over 14771.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.004594, over 42.00 utterances.], tot_loss[loss=0.3536, over 2638381.95 frames; utt_duration=1296 frames, utt_pad_proportion=0.04381, over 8149.57 utterances.], batch size: 42, lr: 2.87e-04
|
| 282 |
+
2022-08-12 10:55:23,802 INFO [train.py:737] (0/3) Epoch 19, batch 600, loss[loss=0.3299, over 14051.00 frames; utt_duration=1814 frames, utt_pad_proportion=0.03901, over 31.00 utterances.], tot_loss[loss=0.3574, over 2707935.45 frames; utt_duration=1246 frames, utt_pad_proportion=0.05837, over 8703.15 utterances.], batch size: 31, lr: 2.87e-04
|
| 283 |
+
2022-08-12 10:58:04,976 INFO [train.py:737] (0/3) Epoch 19, batch 700, loss[loss=0.3729, over 14734.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.007303, over 43.00 utterances.], tot_loss[loss=0.3591, over 2783368.54 frames; utt_duration=1284 frames, utt_pad_proportion=0.04739, over 8679.74 utterances.], batch size: 43, lr: 2.86e-04
|
| 284 |
+
2022-08-12 11:00:28,079 INFO [train.py:737] (0/3) Epoch 19, batch 800, loss[loss=0.3383, over 14587.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.008388, over 36.00 utterances.], tot_loss[loss=0.3601, over 2830461.94 frames; utt_duration=1324 frames, utt_pad_proportion=0.0391, over 8560.05 utterances.], batch size: 36, lr: 2.86e-04
|
| 285 |
+
2022-08-12 11:02:57,819 INFO [train.py:737] (0/3) Epoch 19, batch 900, loss[loss=0.3735, over 14630.00 frames; utt_duration=1627 frames, utt_pad_proportion=0.005484, over 36.00 utterances.], tot_loss[loss=0.3604, over 2847163.98 frames; utt_duration=1308 frames, utt_pad_proportion=0.04081, over 8715.79 utterances.], batch size: 36, lr: 2.86e-04
|
| 286 |
+
2022-08-12 11:05:28,862 INFO [train.py:737] (0/3) Epoch 19, batch 1000, loss[loss=0.3316, over 14555.00 frames; utt_duration=1575 frames, utt_pad_proportion=0.00763, over 37.00 utterances.], tot_loss[loss=0.3632, over 2858343.23 frames; utt_duration=1303 frames, utt_pad_proportion=0.04057, over 8782.09 utterances.], batch size: 37, lr: 2.86e-04
|
| 287 |
+
2022-08-12 11:07:58,551 INFO [train.py:737] (0/3) Epoch 19, batch 1100, loss[loss=0.3934, over 13822.00 frames; utt_duration=971.4 frames, utt_pad_proportion=0.06508, over 57.00 utterances.], tot_loss[loss=0.3634, over 2859475.49 frames; utt_duration=1285 frames, utt_pad_proportion=0.04421, over 8913.61 utterances.], batch size: 57, lr: 2.86e-04
|
| 288 |
+
2022-08-12 11:10:10,292 INFO [train.py:737] (0/3) Epoch 19, batch 1200, loss[loss=0.3712, over 14696.00 frames; utt_duration=1252 frames, utt_pad_proportion=0.01023, over 47.00 utterances.], tot_loss[loss=0.3626, over 2867145.60 frames; utt_duration=1296 frames, utt_pad_proportion=0.04251, over 8860.77 utterances.], batch size: 47, lr: 2.86e-04
|
| 289 |
+
2022-08-12 11:12:02,655 INFO [train.py:737] (0/3) Epoch 19, batch 1300, loss[loss=0.3651, over 14645.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.007632, over 44.00 utterances.], tot_loss[loss=0.3608, over 2878814.04 frames; utt_duration=1319 frames, utt_pad_proportion=0.03693, over 8738.41 utterances.], batch size: 44, lr: 2.86e-04
|
| 290 |
+
2022-08-12 11:14:19,755 INFO [train.py:737] (0/3) Epoch 19, batch 1400, loss[loss=0.3698, over 14701.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.008923, over 45.00 utterances.], tot_loss[loss=0.3592, over 2870111.02 frames; utt_duration=1306 frames, utt_pad_proportion=0.0438, over 8802.86 utterances.], batch size: 45, lr: 2.86e-04
|
| 291 |
+
2022-08-12 11:16:17,937 INFO [train.py:737] (0/3) Epoch 19, batch 1500, loss[loss=0.4314, over 14578.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.006536, over 41.00 utterances.], tot_loss[loss=0.3604, over 2880203.10 frames; utt_duration=1328 frames, utt_pad_proportion=0.03763, over 8686.31 utterances.], batch size: 41, lr: 2.86e-04
|
| 292 |
+
2022-08-12 11:18:46,498 INFO [train.py:737] (0/3) Epoch 19, batch 1600, loss[loss=0.2895, over 14725.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.01044, over 35.00 utterances.], tot_loss[loss=0.3604, over 2879235.87 frames; utt_duration=1317 frames, utt_pad_proportion=0.03868, over 8756.36 utterances.], batch size: 35, lr: 2.86e-04
|
| 293 |
+
2022-08-12 11:21:10,267 INFO [train.py:737] (0/3) Epoch 19, batch 1700, loss[loss=0.3216, over 14556.00 frames; utt_duration=1214 frames, utt_pad_proportion=0.01426, over 48.00 utterances.], tot_loss[loss=0.3589, over 2879470.26 frames; utt_duration=1316 frames, utt_pad_proportion=0.03777, over 8759.42 utterances.], batch size: 48, lr: 2.86e-04
|
| 294 |
+
2022-08-12 11:23:35,420 INFO [train.py:737] (0/3) Epoch 19, batch 1800, loss[loss=0.3891, over 14759.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.00649, over 40.00 utterances.], tot_loss[loss=0.3609, over 2875642.12 frames; utt_duration=1304 frames, utt_pad_proportion=0.04106, over 8832.08 utterances.], batch size: 40, lr: 2.86e-04
|
| 295 |
+
2022-08-12 11:25:45,260 INFO [train.py:737] (0/3) Epoch 19, batch 1900, loss[loss=0.342, over 14626.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.008952, over 44.00 utterances.], tot_loss[loss=0.3576, over 2883860.94 frames; utt_duration=1336 frames, utt_pad_proportion=0.03492, over 8642.58 utterances.], batch size: 44, lr: 2.86e-04
|
| 296 |
+
2022-08-12 11:27:05,703 INFO [train.py:737] (0/3) Epoch 19, batch 2000, loss[loss=0.3733, over 13805.00 frames; utt_duration=970.4 frames, utt_pad_proportion=0.06427, over 57.00 utterances.], tot_loss[loss=0.3583, over 2873564.76 frames; utt_duration=1306 frames, utt_pad_proportion=0.04182, over 8809.94 utterances.], batch size: 57, lr: 2.85e-04
|
| 297 |
+
2022-08-12 11:29:07,396 INFO [train.py:737] (0/3) Epoch 19, batch 2100, loss[loss=0.351, over 14789.00 frames; utt_duration=1600 frames, utt_pad_proportion=0.006576, over 37.00 utterances.], tot_loss[loss=0.3568, over 2869200.45 frames; utt_duration=1297 frames, utt_pad_proportion=0.04427, over 8861.06 utterances.], batch size: 37, lr: 2.85e-04
|
| 298 |
+
2022-08-12 11:31:17,547 INFO [train.py:737] (0/3) Epoch 19, batch 2200, loss[loss=0.3446, over 14530.00 frames; utt_duration=1213 frames, utt_pad_proportion=0.01573, over 48.00 utterances.], tot_loss[loss=0.3581, over 2865795.59 frames; utt_duration=1285 frames, utt_pad_proportion=0.04724, over 8928.65 utterances.], batch size: 48, lr: 2.85e-04
|
| 299 |
+
2022-08-12 11:33:35,213 INFO [train.py:737] (0/3) Epoch 19, batch 2300, loss[loss=0.4636, over 10124.00 frames; utt_duration=446.3 frames, utt_pad_proportion=0.3176, over 91.00 utterances.], tot_loss[loss=0.358, over 2864889.89 frames; utt_duration=1291 frames, utt_pad_proportion=0.04859, over 8884.28 utterances.], batch size: 91, lr: 2.85e-04
|
| 300 |
+
2022-08-12 11:35:43,152 INFO [train.py:737] (0/3) Epoch 19, batch 2400, loss[loss=0.4013, over 14463.00 frames; utt_duration=1159 frames, utt_pad_proportion=0.0247, over 50.00 utterances.], tot_loss[loss=0.3553, over 2875877.03 frames; utt_duration=1317 frames, utt_pad_proportion=0.04241, over 8746.34 utterances.], batch size: 50, lr: 2.85e-04
|
| 301 |
+
2022-08-12 11:38:20,354 INFO [train.py:737] (0/3) Epoch 19, batch 2500, loss[loss=0.3557, over 14766.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.00657, over 39.00 utterances.], tot_loss[loss=0.3552, over 2863007.59 frames; utt_duration=1294 frames, utt_pad_proportion=0.05004, over 8858.28 utterances.], batch size: 39, lr: 2.85e-04
|
| 302 |
+
2022-08-12 11:40:28,213 INFO [train.py:737] (0/3) Epoch 19, batch 2600, loss[loss=0.3236, over 14742.00 frames; utt_duration=1475 frames, utt_pad_proportion=0.007767, over 40.00 utterances.], tot_loss[loss=0.3553, over 2851979.14 frames; utt_duration=1278 frames, utt_pad_proportion=0.0572, over 8935.90 utterances.], batch size: 40, lr: 2.85e-04
|
| 303 |
+
2022-08-12 11:42:36,346 INFO [train.py:737] (0/3) Epoch 19, batch 2700, loss[loss=0.3012, over 14611.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.008962, over 34.00 utterances.], tot_loss[loss=0.3561, over 2869373.85 frames; utt_duration=1309 frames, utt_pad_proportion=0.04552, over 8781.52 utterances.], batch size: 34, lr: 2.85e-04
|
| 304 |
+
2022-08-12 11:44:20,898 INFO [train.py:737] (0/3) Epoch 19, batch 2800, loss[loss=0.4417, over 14593.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.00543, over 41.00 utterances.], tot_loss[loss=0.3592, over 2866838.15 frames; utt_duration=1297 frames, utt_pad_proportion=0.04787, over 8850.66 utterances.], batch size: 41, lr: 2.85e-04
|
| 305 |
+
2022-08-12 11:46:57,783 INFO [train.py:737] (0/3) Epoch 19, batch 2900, loss[loss=0.4313, over 14724.00 frames; utt_duration=1282 frames, utt_pad_proportion=0.009274, over 46.00 utterances.], tot_loss[loss=0.3747, over 2862806.25 frames; utt_duration=1290 frames, utt_pad_proportion=0.05033, over 8885.43 utterances.], batch size: 46, lr: 2.85e-04
|
| 306 |
+
2022-08-12 11:48:38,710 INFO [train.py:737] (0/3) Epoch 19, batch 3000, loss[loss=0.4241, over 14708.00 frames; utt_duration=1309 frames, utt_pad_proportion=0.00867, over 45.00 utterances.], tot_loss[loss=0.3912, over 2855324.73 frames; utt_duration=1281 frames, utt_pad_proportion=0.05569, over 8928.98 utterances.], batch size: 45, lr: 2.85e-04
|
| 307 |
+
2022-08-12 11:48:38,711 INFO [train.py:761] (0/3) Computing validation loss
|
| 308 |
+
2022-08-12 11:49:10,902 INFO [train.py:770] (0/3) Epoch 19, validation: loss=0.2796, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 309 |
+
2022-08-12 11:51:42,711 INFO [train.py:737] (0/3) Epoch 19, batch 3100, loss[loss=0.4283, over 14510.00 frames; utt_duration=1162 frames, utt_pad_proportion=0.02167, over 50.00 utterances.], tot_loss[loss=0.4064, over 2852711.19 frames; utt_duration=1277 frames, utt_pad_proportion=0.05757, over 8948.31 utterances.], batch size: 50, lr: 2.85e-04
|
| 310 |
+
2022-08-12 11:53:58,741 INFO [train.py:737] (0/3) Epoch 19, batch 3200, loss[loss=0.4801, over 9960.00 frames; utt_duration=430 frames, utt_pad_proportion=0.3271, over 93.00 utterances.], tot_loss[loss=0.4182, over 2844190.76 frames; utt_duration=1251 frames, utt_pad_proportion=0.06139, over 9103.70 utterances.], batch size: 93, lr: 2.84e-04
|
| 311 |
+
2022-08-12 11:56:02,101 INFO [train.py:737] (0/3) Epoch 19, batch 3300, loss[loss=0.3554, over 14596.00 frames; utt_duration=1623 frames, utt_pad_proportion=0.00781, over 36.00 utterances.], tot_loss[loss=0.4242, over 2837661.44 frames; utt_duration=1232 frames, utt_pad_proportion=0.06393, over 9223.22 utterances.], batch size: 36, lr: 2.84e-04
|
| 312 |
+
2022-08-12 11:58:26,092 INFO [train.py:737] (0/3) Epoch 19, batch 3400, loss[loss=0.4059, over 14700.00 frames; utt_duration=1682 frames, utt_pad_proportion=0.01137, over 35.00 utterances.], tot_loss[loss=0.4295, over 2831718.16 frames; utt_duration=1228 frames, utt_pad_proportion=0.06738, over 9234.63 utterances.], batch size: 35, lr: 2.84e-04
|
| 313 |
+
2022-08-12 11:58:33,931 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/checkpoint-176000.pt
|
| 314 |
+
2022-08-12 12:00:33,869 INFO [train.py:737] (0/3) Epoch 19, batch 3500, loss[loss=0.4962, over 14521.00 frames; utt_duration=1212 frames, utt_pad_proportion=0.01656, over 48.00 utterances.], tot_loss[loss=0.4367, over 2819455.03 frames; utt_duration=1190 frames, utt_pad_proportion=0.07442, over 9491.93 utterances.], batch size: 48, lr: 2.84e-04
|
| 315 |
+
2022-08-12 12:02:46,753 INFO [train.py:737] (0/3) Epoch 19, batch 3600, loss[loss=0.3974, over 14774.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.006049, over 39.00 utterances.], tot_loss[loss=0.4375, over 2830415.28 frames; utt_duration=1216 frames, utt_pad_proportion=0.06865, over 9323.08 utterances.], batch size: 39, lr: 2.84e-04
|
| 316 |
+
2022-08-12 12:04:15,350 INFO [train.py:737] (0/3) Epoch 19, batch 3700, loss[loss=0.4043, over 14892.00 frames; utt_duration=1355 frames, utt_pad_proportion=0.005703, over 44.00 utterances.], tot_loss[loss=0.4411, over 2830478.04 frames; utt_duration=1212 frames, utt_pad_proportion=0.06863, over 9349.34 utterances.], batch size: 44, lr: 2.84e-04
|
| 317 |
+
2022-08-12 12:06:22,784 INFO [train.py:737] (0/3) Epoch 19, batch 3800, loss[loss=0.5174, over 13005.00 frames; utt_duration=777.9 frames, utt_pad_proportion=0.121, over 67.00 utterances.], tot_loss[loss=0.4403, over 2826344.35 frames; utt_duration=1221 frames, utt_pad_proportion=0.07119, over 9274.12 utterances.], batch size: 67, lr: 2.84e-04
|
| 318 |
+
2022-08-12 12:08:47,677 INFO [train.py:737] (0/3) Epoch 19, batch 3900, loss[loss=0.3652, over 14594.00 frames; utt_duration=1718 frames, utt_pad_proportion=0.01027, over 34.00 utterances.], tot_loss[loss=0.4429, over 2842650.76 frames; utt_duration=1240 frames, utt_pad_proportion=0.06174, over 9182.99 utterances.], batch size: 34, lr: 2.84e-04
|
| 319 |
+
2022-08-12 12:10:40,016 INFO [train.py:737] (0/3) Epoch 19, batch 4000, loss[loss=0.4514, over 14578.00 frames; utt_duration=1578 frames, utt_pad_proportion=0.005978, over 37.00 utterances.], tot_loss[loss=0.4404, over 2852819.54 frames; utt_duration=1270 frames, utt_pad_proportion=0.05462, over 8993.92 utterances.], batch size: 37, lr: 2.84e-04
|
| 320 |
+
2022-08-12 12:11:52,486 INFO [train.py:737] (0/3) Epoch 19, batch 4100, loss[loss=0.5518, over 14746.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.006178, over 41.00 utterances.], tot_loss[loss=0.4383, over 2833453.31 frames; utt_duration=1235 frames, utt_pad_proportion=0.06823, over 9190.44 utterances.], batch size: 41, lr: 2.84e-04
|
| 321 |
+
2022-08-12 12:12:51,328 INFO [train.py:737] (0/3) Epoch 19, batch 4200, loss[loss=0.44, over 14868.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.007403, over 44.00 utterances.], tot_loss[loss=0.44, over 2827376.95 frames; utt_duration=1214 frames, utt_pad_proportion=0.0711, over 9329.81 utterances.], batch size: 44, lr: 2.84e-04
|
| 322 |
+
2022-08-12 12:14:56,496 INFO [train.py:737] (0/3) Epoch 19, batch 4300, loss[loss=0.4299, over 14764.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005952, over 42.00 utterances.], tot_loss[loss=0.4381, over 2845791.47 frames; utt_duration=1251 frames, utt_pad_proportion=0.06088, over 9107.44 utterances.], batch size: 42, lr: 2.84e-04
|
| 323 |
+
2022-08-12 12:17:11,263 INFO [train.py:737] (0/3) Epoch 19, batch 4400, loss[loss=0.4014, over 14710.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.009552, over 35.00 utterances.], tot_loss[loss=0.4384, over 2835105.65 frames; utt_duration=1232 frames, utt_pad_proportion=0.06574, over 9218.80 utterances.], batch size: 35, lr: 2.83e-04
|
| 324 |
+
2022-08-12 12:19:20,857 INFO [train.py:737] (0/3) Epoch 19, batch 4500, loss[loss=0.4105, over 14469.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.0104, over 33.00 utterances.], tot_loss[loss=0.4372, over 2815561.52 frames; utt_duration=1198 frames, utt_pad_proportion=0.07803, over 9414.60 utterances.], batch size: 33, lr: 2.83e-04
|
| 325 |
+
2022-08-12 12:21:26,375 INFO [train.py:737] (0/3) Epoch 19, batch 4600, loss[loss=0.4252, over 14762.00 frames; utt_duration=1515 frames, utt_pad_proportion=0.00694, over 39.00 utterances.], tot_loss[loss=0.4366, over 2823291.23 frames; utt_duration=1212 frames, utt_pad_proportion=0.07474, over 9326.20 utterances.], batch size: 39, lr: 2.83e-04
|
| 326 |
+
2022-08-12 12:23:39,331 INFO [train.py:737] (0/3) Epoch 19, batch 4700, loss[loss=0.4884, over 14493.00 frames; utt_duration=1209 frames, utt_pad_proportion=0.01842, over 48.00 utterances.], tot_loss[loss=0.4363, over 2820046.00 frames; utt_duration=1208 frames, utt_pad_proportion=0.07622, over 9345.81 utterances.], batch size: 48, lr: 2.83e-04
|
| 327 |
+
2022-08-12 12:26:00,560 INFO [train.py:737] (0/3) Epoch 19, batch 4800, loss[loss=0.418, over 14725.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.009759, over 35.00 utterances.], tot_loss[loss=0.4358, over 2819857.00 frames; utt_duration=1206 frames, utt_pad_proportion=0.07577, over 9365.14 utterances.], batch size: 35, lr: 2.83e-04
|
| 328 |
+
2022-08-12 12:27:27,855 INFO [train.py:737] (0/3) Epoch 19, batch 4900, loss[loss=0.4021, over 14572.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006386, over 37.00 utterances.], tot_loss[loss=0.4362, over 2831885.49 frames; utt_duration=1229 frames, utt_pad_proportion=0.06949, over 9224.91 utterances.], batch size: 37, lr: 2.83e-04
|
| 329 |
+
2022-08-12 12:29:00,608 INFO [train.py:737] (0/3) Epoch 19, batch 5000, loss[loss=0.5169, over 9858.00 frames; utt_duration=434.8 frames, utt_pad_proportion=0.3351, over 91.00 utterances.], tot_loss[loss=0.4378, over 2825650.03 frames; utt_duration=1208 frames, utt_pad_proportion=0.07237, over 9364.67 utterances.], batch size: 91, lr: 2.83e-04
|
| 330 |
+
2022-08-12 12:30:53,857 INFO [train.py:737] (0/3) Epoch 19, batch 5100, loss[loss=0.4839, over 14467.00 frames; utt_duration=1159 frames, utt_pad_proportion=0.02468, over 50.00 utterances.], tot_loss[loss=0.4352, over 2837597.97 frames; utt_duration=1232 frames, utt_pad_proportion=0.06444, over 9221.08 utterances.], batch size: 50, lr: 2.83e-04
|
| 331 |
+
2022-08-12 12:33:24,375 INFO [train.py:737] (0/3) Epoch 19, batch 5200, loss[loss=0.4915, over 14287.00 frames; utt_duration=1080 frames, utt_pad_proportion=0.04027, over 53.00 utterances.], tot_loss[loss=0.4382, over 2832402.93 frames; utt_duration=1224 frames, utt_pad_proportion=0.06841, over 9270.97 utterances.], batch size: 53, lr: 2.83e-04
|
| 332 |
+
2022-08-12 12:35:41,952 INFO [train.py:737] (0/3) Epoch 19, batch 5300, loss[loss=0.4136, over 14895.00 frames; utt_duration=1356 frames, utt_pad_proportion=0.005269, over 44.00 utterances.], tot_loss[loss=0.4369, over 2838734.83 frames; utt_duration=1230 frames, utt_pad_proportion=0.06478, over 9241.52 utterances.], batch size: 44, lr: 2.83e-04
|
| 333 |
+
2022-08-12 12:37:57,202 INFO [train.py:737] (0/3) Epoch 19, batch 5400, loss[loss=0.4917, over 14883.00 frames; utt_duration=1355 frames, utt_pad_proportion=0.00622, over 44.00 utterances.], tot_loss[loss=0.4372, over 2831455.03 frames; utt_duration=1219 frames, utt_pad_proportion=0.06935, over 9306.24 utterances.], batch size: 44, lr: 2.83e-04
|
| 334 |
+
2022-08-12 12:40:13,259 INFO [train.py:737] (0/3) Epoch 19, batch 5500, loss[loss=0.5396, over 14744.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.006563, over 43.00 utterances.], tot_loss[loss=0.436, over 2843554.46 frames; utt_duration=1238 frames, utt_pad_proportion=0.06236, over 9198.36 utterances.], batch size: 43, lr: 2.83e-04
|
| 335 |
+
2022-08-12 12:42:24,007 INFO [train.py:737] (0/3) Epoch 19, batch 5600, loss[loss=0.4283, over 14759.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.006272, over 42.00 utterances.], tot_loss[loss=0.4364, over 2833839.48 frames; utt_duration=1223 frames, utt_pad_proportion=0.06758, over 9276.47 utterances.], batch size: 42, lr: 2.83e-04
|
| 336 |
+
2022-08-12 12:43:43,252 INFO [train.py:737] (0/3) Epoch 19, batch 5700, loss[loss=0.4846, over 13672.00 frames; utt_duration=960.9 frames, utt_pad_proportion=0.07512, over 57.00 utterances.], tot_loss[loss=0.4365, over 2823179.21 frames; utt_duration=1208 frames, utt_pad_proportion=0.07351, over 9357.72 utterances.], batch size: 57, lr: 2.82e-04
|
| 337 |
+
2022-08-12 12:44:43,218 INFO [train.py:737] (0/3) Epoch 19, batch 5800, loss[loss=0.4303, over 14735.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.006416, over 43.00 utterances.], tot_loss[loss=0.4388, over 2819626.21 frames; utt_duration=1194 frames, utt_pad_proportion=0.07438, over 9457.73 utterances.], batch size: 43, lr: 2.82e-04
|
| 338 |
+
2022-08-12 12:46:17,972 INFO [train.py:737] (0/3) Epoch 19, batch 5900, loss[loss=0.3876, over 14062.00 frames; utt_duration=1816 frames, utt_pad_proportion=0.03349, over 31.00 utterances.], tot_loss[loss=0.4377, over 2825141.33 frames; utt_duration=1212 frames, utt_pad_proportion=0.07055, over 9333.53 utterances.], batch size: 31, lr: 2.82e-04
|
| 339 |
+
2022-08-12 12:48:42,416 INFO [train.py:737] (0/3) Epoch 19, batch 6000, loss[loss=0.4079, over 14615.00 frames; utt_duration=1721 frames, utt_pad_proportion=0.009431, over 34.00 utterances.], tot_loss[loss=0.4331, over 2826752.00 frames; utt_duration=1214 frames, utt_pad_proportion=0.06934, over 9325.32 utterances.], batch size: 34, lr: 2.82e-04
|
| 340 |
+
2022-08-12 12:48:42,417 INFO [train.py:761] (0/3) Computing validation loss
|
| 341 |
+
2022-08-12 12:49:17,789 INFO [train.py:770] (0/3) Epoch 19, validation: loss=0.2887, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 342 |
+
2022-08-12 12:51:45,151 INFO [train.py:737] (0/3) Epoch 19, batch 6100, loss[loss=0.5028, over 9574.00 frames; utt_duration=413.2 frames, utt_pad_proportion=0.3584, over 93.00 utterances.], tot_loss[loss=0.4387, over 2760019.67 frames; utt_duration=1102 frames, utt_pad_proportion=0.1074, over 10029.84 utterances.], batch size: 93, lr: 2.82e-04
|
| 343 |
+
2022-08-12 12:52:04,408 INFO [checkpoint.py:75] (0/3) Saving checkpoint to cnn_ctc/exp_phone/epoch-19.pt
|
| 344 |
+
2022-08-12 12:52:43,852 INFO [train.py:737] (0/3) Epoch 20, batch 0, loss[loss=0.3405, over 14784.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.005411, over 38.00 utterances.], tot_loss[loss=0.3405, over 14784.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.005411, over 38.00 utterances.], batch size: 38, lr: 2.75e-04
|
exp/log/log-train-2022-08-11-18-53-52-1
ADDED
|
@@ -0,0 +1,290 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2022-08-11 18:53:52,369 INFO [train.py:805] (1/3) Training started
|
| 2 |
+
2022-08-11 18:53:52,369 INFO [train.py:806] (1/3) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'env_info': {'k2-version': '1.17', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '7dcabf85e8bf06984c4abab0400ef1322b5ff3df', 'k2-git-date': 'Tue Aug 2 21:22:39 2022', 'lhotse-version': '1.4.0.dev+git.59318ed.clean', 'torch-version': '1.11.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.3', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '6af45c3-dirty', 'icefall-git-date': 'Tue Aug 2 14:47:09 2022', 'icefall-path': '/thspfs/home/acct-hpc/thspzh-hpc1/git/icefall_dev', 'k2-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/k2-1.17.dev20220805+cuda11.3.torch1.11.0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'dgxs-01.cluster.thspzh.com', 'IP address': '192.168.200.101'}, 'world_size': 3, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 25, 'start_epoch': 16, 'start_batch': 0, 'exp_dir': PosixPath('cnn_ctc/exp_phone'), 'lang_dir': 'data/lang_phone', 'bpe_model': 'data/lang_bpe_500/bpe.model', 'initial_lr': 0.003, 'lr_batches': 5000, 'lr_epochs': 6, 'seed': 42, 'print_diagnostics': False, 'save_every_n': 8000, 'keep_last_k': 20, 'average_period': 100, 'use_fp16': False, 'first_out_channels': 8, 'conv_blocks': 3, 'rnn_dim': 0, 'full_libri': True, 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 600, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'input_strategy': 'PrecomputedFeatures'}
|
| 3 |
+
2022-08-11 18:53:52,750 INFO [lexicon.py:176] (1/3) Loading pre-compiled data/lang_phone/Linv.pt
|
| 4 |
+
2022-08-11 18:54:04,903 INFO [train.py:843] (1/3) About to create model
|
| 5 |
+
2022-08-11 18:54:04,935 INFO [checkpoint.py:112] (1/3) Loading checkpoint from cnn_ctc/exp_phone/epoch-15.pt
|
| 6 |
+
2022-08-11 18:54:07,781 INFO [train.py:876] (1/3) Using DDP
|
| 7 |
+
2022-08-11 18:54:11,800 INFO [train.py:884] (1/3) Loading optimizer state dict
|
| 8 |
+
2022-08-11 18:54:12,337 INFO [train.py:892] (1/3) Loading scheduler state dict
|
| 9 |
+
2022-08-11 18:54:12,337 INFO [asr_datamodule.py:409] (1/3) About to get train-clean-100 cuts
|
| 10 |
+
2022-08-11 18:54:12,339 INFO [asr_datamodule.py:416] (1/3) About to get train-clean-360 cuts
|
| 11 |
+
2022-08-11 18:54:12,343 INFO [asr_datamodule.py:423] (1/3) About to get train-other-500 cuts
|
| 12 |
+
2022-08-11 18:54:12,345 INFO [asr_datamodule.py:225] (1/3) Enable MUSAN
|
| 13 |
+
2022-08-11 18:54:12,345 INFO [asr_datamodule.py:226] (1/3) About to get Musan cuts
|
| 14 |
+
2022-08-11 18:54:14,379 INFO [asr_datamodule.py:254] (1/3) Enable SpecAugment
|
| 15 |
+
2022-08-11 18:54:14,379 INFO [asr_datamodule.py:255] (1/3) Time warp factor: 80
|
| 16 |
+
2022-08-11 18:54:14,380 INFO [asr_datamodule.py:267] (1/3) Num frame mask: 10
|
| 17 |
+
2022-08-11 18:54:14,380 INFO [asr_datamodule.py:280] (1/3) About to create train dataset
|
| 18 |
+
2022-08-11 18:54:14,380 INFO [asr_datamodule.py:309] (1/3) Using DynamicBucketingSampler.
|
| 19 |
+
2022-08-11 18:54:16,680 INFO [asr_datamodule.py:324] (1/3) About to create train dataloader
|
| 20 |
+
2022-08-11 18:54:16,680 INFO [asr_datamodule.py:430] (1/3) About to get dev-clean cuts
|
| 21 |
+
2022-08-11 18:54:16,683 INFO [asr_datamodule.py:437] (1/3) About to get dev-other cuts
|
| 22 |
+
2022-08-11 18:54:16,684 INFO [asr_datamodule.py:355] (1/3) About to create dev dataset
|
| 23 |
+
2022-08-11 18:54:16,984 INFO [asr_datamodule.py:374] (1/3) About to create dev dataloader
|
| 24 |
+
2022-08-11 18:54:16,984 INFO [train.py:959] (1/3) Loading grad scaler state dict
|
| 25 |
+
2022-08-11 18:54:39,536 INFO [train.py:737] (1/3) Epoch 16, batch 0, loss[loss=0.438, over 9397.00 frames; utt_duration=410.2 frames, utt_pad_proportion=0.368, over 92.00 utterances.], tot_loss[loss=0.438, over 9397.00 frames; utt_duration=410.2 frames, utt_pad_proportion=0.368, over 92.00 utterances.], batch size: 92, lr: 3.29e-04
|
| 26 |
+
2022-08-11 18:54:39,572 INFO [distributed.py:948] (1/3) Reducer buckets have been rebuilt in this iteration.
|
| 27 |
+
2022-08-11 19:00:36,848 INFO [train.py:737] (1/3) Epoch 16, batch 100, loss[loss=0.4313, over 10034.00 frames; utt_duration=442.6 frames, utt_pad_proportion=0.3221, over 91.00 utterances.], tot_loss[loss=0.366, over 1126386.17 frames; utt_duration=1256 frames, utt_pad_proportion=0.06815, over 3592.53 utterances.], batch size: 91, lr: 3.29e-04
|
| 28 |
+
2022-08-11 19:06:33,773 INFO [train.py:737] (1/3) Epoch 16, batch 200, loss[loss=0.3787, over 14606.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.01056, over 34.00 utterances.], tot_loss[loss=0.3607, over 1824020.41 frames; utt_duration=1316 frames, utt_pad_proportion=0.04582, over 5551.32 utterances.], batch size: 34, lr: 3.29e-04
|
| 29 |
+
2022-08-11 19:12:27,341 INFO [train.py:737] (1/3) Epoch 16, batch 300, loss[loss=0.3926, over 14574.00 frames; utt_duration=1496 frames, utt_pad_proportion=0.006082, over 39.00 utterances.], tot_loss[loss=0.3584, over 2232241.86 frames; utt_duration=1294 frames, utt_pad_proportion=0.04829, over 6909.01 utterances.], batch size: 39, lr: 3.29e-04
|
| 30 |
+
2022-08-11 19:18:24,972 INFO [train.py:737] (1/3) Epoch 16, batch 400, loss[loss=0.3744, over 14759.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005201, over 41.00 utterances.], tot_loss[loss=0.3573, over 2492730.21 frames; utt_duration=1327 frames, utt_pad_proportion=0.04071, over 7524.36 utterances.], batch size: 41, lr: 3.29e-04
|
| 31 |
+
2022-08-11 19:24:20,561 INFO [train.py:737] (1/3) Epoch 16, batch 500, loss[loss=0.377, over 14764.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006788, over 39.00 utterances.], tot_loss[loss=0.355, over 2647482.59 frames; utt_duration=1330 frames, utt_pad_proportion=0.03991, over 7973.78 utterances.], batch size: 39, lr: 3.29e-04
|
| 32 |
+
2022-08-11 19:30:14,321 INFO [train.py:737] (1/3) Epoch 16, batch 600, loss[loss=0.3831, over 14566.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006744, over 40.00 utterances.], tot_loss[loss=0.3549, over 2744830.44 frames; utt_duration=1339 frames, utt_pad_proportion=0.03666, over 8208.26 utterances.], batch size: 40, lr: 3.28e-04
|
| 33 |
+
2022-08-11 19:36:07,466 INFO [train.py:737] (1/3) Epoch 16, batch 700, loss[loss=0.3551, over 14714.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.01069, over 46.00 utterances.], tot_loss[loss=0.358, over 2788426.29 frames; utt_duration=1315 frames, utt_pad_proportion=0.04411, over 8493.43 utterances.], batch size: 46, lr: 3.28e-04
|
| 34 |
+
2022-08-11 19:42:04,427 INFO [train.py:737] (1/3) Epoch 16, batch 800, loss[loss=0.3786, over 14877.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.007481, over 44.00 utterances.], tot_loss[loss=0.3617, over 2813079.58 frames; utt_duration=1292 frames, utt_pad_proportion=0.05005, over 8716.42 utterances.], batch size: 44, lr: 3.28e-04
|
| 35 |
+
2022-08-11 19:48:01,465 INFO [train.py:737] (1/3) Epoch 16, batch 900, loss[loss=0.3301, over 14756.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.00654, over 40.00 utterances.], tot_loss[loss=0.3614, over 2839157.16 frames; utt_duration=1297 frames, utt_pad_proportion=0.04735, over 8765.15 utterances.], batch size: 40, lr: 3.28e-04
|
| 36 |
+
2022-08-11 19:54:00,644 INFO [train.py:737] (1/3) Epoch 16, batch 1000, loss[loss=0.3958, over 14567.00 frames; utt_duration=1620 frames, utt_pad_proportion=0.008097, over 36.00 utterances.], tot_loss[loss=0.361, over 2857040.31 frames; utt_duration=1310 frames, utt_pad_proportion=0.04353, over 8734.37 utterances.], batch size: 36, lr: 3.28e-04
|
| 37 |
+
2022-08-11 19:59:54,048 INFO [train.py:737] (1/3) Epoch 16, batch 1100, loss[loss=0.3261, over 14559.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.005781, over 40.00 utterances.], tot_loss[loss=0.3618, over 2859679.60 frames; utt_duration=1292 frames, utt_pad_proportion=0.04601, over 8861.45 utterances.], batch size: 40, lr: 3.28e-04
|
| 38 |
+
2022-08-11 20:05:56,451 INFO [train.py:737] (1/3) Epoch 16, batch 1200, loss[loss=0.3782, over 14727.00 frames; utt_duration=1282 frames, utt_pad_proportion=0.00919, over 46.00 utterances.], tot_loss[loss=0.3623, over 2864114.94 frames; utt_duration=1293 frames, utt_pad_proportion=0.0463, over 8868.58 utterances.], batch size: 46, lr: 3.28e-04
|
| 39 |
+
2022-08-11 20:11:53,135 INFO [train.py:737] (1/3) Epoch 16, batch 1300, loss[loss=0.3988, over 14690.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.008879, over 45.00 utterances.], tot_loss[loss=0.3619, over 2870973.73 frames; utt_duration=1302 frames, utt_pad_proportion=0.04291, over 8830.69 utterances.], batch size: 45, lr: 3.28e-04
|
| 40 |
+
2022-08-11 20:17:51,405 INFO [train.py:737] (1/3) Epoch 16, batch 1400, loss[loss=0.3078, over 14462.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01088, over 33.00 utterances.], tot_loss[loss=0.3632, over 2860129.78 frames; utt_duration=1279 frames, utt_pad_proportion=0.05173, over 8957.37 utterances.], batch size: 33, lr: 3.28e-04
|
| 41 |
+
2022-08-11 20:23:48,378 INFO [train.py:737] (1/3) Epoch 16, batch 1500, loss[loss=0.3132, over 14820.00 frames; utt_duration=1648 frames, utt_pad_proportion=0.01054, over 36.00 utterances.], tot_loss[loss=0.3628, over 2857513.37 frames; utt_duration=1265 frames, utt_pad_proportion=0.05289, over 9046.39 utterances.], batch size: 36, lr: 3.27e-04
|
| 42 |
+
2022-08-11 20:29:43,720 INFO [train.py:737] (1/3) Epoch 16, batch 1600, loss[loss=0.3565, over 14661.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01245, over 47.00 utterances.], tot_loss[loss=0.3642, over 2846229.86 frames; utt_duration=1237 frames, utt_pad_proportion=0.06087, over 9214.58 utterances.], batch size: 47, lr: 3.27e-04
|
| 43 |
+
2022-08-11 20:35:44,437 INFO [train.py:737] (1/3) Epoch 16, batch 1700, loss[loss=0.3552, over 14562.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.006407, over 37.00 utterances.], tot_loss[loss=0.3611, over 2856646.65 frames; utt_duration=1264 frames, utt_pad_proportion=0.05579, over 9053.39 utterances.], batch size: 37, lr: 3.27e-04
|
| 44 |
+
2022-08-11 20:41:36,433 INFO [train.py:737] (1/3) Epoch 16, batch 1800, loss[loss=0.3292, over 14734.00 frames; utt_duration=1439 frames, utt_pad_proportion=0.006834, over 41.00 utterances.], tot_loss[loss=0.3582, over 2865608.19 frames; utt_duration=1282 frames, utt_pad_proportion=0.04802, over 8950.47 utterances.], batch size: 41, lr: 3.27e-04
|
| 45 |
+
2022-08-11 20:47:35,924 INFO [train.py:737] (1/3) Epoch 16, batch 1900, loss[loss=0.3355, over 14774.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.006167, over 39.00 utterances.], tot_loss[loss=0.3595, over 2868885.62 frames; utt_duration=1288 frames, utt_pad_proportion=0.04593, over 8917.95 utterances.], batch size: 39, lr: 3.27e-04
|
| 46 |
+
2022-08-11 20:53:36,131 INFO [train.py:737] (1/3) Epoch 16, batch 2000, loss[loss=0.3317, over 14768.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006267, over 39.00 utterances.], tot_loss[loss=0.3582, over 2862998.99 frames; utt_duration=1285 frames, utt_pad_proportion=0.04935, over 8919.15 utterances.], batch size: 39, lr: 3.27e-04
|
| 47 |
+
2022-08-11 20:59:32,538 INFO [train.py:737] (1/3) Epoch 16, batch 2100, loss[loss=0.3186, over 14597.00 frames; utt_duration=1623 frames, utt_pad_proportion=0.007946, over 36.00 utterances.], tot_loss[loss=0.3612, over 2865958.29 frames; utt_duration=1285 frames, utt_pad_proportion=0.04791, over 8934.86 utterances.], batch size: 36, lr: 3.27e-04
|
| 48 |
+
2022-08-11 21:05:29,583 INFO [train.py:737] (1/3) Epoch 16, batch 2200, loss[loss=0.361, over 14767.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005166, over 42.00 utterances.], tot_loss[loss=0.3617, over 2866519.89 frames; utt_duration=1273 frames, utt_pad_proportion=0.04798, over 9019.33 utterances.], batch size: 42, lr: 3.27e-04
|
| 49 |
+
2022-08-11 21:11:25,897 INFO [train.py:737] (1/3) Epoch 16, batch 2300, loss[loss=0.3833, over 14361.00 frames; utt_duration=1085 frames, utt_pad_proportion=0.03618, over 53.00 utterances.], tot_loss[loss=0.3614, over 2857465.57 frames; utt_duration=1263 frames, utt_pad_proportion=0.054, over 9057.62 utterances.], batch size: 53, lr: 3.27e-04
|
| 50 |
+
2022-08-11 21:17:21,665 INFO [train.py:737] (1/3) Epoch 16, batch 2400, loss[loss=0.3623, over 14647.00 frames; utt_duration=1248 frames, utt_pad_proportion=0.01193, over 47.00 utterances.], tot_loss[loss=0.3623, over 2860378.84 frames; utt_duration=1265 frames, utt_pad_proportion=0.05092, over 9053.18 utterances.], batch size: 47, lr: 3.27e-04
|
| 51 |
+
2022-08-11 21:23:16,092 INFO [train.py:737] (1/3) Epoch 16, batch 2500, loss[loss=0.3633, over 14573.00 frames; utt_duration=1496 frames, utt_pad_proportion=0.005878, over 39.00 utterances.], tot_loss[loss=0.362, over 2854211.60 frames; utt_duration=1264 frames, utt_pad_proportion=0.05521, over 9045.26 utterances.], batch size: 39, lr: 3.26e-04
|
| 52 |
+
2022-08-11 21:29:14,051 INFO [train.py:737] (1/3) Epoch 16, batch 2600, loss[loss=0.4054, over 14583.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.006178, over 41.00 utterances.], tot_loss[loss=0.3609, over 2866055.48 frames; utt_duration=1294 frames, utt_pad_proportion=0.04775, over 8871.32 utterances.], batch size: 41, lr: 3.26e-04
|
| 53 |
+
2022-08-11 21:35:11,491 INFO [train.py:737] (1/3) Epoch 16, batch 2700, loss[loss=0.3469, over 14717.00 frames; utt_duration=1370 frames, utt_pad_proportion=0.008397, over 43.00 utterances.], tot_loss[loss=0.3636, over 2872373.18 frames; utt_duration=1294 frames, utt_pad_proportion=0.04404, over 8888.91 utterances.], batch size: 43, lr: 3.26e-04
|
| 54 |
+
2022-08-11 21:41:06,000 INFO [train.py:737] (1/3) Epoch 16, batch 2800, loss[loss=0.4284, over 14565.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.006897, over 37.00 utterances.], tot_loss[loss=0.3661, over 2853087.55 frames; utt_duration=1260 frames, utt_pad_proportion=0.05621, over 9069.84 utterances.], batch size: 37, lr: 3.26e-04
|
| 55 |
+
2022-08-11 21:46:58,170 INFO [train.py:737] (1/3) Epoch 16, batch 2900, loss[loss=0.4331, over 14710.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.01098, over 46.00 utterances.], tot_loss[loss=0.382, over 2863909.00 frames; utt_duration=1287 frames, utt_pad_proportion=0.04954, over 8912.73 utterances.], batch size: 46, lr: 3.26e-04
|
| 56 |
+
2022-08-11 21:52:53,677 INFO [train.py:737] (1/3) Epoch 16, batch 3000, loss[loss=0.5314, over 14704.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01163, over 46.00 utterances.], tot_loss[loss=0.3962, over 2867137.66 frames; utt_duration=1294 frames, utt_pad_proportion=0.04894, over 8874.48 utterances.], batch size: 46, lr: 3.26e-04
|
| 57 |
+
2022-08-11 21:52:53,677 INFO [train.py:761] (1/3) Computing validation loss
|
| 58 |
+
2022-08-11 21:54:35,585 INFO [train.py:770] (1/3) Epoch 16, validation: loss=0.2879, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 59 |
+
2022-08-11 22:00:26,547 INFO [train.py:737] (1/3) Epoch 16, batch 3100, loss[loss=0.396, over 14784.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.005327, over 38.00 utterances.], tot_loss[loss=0.4093, over 2862675.93 frames; utt_duration=1285 frames, utt_pad_proportion=0.05086, over 8920.29 utterances.], batch size: 38, lr: 3.26e-04
|
| 60 |
+
2022-08-11 22:06:13,235 INFO [train.py:737] (1/3) Epoch 16, batch 3200, loss[loss=0.4779, over 14648.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.006622, over 44.00 utterances.], tot_loss[loss=0.4188, over 2850721.39 frames; utt_duration=1263 frames, utt_pad_proportion=0.05742, over 9037.77 utterances.], batch size: 44, lr: 3.26e-04
|
| 61 |
+
2022-08-11 22:12:06,479 INFO [train.py:737] (1/3) Epoch 16, batch 3300, loss[loss=0.3724, over 14613.00 frames; utt_duration=1721 frames, utt_pad_proportion=0.009465, over 34.00 utterances.], tot_loss[loss=0.4278, over 2820855.43 frames; utt_duration=1202 frames, utt_pad_proportion=0.07376, over 9402.38 utterances.], batch size: 34, lr: 3.26e-04
|
| 62 |
+
2022-08-11 22:17:57,675 INFO [train.py:737] (1/3) Epoch 16, batch 3400, loss[loss=0.4005, over 14443.00 frames; utt_duration=1652 frames, utt_pad_proportion=0.007997, over 35.00 utterances.], tot_loss[loss=0.4305, over 2832411.97 frames; utt_duration=1218 frames, utt_pad_proportion=0.06744, over 9309.65 utterances.], batch size: 35, lr: 3.26e-04
|
| 63 |
+
2022-08-11 22:23:49,964 INFO [train.py:737] (1/3) Epoch 16, batch 3500, loss[loss=0.4617, over 13708.00 frames; utt_duration=963.5 frames, utt_pad_proportion=0.07266, over 57.00 utterances.], tot_loss[loss=0.4363, over 2830558.02 frames; utt_duration=1210 frames, utt_pad_proportion=0.06812, over 9367.17 utterances.], batch size: 57, lr: 3.25e-04
|
| 64 |
+
2022-08-11 22:29:40,562 INFO [train.py:737] (1/3) Epoch 16, batch 3600, loss[loss=0.4849, over 14658.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01341, over 47.00 utterances.], tot_loss[loss=0.4381, over 2821513.91 frames; utt_duration=1201 frames, utt_pad_proportion=0.07334, over 9407.19 utterances.], batch size: 47, lr: 3.25e-04
|
| 65 |
+
2022-08-11 22:35:33,288 INFO [train.py:737] (1/3) Epoch 16, batch 3700, loss[loss=0.4401, over 14737.00 frames; utt_duration=1283 frames, utt_pad_proportion=0.009317, over 46.00 utterances.], tot_loss[loss=0.441, over 2836427.71 frames; utt_duration=1228 frames, utt_pad_proportion=0.06567, over 9251.68 utterances.], batch size: 46, lr: 3.25e-04
|
| 66 |
+
2022-08-11 22:41:20,225 INFO [train.py:737] (1/3) Epoch 16, batch 3800, loss[loss=0.4837, over 13905.00 frames; utt_duration=977.2 frames, utt_pad_proportion=0.05952, over 57.00 utterances.], tot_loss[loss=0.4435, over 2825828.64 frames; utt_duration=1206 frames, utt_pad_proportion=0.07119, over 9383.23 utterances.], batch size: 57, lr: 3.25e-04
|
| 67 |
+
2022-08-11 22:47:10,486 INFO [train.py:737] (1/3) Epoch 16, batch 3900, loss[loss=0.5402, over 9890.00 frames; utt_duration=431.8 frames, utt_pad_proportion=0.3358, over 92.00 utterances.], tot_loss[loss=0.4452, over 2833324.43 frames; utt_duration=1217 frames, utt_pad_proportion=0.06777, over 9327.78 utterances.], batch size: 92, lr: 3.25e-04
|
| 68 |
+
2022-08-11 22:53:00,685 INFO [train.py:737] (1/3) Epoch 16, batch 4000, loss[loss=0.4649, over 14640.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.00792, over 44.00 utterances.], tot_loss[loss=0.4463, over 2824294.56 frames; utt_duration=1201 frames, utt_pad_proportion=0.07396, over 9418.33 utterances.], batch size: 44, lr: 3.25e-04
|
| 69 |
+
2022-08-11 22:58:51,025 INFO [train.py:737] (1/3) Epoch 16, batch 4100, loss[loss=0.3861, over 14424.00 frames; utt_duration=1650 frames, utt_pad_proportion=0.01023, over 35.00 utterances.], tot_loss[loss=0.4445, over 2823380.94 frames; utt_duration=1204 frames, utt_pad_proportion=0.07541, over 9394.23 utterances.], batch size: 35, lr: 3.25e-04
|
| 70 |
+
2022-08-11 23:04:44,507 INFO [train.py:737] (1/3) Epoch 16, batch 4200, loss[loss=0.4193, over 14583.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.006515, over 38.00 utterances.], tot_loss[loss=0.4418, over 2830434.57 frames; utt_duration=1227 frames, utt_pad_proportion=0.06999, over 9237.04 utterances.], batch size: 38, lr: 3.25e-04
|
| 71 |
+
2022-08-11 23:10:34,629 INFO [train.py:737] (1/3) Epoch 16, batch 4300, loss[loss=0.4733, over 14538.00 frames; utt_duration=1213 frames, utt_pad_proportion=0.01547, over 48.00 utterances.], tot_loss[loss=0.4427, over 2832293.74 frames; utt_duration=1227 frames, utt_pad_proportion=0.06732, over 9247.96 utterances.], batch size: 48, lr: 3.25e-04
|
| 72 |
+
2022-08-11 23:16:26,642 INFO [train.py:737] (1/3) Epoch 16, batch 4400, loss[loss=0.511, over 10033.00 frames; utt_duration=433.1 frames, utt_pad_proportion=0.3222, over 93.00 utterances.], tot_loss[loss=0.4431, over 2823157.80 frames; utt_duration=1208 frames, utt_pad_proportion=0.07366, over 9358.93 utterances.], batch size: 93, lr: 3.24e-04
|
| 73 |
+
2022-08-11 23:22:14,315 INFO [train.py:737] (1/3) Epoch 16, batch 4500, loss[loss=0.366, over 14437.00 frames; utt_duration=1806 frames, utt_pad_proportion=0.03112, over 32.00 utterances.], tot_loss[loss=0.4418, over 2829683.36 frames; utt_duration=1228 frames, utt_pad_proportion=0.0695, over 9231.08 utterances.], batch size: 32, lr: 3.24e-04
|
| 74 |
+
2022-08-11 23:28:03,896 INFO [train.py:737] (1/3) Epoch 16, batch 4600, loss[loss=0.3903, over 14044.00 frames; utt_duration=1814 frames, utt_pad_proportion=0.04628, over 31.00 utterances.], tot_loss[loss=0.4408, over 2835917.50 frames; utt_duration=1236 frames, utt_pad_proportion=0.0662, over 9186.22 utterances.], batch size: 31, lr: 3.24e-04
|
| 75 |
+
2022-08-11 23:33:56,641 INFO [train.py:737] (1/3) Epoch 16, batch 4700, loss[loss=0.3879, over 14100.00 frames; utt_duration=1821 frames, utt_pad_proportion=0.04874, over 31.00 utterances.], tot_loss[loss=0.4382, over 2838179.95 frames; utt_duration=1243 frames, utt_pad_proportion=0.06561, over 9142.72 utterances.], batch size: 31, lr: 3.24e-04
|
| 76 |
+
2022-08-11 23:39:49,292 INFO [train.py:737] (1/3) Epoch 16, batch 4800, loss[loss=0.4364, over 14695.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.008441, over 45.00 utterances.], tot_loss[loss=0.4378, over 2843077.18 frames; utt_duration=1250 frames, utt_pad_proportion=0.063, over 9109.40 utterances.], batch size: 45, lr: 3.24e-04
|
| 77 |
+
2022-08-11 23:45:40,798 INFO [train.py:737] (1/3) Epoch 16, batch 4900, loss[loss=0.4533, over 14447.00 frames; utt_duration=1653 frames, utt_pad_proportion=0.008587, over 35.00 utterances.], tot_loss[loss=0.4371, over 2835476.43 frames; utt_duration=1238 frames, utt_pad_proportion=0.06806, over 9175.63 utterances.], batch size: 35, lr: 3.24e-04
|
| 78 |
+
2022-08-11 23:51:33,920 INFO [train.py:737] (1/3) Epoch 16, batch 5000, loss[loss=0.4102, over 14574.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006216, over 37.00 utterances.], tot_loss[loss=0.4371, over 2832999.08 frames; utt_duration=1226 frames, utt_pad_proportion=0.07003, over 9255.30 utterances.], batch size: 37, lr: 3.24e-04
|
| 79 |
+
2022-08-11 23:57:27,495 INFO [train.py:737] (1/3) Epoch 16, batch 5100, loss[loss=0.4436, over 13096.00 frames; utt_duration=772 frames, utt_pad_proportion=0.1167, over 68.00 utterances.], tot_loss[loss=0.4384, over 2836272.52 frames; utt_duration=1230 frames, utt_pad_proportion=0.06725, over 9236.93 utterances.], batch size: 68, lr: 3.24e-04
|
| 80 |
+
2022-08-12 00:03:19,189 INFO [train.py:737] (1/3) Epoch 16, batch 5200, loss[loss=0.3946, over 14595.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01112, over 34.00 utterances.], tot_loss[loss=0.4355, over 2844126.38 frames; utt_duration=1247 frames, utt_pad_proportion=0.0618, over 9133.29 utterances.], batch size: 34, lr: 3.24e-04
|
| 81 |
+
2022-08-12 00:09:07,289 INFO [train.py:737] (1/3) Epoch 16, batch 5300, loss[loss=0.4914, over 12849.00 frames; utt_duration=768.7 frames, utt_pad_proportion=0.1324, over 67.00 utterances.], tot_loss[loss=0.4384, over 2839308.41 frames; utt_duration=1235 frames, utt_pad_proportion=0.06318, over 9206.51 utterances.], batch size: 67, lr: 3.24e-04
|
| 82 |
+
2022-08-12 00:15:01,428 INFO [train.py:737] (1/3) Epoch 16, batch 5400, loss[loss=0.4641, over 14881.00 frames; utt_duration=1355 frames, utt_pad_proportion=0.00617, over 44.00 utterances.], tot_loss[loss=0.439, over 2823471.70 frames; utt_duration=1216 frames, utt_pad_proportion=0.07379, over 9301.89 utterances.], batch size: 44, lr: 3.23e-04
|
| 83 |
+
2022-08-12 00:20:52,398 INFO [train.py:737] (1/3) Epoch 16, batch 5500, loss[loss=0.5128, over 14077.00 frames; utt_duration=1818 frames, utt_pad_proportion=0.036, over 31.00 utterances.], tot_loss[loss=0.4405, over 2819269.06 frames; utt_duration=1208 frames, utt_pad_proportion=0.0756, over 9344.12 utterances.], batch size: 31, lr: 3.23e-04
|
| 84 |
+
2022-08-12 00:26:40,782 INFO [train.py:737] (1/3) Epoch 16, batch 5600, loss[loss=0.4491, over 14352.00 frames; utt_duration=1084 frames, utt_pad_proportion=0.03606, over 53.00 utterances.], tot_loss[loss=0.4431, over 2809101.98 frames; utt_duration=1182 frames, utt_pad_proportion=0.08134, over 9518.15 utterances.], batch size: 53, lr: 3.23e-04
|
| 85 |
+
2022-08-12 00:32:31,458 INFO [train.py:737] (1/3) Epoch 16, batch 5700, loss[loss=0.4558, over 14690.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009478, over 45.00 utterances.], tot_loss[loss=0.442, over 2817870.37 frames; utt_duration=1191 frames, utt_pad_proportion=0.07637, over 9477.06 utterances.], batch size: 45, lr: 3.23e-04
|
| 86 |
+
2022-08-12 00:38:21,055 INFO [train.py:737] (1/3) Epoch 16, batch 5800, loss[loss=0.4402, over 14749.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.005875, over 41.00 utterances.], tot_loss[loss=0.4412, over 2815134.83 frames; utt_duration=1190 frames, utt_pad_proportion=0.07805, over 9477.88 utterances.], batch size: 41, lr: 3.23e-04
|
| 87 |
+
2022-08-12 00:44:13,575 INFO [train.py:737] (1/3) Epoch 16, batch 5900, loss[loss=0.5286, over 14687.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.008862, over 45.00 utterances.], tot_loss[loss=0.4416, over 2827049.45 frames; utt_duration=1211 frames, utt_pad_proportion=0.07148, over 9352.83 utterances.], batch size: 45, lr: 3.23e-04
|
| 88 |
+
2022-08-12 00:50:02,629 INFO [train.py:737] (1/3) Epoch 16, batch 6000, loss[loss=0.4511, over 14397.00 frames; utt_duration=1088 frames, utt_pad_proportion=0.03376, over 53.00 utterances.], tot_loss[loss=0.4386, over 2836588.28 frames; utt_duration=1229 frames, utt_pad_proportion=0.06542, over 9245.87 utterances.], batch size: 53, lr: 3.23e-04
|
| 89 |
+
2022-08-12 00:50:02,629 INFO [train.py:761] (1/3) Computing validation loss
|
| 90 |
+
2022-08-12 00:51:42,069 INFO [train.py:770] (1/3) Epoch 16, validation: loss=0.2945, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 91 |
+
2022-08-12 00:57:06,529 INFO [train.py:737] (1/3) Epoch 16, batch 6100, loss[loss=0.5516, over 8946.00 frames; utt_duration=394.7 frames, utt_pad_proportion=0.3956, over 91.00 utterances.], tot_loss[loss=0.4451, over 2774716.39 frames; utt_duration=1119 frames, utt_pad_proportion=0.1015, over 9928.97 utterances.], batch size: 91, lr: 3.23e-04
|
| 92 |
+
2022-08-12 00:58:41,882 INFO [train.py:737] (1/3) Epoch 17, batch 0, loss[loss=0.3986, over 14760.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.005168, over 41.00 utterances.], tot_loss[loss=0.3986, over 14760.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.005168, over 41.00 utterances.], batch size: 41, lr: 3.14e-04
|
| 93 |
+
2022-08-12 01:04:37,183 INFO [train.py:737] (1/3) Epoch 17, batch 100, loss[loss=0.3183, over 14760.00 frames; utt_duration=1478 frames, utt_pad_proportion=0.006254, over 40.00 utterances.], tot_loss[loss=0.3669, over 1133663.13 frames; utt_duration=1252 frames, utt_pad_proportion=0.05542, over 3625.20 utterances.], batch size: 40, lr: 3.14e-04
|
| 94 |
+
2022-08-12 01:10:31,074 INFO [train.py:737] (1/3) Epoch 17, batch 200, loss[loss=0.3531, over 14710.00 frames; utt_duration=1309 frames, utt_pad_proportion=0.007447, over 45.00 utterances.], tot_loss[loss=0.3613, over 1807133.63 frames; utt_duration=1249 frames, utt_pad_proportion=0.06141, over 5793.14 utterances.], batch size: 45, lr: 3.14e-04
|
| 95 |
+
2022-08-12 01:16:27,828 INFO [train.py:737] (1/3) Epoch 17, batch 300, loss[loss=0.3855, over 13814.00 frames; utt_duration=970.9 frames, utt_pad_proportion=0.06195, over 57.00 utterances.], tot_loss[loss=0.3578, over 2229197.86 frames; utt_duration=1279 frames, utt_pad_proportion=0.05095, over 6978.14 utterances.], batch size: 57, lr: 3.14e-04
|
| 96 |
+
2022-08-12 01:22:20,487 INFO [train.py:737] (1/3) Epoch 17, batch 400, loss[loss=0.3551, over 14716.00 frames; utt_duration=1282 frames, utt_pad_proportion=0.01042, over 46.00 utterances.], tot_loss[loss=0.3572, over 2474186.45 frames; utt_duration=1276 frames, utt_pad_proportion=0.056, over 7762.15 utterances.], batch size: 46, lr: 3.13e-04
|
| 97 |
+
2022-08-12 01:28:17,405 INFO [train.py:737] (1/3) Epoch 17, batch 500, loss[loss=0.2782, over 14725.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.01041, over 35.00 utterances.], tot_loss[loss=0.3546, over 2628362.50 frames; utt_duration=1283 frames, utt_pad_proportion=0.05282, over 8206.93 utterances.], batch size: 35, lr: 3.13e-04
|
| 98 |
+
2022-08-12 01:34:10,598 INFO [train.py:737] (1/3) Epoch 17, batch 600, loss[loss=0.3514, over 14591.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005025, over 41.00 utterances.], tot_loss[loss=0.3576, over 2724771.37 frames; utt_duration=1286 frames, utt_pad_proportion=0.04838, over 8487.78 utterances.], batch size: 41, lr: 3.13e-04
|
| 99 |
+
2022-08-12 01:40:07,111 INFO [train.py:737] (1/3) Epoch 17, batch 700, loss[loss=0.3885, over 14578.00 frames; utt_duration=1578 frames, utt_pad_proportion=0.005978, over 37.00 utterances.], tot_loss[loss=0.3581, over 2795526.33 frames; utt_duration=1318 frames, utt_pad_proportion=0.03912, over 8496.64 utterances.], batch size: 37, lr: 3.13e-04
|
| 100 |
+
2022-08-12 01:46:04,164 INFO [train.py:737] (1/3) Epoch 17, batch 800, loss[loss=0.3698, over 12928.00 frames; utt_duration=773.3 frames, utt_pad_proportion=0.1292, over 67.00 utterances.], tot_loss[loss=0.3566, over 2825889.10 frames; utt_duration=1327 frames, utt_pad_proportion=0.0404, over 8527.81 utterances.], batch size: 67, lr: 3.13e-04
|
| 101 |
+
2022-08-12 01:52:03,017 INFO [train.py:737] (1/3) Epoch 17, batch 900, loss[loss=0.4379, over 14350.00 frames; utt_duration=1084 frames, utt_pad_proportion=0.03698, over 53.00 utterances.], tot_loss[loss=0.3602, over 2848586.68 frames; utt_duration=1316 frames, utt_pad_proportion=0.03867, over 8667.58 utterances.], batch size: 53, lr: 3.13e-04
|
| 102 |
+
2022-08-12 01:57:59,864 INFO [train.py:737] (1/3) Epoch 17, batch 1000, loss[loss=0.3736, over 14576.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006467, over 42.00 utterances.], tot_loss[loss=0.3601, over 2855445.25 frames; utt_duration=1317 frames, utt_pad_proportion=0.04218, over 8680.43 utterances.], batch size: 42, lr: 3.13e-04
|
| 103 |
+
2022-08-12 02:03:56,401 INFO [train.py:737] (1/3) Epoch 17, batch 1100, loss[loss=0.3624, over 14777.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.006883, over 37.00 utterances.], tot_loss[loss=0.3613, over 2870081.47 frames; utt_duration=1321 frames, utt_pad_proportion=0.03851, over 8702.75 utterances.], batch size: 37, lr: 3.13e-04
|
| 104 |
+
2022-08-12 02:09:56,096 INFO [train.py:737] (1/3) Epoch 17, batch 1200, loss[loss=0.3967, over 14571.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005031, over 40.00 utterances.], tot_loss[loss=0.3594, over 2877741.75 frames; utt_duration=1338 frames, utt_pad_proportion=0.03472, over 8610.33 utterances.], batch size: 40, lr: 3.13e-04
|
| 105 |
+
2022-08-12 02:15:53,094 INFO [train.py:737] (1/3) Epoch 17, batch 1300, loss[loss=0.3934, over 14695.00 frames; utt_duration=1279 frames, utt_pad_proportion=0.01212, over 46.00 utterances.], tot_loss[loss=0.3603, over 2877825.29 frames; utt_duration=1327 frames, utt_pad_proportion=0.03795, over 8682.20 utterances.], batch size: 46, lr: 3.13e-04
|
| 106 |
+
2022-08-12 02:21:48,044 INFO [train.py:737] (1/3) Epoch 17, batch 1400, loss[loss=0.309, over 14588.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.007225, over 38.00 utterances.], tot_loss[loss=0.3613, over 2880875.82 frames; utt_duration=1317 frames, utt_pad_proportion=0.03702, over 8757.14 utterances.], batch size: 38, lr: 3.12e-04
|
| 107 |
+
2022-08-12 02:27:45,481 INFO [train.py:737] (1/3) Epoch 17, batch 1500, loss[loss=0.4049, over 13119.00 frames; utt_duration=784.8 frames, utt_pad_proportion=0.1142, over 67.00 utterances.], tot_loss[loss=0.361, over 2884186.82 frames; utt_duration=1320 frames, utt_pad_proportion=0.0363, over 8748.05 utterances.], batch size: 67, lr: 3.12e-04
|
| 108 |
+
2022-08-12 02:33:41,558 INFO [train.py:737] (1/3) Epoch 17, batch 1600, loss[loss=0.3486, over 14766.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.004713, over 41.00 utterances.], tot_loss[loss=0.3616, over 2875863.27 frames; utt_duration=1292 frames, utt_pad_proportion=0.04309, over 8912.51 utterances.], batch size: 41, lr: 3.12e-04
|
| 109 |
+
2022-08-12 02:39:40,259 INFO [train.py:737] (1/3) Epoch 17, batch 1700, loss[loss=0.4617, over 14776.00 frames; utt_duration=1409 frames, utt_pad_proportion=0.005229, over 42.00 utterances.], tot_loss[loss=0.361, over 2877263.95 frames; utt_duration=1298 frames, utt_pad_proportion=0.04201, over 8875.23 utterances.], batch size: 42, lr: 3.12e-04
|
| 110 |
+
2022-08-12 02:45:36,012 INFO [train.py:737] (1/3) Epoch 17, batch 1800, loss[loss=0.331, over 13654.00 frames; utt_duration=1822 frames, utt_pad_proportion=0.07462, over 30.00 utterances.], tot_loss[loss=0.3614, over 2864510.38 frames; utt_duration=1276 frames, utt_pad_proportion=0.0502, over 8990.43 utterances.], batch size: 30, lr: 3.12e-04
|
| 111 |
+
2022-08-12 02:51:32,198 INFO [train.py:737] (1/3) Epoch 17, batch 1900, loss[loss=0.3724, over 12990.00 frames; utt_duration=777 frames, utt_pad_proportion=0.1231, over 67.00 utterances.], tot_loss[loss=0.3613, over 2848452.32 frames; utt_duration=1249 frames, utt_pad_proportion=0.05971, over 9131.69 utterances.], batch size: 67, lr: 3.12e-04
|
| 112 |
+
2022-08-12 02:57:29,849 INFO [train.py:737] (1/3) Epoch 17, batch 2000, loss[loss=0.4231, over 14536.00 frames; utt_duration=1213 frames, utt_pad_proportion=0.01569, over 48.00 utterances.], tot_loss[loss=0.3597, over 2861121.74 frames; utt_duration=1281 frames, utt_pad_proportion=0.05196, over 8945.75 utterances.], batch size: 48, lr: 3.12e-04
|
| 113 |
+
2022-08-12 03:03:23,357 INFO [train.py:737] (1/3) Epoch 17, batch 2100, loss[loss=0.382, over 14648.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.00748, over 44.00 utterances.], tot_loss[loss=0.3613, over 2862918.30 frames; utt_duration=1276 frames, utt_pad_proportion=0.05026, over 8988.59 utterances.], batch size: 44, lr: 3.12e-04
|
| 114 |
+
2022-08-12 03:09:19,730 INFO [train.py:737] (1/3) Epoch 17, batch 2200, loss[loss=0.367, over 13187.00 frames; utt_duration=776.9 frames, utt_pad_proportion=0.1141, over 68.00 utterances.], tot_loss[loss=0.3631, over 2856493.18 frames; utt_duration=1260 frames, utt_pad_proportion=0.05423, over 9081.88 utterances.], batch size: 68, lr: 3.12e-04
|
| 115 |
+
2022-08-12 03:15:11,265 INFO [train.py:737] (1/3) Epoch 17, batch 2300, loss[loss=0.403, over 14522.00 frames; utt_duration=1212 frames, utt_pad_proportion=0.01569, over 48.00 utterances.], tot_loss[loss=0.3636, over 2854242.89 frames; utt_duration=1259 frames, utt_pad_proportion=0.05636, over 9080.49 utterances.], batch size: 48, lr: 3.12e-04
|
| 116 |
+
2022-08-12 03:21:06,346 INFO [train.py:737] (1/3) Epoch 17, batch 2400, loss[loss=0.3131, over 14600.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.008532, over 34.00 utterances.], tot_loss[loss=0.3612, over 2861297.69 frames; utt_duration=1279 frames, utt_pad_proportion=0.05208, over 8958.83 utterances.], batch size: 34, lr: 3.12e-04
|
| 117 |
+
2022-08-12 03:27:05,204 INFO [train.py:737] (1/3) Epoch 17, batch 2500, loss[loss=0.35, over 14773.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.005448, over 38.00 utterances.], tot_loss[loss=0.3583, over 2869937.73 frames; utt_duration=1294 frames, utt_pad_proportion=0.04619, over 8884.28 utterances.], batch size: 38, lr: 3.11e-04
|
| 118 |
+
2022-08-12 03:33:01,499 INFO [train.py:737] (1/3) Epoch 17, batch 2600, loss[loss=0.4027, over 9615.00 frames; utt_duration=424.4 frames, utt_pad_proportion=0.3472, over 91.00 utterances.], tot_loss[loss=0.3608, over 2860617.62 frames; utt_duration=1273 frames, utt_pad_proportion=0.05217, over 9002.43 utterances.], batch size: 91, lr: 3.11e-04
|
| 119 |
+
2022-08-12 03:38:55,048 INFO [train.py:737] (1/3) Epoch 17, batch 2700, loss[loss=0.3539, over 14477.00 frames; utt_duration=1811 frames, utt_pad_proportion=0.03006, over 32.00 utterances.], tot_loss[loss=0.3632, over 2864715.07 frames; utt_duration=1267 frames, utt_pad_proportion=0.04923, over 9056.41 utterances.], batch size: 32, lr: 3.11e-04
|
| 120 |
+
2022-08-12 03:44:52,668 INFO [train.py:737] (1/3) Epoch 17, batch 2800, loss[loss=0.3204, over 14583.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.005384, over 39.00 utterances.], tot_loss[loss=0.3613, over 2857384.33 frames; utt_duration=1263 frames, utt_pad_proportion=0.05395, over 9061.50 utterances.], batch size: 39, lr: 3.11e-04
|
| 121 |
+
2022-08-12 03:50:46,295 INFO [train.py:737] (1/3) Epoch 17, batch 2900, loss[loss=0.4171, over 14572.00 frames; utt_duration=1423 frames, utt_pad_proportion=0.006995, over 41.00 utterances.], tot_loss[loss=0.378, over 2857989.61 frames; utt_duration=1267 frames, utt_pad_proportion=0.05212, over 9034.08 utterances.], batch size: 41, lr: 3.11e-04
|
| 122 |
+
2022-08-12 03:56:37,635 INFO [train.py:737] (1/3) Epoch 17, batch 3000, loss[loss=0.4732, over 14594.00 frames; utt_duration=1392 frames, utt_pad_proportion=0.005361, over 42.00 utterances.], tot_loss[loss=0.3969, over 2856037.07 frames; utt_duration=1263 frames, utt_pad_proportion=0.05426, over 9058.55 utterances.], batch size: 42, lr: 3.11e-04
|
| 123 |
+
2022-08-12 03:56:37,635 INFO [train.py:761] (1/3) Computing validation loss
|
| 124 |
+
2022-08-12 03:58:13,087 INFO [train.py:770] (1/3) Epoch 17, validation: loss=0.2805, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 125 |
+
2022-08-12 04:04:03,705 INFO [train.py:737] (1/3) Epoch 17, batch 3100, loss[loss=0.3727, over 14602.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.006205, over 38.00 utterances.], tot_loss[loss=0.4085, over 2834687.57 frames; utt_duration=1221 frames, utt_pad_proportion=0.06735, over 9298.81 utterances.], batch size: 38, lr: 3.11e-04
|
| 126 |
+
2022-08-12 04:09:52,795 INFO [train.py:737] (1/3) Epoch 17, batch 3200, loss[loss=0.4658, over 14738.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.006214, over 43.00 utterances.], tot_loss[loss=0.4184, over 2824546.31 frames; utt_duration=1202 frames, utt_pad_proportion=0.07472, over 9407.69 utterances.], batch size: 43, lr: 3.11e-04
|
| 127 |
+
2022-08-12 04:15:46,144 INFO [train.py:737] (1/3) Epoch 17, batch 3300, loss[loss=0.3853, over 14595.00 frames; utt_duration=1623 frames, utt_pad_proportion=0.006294, over 36.00 utterances.], tot_loss[loss=0.4263, over 2830351.93 frames; utt_duration=1212 frames, utt_pad_proportion=0.07308, over 9353.86 utterances.], batch size: 36, lr: 3.11e-04
|
| 128 |
+
2022-08-12 04:21:38,717 INFO [train.py:737] (1/3) Epoch 17, batch 3400, loss[loss=0.365, over 14564.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.006135, over 37.00 utterances.], tot_loss[loss=0.4315, over 2829059.01 frames; utt_duration=1213 frames, utt_pad_proportion=0.07059, over 9337.85 utterances.], batch size: 37, lr: 3.11e-04
|
| 129 |
+
2022-08-12 04:27:31,853 INFO [train.py:737] (1/3) Epoch 17, batch 3500, loss[loss=0.468, over 14637.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.007835, over 44.00 utterances.], tot_loss[loss=0.4348, over 2848766.35 frames; utt_duration=1239 frames, utt_pad_proportion=0.05729, over 9205.07 utterances.], batch size: 44, lr: 3.10e-04
|
| 130 |
+
2022-08-12 04:33:19,372 INFO [train.py:737] (1/3) Epoch 17, batch 3600, loss[loss=0.4969, over 14748.00 frames; utt_duration=1406 frames, utt_pad_proportion=0.006293, over 42.00 utterances.], tot_loss[loss=0.4372, over 2834000.63 frames; utt_duration=1215 frames, utt_pad_proportion=0.06844, over 9343.77 utterances.], batch size: 42, lr: 3.10e-04
|
| 131 |
+
2022-08-12 04:39:12,741 INFO [train.py:737] (1/3) Epoch 17, batch 3700, loss[loss=0.4577, over 13054.00 frames; utt_duration=781 frames, utt_pad_proportion=0.1195, over 67.00 utterances.], tot_loss[loss=0.4373, over 2834816.16 frames; utt_duration=1227 frames, utt_pad_proportion=0.06736, over 9254.03 utterances.], batch size: 67, lr: 3.10e-04
|
| 132 |
+
2022-08-12 04:45:00,418 INFO [train.py:737] (1/3) Epoch 17, batch 3800, loss[loss=0.3707, over 13754.00 frames; utt_duration=966.8 frames, utt_pad_proportion=0.0695, over 57.00 utterances.], tot_loss[loss=0.4387, over 2821519.52 frames; utt_duration=1202 frames, utt_pad_proportion=0.07557, over 9398.33 utterances.], batch size: 57, lr: 3.10e-04
|
| 133 |
+
2022-08-12 04:50:52,732 INFO [train.py:737] (1/3) Epoch 17, batch 3900, loss[loss=0.4474, over 14706.00 frames; utt_duration=1682 frames, utt_pad_proportion=0.009787, over 35.00 utterances.], tot_loss[loss=0.4391, over 2839569.34 frames; utt_duration=1226 frames, utt_pad_proportion=0.06513, over 9278.25 utterances.], batch size: 35, lr: 3.10e-04
|
| 134 |
+
2022-08-12 04:56:42,375 INFO [train.py:737] (1/3) Epoch 17, batch 4000, loss[loss=0.5613, over 9894.00 frames; utt_duration=431.5 frames, utt_pad_proportion=0.3371, over 92.00 utterances.], tot_loss[loss=0.44, over 2831851.31 frames; utt_duration=1207 frames, utt_pad_proportion=0.06957, over 9394.39 utterances.], batch size: 92, lr: 3.10e-04
|
| 135 |
+
2022-08-12 05:02:33,028 INFO [train.py:737] (1/3) Epoch 17, batch 4100, loss[loss=0.4561, over 14689.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.008862, over 45.00 utterances.], tot_loss[loss=0.4388, over 2827172.65 frames; utt_duration=1216 frames, utt_pad_proportion=0.07261, over 9311.91 utterances.], batch size: 45, lr: 3.10e-04
|
| 136 |
+
2022-08-12 05:08:23,750 INFO [train.py:737] (1/3) Epoch 17, batch 4200, loss[loss=0.4327, over 14747.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.005793, over 43.00 utterances.], tot_loss[loss=0.4428, over 2816079.19 frames; utt_duration=1189 frames, utt_pad_proportion=0.07652, over 9487.42 utterances.], batch size: 43, lr: 3.10e-04
|
| 137 |
+
2022-08-12 05:14:14,534 INFO [train.py:737] (1/3) Epoch 17, batch 4300, loss[loss=0.4097, over 14736.00 frames; utt_duration=1686 frames, utt_pad_proportion=0.009585, over 35.00 utterances.], tot_loss[loss=0.4372, over 2836509.50 frames; utt_duration=1237 frames, utt_pad_proportion=0.06629, over 9186.78 utterances.], batch size: 35, lr: 3.10e-04
|
| 138 |
+
2022-08-12 05:20:08,615 INFO [train.py:737] (1/3) Epoch 17, batch 4400, loss[loss=0.481, over 14869.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.00722, over 44.00 utterances.], tot_loss[loss=0.4373, over 2837374.35 frames; utt_duration=1240 frames, utt_pad_proportion=0.06607, over 9160.51 utterances.], batch size: 44, lr: 3.10e-04
|
| 139 |
+
2022-08-12 05:26:02,687 INFO [train.py:737] (1/3) Epoch 17, batch 4500, loss[loss=0.4741, over 14778.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.005713, over 39.00 utterances.], tot_loss[loss=0.4376, over 2844116.65 frames; utt_duration=1256 frames, utt_pad_proportion=0.06194, over 9066.38 utterances.], batch size: 39, lr: 3.10e-04
|
| 140 |
+
2022-08-12 05:31:53,044 INFO [train.py:737] (1/3) Epoch 17, batch 4600, loss[loss=0.4375, over 14751.00 frames; utt_duration=1596 frames, utt_pad_proportion=0.008113, over 37.00 utterances.], tot_loss[loss=0.4386, over 2824437.19 frames; utt_duration=1209 frames, utt_pad_proportion=0.0733, over 9359.86 utterances.], batch size: 37, lr: 3.09e-04
|
| 141 |
+
2022-08-12 05:37:40,348 INFO [train.py:737] (1/3) Epoch 17, batch 4700, loss[loss=0.3992, over 14109.00 frames; utt_duration=1822 frames, utt_pad_proportion=0.0406, over 31.00 utterances.], tot_loss[loss=0.4413, over 2813232.78 frames; utt_duration=1180 frames, utt_pad_proportion=0.07992, over 9545.97 utterances.], batch size: 31, lr: 3.09e-04
|
| 142 |
+
2022-08-12 05:43:30,042 INFO [train.py:737] (1/3) Epoch 17, batch 4800, loss[loss=0.4461, over 14753.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.006659, over 42.00 utterances.], tot_loss[loss=0.4431, over 2825944.45 frames; utt_duration=1208 frames, utt_pad_proportion=0.07242, over 9371.53 utterances.], batch size: 42, lr: 3.09e-04
|
| 143 |
+
2022-08-12 05:49:21,887 INFO [train.py:737] (1/3) Epoch 17, batch 4900, loss[loss=0.4306, over 14753.00 frames; utt_duration=1406 frames, utt_pad_proportion=0.006759, over 42.00 utterances.], tot_loss[loss=0.4401, over 2837078.95 frames; utt_duration=1232 frames, utt_pad_proportion=0.06645, over 9226.06 utterances.], batch size: 42, lr: 3.09e-04
|
| 144 |
+
2022-08-12 05:55:09,870 INFO [train.py:737] (1/3) Epoch 17, batch 5000, loss[loss=0.378, over 14770.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.006268, over 38.00 utterances.], tot_loss[loss=0.4403, over 2831317.00 frames; utt_duration=1220 frames, utt_pad_proportion=0.06869, over 9292.42 utterances.], batch size: 38, lr: 3.09e-04
|
| 145 |
+
2022-08-12 06:00:59,659 INFO [train.py:737] (1/3) Epoch 17, batch 5100, loss[loss=0.4972, over 14406.00 frames; utt_duration=1089 frames, utt_pad_proportion=0.03293, over 53.00 utterances.], tot_loss[loss=0.4389, over 2825389.79 frames; utt_duration=1215 frames, utt_pad_proportion=0.0723, over 9315.03 utterances.], batch size: 53, lr: 3.09e-04
|
| 146 |
+
2022-08-12 06:06:51,514 INFO [train.py:737] (1/3) Epoch 17, batch 5200, loss[loss=0.4067, over 14110.00 frames; utt_duration=1823 frames, utt_pad_proportion=0.0473, over 31.00 utterances.], tot_loss[loss=0.4412, over 2814937.07 frames; utt_duration=1199 frames, utt_pad_proportion=0.0783, over 9405.56 utterances.], batch size: 31, lr: 3.09e-04
|
| 147 |
+
2022-08-12 06:12:42,099 INFO [train.py:737] (1/3) Epoch 17, batch 5300, loss[loss=0.4479, over 14494.00 frames; utt_duration=1209 frames, utt_pad_proportion=0.01845, over 48.00 utterances.], tot_loss[loss=0.4398, over 2829499.89 frames; utt_duration=1220 frames, utt_pad_proportion=0.07004, over 9284.87 utterances.], batch size: 48, lr: 3.09e-04
|
| 148 |
+
2022-08-12 06:18:31,727 INFO [train.py:737] (1/3) Epoch 17, batch 5400, loss[loss=0.4326, over 14755.00 frames; utt_duration=1688 frames, utt_pad_proportion=0.007609, over 35.00 utterances.], tot_loss[loss=0.4391, over 2831078.01 frames; utt_duration=1219 frames, utt_pad_proportion=0.06981, over 9302.72 utterances.], batch size: 35, lr: 3.09e-04
|
| 149 |
+
2022-08-12 06:24:22,453 INFO [train.py:737] (1/3) Epoch 17, batch 5500, loss[loss=0.4728, over 14583.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.006127, over 41.00 utterances.], tot_loss[loss=0.4399, over 2823566.95 frames; utt_duration=1206 frames, utt_pad_proportion=0.07425, over 9380.62 utterances.], batch size: 41, lr: 3.09e-04
|
| 150 |
+
2022-08-12 06:30:10,219 INFO [train.py:737] (1/3) Epoch 17, batch 5600, loss[loss=0.4908, over 13811.00 frames; utt_duration=970.6 frames, utt_pad_proportion=0.0649, over 57.00 utterances.], tot_loss[loss=0.4383, over 2837774.40 frames; utt_duration=1234 frames, utt_pad_proportion=0.06648, over 9209.11 utterances.], batch size: 57, lr: 3.08e-04
|
| 151 |
+
2022-08-12 06:35:59,299 INFO [train.py:737] (1/3) Epoch 17, batch 5700, loss[loss=0.421, over 14886.00 frames; utt_duration=1355 frames, utt_pad_proportion=0.006665, over 44.00 utterances.], tot_loss[loss=0.44, over 2823998.94 frames; utt_duration=1208 frames, utt_pad_proportion=0.07336, over 9361.71 utterances.], batch size: 44, lr: 3.08e-04
|
| 152 |
+
2022-08-12 06:41:51,110 INFO [train.py:737] (1/3) Epoch 17, batch 5800, loss[loss=0.4443, over 14481.00 frames; utt_duration=1160 frames, utt_pad_proportion=0.02283, over 50.00 utterances.], tot_loss[loss=0.4412, over 2851407.99 frames; utt_duration=1252 frames, utt_pad_proportion=0.05669, over 9119.38 utterances.], batch size: 50, lr: 3.08e-04
|
| 153 |
+
2022-08-12 06:47:36,189 INFO [train.py:737] (1/3) Epoch 17, batch 5900, loss[loss=0.4415, over 14770.00 frames; utt_duration=1409 frames, utt_pad_proportion=0.00528, over 42.00 utterances.], tot_loss[loss=0.4386, over 2832785.20 frames; utt_duration=1227 frames, utt_pad_proportion=0.06889, over 9245.01 utterances.], batch size: 42, lr: 3.08e-04
|
| 154 |
+
2022-08-12 06:53:24,450 INFO [train.py:737] (1/3) Epoch 17, batch 6000, loss[loss=0.5343, over 10062.00 frames; utt_duration=434.5 frames, utt_pad_proportion=0.3243, over 93.00 utterances.], tot_loss[loss=0.4376, over 2827013.92 frames; utt_duration=1222 frames, utt_pad_proportion=0.0708, over 9261.98 utterances.], batch size: 93, lr: 3.08e-04
|
| 155 |
+
2022-08-12 06:53:24,450 INFO [train.py:761] (1/3) Computing validation loss
|
| 156 |
+
2022-08-12 06:55:07,777 INFO [train.py:770] (1/3) Epoch 17, validation: loss=0.3115, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 157 |
+
2022-08-12 07:00:34,846 INFO [train.py:737] (1/3) Epoch 17, batch 6100, loss[loss=0.5135, over 9697.00 frames; utt_duration=422.9 frames, utt_pad_proportion=0.3503, over 92.00 utterances.], tot_loss[loss=0.4426, over 2763165.99 frames; utt_duration=1110 frames, utt_pad_proportion=0.1079, over 9967.97 utterances.], batch size: 92, lr: 3.08e-04
|
| 158 |
+
2022-08-12 07:02:15,428 INFO [train.py:737] (1/3) Epoch 18, batch 0, loss[loss=0.3682, over 14765.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.006154, over 42.00 utterances.], tot_loss[loss=0.3682, over 14765.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.006154, over 42.00 utterances.], batch size: 42, lr: 3.00e-04
|
| 159 |
+
2022-08-12 07:08:15,725 INFO [train.py:737] (1/3) Epoch 18, batch 100, loss[loss=0.36, over 14586.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.008473, over 36.00 utterances.], tot_loss[loss=0.3556, over 1145836.84 frames; utt_duration=1326 frames, utt_pad_proportion=0.03503, over 3459.91 utterances.], batch size: 36, lr: 3.00e-04
|
| 160 |
+
2022-08-12 07:14:10,523 INFO [train.py:737] (1/3) Epoch 18, batch 200, loss[loss=0.4541, over 14580.00 frames; utt_duration=1536 frames, utt_pad_proportion=0.007582, over 38.00 utterances.], tot_loss[loss=0.3589, over 1824723.85 frames; utt_duration=1296 frames, utt_pad_proportion=0.04209, over 5638.08 utterances.], batch size: 38, lr: 3.00e-04
|
| 161 |
+
2022-08-12 07:20:03,956 INFO [train.py:737] (1/3) Epoch 18, batch 300, loss[loss=0.4021, over 14367.00 frames; utt_duration=1086 frames, utt_pad_proportion=0.03561, over 53.00 utterances.], tot_loss[loss=0.3574, over 2237823.25 frames; utt_duration=1300 frames, utt_pad_proportion=0.04211, over 6894.84 utterances.], batch size: 53, lr: 3.00e-04
|
| 162 |
+
2022-08-12 07:25:56,871 INFO [train.py:737] (1/3) Epoch 18, batch 400, loss[loss=0.3407, over 14491.00 frames; utt_duration=1209 frames, utt_pad_proportion=0.01684, over 48.00 utterances.], tot_loss[loss=0.3581, over 2481109.11 frames; utt_duration=1276 frames, utt_pad_proportion=0.04794, over 7789.53 utterances.], batch size: 48, lr: 3.00e-04
|
| 163 |
+
2022-08-12 07:31:51,513 INFO [train.py:737] (1/3) Epoch 18, batch 500, loss[loss=0.3262, over 14853.00 frames; utt_duration=1652 frames, utt_pad_proportion=0.008587, over 36.00 utterances.], tot_loss[loss=0.3541, over 2637135.00 frames; utt_duration=1288 frames, utt_pad_proportion=0.04336, over 8198.74 utterances.], batch size: 36, lr: 2.99e-04
|
| 164 |
+
2022-08-12 07:37:46,288 INFO [train.py:737] (1/3) Epoch 18, batch 600, loss[loss=0.3597, over 14764.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.004881, over 41.00 utterances.], tot_loss[loss=0.3557, over 2729746.81 frames; utt_duration=1290 frames, utt_pad_proportion=0.04287, over 8474.73 utterances.], batch size: 41, lr: 2.99e-04
|
| 165 |
+
2022-08-12 07:43:44,663 INFO [train.py:737] (1/3) Epoch 18, batch 700, loss[loss=0.3174, over 14766.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007521, over 37.00 utterances.], tot_loss[loss=0.3587, over 2784998.62 frames; utt_duration=1287 frames, utt_pad_proportion=0.04454, over 8667.96 utterances.], batch size: 37, lr: 2.99e-04
|
| 166 |
+
2022-08-12 07:49:44,656 INFO [train.py:737] (1/3) Epoch 18, batch 800, loss[loss=0.3649, over 13973.00 frames; utt_duration=965.1 frames, utt_pad_proportion=0.06661, over 58.00 utterances.], tot_loss[loss=0.3587, over 2826944.87 frames; utt_duration=1309 frames, utt_pad_proportion=0.03948, over 8645.12 utterances.], batch size: 58, lr: 2.99e-04
|
| 167 |
+
2022-08-12 07:55:42,381 INFO [train.py:737] (1/3) Epoch 18, batch 900, loss[loss=0.3227, over 14603.00 frames; utt_duration=1624 frames, utt_pad_proportion=0.007403, over 36.00 utterances.], tot_loss[loss=0.3586, over 2844642.63 frames; utt_duration=1311 frames, utt_pad_proportion=0.04045, over 8692.14 utterances.], batch size: 36, lr: 2.99e-04
|
| 168 |
+
2022-08-12 08:01:36,735 INFO [train.py:737] (1/3) Epoch 18, batch 1000, loss[loss=0.3734, over 14878.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.006736, over 44.00 utterances.], tot_loss[loss=0.3597, over 2855074.98 frames; utt_duration=1303 frames, utt_pad_proportion=0.04301, over 8773.10 utterances.], batch size: 44, lr: 2.99e-04
|
| 169 |
+
2022-08-12 08:07:31,602 INFO [train.py:737] (1/3) Epoch 18, batch 1100, loss[loss=0.3641, over 13797.00 frames; utt_duration=969.6 frames, utt_pad_proportion=0.06675, over 57.00 utterances.], tot_loss[loss=0.3584, over 2875680.28 frames; utt_duration=1337 frames, utt_pad_proportion=0.03389, over 8610.53 utterances.], batch size: 57, lr: 2.99e-04
|
| 170 |
+
2022-08-12 08:13:30,146 INFO [train.py:737] (1/3) Epoch 18, batch 1200, loss[loss=0.3799, over 14582.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.005553, over 41.00 utterances.], tot_loss[loss=0.3587, over 2875637.87 frames; utt_duration=1330 frames, utt_pad_proportion=0.03763, over 8659.05 utterances.], batch size: 41, lr: 2.99e-04
|
| 171 |
+
2022-08-12 08:19:25,569 INFO [train.py:737] (1/3) Epoch 18, batch 1300, loss[loss=0.3483, over 14712.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.01095, over 46.00 utterances.], tot_loss[loss=0.3628, over 2854831.73 frames; utt_duration=1270 frames, utt_pad_proportion=0.05298, over 9001.79 utterances.], batch size: 46, lr: 2.99e-04
|
| 172 |
+
2022-08-12 08:25:25,694 INFO [train.py:737] (1/3) Epoch 18, batch 1400, loss[loss=0.3875, over 14743.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.006428, over 43.00 utterances.], tot_loss[loss=0.3625, over 2859476.01 frames; utt_duration=1284 frames, utt_pad_proportion=0.05112, over 8917.82 utterances.], batch size: 43, lr: 2.99e-04
|
| 173 |
+
2022-08-12 08:31:23,720 INFO [train.py:737] (1/3) Epoch 18, batch 1500, loss[loss=0.4025, over 14483.00 frames; utt_duration=1160 frames, utt_pad_proportion=0.02234, over 50.00 utterances.], tot_loss[loss=0.3626, over 2857573.95 frames; utt_duration=1270 frames, utt_pad_proportion=0.05321, over 9012.27 utterances.], batch size: 50, lr: 2.99e-04
|
| 174 |
+
2022-08-12 08:37:25,240 INFO [train.py:737] (1/3) Epoch 18, batch 1600, loss[loss=0.3134, over 14569.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.005661, over 37.00 utterances.], tot_loss[loss=0.3605, over 2869405.00 frames; utt_duration=1293 frames, utt_pad_proportion=0.04441, over 8886.70 utterances.], batch size: 37, lr: 2.98e-04
|
| 175 |
+
2022-08-12 08:43:19,482 INFO [train.py:737] (1/3) Epoch 18, batch 1700, loss[loss=0.3621, over 14701.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01172, over 46.00 utterances.], tot_loss[loss=0.3606, over 2868068.21 frames; utt_duration=1283 frames, utt_pad_proportion=0.04619, over 8953.19 utterances.], batch size: 46, lr: 2.98e-04
|
| 176 |
+
2022-08-12 08:49:17,872 INFO [train.py:737] (1/3) Epoch 18, batch 1800, loss[loss=0.3819, over 13750.00 frames; utt_duration=966.3 frames, utt_pad_proportion=0.06999, over 57.00 utterances.], tot_loss[loss=0.3618, over 2869138.73 frames; utt_duration=1286 frames, utt_pad_proportion=0.04553, over 8931.39 utterances.], batch size: 57, lr: 2.98e-04
|
| 177 |
+
2022-08-12 08:55:13,602 INFO [train.py:737] (1/3) Epoch 18, batch 1900, loss[loss=0.4208, over 14770.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.004494, over 41.00 utterances.], tot_loss[loss=0.361, over 2865389.83 frames; utt_duration=1279 frames, utt_pad_proportion=0.04857, over 8970.61 utterances.], batch size: 41, lr: 2.98e-04
|
| 178 |
+
2022-08-12 08:59:37,387 INFO [train.py:737] (1/3) Epoch 18, batch 2000, loss[loss=0.3324, over 14722.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.01029, over 35.00 utterances.], tot_loss[loss=0.3592, over 2855542.39 frames; utt_duration=1265 frames, utt_pad_proportion=0.05391, over 9043.34 utterances.], batch size: 35, lr: 2.98e-04
|
| 179 |
+
2022-08-12 09:02:23,584 INFO [train.py:737] (1/3) Epoch 18, batch 2100, loss[loss=0.3342, over 14438.00 frames; utt_duration=1652 frames, utt_pad_proportion=0.009135, over 35.00 utterances.], tot_loss[loss=0.3619, over 2860083.31 frames; utt_duration=1274 frames, utt_pad_proportion=0.05184, over 8990.09 utterances.], batch size: 35, lr: 2.98e-04
|
| 180 |
+
2022-08-12 09:05:26,496 INFO [train.py:737] (1/3) Epoch 18, batch 2200, loss[loss=0.4609, over 9763.00 frames; utt_duration=425.8 frames, utt_pad_proportion=0.3469, over 92.00 utterances.], tot_loss[loss=0.3627, over 2847201.20 frames; utt_duration=1246 frames, utt_pad_proportion=0.05913, over 9148.34 utterances.], batch size: 92, lr: 2.98e-04
|
| 181 |
+
2022-08-12 09:07:43,371 INFO [train.py:737] (1/3) Epoch 18, batch 2300, loss[loss=0.3827, over 14702.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01068, over 46.00 utterances.], tot_loss[loss=0.3629, over 2851051.17 frames; utt_duration=1256 frames, utt_pad_proportion=0.05743, over 9090.09 utterances.], batch size: 46, lr: 2.98e-04
|
| 182 |
+
2022-08-12 09:10:16,147 INFO [train.py:737] (1/3) Epoch 18, batch 2400, loss[loss=0.332, over 14739.00 frames; utt_duration=1686 frames, utt_pad_proportion=0.009485, over 35.00 utterances.], tot_loss[loss=0.3635, over 2853667.13 frames; utt_duration=1272 frames, utt_pad_proportion=0.05635, over 8987.21 utterances.], batch size: 35, lr: 2.98e-04
|
| 183 |
+
2022-08-12 09:12:54,438 INFO [train.py:737] (1/3) Epoch 18, batch 2500, loss[loss=0.3629, over 14579.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.005467, over 42.00 utterances.], tot_loss[loss=0.3618, over 2845582.37 frames; utt_duration=1261 frames, utt_pad_proportion=0.06228, over 9039.24 utterances.], batch size: 42, lr: 2.98e-04
|
| 184 |
+
2022-08-12 09:16:00,773 INFO [train.py:737] (1/3) Epoch 18, batch 2600, loss[loss=0.3356, over 14462.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01078, over 33.00 utterances.], tot_loss[loss=0.3607, over 2847045.76 frames; utt_duration=1260 frames, utt_pad_proportion=0.0611, over 9046.29 utterances.], batch size: 33, lr: 2.98e-04
|
| 185 |
+
2022-08-12 09:18:22,666 INFO [train.py:737] (1/3) Epoch 18, batch 2700, loss[loss=0.2907, over 14451.00 frames; utt_duration=1753 frames, utt_pad_proportion=0.01118, over 33.00 utterances.], tot_loss[loss=0.3613, over 2866687.64 frames; utt_duration=1290 frames, utt_pad_proportion=0.04868, over 8897.48 utterances.], batch size: 33, lr: 2.97e-04
|
| 186 |
+
2022-08-12 09:21:19,620 INFO [train.py:737] (1/3) Epoch 18, batch 2800, loss[loss=0.3318, over 14572.00 frames; utt_duration=1621 frames, utt_pad_proportion=0.009457, over 36.00 utterances.], tot_loss[loss=0.3634, over 2859092.13 frames; utt_duration=1275 frames, utt_pad_proportion=0.05365, over 8976.83 utterances.], batch size: 36, lr: 2.97e-04
|
| 187 |
+
2022-08-12 09:24:12,307 INFO [train.py:737] (1/3) Epoch 18, batch 2900, loss[loss=0.3539, over 14132.00 frames; utt_duration=1825 frames, utt_pad_proportion=0.03485, over 31.00 utterances.], tot_loss[loss=0.3768, over 2868441.58 frames; utt_duration=1303 frames, utt_pad_proportion=0.04682, over 8817.95 utterances.], batch size: 31, lr: 2.97e-04
|
| 188 |
+
2022-08-12 09:26:27,704 INFO [train.py:737] (1/3) Epoch 18, batch 3000, loss[loss=0.4143, over 14587.00 frames; utt_duration=1498 frames, utt_pad_proportion=0.004941, over 39.00 utterances.], tot_loss[loss=0.3899, over 2865062.18 frames; utt_duration=1293 frames, utt_pad_proportion=0.04969, over 8874.47 utterances.], batch size: 39, lr: 2.97e-04
|
| 189 |
+
2022-08-12 09:26:27,704 INFO [train.py:761] (1/3) Computing validation loss
|
| 190 |
+
2022-08-12 09:27:08,102 INFO [train.py:770] (1/3) Epoch 18, validation: loss=0.2876, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 191 |
+
2022-08-12 09:29:25,969 INFO [train.py:737] (1/3) Epoch 18, batch 3100, loss[loss=0.4076, over 14592.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.005971, over 38.00 utterances.], tot_loss[loss=0.4052, over 2852161.48 frames; utt_duration=1263 frames, utt_pad_proportion=0.05775, over 9043.43 utterances.], batch size: 38, lr: 2.97e-04
|
| 192 |
+
2022-08-12 09:32:24,407 INFO [train.py:737] (1/3) Epoch 18, batch 3200, loss[loss=0.4843, over 14880.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.007315, over 44.00 utterances.], tot_loss[loss=0.413, over 2835995.72 frames; utt_duration=1235 frames, utt_pad_proportion=0.06764, over 9199.14 utterances.], batch size: 44, lr: 2.97e-04
|
| 193 |
+
2022-08-12 09:34:49,898 INFO [train.py:737] (1/3) Epoch 18, batch 3300, loss[loss=0.5163, over 12870.00 frames; utt_duration=769.8 frames, utt_pad_proportion=0.1302, over 67.00 utterances.], tot_loss[loss=0.4232, over 2835843.05 frames; utt_duration=1230 frames, utt_pad_proportion=0.06766, over 9236.56 utterances.], batch size: 67, lr: 2.97e-04
|
| 194 |
+
2022-08-12 09:37:01,565 INFO [train.py:737] (1/3) Epoch 18, batch 3400, loss[loss=0.4673, over 14762.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.00505, over 41.00 utterances.], tot_loss[loss=0.43, over 2831259.84 frames; utt_duration=1219 frames, utt_pad_proportion=0.07024, over 9301.88 utterances.], batch size: 41, lr: 2.97e-04
|
| 195 |
+
2022-08-12 09:39:45,278 INFO [train.py:737] (1/3) Epoch 18, batch 3500, loss[loss=0.4632, over 14317.00 frames; utt_duration=1082 frames, utt_pad_proportion=0.03814, over 53.00 utterances.], tot_loss[loss=0.4342, over 2834900.13 frames; utt_duration=1225 frames, utt_pad_proportion=0.06689, over 9266.31 utterances.], batch size: 53, lr: 2.97e-04
|
| 196 |
+
2022-08-12 09:41:58,170 INFO [train.py:737] (1/3) Epoch 18, batch 3600, loss[loss=0.3891, over 14763.00 frames; utt_duration=1478 frames, utt_pad_proportion=0.006002, over 40.00 utterances.], tot_loss[loss=0.4388, over 2834612.95 frames; utt_duration=1232 frames, utt_pad_proportion=0.06696, over 9211.27 utterances.], batch size: 40, lr: 2.97e-04
|
| 197 |
+
2022-08-12 09:44:05,887 INFO [train.py:737] (1/3) Epoch 18, batch 3700, loss[loss=0.4226, over 14678.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.009721, over 45.00 utterances.], tot_loss[loss=0.4406, over 2829068.39 frames; utt_duration=1219 frames, utt_pad_proportion=0.07034, over 9294.23 utterances.], batch size: 45, lr: 2.97e-04
|
| 198 |
+
2022-08-12 09:46:59,242 INFO [train.py:737] (1/3) Epoch 18, batch 3800, loss[loss=0.4251, over 14672.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.0126, over 47.00 utterances.], tot_loss[loss=0.4398, over 2838039.00 frames; utt_duration=1237 frames, utt_pad_proportion=0.06611, over 9186.12 utterances.], batch size: 47, lr: 2.97e-04
|
| 199 |
+
2022-08-12 09:49:10,332 INFO [train.py:737] (1/3) Epoch 18, batch 3900, loss[loss=0.4171, over 14692.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.00936, over 45.00 utterances.], tot_loss[loss=0.4423, over 2821825.24 frames; utt_duration=1200 frames, utt_pad_proportion=0.07575, over 9417.00 utterances.], batch size: 45, lr: 2.96e-04
|
| 200 |
+
2022-08-12 09:51:49,476 INFO [train.py:737] (1/3) Epoch 18, batch 4000, loss[loss=0.4442, over 14702.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01085, over 46.00 utterances.], tot_loss[loss=0.442, over 2808079.53 frames; utt_duration=1181 frames, utt_pad_proportion=0.08309, over 9523.31 utterances.], batch size: 46, lr: 2.96e-04
|
| 201 |
+
2022-08-12 09:54:08,257 INFO [train.py:737] (1/3) Epoch 18, batch 4100, loss[loss=0.3796, over 14778.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.005915, over 39.00 utterances.], tot_loss[loss=0.4426, over 2804313.11 frames; utt_duration=1167 frames, utt_pad_proportion=0.08325, over 9624.78 utterances.], batch size: 39, lr: 2.96e-04
|
| 202 |
+
2022-08-12 09:56:28,402 INFO [train.py:737] (1/3) Epoch 18, batch 4200, loss[loss=0.3869, over 14619.00 frames; utt_duration=1721 frames, utt_pad_proportion=0.009561, over 34.00 utterances.], tot_loss[loss=0.4409, over 2814946.60 frames; utt_duration=1193 frames, utt_pad_proportion=0.07856, over 9450.62 utterances.], batch size: 34, lr: 2.96e-04
|
| 203 |
+
2022-08-12 09:58:21,328 INFO [train.py:737] (1/3) Epoch 18, batch 4300, loss[loss=0.3447, over 14585.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.00773, over 36.00 utterances.], tot_loss[loss=0.44, over 2831291.63 frames; utt_duration=1218 frames, utt_pad_proportion=0.06916, over 9307.75 utterances.], batch size: 36, lr: 2.96e-04
|
| 204 |
+
2022-08-12 10:01:09,185 INFO [train.py:737] (1/3) Epoch 18, batch 4400, loss[loss=0.3966, over 14728.00 frames; utt_duration=1685 frames, utt_pad_proportion=0.009574, over 35.00 utterances.], tot_loss[loss=0.441, over 2832431.11 frames; utt_duration=1219 frames, utt_pad_proportion=0.06709, over 9305.18 utterances.], batch size: 35, lr: 2.96e-04
|
| 205 |
+
2022-08-12 10:04:01,950 INFO [train.py:737] (1/3) Epoch 18, batch 4500, loss[loss=0.4018, over 14034.00 frames; utt_duration=1812 frames, utt_pad_proportion=0.05409, over 31.00 utterances.], tot_loss[loss=0.4418, over 2812821.45 frames; utt_duration=1182 frames, utt_pad_proportion=0.07915, over 9529.56 utterances.], batch size: 31, lr: 2.96e-04
|
| 206 |
+
2022-08-12 10:07:00,421 INFO [train.py:737] (1/3) Epoch 18, batch 4600, loss[loss=0.4281, over 14710.00 frames; utt_duration=1281 frames, utt_pad_proportion=0.009634, over 46.00 utterances.], tot_loss[loss=0.4393, over 2825770.83 frames; utt_duration=1206 frames, utt_pad_proportion=0.07249, over 9382.68 utterances.], batch size: 46, lr: 2.96e-04
|
| 207 |
+
2022-08-12 10:09:39,662 INFO [train.py:737] (1/3) Epoch 18, batch 4700, loss[loss=0.4395, over 14767.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.00657, over 39.00 utterances.], tot_loss[loss=0.4394, over 2838658.96 frames; utt_duration=1226 frames, utt_pad_proportion=0.06555, over 9273.30 utterances.], batch size: 39, lr: 2.96e-04
|
| 208 |
+
2022-08-12 10:12:23,883 INFO [train.py:737] (1/3) Epoch 18, batch 4800, loss[loss=0.4692, over 14880.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.006253, over 44.00 utterances.], tot_loss[loss=0.4395, over 2831019.69 frames; utt_duration=1215 frames, utt_pad_proportion=0.07018, over 9332.63 utterances.], batch size: 44, lr: 2.96e-04
|
| 209 |
+
2022-08-12 10:15:16,744 INFO [train.py:737] (1/3) Epoch 18, batch 4900, loss[loss=0.3536, over 14471.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.00993, over 33.00 utterances.], tot_loss[loss=0.4403, over 2825306.37 frames; utt_duration=1204 frames, utt_pad_proportion=0.07372, over 9394.29 utterances.], batch size: 33, lr: 2.96e-04
|
| 210 |
+
2022-08-12 10:17:43,026 INFO [train.py:737] (1/3) Epoch 18, batch 5000, loss[loss=0.4095, over 14754.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.005755, over 43.00 utterances.], tot_loss[loss=0.4379, over 2832704.38 frames; utt_duration=1216 frames, utt_pad_proportion=0.06759, over 9326.06 utterances.], batch size: 43, lr: 2.95e-04
|
| 211 |
+
2022-08-12 10:19:42,979 INFO [train.py:737] (1/3) Epoch 18, batch 5100, loss[loss=0.4867, over 14880.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.006453, over 44.00 utterances.], tot_loss[loss=0.4357, over 2833241.99 frames; utt_duration=1227 frames, utt_pad_proportion=0.06838, over 9247.11 utterances.], batch size: 44, lr: 2.95e-04
|
| 212 |
+
2022-08-12 10:22:03,880 INFO [train.py:737] (1/3) Epoch 18, batch 5200, loss[loss=0.4852, over 14681.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.009519, over 45.00 utterances.], tot_loss[loss=0.4349, over 2833671.80 frames; utt_duration=1225 frames, utt_pad_proportion=0.06967, over 9267.58 utterances.], batch size: 45, lr: 2.95e-04
|
| 213 |
+
2022-08-12 10:24:26,637 INFO [train.py:737] (1/3) Epoch 18, batch 5300, loss[loss=0.4007, over 14773.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.006534, over 37.00 utterances.], tot_loss[loss=0.4343, over 2815689.09 frames; utt_duration=1196 frames, utt_pad_proportion=0.07774, over 9428.49 utterances.], batch size: 37, lr: 2.95e-04
|
| 214 |
+
2022-08-12 10:26:43,153 INFO [train.py:737] (1/3) Epoch 18, batch 5400, loss[loss=0.3978, over 14725.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.01048, over 35.00 utterances.], tot_loss[loss=0.4343, over 2812401.27 frames; utt_duration=1194 frames, utt_pad_proportion=0.07964, over 9435.71 utterances.], batch size: 35, lr: 2.95e-04
|
| 215 |
+
2022-08-12 10:28:50,808 INFO [train.py:737] (1/3) Epoch 18, batch 5500, loss[loss=0.4791, over 14478.00 frames; utt_duration=1160 frames, utt_pad_proportion=0.02359, over 50.00 utterances.], tot_loss[loss=0.435, over 2796751.64 frames; utt_duration=1161 frames, utt_pad_proportion=0.08885, over 9645.74 utterances.], batch size: 50, lr: 2.95e-04
|
| 216 |
+
2022-08-12 10:30:53,582 INFO [train.py:737] (1/3) Epoch 18, batch 5600, loss[loss=0.3693, over 14627.00 frames; utt_duration=1722 frames, utt_pad_proportion=0.009223, over 34.00 utterances.], tot_loss[loss=0.4369, over 2802118.71 frames; utt_duration=1166 frames, utt_pad_proportion=0.08556, over 9628.71 utterances.], batch size: 34, lr: 2.95e-04
|
| 217 |
+
2022-08-12 10:33:04,876 INFO [train.py:737] (1/3) Epoch 18, batch 5700, loss[loss=0.4261, over 14646.00 frames; utt_duration=1248 frames, utt_pad_proportion=0.01346, over 47.00 utterances.], tot_loss[loss=0.437, over 2819980.40 frames; utt_duration=1194 frames, utt_pad_proportion=0.07512, over 9460.64 utterances.], batch size: 47, lr: 2.95e-04
|
| 218 |
+
2022-08-12 10:35:25,142 INFO [train.py:737] (1/3) Epoch 18, batch 5800, loss[loss=0.3828, over 14588.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.008507, over 36.00 utterances.], tot_loss[loss=0.4376, over 2832942.80 frames; utt_duration=1217 frames, utt_pad_proportion=0.06687, over 9325.17 utterances.], batch size: 36, lr: 2.95e-04
|
| 219 |
+
2022-08-12 10:37:04,067 INFO [train.py:737] (1/3) Epoch 18, batch 5900, loss[loss=0.3781, over 14145.00 frames; utt_duration=1826 frames, utt_pad_proportion=0.03778, over 31.00 utterances.], tot_loss[loss=0.4348, over 2838741.54 frames; utt_duration=1237 frames, utt_pad_proportion=0.06562, over 9187.73 utterances.], batch size: 31, lr: 2.95e-04
|
| 220 |
+
2022-08-12 10:38:07,037 INFO [train.py:737] (1/3) Epoch 18, batch 6000, loss[loss=0.363, over 14485.00 frames; utt_duration=1757 frames, utt_pad_proportion=0.009463, over 33.00 utterances.], tot_loss[loss=0.4345, over 2826613.50 frames; utt_duration=1217 frames, utt_pad_proportion=0.07365, over 9303.00 utterances.], batch size: 33, lr: 2.95e-04
|
| 221 |
+
2022-08-12 10:38:07,037 INFO [train.py:761] (1/3) Computing validation loss
|
| 222 |
+
2022-08-12 10:38:38,844 INFO [train.py:770] (1/3) Epoch 18, validation: loss=0.3055, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 223 |
+
2022-08-12 10:40:35,063 INFO [train.py:737] (1/3) Epoch 18, batch 6100, loss[loss=0.544, over 9757.00 frames; utt_duration=425.8 frames, utt_pad_proportion=0.3459, over 92.00 utterances.], tot_loss[loss=0.4409, over 2782661.91 frames; utt_duration=1141 frames, utt_pad_proportion=0.0996, over 9770.66 utterances.], batch size: 92, lr: 2.95e-04
|
| 224 |
+
2022-08-12 10:41:44,280 INFO [train.py:737] (1/3) Epoch 19, batch 0, loss[loss=0.4154, over 14350.00 frames; utt_duration=1084 frames, utt_pad_proportion=0.03691, over 53.00 utterances.], tot_loss[loss=0.4154, over 14350.00 frames; utt_duration=1084 frames, utt_pad_proportion=0.03691, over 53.00 utterances.], batch size: 53, lr: 2.87e-04
|
| 225 |
+
2022-08-12 10:44:11,840 INFO [train.py:737] (1/3) Epoch 19, batch 100, loss[loss=0.3451, over 13887.00 frames; utt_duration=976.1 frames, utt_pad_proportion=0.06057, over 57.00 utterances.], tot_loss[loss=0.3644, over 1148625.58 frames; utt_duration=1348 frames, utt_pad_proportion=0.03638, over 3412.98 utterances.], batch size: 57, lr: 2.87e-04
|
| 226 |
+
2022-08-12 10:46:22,385 INFO [train.py:737] (1/3) Epoch 19, batch 200, loss[loss=0.418, over 14752.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.006125, over 43.00 utterances.], tot_loss[loss=0.3596, over 1834424.57 frames; utt_duration=1344 frames, utt_pad_proportion=0.03408, over 5463.91 utterances.], batch size: 43, lr: 2.87e-04
|
| 227 |
+
2022-08-12 10:48:43,569 INFO [train.py:737] (1/3) Epoch 19, batch 300, loss[loss=0.3206, over 14790.00 frames; utt_duration=1559 frames, utt_pad_proportion=0.004688, over 38.00 utterances.], tot_loss[loss=0.3578, over 2243287.43 frames; utt_duration=1323 frames, utt_pad_proportion=0.04081, over 6788.08 utterances.], batch size: 38, lr: 2.87e-04
|
| 228 |
+
2022-08-12 10:51:23,916 INFO [train.py:737] (1/3) Epoch 19, batch 400, loss[loss=0.3746, over 12710.00 frames; utt_duration=760.3 frames, utt_pad_proportion=0.1409, over 67.00 utterances.], tot_loss[loss=0.3553, over 2479543.10 frames; utt_duration=1296 frames, utt_pad_proportion=0.04958, over 7660.24 utterances.], batch size: 67, lr: 2.87e-04
|
| 229 |
+
2022-08-12 10:53:02,775 INFO [train.py:737] (1/3) Epoch 19, batch 500, loss[loss=0.3152, over 14602.00 frames; utt_duration=1624 frames, utt_pad_proportion=0.006796, over 36.00 utterances.], tot_loss[loss=0.3539, over 2629046.71 frames; utt_duration=1293 frames, utt_pad_proportion=0.05131, over 8140.93 utterances.], batch size: 36, lr: 2.87e-04
|
| 230 |
+
2022-08-12 10:55:23,803 INFO [train.py:737] (1/3) Epoch 19, batch 600, loss[loss=0.3472, over 14641.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.007869, over 44.00 utterances.], tot_loss[loss=0.353, over 2731692.68 frames; utt_duration=1306 frames, utt_pad_proportion=0.04428, over 8376.50 utterances.], batch size: 44, lr: 2.87e-04
|
| 231 |
+
2022-08-12 10:58:04,976 INFO [train.py:737] (1/3) Epoch 19, batch 700, loss[loss=0.3115, over 14575.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006012, over 37.00 utterances.], tot_loss[loss=0.3546, over 2785175.91 frames; utt_duration=1310 frames, utt_pad_proportion=0.04439, over 8516.03 utterances.], batch size: 37, lr: 2.86e-04
|
| 232 |
+
2022-08-12 11:00:28,080 INFO [train.py:737] (1/3) Epoch 19, batch 800, loss[loss=0.3698, over 14698.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.009108, over 45.00 utterances.], tot_loss[loss=0.3588, over 2816062.68 frames; utt_duration=1285 frames, utt_pad_proportion=0.04541, over 8773.67 utterances.], batch size: 45, lr: 2.86e-04
|
| 233 |
+
2022-08-12 11:02:57,820 INFO [train.py:737] (1/3) Epoch 19, batch 900, loss[loss=0.3444, over 14717.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.009849, over 35.00 utterances.], tot_loss[loss=0.3578, over 2843485.81 frames; utt_duration=1306 frames, utt_pad_proportion=0.04201, over 8716.66 utterances.], batch size: 35, lr: 2.86e-04
|
| 234 |
+
2022-08-12 11:05:28,862 INFO [train.py:737] (1/3) Epoch 19, batch 1000, loss[loss=0.3824, over 14759.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005218, over 41.00 utterances.], tot_loss[loss=0.3595, over 2854029.08 frames; utt_duration=1303 frames, utt_pad_proportion=0.0437, over 8770.47 utterances.], batch size: 41, lr: 2.86e-04
|
| 235 |
+
2022-08-12 11:07:58,552 INFO [train.py:737] (1/3) Epoch 19, batch 1100, loss[loss=0.3245, over 14569.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.006625, over 37.00 utterances.], tot_loss[loss=0.3596, over 2854586.54 frames; utt_duration=1288 frames, utt_pad_proportion=0.04956, over 8873.75 utterances.], batch size: 37, lr: 2.86e-04
|
| 236 |
+
2022-08-12 11:10:10,293 INFO [train.py:737] (1/3) Epoch 19, batch 1200, loss[loss=0.3176, over 14465.00 frames; utt_duration=1655 frames, utt_pad_proportion=0.007439, over 35.00 utterances.], tot_loss[loss=0.3594, over 2864111.66 frames; utt_duration=1297 frames, utt_pad_proportion=0.0446, over 8845.14 utterances.], batch size: 35, lr: 2.86e-04
|
| 237 |
+
2022-08-12 11:12:02,656 INFO [train.py:737] (1/3) Epoch 19, batch 1300, loss[loss=0.3317, over 13678.00 frames; utt_duration=1825 frames, utt_pad_proportion=0.06259, over 30.00 utterances.], tot_loss[loss=0.3596, over 2851826.23 frames; utt_duration=1266 frames, utt_pad_proportion=0.05338, over 9020.03 utterances.], batch size: 30, lr: 2.86e-04
|
| 238 |
+
2022-08-12 11:14:19,755 INFO [train.py:737] (1/3) Epoch 19, batch 1400, loss[loss=0.3226, over 14743.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.005911, over 43.00 utterances.], tot_loss[loss=0.3588, over 2845671.48 frames; utt_duration=1252 frames, utt_pad_proportion=0.05863, over 9103.48 utterances.], batch size: 43, lr: 2.86e-04
|
| 239 |
+
2022-08-12 11:16:17,938 INFO [train.py:737] (1/3) Epoch 19, batch 1500, loss[loss=0.3329, over 14758.00 frames; utt_duration=1597 frames, utt_pad_proportion=0.007525, over 37.00 utterances.], tot_loss[loss=0.3599, over 2840929.53 frames; utt_duration=1241 frames, utt_pad_proportion=0.06314, over 9169.50 utterances.], batch size: 37, lr: 2.86e-04
|
| 240 |
+
2022-08-12 11:18:46,500 INFO [train.py:737] (1/3) Epoch 19, batch 1600, loss[loss=0.292, over 14602.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.009973, over 34.00 utterances.], tot_loss[loss=0.3591, over 2842424.41 frames; utt_duration=1243 frames, utt_pad_proportion=0.06235, over 9158.96 utterances.], batch size: 34, lr: 2.86e-04
|
| 241 |
+
2022-08-12 11:21:10,268 INFO [train.py:737] (1/3) Epoch 19, batch 1700, loss[loss=0.4134, over 14864.00 frames; utt_duration=1352 frames, utt_pad_proportion=0.006975, over 44.00 utterances.], tot_loss[loss=0.3593, over 2865210.81 frames; utt_duration=1288 frames, utt_pad_proportion=0.049, over 8908.22 utterances.], batch size: 44, lr: 2.86e-04
|
| 242 |
+
2022-08-12 11:23:35,421 INFO [train.py:737] (1/3) Epoch 19, batch 1800, loss[loss=0.4285, over 14582.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006314, over 42.00 utterances.], tot_loss[loss=0.3611, over 2867178.96 frames; utt_duration=1280 frames, utt_pad_proportion=0.04729, over 8973.76 utterances.], batch size: 42, lr: 2.86e-04
|
| 243 |
+
2022-08-12 11:25:45,261 INFO [train.py:737] (1/3) Epoch 19, batch 1900, loss[loss=0.3167, over 14572.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.005828, over 37.00 utterances.], tot_loss[loss=0.3598, over 2873242.54 frames; utt_duration=1293 frames, utt_pad_proportion=0.0458, over 8897.19 utterances.], batch size: 37, lr: 2.86e-04
|
| 244 |
+
2022-08-12 11:27:05,703 INFO [train.py:737] (1/3) Epoch 19, batch 2000, loss[loss=0.3104, over 13789.00 frames; utt_duration=969.2 frames, utt_pad_proportion=0.0672, over 57.00 utterances.], tot_loss[loss=0.3582, over 2865584.41 frames; utt_duration=1287 frames, utt_pad_proportion=0.04886, over 8914.95 utterances.], batch size: 57, lr: 2.85e-04
|
| 245 |
+
2022-08-12 11:29:07,397 INFO [train.py:737] (1/3) Epoch 19, batch 2100, loss[loss=0.3498, over 14582.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.008053, over 36.00 utterances.], tot_loss[loss=0.3604, over 2866381.13 frames; utt_duration=1296 frames, utt_pad_proportion=0.04911, over 8858.82 utterances.], batch size: 36, lr: 2.85e-04
|
| 246 |
+
2022-08-12 11:31:17,548 INFO [train.py:737] (1/3) Epoch 19, batch 2200, loss[loss=0.3944, over 14455.00 frames; utt_duration=1158 frames, utt_pad_proportion=0.02549, over 50.00 utterances.], tot_loss[loss=0.3581, over 2881207.20 frames; utt_duration=1324 frames, utt_pad_proportion=0.03907, over 8717.10 utterances.], batch size: 50, lr: 2.85e-04
|
| 247 |
+
2022-08-12 11:33:35,214 INFO [train.py:737] (1/3) Epoch 19, batch 2300, loss[loss=0.4353, over 9932.00 frames; utt_duration=438.1 frames, utt_pad_proportion=0.329, over 91.00 utterances.], tot_loss[loss=0.3597, over 2872808.22 frames; utt_duration=1300 frames, utt_pad_proportion=0.04363, over 8851.29 utterances.], batch size: 91, lr: 2.85e-04
|
| 248 |
+
2022-08-12 11:35:43,153 INFO [train.py:737] (1/3) Epoch 19, batch 2400, loss[loss=0.3774, over 14864.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.007453, over 44.00 utterances.], tot_loss[loss=0.3614, over 2856410.29 frames; utt_duration=1255 frames, utt_pad_proportion=0.05495, over 9117.97 utterances.], batch size: 44, lr: 2.85e-04
|
| 249 |
+
2022-08-12 11:38:20,355 INFO [train.py:737] (1/3) Epoch 19, batch 2500, loss[loss=0.3201, over 14597.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.00986, over 34.00 utterances.], tot_loss[loss=0.3625, over 2846910.15 frames; utt_duration=1240 frames, utt_pad_proportion=0.06094, over 9191.39 utterances.], batch size: 34, lr: 2.85e-04
|
| 250 |
+
2022-08-12 11:40:28,214 INFO [train.py:737] (1/3) Epoch 19, batch 2600, loss[loss=0.395, over 14570.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006574, over 40.00 utterances.], tot_loss[loss=0.3616, over 2852195.46 frames; utt_duration=1254 frames, utt_pad_proportion=0.05772, over 9110.80 utterances.], batch size: 40, lr: 2.85e-04
|
| 251 |
+
2022-08-12 11:42:36,347 INFO [train.py:737] (1/3) Epoch 19, batch 2700, loss[loss=0.3708, over 13832.00 frames; utt_duration=955.6 frames, utt_pad_proportion=0.07496, over 58.00 utterances.], tot_loss[loss=0.3601, over 2852987.21 frames; utt_duration=1258 frames, utt_pad_proportion=0.05688, over 9081.22 utterances.], batch size: 58, lr: 2.85e-04
|
| 252 |
+
2022-08-12 11:44:20,899 INFO [train.py:737] (1/3) Epoch 19, batch 2800, loss[loss=0.2736, over 14575.00 frames; utt_duration=1717 frames, utt_pad_proportion=0.01172, over 34.00 utterances.], tot_loss[loss=0.3602, over 2859449.67 frames; utt_duration=1274 frames, utt_pad_proportion=0.05343, over 8985.50 utterances.], batch size: 34, lr: 2.85e-04
|
| 253 |
+
2022-08-12 11:46:57,784 INFO [train.py:737] (1/3) Epoch 19, batch 2900, loss[loss=0.4667, over 13066.00 frames; utt_duration=781.5 frames, utt_pad_proportion=0.118, over 67.00 utterances.], tot_loss[loss=0.3735, over 2864476.25 frames; utt_duration=1284 frames, utt_pad_proportion=0.04956, over 8937.46 utterances.], batch size: 67, lr: 2.85e-04
|
| 254 |
+
2022-08-12 11:48:38,711 INFO [train.py:737] (1/3) Epoch 19, batch 3000, loss[loss=0.4035, over 14572.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.006369, over 40.00 utterances.], tot_loss[loss=0.3914, over 2834412.73 frames; utt_duration=1222 frames, utt_pad_proportion=0.06651, over 9287.24 utterances.], batch size: 40, lr: 2.85e-04
|
| 255 |
+
2022-08-12 11:48:38,711 INFO [train.py:761] (1/3) Computing validation loss
|
| 256 |
+
2022-08-12 11:49:10,905 INFO [train.py:770] (1/3) Epoch 19, validation: loss=0.2796, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 257 |
+
2022-08-12 11:51:42,713 INFO [train.py:737] (1/3) Epoch 19, batch 3100, loss[loss=0.3944, over 14843.00 frames; utt_duration=1651 frames, utt_pad_proportion=0.009798, over 36.00 utterances.], tot_loss[loss=0.4013, over 2844651.70 frames; utt_duration=1243 frames, utt_pad_proportion=0.06128, over 9165.18 utterances.], batch size: 36, lr: 2.85e-04
|
| 258 |
+
2022-08-12 11:53:58,742 INFO [train.py:737] (1/3) Epoch 19, batch 3200, loss[loss=0.3924, over 14768.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.006183, over 39.00 utterances.], tot_loss[loss=0.4152, over 2831695.35 frames; utt_duration=1218 frames, utt_pad_proportion=0.06918, over 9312.77 utterances.], batch size: 39, lr: 2.84e-04
|
| 259 |
+
2022-08-12 11:56:02,102 INFO [train.py:737] (1/3) Epoch 19, batch 3300, loss[loss=0.3929, over 14585.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.005111, over 39.00 utterances.], tot_loss[loss=0.4237, over 2836472.94 frames; utt_duration=1227 frames, utt_pad_proportion=0.06686, over 9261.82 utterances.], batch size: 39, lr: 2.84e-04
|
| 260 |
+
2022-08-12 11:58:26,093 INFO [train.py:737] (1/3) Epoch 19, batch 3400, loss[loss=0.4402, over 14409.00 frames; utt_duration=1154 frames, utt_pad_proportion=0.02755, over 50.00 utterances.], tot_loss[loss=0.4343, over 2820757.56 frames; utt_duration=1190 frames, utt_pad_proportion=0.07668, over 9492.98 utterances.], batch size: 50, lr: 2.84e-04
|
| 261 |
+
2022-08-12 12:00:33,870 INFO [train.py:737] (1/3) Epoch 19, batch 3500, loss[loss=0.3437, over 14613.00 frames; utt_duration=1721 frames, utt_pad_proportion=0.009313, over 34.00 utterances.], tot_loss[loss=0.4356, over 2823989.07 frames; utt_duration=1204 frames, utt_pad_proportion=0.07372, over 9396.41 utterances.], batch size: 34, lr: 2.84e-04
|
| 262 |
+
2022-08-12 12:02:46,755 INFO [train.py:737] (1/3) Epoch 19, batch 3600, loss[loss=0.4542, over 14481.00 frames; utt_duration=1757 frames, utt_pad_proportion=0.009076, over 33.00 utterances.], tot_loss[loss=0.437, over 2812670.71 frames; utt_duration=1191 frames, utt_pad_proportion=0.08138, over 9460.43 utterances.], batch size: 33, lr: 2.84e-04
|
| 263 |
+
2022-08-12 12:04:15,351 INFO [train.py:737] (1/3) Epoch 19, batch 3700, loss[loss=0.408, over 14577.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.005436, over 37.00 utterances.], tot_loss[loss=0.438, over 2811557.26 frames; utt_duration=1193 frames, utt_pad_proportion=0.08254, over 9440.05 utterances.], batch size: 37, lr: 2.84e-04
|
| 264 |
+
2022-08-12 12:06:22,784 INFO [train.py:737] (1/3) Epoch 19, batch 3800, loss[loss=0.4635, over 14756.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005403, over 41.00 utterances.], tot_loss[loss=0.4411, over 2809485.56 frames; utt_duration=1178 frames, utt_pad_proportion=0.08108, over 9552.58 utterances.], batch size: 41, lr: 2.84e-04
|
| 265 |
+
2022-08-12 12:08:47,678 INFO [train.py:737] (1/3) Epoch 19, batch 3900, loss[loss=0.3977, over 14752.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006876, over 40.00 utterances.], tot_loss[loss=0.4413, over 2809429.08 frames; utt_duration=1189 frames, utt_pad_proportion=0.08323, over 9463.99 utterances.], batch size: 40, lr: 2.84e-04
|
| 266 |
+
2022-08-12 12:10:40,016 INFO [train.py:737] (1/3) Epoch 19, batch 4000, loss[loss=0.5094, over 10220.00 frames; utt_duration=436.4 frames, utt_pad_proportion=0.3159, over 94.00 utterances.], tot_loss[loss=0.4384, over 2813043.90 frames; utt_duration=1193 frames, utt_pad_proportion=0.0811, over 9445.38 utterances.], batch size: 94, lr: 2.84e-04
|
| 267 |
+
2022-08-12 12:11:52,487 INFO [train.py:737] (1/3) Epoch 19, batch 4100, loss[loss=0.4103, over 14595.00 frames; utt_duration=1623 frames, utt_pad_proportion=0.007238, over 36.00 utterances.], tot_loss[loss=0.4368, over 2818509.94 frames; utt_duration=1197 frames, utt_pad_proportion=0.07647, over 9431.41 utterances.], batch size: 36, lr: 2.84e-04
|
| 268 |
+
2022-08-12 12:12:51,329 INFO [train.py:737] (1/3) Epoch 19, batch 4200, loss[loss=0.3988, over 14576.00 frames; utt_duration=1536 frames, utt_pad_proportion=0.007145, over 38.00 utterances.], tot_loss[loss=0.4394, over 2802413.01 frames; utt_duration=1170 frames, utt_pad_proportion=0.08474, over 9595.71 utterances.], batch size: 38, lr: 2.84e-04
|
| 269 |
+
2022-08-12 12:14:56,498 INFO [train.py:737] (1/3) Epoch 19, batch 4300, loss[loss=0.3892, over 14574.00 frames; utt_duration=1389 frames, utt_pad_proportion=0.006876, over 42.00 utterances.], tot_loss[loss=0.4398, over 2811515.20 frames; utt_duration=1187 frames, utt_pad_proportion=0.08179, over 9485.75 utterances.], batch size: 42, lr: 2.84e-04
|
| 270 |
+
2022-08-12 12:17:11,264 INFO [train.py:737] (1/3) Epoch 19, batch 4400, loss[loss=0.3731, over 14598.00 frames; utt_duration=1624 frames, utt_pad_proportion=0.006966, over 36.00 utterances.], tot_loss[loss=0.439, over 2825417.99 frames; utt_duration=1210 frames, utt_pad_proportion=0.07234, over 9348.71 utterances.], batch size: 36, lr: 2.83e-04
|
| 271 |
+
2022-08-12 12:19:20,857 INFO [train.py:737] (1/3) Epoch 19, batch 4500, loss[loss=0.4494, over 14862.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.00757, over 44.00 utterances.], tot_loss[loss=0.4381, over 2820572.82 frames; utt_duration=1205 frames, utt_pad_proportion=0.07494, over 9374.65 utterances.], batch size: 44, lr: 2.83e-04
|
| 272 |
+
2022-08-12 12:21:26,375 INFO [train.py:737] (1/3) Epoch 19, batch 4600, loss[loss=0.5155, over 12864.00 frames; utt_duration=769.6 frames, utt_pad_proportion=0.1324, over 67.00 utterances.], tot_loss[loss=0.4381, over 2816816.26 frames; utt_duration=1191 frames, utt_pad_proportion=0.07732, over 9468.41 utterances.], batch size: 67, lr: 2.83e-04
|
| 273 |
+
2022-08-12 12:23:39,331 INFO [train.py:737] (1/3) Epoch 19, batch 4700, loss[loss=0.4565, over 14523.00 frames; utt_duration=1212 frames, utt_pad_proportion=0.01639, over 48.00 utterances.], tot_loss[loss=0.4402, over 2819498.78 frames; utt_duration=1197 frames, utt_pad_proportion=0.07533, over 9436.78 utterances.], batch size: 48, lr: 2.83e-04
|
| 274 |
+
2022-08-12 12:26:00,560 INFO [train.py:737] (1/3) Epoch 19, batch 4800, loss[loss=0.437, over 14450.00 frames; utt_duration=1158 frames, utt_pad_proportion=0.02321, over 50.00 utterances.], tot_loss[loss=0.4365, over 2841237.68 frames; utt_duration=1234 frames, utt_pad_proportion=0.0633, over 9223.65 utterances.], batch size: 50, lr: 2.83e-04
|
| 275 |
+
2022-08-12 12:27:27,856 INFO [train.py:737] (1/3) Epoch 19, batch 4900, loss[loss=0.4191, over 14347.00 frames; utt_duration=1084 frames, utt_pad_proportion=0.03728, over 53.00 utterances.], tot_loss[loss=0.4368, over 2827844.58 frames; utt_duration=1214 frames, utt_pad_proportion=0.07212, over 9331.66 utterances.], batch size: 53, lr: 2.83e-04
|
| 276 |
+
2022-08-12 12:29:00,609 INFO [train.py:737] (1/3) Epoch 19, batch 5000, loss[loss=0.552, over 10295.00 frames; utt_duration=449.1 frames, utt_pad_proportion=0.3101, over 92.00 utterances.], tot_loss[loss=0.4344, over 2821616.07 frames; utt_duration=1208 frames, utt_pad_proportion=0.07562, over 9357.57 utterances.], batch size: 92, lr: 2.83e-04
|
| 277 |
+
2022-08-12 12:30:53,858 INFO [train.py:737] (1/3) Epoch 19, batch 5100, loss[loss=0.4155, over 14630.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.008444, over 44.00 utterances.], tot_loss[loss=0.4337, over 2819678.31 frames; utt_duration=1200 frames, utt_pad_proportion=0.07654, over 9411.42 utterances.], batch size: 44, lr: 2.83e-04
|
| 278 |
+
2022-08-12 12:33:24,376 INFO [train.py:737] (1/3) Epoch 19, batch 5200, loss[loss=0.511, over 9803.00 frames; utt_duration=427.8 frames, utt_pad_proportion=0.3419, over 92.00 utterances.], tot_loss[loss=0.434, over 2823598.48 frames; utt_duration=1209 frames, utt_pad_proportion=0.07344, over 9356.50 utterances.], batch size: 92, lr: 2.83e-04
|
| 279 |
+
2022-08-12 12:35:41,952 INFO [train.py:737] (1/3) Epoch 19, batch 5300, loss[loss=0.3803, over 14555.00 frames; utt_duration=1575 frames, utt_pad_proportion=0.007595, over 37.00 utterances.], tot_loss[loss=0.4349, over 2820695.70 frames; utt_duration=1207 frames, utt_pad_proportion=0.0748, over 9356.16 utterances.], batch size: 37, lr: 2.83e-04
|
| 280 |
+
2022-08-12 12:37:57,204 INFO [train.py:737] (1/3) Epoch 19, batch 5400, loss[loss=0.3397, over 14745.00 frames; utt_duration=1687 frames, utt_pad_proportion=0.009031, over 35.00 utterances.], tot_loss[loss=0.4355, over 2822197.69 frames; utt_duration=1207 frames, utt_pad_proportion=0.07373, over 9362.77 utterances.], batch size: 35, lr: 2.83e-04
|
| 281 |
+
2022-08-12 12:40:13,260 INFO [train.py:737] (1/3) Epoch 19, batch 5500, loss[loss=0.3799, over 14765.00 frames; utt_duration=1478 frames, utt_pad_proportion=0.005968, over 40.00 utterances.], tot_loss[loss=0.4362, over 2813039.26 frames; utt_duration=1193 frames, utt_pad_proportion=0.07935, over 9443.31 utterances.], batch size: 40, lr: 2.83e-04
|
| 282 |
+
2022-08-12 12:42:24,008 INFO [train.py:737] (1/3) Epoch 19, batch 5600, loss[loss=0.4609, over 14483.00 frames; utt_duration=1757 frames, utt_pad_proportion=0.009361, over 33.00 utterances.], tot_loss[loss=0.4338, over 2840025.68 frames; utt_duration=1240 frames, utt_pad_proportion=0.06474, over 9175.12 utterances.], batch size: 33, lr: 2.83e-04
|
| 283 |
+
2022-08-12 12:43:43,252 INFO [train.py:737] (1/3) Epoch 19, batch 5700, loss[loss=0.4445, over 14711.00 frames; utt_duration=1683 frames, utt_pad_proportion=0.01065, over 35.00 utterances.], tot_loss[loss=0.4355, over 2852013.56 frames; utt_duration=1255 frames, utt_pad_proportion=0.05668, over 9101.46 utterances.], batch size: 35, lr: 2.82e-04
|
| 284 |
+
2022-08-12 12:44:43,219 INFO [train.py:737] (1/3) Epoch 19, batch 5800, loss[loss=0.4331, over 14879.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.007315, over 44.00 utterances.], tot_loss[loss=0.4379, over 2841623.62 frames; utt_duration=1232 frames, utt_pad_proportion=0.06335, over 9239.50 utterances.], batch size: 44, lr: 2.82e-04
|
| 285 |
+
2022-08-12 12:46:17,973 INFO [train.py:737] (1/3) Epoch 19, batch 5900, loss[loss=0.3598, over 14440.00 frames; utt_duration=1752 frames, utt_pad_proportion=0.01237, over 33.00 utterances.], tot_loss[loss=0.4389, over 2846875.96 frames; utt_duration=1244 frames, utt_pad_proportion=0.06108, over 9161.62 utterances.], batch size: 33, lr: 2.82e-04
|
| 286 |
+
2022-08-12 12:48:42,418 INFO [train.py:737] (1/3) Epoch 19, batch 6000, loss[loss=0.3932, over 14771.00 frames; utt_duration=1516 frames, utt_pad_proportion=0.006284, over 39.00 utterances.], tot_loss[loss=0.4382, over 2819813.46 frames; utt_duration=1209 frames, utt_pad_proportion=0.07693, over 9343.13 utterances.], batch size: 39, lr: 2.82e-04
|
| 287 |
+
2022-08-12 12:48:42,419 INFO [train.py:761] (1/3) Computing validation loss
|
| 288 |
+
2022-08-12 12:49:17,792 INFO [train.py:770] (1/3) Epoch 19, validation: loss=0.2887, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 289 |
+
2022-08-12 12:51:45,152 INFO [train.py:737] (1/3) Epoch 19, batch 6100, loss[loss=0.5328, over 10201.00 frames; utt_duration=445.1 frames, utt_pad_proportion=0.3153, over 92.00 utterances.], tot_loss[loss=0.441, over 2774946.59 frames; utt_duration=1131 frames, utt_pad_proportion=0.103, over 9823.40 utterances.], batch size: 92, lr: 2.82e-04
|
| 290 |
+
2022-08-12 12:52:43,856 INFO [train.py:737] (1/3) Epoch 20, batch 0, loss[loss=0.4224, over 12940.00 frames; utt_duration=774.4 frames, utt_pad_proportion=0.127, over 67.00 utterances.], tot_loss[loss=0.4224, over 12940.00 frames; utt_duration=774.4 frames, utt_pad_proportion=0.127, over 67.00 utterances.], batch size: 67, lr: 2.75e-04
|
exp/log/log-train-2022-08-11-18-53-52-2
ADDED
|
@@ -0,0 +1,290 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2022-08-11 18:53:52,371 INFO [train.py:805] (2/3) Training started
|
| 2 |
+
2022-08-11 18:53:52,372 INFO [train.py:806] (2/3) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'env_info': {'k2-version': '1.17', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '7dcabf85e8bf06984c4abab0400ef1322b5ff3df', 'k2-git-date': 'Tue Aug 2 21:22:39 2022', 'lhotse-version': '1.4.0.dev+git.59318ed.clean', 'torch-version': '1.11.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.3', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '6af45c3-dirty', 'icefall-git-date': 'Tue Aug 2 14:47:09 2022', 'icefall-path': '/thspfs/home/acct-hpc/thspzh-hpc1/git/icefall_dev', 'k2-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/k2-1.17.dev20220805+cuda11.3.torch1.11.0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/thspfs/home/acct-hpc/thspzh-hpc1/.conda/envs/k2/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'dgxs-01.cluster.thspzh.com', 'IP address': '192.168.200.101'}, 'world_size': 3, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 25, 'start_epoch': 16, 'start_batch': 0, 'exp_dir': PosixPath('cnn_ctc/exp_phone'), 'lang_dir': 'data/lang_phone', 'bpe_model': 'data/lang_bpe_500/bpe.model', 'initial_lr': 0.003, 'lr_batches': 5000, 'lr_epochs': 6, 'seed': 42, 'print_diagnostics': False, 'save_every_n': 8000, 'keep_last_k': 20, 'average_period': 100, 'use_fp16': False, 'first_out_channels': 8, 'conv_blocks': 3, 'rnn_dim': 0, 'full_libri': True, 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 600, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'input_strategy': 'PrecomputedFeatures'}
|
| 3 |
+
2022-08-11 18:53:52,751 INFO [lexicon.py:176] (2/3) Loading pre-compiled data/lang_phone/Linv.pt
|
| 4 |
+
2022-08-11 18:54:05,147 INFO [train.py:843] (2/3) About to create model
|
| 5 |
+
2022-08-11 18:54:05,170 INFO [checkpoint.py:112] (2/3) Loading checkpoint from cnn_ctc/exp_phone/epoch-15.pt
|
| 6 |
+
2022-08-11 18:54:07,781 INFO [train.py:876] (2/3) Using DDP
|
| 7 |
+
2022-08-11 18:54:11,801 INFO [train.py:884] (2/3) Loading optimizer state dict
|
| 8 |
+
2022-08-11 18:54:12,339 INFO [train.py:892] (2/3) Loading scheduler state dict
|
| 9 |
+
2022-08-11 18:54:12,339 INFO [asr_datamodule.py:409] (2/3) About to get train-clean-100 cuts
|
| 10 |
+
2022-08-11 18:54:12,341 INFO [asr_datamodule.py:416] (2/3) About to get train-clean-360 cuts
|
| 11 |
+
2022-08-11 18:54:12,343 INFO [asr_datamodule.py:423] (2/3) About to get train-other-500 cuts
|
| 12 |
+
2022-08-11 18:54:12,345 INFO [asr_datamodule.py:225] (2/3) Enable MUSAN
|
| 13 |
+
2022-08-11 18:54:12,345 INFO [asr_datamodule.py:226] (2/3) About to get Musan cuts
|
| 14 |
+
2022-08-11 18:54:14,353 INFO [asr_datamodule.py:254] (2/3) Enable SpecAugment
|
| 15 |
+
2022-08-11 18:54:14,354 INFO [asr_datamodule.py:255] (2/3) Time warp factor: 80
|
| 16 |
+
2022-08-11 18:54:14,354 INFO [asr_datamodule.py:267] (2/3) Num frame mask: 10
|
| 17 |
+
2022-08-11 18:54:14,354 INFO [asr_datamodule.py:280] (2/3) About to create train dataset
|
| 18 |
+
2022-08-11 18:54:14,354 INFO [asr_datamodule.py:309] (2/3) Using DynamicBucketingSampler.
|
| 19 |
+
2022-08-11 18:54:16,725 INFO [asr_datamodule.py:324] (2/3) About to create train dataloader
|
| 20 |
+
2022-08-11 18:54:16,725 INFO [asr_datamodule.py:430] (2/3) About to get dev-clean cuts
|
| 21 |
+
2022-08-11 18:54:16,727 INFO [asr_datamodule.py:437] (2/3) About to get dev-other cuts
|
| 22 |
+
2022-08-11 18:54:16,728 INFO [asr_datamodule.py:355] (2/3) About to create dev dataset
|
| 23 |
+
2022-08-11 18:54:17,039 INFO [asr_datamodule.py:374] (2/3) About to create dev dataloader
|
| 24 |
+
2022-08-11 18:54:17,040 INFO [train.py:959] (2/3) Loading grad scaler state dict
|
| 25 |
+
2022-08-11 18:54:39,535 INFO [train.py:737] (2/3) Epoch 16, batch 0, loss[loss=0.3967, over 14557.00 frames; utt_duration=1457 frames, utt_pad_proportion=0.006003, over 40.00 utterances.], tot_loss[loss=0.3967, over 14557.00 frames; utt_duration=1457 frames, utt_pad_proportion=0.006003, over 40.00 utterances.], batch size: 40, lr: 3.29e-04
|
| 26 |
+
2022-08-11 18:54:39,583 INFO [distributed.py:948] (2/3) Reducer buckets have been rebuilt in this iteration.
|
| 27 |
+
2022-08-11 19:00:36,847 INFO [train.py:737] (2/3) Epoch 16, batch 100, loss[loss=0.3823, over 14701.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.008889, over 45.00 utterances.], tot_loss[loss=0.3634, over 1147951.84 frames; utt_duration=1362 frames, utt_pad_proportion=0.03543, over 3374.94 utterances.], batch size: 45, lr: 3.29e-04
|
| 28 |
+
2022-08-11 19:06:33,773 INFO [train.py:737] (2/3) Epoch 16, batch 200, loss[loss=0.4004, over 14757.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006691, over 40.00 utterances.], tot_loss[loss=0.3596, over 1829030.05 frames; utt_duration=1332 frames, utt_pad_proportion=0.03961, over 5499.02 utterances.], batch size: 40, lr: 3.29e-04
|
| 29 |
+
2022-08-11 19:12:27,340 INFO [train.py:737] (2/3) Epoch 16, batch 300, loss[loss=0.3645, over 14410.00 frames; utt_duration=1649 frames, utt_pad_proportion=0.01105, over 35.00 utterances.], tot_loss[loss=0.3576, over 2239053.07 frames; utt_duration=1311 frames, utt_pad_proportion=0.04337, over 6837.78 utterances.], batch size: 35, lr: 3.29e-04
|
| 30 |
+
2022-08-11 19:18:24,971 INFO [train.py:737] (2/3) Epoch 16, batch 400, loss[loss=0.2836, over 14020.00 frames; utt_duration=1811 frames, utt_pad_proportion=0.03989, over 31.00 utterances.], tot_loss[loss=0.3532, over 2497302.56 frames; utt_duration=1337 frames, utt_pad_proportion=0.03808, over 7478.96 utterances.], batch size: 31, lr: 3.29e-04
|
| 31 |
+
2022-08-11 19:24:20,560 INFO [train.py:737] (2/3) Epoch 16, batch 500, loss[loss=0.3595, over 14786.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.004957, over 38.00 utterances.], tot_loss[loss=0.3573, over 2635990.03 frames; utt_duration=1294 frames, utt_pad_proportion=0.0474, over 8160.75 utterances.], batch size: 38, lr: 3.29e-04
|
| 32 |
+
2022-08-11 19:30:14,319 INFO [train.py:737] (2/3) Epoch 16, batch 600, loss[loss=0.5106, over 9599.00 frames; utt_duration=418.7 frames, utt_pad_proportion=0.3578, over 92.00 utterances.], tot_loss[loss=0.3602, over 2724270.24 frames; utt_duration=1274 frames, utt_pad_proportion=0.04791, over 8561.98 utterances.], batch size: 92, lr: 3.28e-04
|
| 33 |
+
2022-08-11 19:36:07,465 INFO [train.py:737] (2/3) Epoch 16, batch 700, loss[loss=0.3604, over 14455.00 frames; utt_duration=1754 frames, utt_pad_proportion=0.009296, over 33.00 utterances.], tot_loss[loss=0.3659, over 2764837.56 frames; utt_duration=1243 frames, utt_pad_proportion=0.05699, over 8910.34 utterances.], batch size: 33, lr: 3.28e-04
|
| 34 |
+
2022-08-11 19:42:04,438 INFO [train.py:737] (2/3) Epoch 16, batch 800, loss[loss=0.4017, over 14671.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.01214, over 47.00 utterances.], tot_loss[loss=0.3635, over 2816239.34 frames; utt_duration=1282 frames, utt_pad_proportion=0.0464, over 8798.57 utterances.], batch size: 47, lr: 3.28e-04
|
| 35 |
+
2022-08-11 19:48:01,467 INFO [train.py:737] (2/3) Epoch 16, batch 900, loss[loss=0.3066, over 14587.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.00854, over 36.00 utterances.], tot_loss[loss=0.3623, over 2834461.64 frames; utt_duration=1284 frames, utt_pad_proportion=0.04597, over 8841.91 utterances.], batch size: 36, lr: 3.28e-04
|
| 36 |
+
2022-08-11 19:54:00,643 INFO [train.py:737] (2/3) Epoch 16, batch 1000, loss[loss=0.3931, over 14595.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.004582, over 41.00 utterances.], tot_loss[loss=0.3618, over 2850864.36 frames; utt_duration=1296 frames, utt_pad_proportion=0.04384, over 8808.56 utterances.], batch size: 41, lr: 3.28e-04
|
| 37 |
+
2022-08-11 19:59:54,048 INFO [train.py:737] (2/3) Epoch 16, batch 1100, loss[loss=0.3917, over 14698.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.009108, over 45.00 utterances.], tot_loss[loss=0.3586, over 2852902.62 frames; utt_duration=1290 frames, utt_pad_proportion=0.04705, over 8858.04 utterances.], batch size: 45, lr: 3.28e-04
|
| 38 |
+
2022-08-11 20:05:56,450 INFO [train.py:737] (2/3) Epoch 16, batch 1200, loss[loss=0.3935, over 14749.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.006209, over 43.00 utterances.], tot_loss[loss=0.3614, over 2855898.28 frames; utt_duration=1290 frames, utt_pad_proportion=0.04814, over 8867.21 utterances.], batch size: 43, lr: 3.28e-04
|
| 39 |
+
2022-08-11 20:11:53,132 INFO [train.py:737] (2/3) Epoch 16, batch 1300, loss[loss=0.316, over 14778.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.005764, over 38.00 utterances.], tot_loss[loss=0.3612, over 2845853.25 frames; utt_duration=1260 frames, utt_pad_proportion=0.05589, over 9043.84 utterances.], batch size: 38, lr: 3.28e-04
|
| 40 |
+
2022-08-11 20:17:51,406 INFO [train.py:737] (2/3) Epoch 16, batch 1400, loss[loss=0.3604, over 14685.00 frames; utt_duration=1279 frames, utt_pad_proportion=0.01194, over 46.00 utterances.], tot_loss[loss=0.3619, over 2858144.33 frames; utt_duration=1278 frames, utt_pad_proportion=0.04866, over 8955.68 utterances.], batch size: 46, lr: 3.28e-04
|
| 41 |
+
2022-08-11 20:23:48,378 INFO [train.py:737] (2/3) Epoch 16, batch 1500, loss[loss=0.4216, over 14516.00 frames; utt_duration=1211 frames, utt_pad_proportion=0.01594, over 48.00 utterances.], tot_loss[loss=0.3624, over 2866187.66 frames; utt_duration=1297 frames, utt_pad_proportion=0.04465, over 8849.94 utterances.], batch size: 48, lr: 3.27e-04
|
| 42 |
+
2022-08-11 20:29:43,722 INFO [train.py:737] (2/3) Epoch 16, batch 1600, loss[loss=0.3418, over 14672.00 frames; utt_duration=1306 frames, utt_pad_proportion=0.01088, over 45.00 utterances.], tot_loss[loss=0.3623, over 2848415.44 frames; utt_duration=1264 frames, utt_pad_proportion=0.05739, over 9026.47 utterances.], batch size: 45, lr: 3.27e-04
|
| 43 |
+
2022-08-11 20:35:44,437 INFO [train.py:737] (2/3) Epoch 16, batch 1700, loss[loss=0.4635, over 9834.00 frames; utt_duration=433.9 frames, utt_pad_proportion=0.3356, over 91.00 utterances.], tot_loss[loss=0.3615, over 2855919.33 frames; utt_duration=1278 frames, utt_pad_proportion=0.05246, over 8950.09 utterances.], batch size: 91, lr: 3.27e-04
|
| 44 |
+
2022-08-11 20:41:36,433 INFO [train.py:737] (2/3) Epoch 16, batch 1800, loss[loss=0.3904, over 14559.00 frames; utt_duration=1215 frames, utt_pad_proportion=0.0139, over 48.00 utterances.], tot_loss[loss=0.3632, over 2844376.08 frames; utt_duration=1252 frames, utt_pad_proportion=0.05986, over 9096.08 utterances.], batch size: 48, lr: 3.27e-04
|
| 45 |
+
2022-08-11 20:47:35,926 INFO [train.py:737] (2/3) Epoch 16, batch 1900, loss[loss=0.3563, over 12608.00 frames; utt_duration=754.2 frames, utt_pad_proportion=0.1459, over 67.00 utterances.], tot_loss[loss=0.3628, over 2852384.37 frames; utt_duration=1271 frames, utt_pad_proportion=0.05694, over 8987.84 utterances.], batch size: 67, lr: 3.27e-04
|
| 46 |
+
2022-08-11 20:53:36,130 INFO [train.py:737] (2/3) Epoch 16, batch 2000, loss[loss=0.3722, over 14582.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.005247, over 39.00 utterances.], tot_loss[loss=0.3646, over 2848380.25 frames; utt_duration=1260 frames, utt_pad_proportion=0.05994, over 9050.75 utterances.], batch size: 39, lr: 3.27e-04
|
| 47 |
+
2022-08-11 20:59:32,537 INFO [train.py:737] (2/3) Epoch 16, batch 2100, loss[loss=0.3485, over 14573.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.006454, over 40.00 utterances.], tot_loss[loss=0.3626, over 2856931.29 frames; utt_duration=1278 frames, utt_pad_proportion=0.05348, over 8949.36 utterances.], batch size: 40, lr: 3.27e-04
|
| 48 |
+
2022-08-11 21:05:29,584 INFO [train.py:737] (2/3) Epoch 16, batch 2200, loss[loss=0.2958, over 14704.00 frames; utt_duration=1682 frames, utt_pad_proportion=0.0116, over 35.00 utterances.], tot_loss[loss=0.3628, over 2862504.53 frames; utt_duration=1291 frames, utt_pad_proportion=0.04984, over 8877.27 utterances.], batch size: 35, lr: 3.27e-04
|
| 49 |
+
2022-08-11 21:11:25,898 INFO [train.py:737] (2/3) Epoch 16, batch 2300, loss[loss=0.3158, over 14466.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.009799, over 33.00 utterances.], tot_loss[loss=0.3608, over 2865288.52 frames; utt_duration=1294 frames, utt_pad_proportion=0.0487, over 8869.11 utterances.], batch size: 33, lr: 3.27e-04
|
| 50 |
+
2022-08-11 21:17:21,664 INFO [train.py:737] (2/3) Epoch 16, batch 2400, loss[loss=0.3698, over 14856.00 frames; utt_duration=1352 frames, utt_pad_proportion=0.00807, over 44.00 utterances.], tot_loss[loss=0.3604, over 2857110.29 frames; utt_duration=1280 frames, utt_pad_proportion=0.05449, over 8940.85 utterances.], batch size: 44, lr: 3.27e-04
|
| 51 |
+
2022-08-11 21:23:16,090 INFO [train.py:737] (2/3) Epoch 16, batch 2500, loss[loss=0.3911, over 14579.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006263, over 42.00 utterances.], tot_loss[loss=0.3627, over 2859525.12 frames; utt_duration=1271 frames, utt_pad_proportion=0.05398, over 9008.84 utterances.], batch size: 42, lr: 3.26e-04
|
| 52 |
+
2022-08-11 21:29:14,050 INFO [train.py:737] (2/3) Epoch 16, batch 2600, loss[loss=0.3446, over 14780.00 frames; utt_duration=1517 frames, utt_pad_proportion=0.005814, over 39.00 utterances.], tot_loss[loss=0.3637, over 2866743.27 frames; utt_duration=1286 frames, utt_pad_proportion=0.04874, over 8928.12 utterances.], batch size: 39, lr: 3.26e-04
|
| 53 |
+
2022-08-11 21:35:11,491 INFO [train.py:737] (2/3) Epoch 16, batch 2700, loss[loss=0.383, over 14705.00 frames; utt_duration=1309 frames, utt_pad_proportion=0.007885, over 45.00 utterances.], tot_loss[loss=0.3603, over 2877939.95 frames; utt_duration=1312 frames, utt_pad_proportion=0.04327, over 8783.98 utterances.], batch size: 45, lr: 3.26e-04
|
| 54 |
+
2022-08-11 21:41:05,997 INFO [train.py:737] (2/3) Epoch 16, batch 2800, loss[loss=0.4024, over 14501.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.01696, over 48.00 utterances.], tot_loss[loss=0.3625, over 2865997.86 frames; utt_duration=1290 frames, utt_pad_proportion=0.05039, over 8896.95 utterances.], batch size: 48, lr: 3.26e-04
|
| 55 |
+
2022-08-11 21:46:58,172 INFO [train.py:737] (2/3) Epoch 16, batch 2900, loss[loss=0.4569, over 14683.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009367, over 45.00 utterances.], tot_loss[loss=0.3747, over 2858698.94 frames; utt_duration=1262 frames, utt_pad_proportion=0.0544, over 9070.14 utterances.], batch size: 45, lr: 3.26e-04
|
| 56 |
+
2022-08-11 21:52:53,677 INFO [train.py:737] (2/3) Epoch 16, batch 3000, loss[loss=0.4405, over 14866.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.007387, over 44.00 utterances.], tot_loss[loss=0.395, over 2845055.53 frames; utt_duration=1239 frames, utt_pad_proportion=0.06191, over 9196.80 utterances.], batch size: 44, lr: 3.26e-04
|
| 57 |
+
2022-08-11 21:52:53,677 INFO [train.py:761] (2/3) Computing validation loss
|
| 58 |
+
2022-08-11 21:54:35,585 INFO [train.py:770] (2/3) Epoch 16, validation: loss=0.2879, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 59 |
+
2022-08-11 22:00:26,547 INFO [train.py:737] (2/3) Epoch 16, batch 3100, loss[loss=0.4342, over 14591.00 frames; utt_duration=1579 frames, utt_pad_proportion=0.005041, over 37.00 utterances.], tot_loss[loss=0.4112, over 2843337.74 frames; utt_duration=1241 frames, utt_pad_proportion=0.06334, over 9175.90 utterances.], batch size: 37, lr: 3.26e-04
|
| 60 |
+
2022-08-11 22:06:13,235 INFO [train.py:737] (2/3) Epoch 16, batch 3200, loss[loss=0.4305, over 14752.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.006575, over 42.00 utterances.], tot_loss[loss=0.4207, over 2814912.31 frames; utt_duration=1182 frames, utt_pad_proportion=0.07914, over 9537.41 utterances.], batch size: 42, lr: 3.26e-04
|
| 61 |
+
2022-08-11 22:12:06,480 INFO [train.py:737] (2/3) Epoch 16, batch 3300, loss[loss=0.4722, over 14683.00 frames; utt_duration=1251 frames, utt_pad_proportion=0.0109, over 47.00 utterances.], tot_loss[loss=0.4242, over 2849599.60 frames; utt_duration=1256 frames, utt_pad_proportion=0.05934, over 9087.42 utterances.], batch size: 47, lr: 3.26e-04
|
| 62 |
+
2022-08-11 22:17:57,672 INFO [train.py:737] (2/3) Epoch 16, batch 3400, loss[loss=0.424, over 14673.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.01247, over 47.00 utterances.], tot_loss[loss=0.4306, over 2832215.34 frames; utt_duration=1221 frames, utt_pad_proportion=0.0697, over 9286.32 utterances.], batch size: 47, lr: 3.26e-04
|
| 63 |
+
2022-08-11 22:23:49,963 INFO [train.py:737] (2/3) Epoch 16, batch 3500, loss[loss=0.4875, over 14535.00 frames; utt_duration=1164 frames, utt_pad_proportion=0.0202, over 50.00 utterances.], tot_loss[loss=0.4317, over 2834593.58 frames; utt_duration=1229 frames, utt_pad_proportion=0.06689, over 9235.89 utterances.], batch size: 50, lr: 3.25e-04
|
| 64 |
+
2022-08-11 22:29:40,570 INFO [train.py:737] (2/3) Epoch 16, batch 3600, loss[loss=0.5361, over 9797.00 frames; utt_duration=427.4 frames, utt_pad_proportion=0.3425, over 92.00 utterances.], tot_loss[loss=0.4385, over 2832801.85 frames; utt_duration=1219 frames, utt_pad_proportion=0.0683, over 9305.92 utterances.], batch size: 92, lr: 3.25e-04
|
| 65 |
+
2022-08-11 22:35:33,290 INFO [train.py:737] (2/3) Epoch 16, batch 3700, loss[loss=0.3691, over 14455.00 frames; utt_duration=1754 frames, utt_pad_proportion=0.01141, over 33.00 utterances.], tot_loss[loss=0.4389, over 2837246.20 frames; utt_duration=1233 frames, utt_pad_proportion=0.06583, over 9215.25 utterances.], batch size: 33, lr: 3.25e-04
|
| 66 |
+
2022-08-11 22:41:20,225 INFO [train.py:737] (2/3) Epoch 16, batch 3800, loss[loss=0.4741, over 12927.00 frames; utt_duration=773.3 frames, utt_pad_proportion=0.1243, over 67.00 utterances.], tot_loss[loss=0.4415, over 2833864.51 frames; utt_duration=1224 frames, utt_pad_proportion=0.06705, over 9275.13 utterances.], batch size: 67, lr: 3.25e-04
|
| 67 |
+
2022-08-11 22:47:10,485 INFO [train.py:737] (2/3) Epoch 16, batch 3900, loss[loss=0.379, over 14582.00 frames; utt_duration=1497 frames, utt_pad_proportion=0.006027, over 39.00 utterances.], tot_loss[loss=0.4385, over 2823099.44 frames; utt_duration=1205 frames, utt_pad_proportion=0.07335, over 9384.19 utterances.], batch size: 39, lr: 3.25e-04
|
| 68 |
+
2022-08-11 22:53:00,686 INFO [train.py:737] (2/3) Epoch 16, batch 4000, loss[loss=0.4351, over 14580.00 frames; utt_duration=1423 frames, utt_pad_proportion=0.006655, over 41.00 utterances.], tot_loss[loss=0.4395, over 2828401.45 frames; utt_duration=1219 frames, utt_pad_proportion=0.0703, over 9289.30 utterances.], batch size: 41, lr: 3.25e-04
|
| 69 |
+
2022-08-11 22:58:51,027 INFO [train.py:737] (2/3) Epoch 16, batch 4100, loss[loss=0.4194, over 14566.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006659, over 40.00 utterances.], tot_loss[loss=0.4422, over 2812232.21 frames; utt_duration=1186 frames, utt_pad_proportion=0.08035, over 9497.56 utterances.], batch size: 40, lr: 3.25e-04
|
| 70 |
+
2022-08-11 23:04:44,506 INFO [train.py:737] (2/3) Epoch 16, batch 4200, loss[loss=0.448, over 14461.00 frames; utt_duration=1159 frames, utt_pad_proportion=0.0247, over 50.00 utterances.], tot_loss[loss=0.4428, over 2818235.11 frames; utt_duration=1190 frames, utt_pad_proportion=0.07566, over 9483.37 utterances.], batch size: 50, lr: 3.25e-04
|
| 71 |
+
2022-08-11 23:10:34,629 INFO [train.py:737] (2/3) Epoch 16, batch 4300, loss[loss=0.4475, over 14577.00 frames; utt_duration=1578 frames, utt_pad_proportion=0.005978, over 37.00 utterances.], tot_loss[loss=0.4425, over 2827867.61 frames; utt_duration=1219 frames, utt_pad_proportion=0.07091, over 9290.70 utterances.], batch size: 37, lr: 3.25e-04
|
| 72 |
+
2022-08-11 23:16:26,641 INFO [train.py:737] (2/3) Epoch 16, batch 4400, loss[loss=0.3593, over 14752.00 frames; utt_duration=1688 frames, utt_pad_proportion=0.007928, over 35.00 utterances.], tot_loss[loss=0.4413, over 2827996.46 frames; utt_duration=1228 frames, utt_pad_proportion=0.07091, over 9221.50 utterances.], batch size: 35, lr: 3.24e-04
|
| 73 |
+
2022-08-11 23:22:14,315 INFO [train.py:737] (2/3) Epoch 16, batch 4500, loss[loss=0.4613, over 14585.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.00714, over 38.00 utterances.], tot_loss[loss=0.445, over 2826095.20 frames; utt_duration=1210 frames, utt_pad_proportion=0.07121, over 9350.59 utterances.], batch size: 38, lr: 3.24e-04
|
| 74 |
+
2022-08-11 23:28:03,895 INFO [train.py:737] (2/3) Epoch 16, batch 4600, loss[loss=0.4416, over 13676.00 frames; utt_duration=961.1 frames, utt_pad_proportion=0.07228, over 57.00 utterances.], tot_loss[loss=0.4452, over 2820792.81 frames; utt_duration=1191 frames, utt_pad_proportion=0.07369, over 9488.01 utterances.], batch size: 57, lr: 3.24e-04
|
| 75 |
+
2022-08-11 23:33:56,639 INFO [train.py:737] (2/3) Epoch 16, batch 4700, loss[loss=0.4218, over 14755.00 frames; utt_duration=1596 frames, utt_pad_proportion=0.007878, over 37.00 utterances.], tot_loss[loss=0.4435, over 2816432.56 frames; utt_duration=1194 frames, utt_pad_proportion=0.07557, over 9444.88 utterances.], batch size: 37, lr: 3.24e-04
|
| 76 |
+
2022-08-11 23:39:49,294 INFO [train.py:737] (2/3) Epoch 16, batch 4800, loss[loss=0.4609, over 14759.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.00639, over 42.00 utterances.], tot_loss[loss=0.444, over 2827966.46 frames; utt_duration=1212 frames, utt_pad_proportion=0.06916, over 9347.70 utterances.], batch size: 42, lr: 3.24e-04
|
| 77 |
+
2022-08-11 23:45:40,798 INFO [train.py:737] (2/3) Epoch 16, batch 4900, loss[loss=0.4627, over 14568.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.005879, over 40.00 utterances.], tot_loss[loss=0.4405, over 2825974.94 frames; utt_duration=1207 frames, utt_pad_proportion=0.07179, over 9378.19 utterances.], batch size: 40, lr: 3.24e-04
|
| 78 |
+
2022-08-11 23:51:33,921 INFO [train.py:737] (2/3) Epoch 16, batch 5000, loss[loss=0.3776, over 14435.00 frames; utt_duration=1651 frames, utt_pad_proportion=0.009495, over 35.00 utterances.], tot_loss[loss=0.4396, over 2816084.63 frames; utt_duration=1197 frames, utt_pad_proportion=0.07648, over 9420.99 utterances.], batch size: 35, lr: 3.24e-04
|
| 79 |
+
2022-08-11 23:57:27,494 INFO [train.py:737] (2/3) Epoch 16, batch 5100, loss[loss=0.3837, over 14615.00 frames; utt_duration=1625 frames, utt_pad_proportion=0.006656, over 36.00 utterances.], tot_loss[loss=0.4395, over 2825261.08 frames; utt_duration=1214 frames, utt_pad_proportion=0.07196, over 9319.79 utterances.], batch size: 36, lr: 3.24e-04
|
| 80 |
+
2022-08-12 00:03:19,189 INFO [train.py:737] (2/3) Epoch 16, batch 5200, loss[loss=0.4729, over 14646.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.007361, over 44.00 utterances.], tot_loss[loss=0.441, over 2819710.52 frames; utt_duration=1198 frames, utt_pad_proportion=0.07601, over 9422.93 utterances.], batch size: 44, lr: 3.24e-04
|
| 81 |
+
2022-08-12 00:09:07,288 INFO [train.py:737] (2/3) Epoch 16, batch 5300, loss[loss=0.4735, over 13730.00 frames; utt_duration=965 frames, utt_pad_proportion=0.07034, over 57.00 utterances.], tot_loss[loss=0.4426, over 2822002.38 frames; utt_duration=1202 frames, utt_pad_proportion=0.07408, over 9403.48 utterances.], batch size: 57, lr: 3.24e-04
|
| 82 |
+
2022-08-12 00:15:01,426 INFO [train.py:737] (2/3) Epoch 16, batch 5400, loss[loss=0.4782, over 14751.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.005908, over 41.00 utterances.], tot_loss[loss=0.4438, over 2825157.85 frames; utt_duration=1212 frames, utt_pad_proportion=0.07388, over 9332.01 utterances.], batch size: 41, lr: 3.23e-04
|
| 83 |
+
2022-08-12 00:20:52,398 INFO [train.py:737] (2/3) Epoch 16, batch 5500, loss[loss=0.4526, over 14367.00 frames; utt_duration=1086 frames, utt_pad_proportion=0.03552, over 53.00 utterances.], tot_loss[loss=0.4446, over 2827971.53 frames; utt_duration=1216 frames, utt_pad_proportion=0.07163, over 9316.83 utterances.], batch size: 53, lr: 3.23e-04
|
| 84 |
+
2022-08-12 00:26:40,779 INFO [train.py:737] (2/3) Epoch 16, batch 5600, loss[loss=0.498, over 12921.00 frames; utt_duration=772.8 frames, utt_pad_proportion=0.1287, over 67.00 utterances.], tot_loss[loss=0.4407, over 2835069.21 frames; utt_duration=1229 frames, utt_pad_proportion=0.06844, over 9237.08 utterances.], batch size: 67, lr: 3.23e-04
|
| 85 |
+
2022-08-12 00:32:31,460 INFO [train.py:737] (2/3) Epoch 16, batch 5700, loss[loss=0.3641, over 14454.00 frames; utt_duration=1753 frames, utt_pad_proportion=0.0105, over 33.00 utterances.], tot_loss[loss=0.4406, over 2836769.22 frames; utt_duration=1240 frames, utt_pad_proportion=0.06709, over 9165.01 utterances.], batch size: 33, lr: 3.23e-04
|
| 86 |
+
2022-08-12 00:38:21,055 INFO [train.py:737] (2/3) Epoch 16, batch 5800, loss[loss=0.3547, over 14783.00 frames; utt_duration=1599 frames, utt_pad_proportion=0.005929, over 37.00 utterances.], tot_loss[loss=0.4412, over 2830552.75 frames; utt_duration=1216 frames, utt_pad_proportion=0.06954, over 9324.48 utterances.], batch size: 37, lr: 3.23e-04
|
| 87 |
+
2022-08-12 00:44:13,575 INFO [train.py:737] (2/3) Epoch 16, batch 5900, loss[loss=0.5077, over 13031.00 frames; utt_duration=768 frames, utt_pad_proportion=0.1292, over 68.00 utterances.], tot_loss[loss=0.4385, over 2815089.46 frames; utt_duration=1190 frames, utt_pad_proportion=0.07846, over 9478.40 utterances.], batch size: 68, lr: 3.23e-04
|
| 88 |
+
2022-08-12 00:50:02,626 INFO [train.py:737] (2/3) Epoch 16, batch 6000, loss[loss=0.4451, over 14646.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.007463, over 44.00 utterances.], tot_loss[loss=0.4399, over 2811969.27 frames; utt_duration=1187 frames, utt_pad_proportion=0.07979, over 9485.64 utterances.], batch size: 44, lr: 3.23e-04
|
| 89 |
+
2022-08-12 00:50:02,627 INFO [train.py:761] (2/3) Computing validation loss
|
| 90 |
+
2022-08-12 00:51:42,069 INFO [train.py:770] (2/3) Epoch 16, validation: loss=0.2945, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 91 |
+
2022-08-12 00:57:06,529 INFO [train.py:737] (2/3) Epoch 16, batch 6100, loss[loss=0.5674, over 10257.00 frames; utt_duration=447.4 frames, utt_pad_proportion=0.3095, over 92.00 utterances.], tot_loss[loss=0.4454, over 2746033.36 frames; utt_duration=1080 frames, utt_pad_proportion=0.1153, over 10185.29 utterances.], batch size: 92, lr: 3.23e-04
|
| 92 |
+
2022-08-12 00:58:41,880 INFO [train.py:737] (2/3) Epoch 17, batch 0, loss[loss=0.4031, over 14748.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.006194, over 41.00 utterances.], tot_loss[loss=0.4031, over 14748.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.006194, over 41.00 utterances.], batch size: 41, lr: 3.14e-04
|
| 93 |
+
2022-08-12 01:04:37,182 INFO [train.py:737] (2/3) Epoch 17, batch 100, loss[loss=0.2913, over 14602.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01071, over 34.00 utterances.], tot_loss[loss=0.3649, over 1139805.18 frames; utt_duration=1287 frames, utt_pad_proportion=0.04738, over 3546.37 utterances.], batch size: 34, lr: 3.14e-04
|
| 94 |
+
2022-08-12 01:10:31,073 INFO [train.py:737] (2/3) Epoch 17, batch 200, loss[loss=0.3289, over 14580.00 frames; utt_duration=1578 frames, utt_pad_proportion=0.005944, over 37.00 utterances.], tot_loss[loss=0.3593, over 1817791.02 frames; utt_duration=1284 frames, utt_pad_proportion=0.05016, over 5670.27 utterances.], batch size: 37, lr: 3.14e-04
|
| 95 |
+
2022-08-12 01:16:27,829 INFO [train.py:737] (2/3) Epoch 17, batch 300, loss[loss=0.3499, over 14308.00 frames; utt_duration=1082 frames, utt_pad_proportion=0.03609, over 53.00 utterances.], tot_loss[loss=0.3595, over 2233327.36 frames; utt_duration=1287 frames, utt_pad_proportion=0.04816, over 6951.77 utterances.], batch size: 53, lr: 3.14e-04
|
| 96 |
+
2022-08-12 01:22:20,487 INFO [train.py:737] (2/3) Epoch 17, batch 400, loss[loss=0.3578, over 13140.00 frames; utt_duration=774.6 frames, utt_pad_proportion=0.1188, over 68.00 utterances.], tot_loss[loss=0.3572, over 2482483.17 frames; utt_duration=1292 frames, utt_pad_proportion=0.04608, over 7693.53 utterances.], batch size: 68, lr: 3.13e-04
|
| 97 |
+
2022-08-12 01:28:17,404 INFO [train.py:737] (2/3) Epoch 17, batch 500, loss[loss=0.3437, over 14598.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.005716, over 38.00 utterances.], tot_loss[loss=0.3533, over 2645363.32 frames; utt_duration=1325 frames, utt_pad_proportion=0.04001, over 7993.37 utterances.], batch size: 38, lr: 3.13e-04
|
| 98 |
+
2022-08-12 01:34:10,598 INFO [train.py:737] (2/3) Epoch 17, batch 600, loss[loss=0.376, over 14559.00 frames; utt_duration=1457 frames, utt_pad_proportion=0.007221, over 40.00 utterances.], tot_loss[loss=0.3558, over 2736528.68 frames; utt_duration=1328 frames, utt_pad_proportion=0.04004, over 8251.13 utterances.], batch size: 40, lr: 3.13e-04
|
| 99 |
+
2022-08-12 01:40:07,110 INFO [train.py:737] (2/3) Epoch 17, batch 700, loss[loss=0.3794, over 14757.00 frames; utt_duration=1374 frames, utt_pad_proportion=0.00589, over 43.00 utterances.], tot_loss[loss=0.3588, over 2788336.22 frames; utt_duration=1313 frames, utt_pad_proportion=0.04329, over 8507.07 utterances.], batch size: 43, lr: 3.13e-04
|
| 100 |
+
2022-08-12 01:46:04,164 INFO [train.py:737] (2/3) Epoch 17, batch 800, loss[loss=0.3696, over 14756.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.006373, over 42.00 utterances.], tot_loss[loss=0.3582, over 2812198.40 frames; utt_duration=1286 frames, utt_pad_proportion=0.04849, over 8754.54 utterances.], batch size: 42, lr: 3.13e-04
|
| 101 |
+
2022-08-12 01:52:03,017 INFO [train.py:737] (2/3) Epoch 17, batch 900, loss[loss=0.3343, over 14735.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.007152, over 43.00 utterances.], tot_loss[loss=0.3569, over 2835947.84 frames; utt_duration=1295 frames, utt_pad_proportion=0.04684, over 8770.73 utterances.], batch size: 43, lr: 3.13e-04
|
| 102 |
+
2022-08-12 01:57:59,864 INFO [train.py:737] (2/3) Epoch 17, batch 1000, loss[loss=0.3905, over 14732.00 frames; utt_duration=1372 frames, utt_pad_proportion=0.00645, over 43.00 utterances.], tot_loss[loss=0.3593, over 2842433.99 frames; utt_duration=1277 frames, utt_pad_proportion=0.05044, over 8912.39 utterances.], batch size: 43, lr: 3.13e-04
|
| 103 |
+
2022-08-12 02:03:56,400 INFO [train.py:737] (2/3) Epoch 17, batch 1100, loss[loss=0.3832, over 13236.00 frames; utt_duration=780 frames, utt_pad_proportion=0.1156, over 68.00 utterances.], tot_loss[loss=0.3571, over 2863452.63 frames; utt_duration=1302 frames, utt_pad_proportion=0.04195, over 8808.64 utterances.], batch size: 68, lr: 3.13e-04
|
| 104 |
+
2022-08-12 02:09:56,095 INFO [train.py:737] (2/3) Epoch 17, batch 1200, loss[loss=0.3417, over 14435.00 frames; utt_duration=1751 frames, utt_pad_proportion=0.01279, over 33.00 utterances.], tot_loss[loss=0.359, over 2866037.55 frames; utt_duration=1301 frames, utt_pad_proportion=0.04299, over 8824.61 utterances.], batch size: 33, lr: 3.13e-04
|
| 105 |
+
2022-08-12 02:15:53,093 INFO [train.py:737] (2/3) Epoch 17, batch 1300, loss[loss=0.3185, over 14478.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.01006, over 33.00 utterances.], tot_loss[loss=0.3602, over 2861385.67 frames; utt_duration=1287 frames, utt_pad_proportion=0.04724, over 8905.33 utterances.], batch size: 33, lr: 3.13e-04
|
| 106 |
+
2022-08-12 02:21:48,046 INFO [train.py:737] (2/3) Epoch 17, batch 1400, loss[loss=0.347, over 14764.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.006053, over 42.00 utterances.], tot_loss[loss=0.3609, over 2868437.38 frames; utt_duration=1293 frames, utt_pad_proportion=0.04457, over 8887.19 utterances.], batch size: 42, lr: 3.12e-04
|
| 107 |
+
2022-08-12 02:27:45,480 INFO [train.py:737] (2/3) Epoch 17, batch 1500, loss[loss=0.3551, over 14651.00 frames; utt_duration=1333 frames, utt_pad_proportion=0.007175, over 44.00 utterances.], tot_loss[loss=0.3618, over 2862941.22 frames; utt_duration=1290 frames, utt_pad_proportion=0.04886, over 8887.75 utterances.], batch size: 44, lr: 3.12e-04
|
| 108 |
+
2022-08-12 02:33:41,560 INFO [train.py:737] (2/3) Epoch 17, batch 1600, loss[loss=0.3954, over 14500.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.01777, over 48.00 utterances.], tot_loss[loss=0.3586, over 2873685.64 frames; utt_duration=1316 frames, utt_pad_proportion=0.04278, over 8742.19 utterances.], batch size: 48, lr: 3.12e-04
|
| 109 |
+
2022-08-12 02:39:40,258 INFO [train.py:737] (2/3) Epoch 17, batch 1700, loss[loss=0.319, over 14474.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.01033, over 33.00 utterances.], tot_loss[loss=0.3598, over 2877536.29 frames; utt_duration=1318 frames, utt_pad_proportion=0.03911, over 8739.68 utterances.], batch size: 33, lr: 3.12e-04
|
| 110 |
+
2022-08-12 02:45:36,012 INFO [train.py:737] (2/3) Epoch 17, batch 1800, loss[loss=0.3955, over 14368.00 frames; utt_duration=1086 frames, utt_pad_proportion=0.03488, over 53.00 utterances.], tot_loss[loss=0.36, over 2871240.82 frames; utt_duration=1305 frames, utt_pad_proportion=0.04341, over 8810.32 utterances.], batch size: 53, lr: 3.12e-04
|
| 111 |
+
2022-08-12 02:51:32,200 INFO [train.py:737] (2/3) Epoch 17, batch 1900, loss[loss=0.3977, over 14748.00 frames; utt_duration=1440 frames, utt_pad_proportion=0.005976, over 41.00 utterances.], tot_loss[loss=0.3612, over 2876271.44 frames; utt_duration=1312 frames, utt_pad_proportion=0.04134, over 8777.24 utterances.], batch size: 41, lr: 3.12e-04
|
| 112 |
+
2022-08-12 02:57:29,847 INFO [train.py:737] (2/3) Epoch 17, batch 2000, loss[loss=0.3673, over 14481.00 frames; utt_duration=1208 frames, utt_pad_proportion=0.01841, over 48.00 utterances.], tot_loss[loss=0.361, over 2871819.13 frames; utt_duration=1295 frames, utt_pad_proportion=0.0445, over 8882.61 utterances.], batch size: 48, lr: 3.12e-04
|
| 113 |
+
2022-08-12 03:03:23,355 INFO [train.py:737] (2/3) Epoch 17, batch 2100, loss[loss=0.409, over 14670.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.01186, over 47.00 utterances.], tot_loss[loss=0.3595, over 2877925.05 frames; utt_duration=1312 frames, utt_pad_proportion=0.04065, over 8786.87 utterances.], batch size: 47, lr: 3.12e-04
|
| 114 |
+
2022-08-12 03:09:19,729 INFO [train.py:737] (2/3) Epoch 17, batch 2200, loss[loss=0.3598, over 14582.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.007344, over 38.00 utterances.], tot_loss[loss=0.3601, over 2854938.32 frames; utt_duration=1275 frames, utt_pad_proportion=0.05566, over 8970.32 utterances.], batch size: 38, lr: 3.12e-04
|
| 115 |
+
2022-08-12 03:15:11,264 INFO [train.py:737] (2/3) Epoch 17, batch 2300, loss[loss=0.3744, over 14726.00 frames; utt_duration=1282 frames, utt_pad_proportion=0.01002, over 46.00 utterances.], tot_loss[loss=0.362, over 2853246.34 frames; utt_duration=1263 frames, utt_pad_proportion=0.05629, over 9049.50 utterances.], batch size: 46, lr: 3.12e-04
|
| 116 |
+
2022-08-12 03:21:06,344 INFO [train.py:737] (2/3) Epoch 17, batch 2400, loss[loss=0.3589, over 14562.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.005576, over 40.00 utterances.], tot_loss[loss=0.3607, over 2866419.82 frames; utt_duration=1281 frames, utt_pad_proportion=0.04766, over 8961.05 utterances.], batch size: 40, lr: 3.12e-04
|
| 117 |
+
2022-08-12 03:27:05,203 INFO [train.py:737] (2/3) Epoch 17, batch 2500, loss[loss=0.3461, over 13723.00 frames; utt_duration=964.5 frames, utt_pad_proportion=0.06996, over 57.00 utterances.], tot_loss[loss=0.3586, over 2872948.14 frames; utt_duration=1304 frames, utt_pad_proportion=0.04348, over 8823.89 utterances.], batch size: 57, lr: 3.11e-04
|
| 118 |
+
2022-08-12 03:33:01,500 INFO [train.py:737] (2/3) Epoch 17, batch 2600, loss[loss=0.3746, over 14378.00 frames; utt_duration=1087 frames, utt_pad_proportion=0.03408, over 53.00 utterances.], tot_loss[loss=0.3579, over 2868751.83 frames; utt_duration=1306 frames, utt_pad_proportion=0.04579, over 8798.67 utterances.], batch size: 53, lr: 3.11e-04
|
| 119 |
+
2022-08-12 03:38:55,047 INFO [train.py:737] (2/3) Epoch 17, batch 2700, loss[loss=0.3135, over 14728.00 frames; utt_duration=1371 frames, utt_pad_proportion=0.007623, over 43.00 utterances.], tot_loss[loss=0.3588, over 2877369.04 frames; utt_duration=1320 frames, utt_pad_proportion=0.04259, over 8729.50 utterances.], batch size: 43, lr: 3.11e-04
|
| 120 |
+
2022-08-12 03:44:52,668 INFO [train.py:737] (2/3) Epoch 17, batch 2800, loss[loss=0.4592, over 14042.00 frames; utt_duration=1813 frames, utt_pad_proportion=0.05408, over 31.00 utterances.], tot_loss[loss=0.3599, over 2876920.30 frames; utt_duration=1318 frames, utt_pad_proportion=0.04286, over 8741.02 utterances.], batch size: 31, lr: 3.11e-04
|
| 121 |
+
2022-08-12 03:50:46,293 INFO [train.py:737] (2/3) Epoch 17, batch 2900, loss[loss=0.378, over 14440.00 frames; utt_duration=1157 frames, utt_pad_proportion=0.02548, over 50.00 utterances.], tot_loss[loss=0.3734, over 2873215.83 frames; utt_duration=1310 frames, utt_pad_proportion=0.04478, over 8784.93 utterances.], batch size: 50, lr: 3.11e-04
|
| 122 |
+
2022-08-12 03:56:37,635 INFO [train.py:737] (2/3) Epoch 17, batch 3000, loss[loss=0.4477, over 14668.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.01269, over 47.00 utterances.], tot_loss[loss=0.3945, over 2854703.91 frames; utt_duration=1270 frames, utt_pad_proportion=0.05527, over 9000.10 utterances.], batch size: 47, lr: 3.11e-04
|
| 123 |
+
2022-08-12 03:56:37,635 INFO [train.py:761] (2/3) Computing validation loss
|
| 124 |
+
2022-08-12 03:58:13,086 INFO [train.py:770] (2/3) Epoch 17, validation: loss=0.2805, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 125 |
+
2022-08-12 04:04:03,703 INFO [train.py:737] (2/3) Epoch 17, batch 3100, loss[loss=0.3706, over 14765.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007739, over 37.00 utterances.], tot_loss[loss=0.4065, over 2857642.61 frames; utt_duration=1278 frames, utt_pad_proportion=0.05333, over 8952.64 utterances.], batch size: 37, lr: 3.11e-04
|
| 126 |
+
2022-08-12 04:09:52,795 INFO [train.py:737] (2/3) Epoch 17, batch 3200, loss[loss=0.4418, over 14596.00 frames; utt_duration=1623 frames, utt_pad_proportion=0.006647, over 36.00 utterances.], tot_loss[loss=0.4189, over 2850406.59 frames; utt_duration=1257 frames, utt_pad_proportion=0.0568, over 9083.77 utterances.], batch size: 36, lr: 3.11e-04
|
| 127 |
+
2022-08-12 04:15:46,143 INFO [train.py:737] (2/3) Epoch 17, batch 3300, loss[loss=0.3883, over 14472.00 frames; utt_duration=1756 frames, utt_pad_proportion=0.01025, over 33.00 utterances.], tot_loss[loss=0.4263, over 2854093.98 frames; utt_duration=1267 frames, utt_pad_proportion=0.05501, over 9020.26 utterances.], batch size: 33, lr: 3.11e-04
|
| 128 |
+
2022-08-12 04:21:38,716 INFO [train.py:737] (2/3) Epoch 17, batch 3400, loss[loss=0.4449, over 14571.00 frames; utt_duration=1496 frames, utt_pad_proportion=0.006657, over 39.00 utterances.], tot_loss[loss=0.4352, over 2839316.79 frames; utt_duration=1233 frames, utt_pad_proportion=0.06439, over 9219.98 utterances.], batch size: 39, lr: 3.11e-04
|
| 129 |
+
2022-08-12 04:27:31,852 INFO [train.py:737] (2/3) Epoch 17, batch 3500, loss[loss=0.445, over 14740.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.00613, over 43.00 utterances.], tot_loss[loss=0.436, over 2820579.18 frames; utt_duration=1212 frames, utt_pad_proportion=0.07615, over 9323.40 utterances.], batch size: 43, lr: 3.10e-04
|
| 130 |
+
2022-08-12 04:33:19,371 INFO [train.py:737] (2/3) Epoch 17, batch 3600, loss[loss=0.4823, over 14489.00 frames; utt_duration=1161 frames, utt_pad_proportion=0.02148, over 50.00 utterances.], tot_loss[loss=0.4385, over 2810902.95 frames; utt_duration=1184 frames, utt_pad_proportion=0.0798, over 9508.39 utterances.], batch size: 50, lr: 3.10e-04
|
| 131 |
+
2022-08-12 04:39:12,740 INFO [train.py:737] (2/3) Epoch 17, batch 3700, loss[loss=0.4062, over 14590.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.006413, over 38.00 utterances.], tot_loss[loss=0.4398, over 2811526.55 frames; utt_duration=1184 frames, utt_pad_proportion=0.07895, over 9513.34 utterances.], batch size: 38, lr: 3.10e-04
|
| 132 |
+
2022-08-12 04:45:00,418 INFO [train.py:737] (2/3) Epoch 17, batch 3800, loss[loss=0.4228, over 14506.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.01767, over 48.00 utterances.], tot_loss[loss=0.4421, over 2792093.28 frames; utt_duration=1155 frames, utt_pad_proportion=0.09039, over 9683.59 utterances.], batch size: 48, lr: 3.10e-04
|
| 133 |
+
2022-08-12 04:50:52,734 INFO [train.py:737] (2/3) Epoch 17, batch 3900, loss[loss=0.5261, over 9859.00 frames; utt_duration=425.4 frames, utt_pad_proportion=0.3363, over 93.00 utterances.], tot_loss[loss=0.4413, over 2803730.25 frames; utt_duration=1170 frames, utt_pad_proportion=0.08323, over 9597.21 utterances.], batch size: 93, lr: 3.10e-04
|
| 134 |
+
2022-08-12 04:56:42,374 INFO [train.py:737] (2/3) Epoch 17, batch 4000, loss[loss=0.418, over 14474.00 frames; utt_duration=1160 frames, utt_pad_proportion=0.02391, over 50.00 utterances.], tot_loss[loss=0.4397, over 2822304.35 frames; utt_duration=1205 frames, utt_pad_proportion=0.07323, over 9379.58 utterances.], batch size: 50, lr: 3.10e-04
|
| 135 |
+
2022-08-12 05:02:33,028 INFO [train.py:737] (2/3) Epoch 17, batch 4100, loss[loss=0.461, over 14657.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01267, over 47.00 utterances.], tot_loss[loss=0.4407, over 2825758.94 frames; utt_duration=1212 frames, utt_pad_proportion=0.07287, over 9337.09 utterances.], batch size: 47, lr: 3.10e-04
|
| 136 |
+
2022-08-12 05:08:23,748 INFO [train.py:737] (2/3) Epoch 17, batch 4200, loss[loss=0.4433, over 14569.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.005896, over 40.00 utterances.], tot_loss[loss=0.437, over 2839570.01 frames; utt_duration=1246 frames, utt_pad_proportion=0.06564, over 9127.99 utterances.], batch size: 40, lr: 3.10e-04
|
| 137 |
+
2022-08-12 05:14:14,533 INFO [train.py:737] (2/3) Epoch 17, batch 4300, loss[loss=0.4014, over 14564.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.006897, over 37.00 utterances.], tot_loss[loss=0.4397, over 2831753.75 frames; utt_duration=1219 frames, utt_pad_proportion=0.06862, over 9306.40 utterances.], batch size: 37, lr: 3.10e-04
|
| 138 |
+
2022-08-12 05:20:08,615 INFO [train.py:737] (2/3) Epoch 17, batch 4400, loss[loss=0.3978, over 14570.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006437, over 37.00 utterances.], tot_loss[loss=0.44, over 2810664.99 frames; utt_duration=1186 frames, utt_pad_proportion=0.08189, over 9491.50 utterances.], batch size: 37, lr: 3.10e-04
|
| 139 |
+
2022-08-12 05:26:02,686 INFO [train.py:737] (2/3) Epoch 17, batch 4500, loss[loss=0.4647, over 14655.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.01365, over 47.00 utterances.], tot_loss[loss=0.4394, over 2821438.52 frames; utt_duration=1199 frames, utt_pad_proportion=0.07628, over 9427.32 utterances.], batch size: 47, lr: 3.10e-04
|
| 140 |
+
2022-08-12 05:31:53,044 INFO [train.py:737] (2/3) Epoch 17, batch 4600, loss[loss=0.4049, over 14611.00 frames; utt_duration=1720 frames, utt_pad_proportion=0.01031, over 34.00 utterances.], tot_loss[loss=0.4377, over 2833933.85 frames; utt_duration=1224 frames, utt_pad_proportion=0.06752, over 9274.96 utterances.], batch size: 34, lr: 3.09e-04
|
| 141 |
+
2022-08-12 05:37:40,347 INFO [train.py:737] (2/3) Epoch 17, batch 4700, loss[loss=0.3469, over 14457.00 frames; utt_duration=1654 frames, utt_pad_proportion=0.008021, over 35.00 utterances.], tot_loss[loss=0.4367, over 2818674.43 frames; utt_duration=1203 frames, utt_pad_proportion=0.07733, over 9384.42 utterances.], batch size: 35, lr: 3.09e-04
|
| 142 |
+
2022-08-12 05:43:30,041 INFO [train.py:737] (2/3) Epoch 17, batch 4800, loss[loss=0.4215, over 14781.00 frames; utt_duration=1600 frames, utt_pad_proportion=0.005143, over 37.00 utterances.], tot_loss[loss=0.4382, over 2822188.52 frames; utt_duration=1200 frames, utt_pad_proportion=0.07444, over 9422.04 utterances.], batch size: 37, lr: 3.09e-04
|
| 143 |
+
2022-08-12 05:49:21,887 INFO [train.py:737] (2/3) Epoch 17, batch 4900, loss[loss=0.4135, over 14743.00 frames; utt_duration=1373 frames, utt_pad_proportion=0.005877, over 43.00 utterances.], tot_loss[loss=0.4347, over 2830627.44 frames; utt_duration=1219 frames, utt_pad_proportion=0.06844, over 9301.46 utterances.], batch size: 43, lr: 3.09e-04
|
| 144 |
+
2022-08-12 05:55:09,869 INFO [train.py:737] (2/3) Epoch 17, batch 5000, loss[loss=0.4093, over 14415.00 frames; utt_duration=1155 frames, utt_pad_proportion=0.02716, over 50.00 utterances.], tot_loss[loss=0.4348, over 2840053.48 frames; utt_duration=1231 frames, utt_pad_proportion=0.06427, over 9242.05 utterances.], batch size: 50, lr: 3.09e-04
|
| 145 |
+
2022-08-12 06:00:59,658 INFO [train.py:737] (2/3) Epoch 17, batch 5100, loss[loss=0.4205, over 14574.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.006233, over 40.00 utterances.], tot_loss[loss=0.4342, over 2824778.99 frames; utt_duration=1208 frames, utt_pad_proportion=0.07437, over 9362.93 utterances.], batch size: 40, lr: 3.09e-04
|
| 146 |
+
2022-08-12 06:06:51,513 INFO [train.py:737] (2/3) Epoch 17, batch 5200, loss[loss=0.3864, over 14579.00 frames; utt_duration=1536 frames, utt_pad_proportion=0.007514, over 38.00 utterances.], tot_loss[loss=0.4368, over 2830427.59 frames; utt_duration=1215 frames, utt_pad_proportion=0.07068, over 9326.68 utterances.], batch size: 38, lr: 3.09e-04
|
| 147 |
+
2022-08-12 06:12:42,099 INFO [train.py:737] (2/3) Epoch 17, batch 5300, loss[loss=0.4576, over 14886.00 frames; utt_duration=1355 frames, utt_pad_proportion=0.005853, over 44.00 utterances.], tot_loss[loss=0.4387, over 2815305.19 frames; utt_duration=1184 frames, utt_pad_proportion=0.07877, over 9526.10 utterances.], batch size: 44, lr: 3.09e-04
|
| 148 |
+
2022-08-12 06:18:31,726 INFO [train.py:737] (2/3) Epoch 17, batch 5400, loss[loss=0.3714, over 14063.00 frames; utt_duration=1816 frames, utt_pad_proportion=0.04019, over 31.00 utterances.], tot_loss[loss=0.4363, over 2822350.10 frames; utt_duration=1198 frames, utt_pad_proportion=0.07449, over 9436.51 utterances.], batch size: 31, lr: 3.09e-04
|
| 149 |
+
2022-08-12 06:24:22,452 INFO [train.py:737] (2/3) Epoch 17, batch 5500, loss[loss=0.4059, over 14050.00 frames; utt_duration=1815 frames, utt_pad_proportion=0.04943, over 31.00 utterances.], tot_loss[loss=0.4363, over 2832589.37 frames; utt_duration=1217 frames, utt_pad_proportion=0.06811, over 9322.30 utterances.], batch size: 31, lr: 3.09e-04
|
| 150 |
+
2022-08-12 06:30:10,219 INFO [train.py:737] (2/3) Epoch 17, batch 5600, loss[loss=0.5181, over 10150.00 frames; utt_duration=443.1 frames, utt_pad_proportion=0.3194, over 92.00 utterances.], tot_loss[loss=0.4365, over 2807905.01 frames; utt_duration=1173 frames, utt_pad_proportion=0.08165, over 9591.21 utterances.], batch size: 92, lr: 3.08e-04
|
| 151 |
+
2022-08-12 06:35:59,297 INFO [train.py:737] (2/3) Epoch 17, batch 5700, loss[loss=0.4582, over 14600.00 frames; utt_duration=1426 frames, utt_pad_proportion=0.005072, over 41.00 utterances.], tot_loss[loss=0.439, over 2821176.32 frames; utt_duration=1193 frames, utt_pad_proportion=0.07395, over 9469.37 utterances.], batch size: 41, lr: 3.08e-04
|
| 152 |
+
2022-08-12 06:41:51,110 INFO [train.py:737] (2/3) Epoch 17, batch 5800, loss[loss=0.5082, over 10009.00 frames; utt_duration=441.4 frames, utt_pad_proportion=0.3241, over 91.00 utterances.], tot_loss[loss=0.4405, over 2812328.82 frames; utt_duration=1185 frames, utt_pad_proportion=0.08057, over 9503.64 utterances.], batch size: 91, lr: 3.08e-04
|
| 153 |
+
2022-08-12 06:47:36,189 INFO [train.py:737] (2/3) Epoch 17, batch 5900, loss[loss=0.4183, over 14563.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.006812, over 37.00 utterances.], tot_loss[loss=0.4408, over 2810782.06 frames; utt_duration=1177 frames, utt_pad_proportion=0.08178, over 9564.45 utterances.], batch size: 37, lr: 3.08e-04
|
| 154 |
+
2022-08-12 06:53:24,449 INFO [train.py:737] (2/3) Epoch 17, batch 6000, loss[loss=0.4114, over 13712.00 frames; utt_duration=963.7 frames, utt_pad_proportion=0.0707, over 57.00 utterances.], tot_loss[loss=0.4371, over 2824084.19 frames; utt_duration=1208 frames, utt_pad_proportion=0.07446, over 9360.98 utterances.], batch size: 57, lr: 3.08e-04
|
| 155 |
+
2022-08-12 06:53:24,450 INFO [train.py:761] (2/3) Computing validation loss
|
| 156 |
+
2022-08-12 06:55:07,776 INFO [train.py:770] (2/3) Epoch 17, validation: loss=0.3115, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 157 |
+
2022-08-12 07:00:34,845 INFO [train.py:737] (2/3) Epoch 17, batch 6100, loss[loss=0.5273, over 9986.00 frames; utt_duration=435.8 frames, utt_pad_proportion=0.3296, over 92.00 utterances.], tot_loss[loss=0.4417, over 2765430.86 frames; utt_duration=1117 frames, utt_pad_proportion=0.1093, over 9914.53 utterances.], batch size: 92, lr: 3.08e-04
|
| 158 |
+
2022-08-12 07:02:15,428 INFO [train.py:737] (2/3) Epoch 18, batch 0, loss[loss=0.3517, over 14440.00 frames; utt_duration=1652 frames, utt_pad_proportion=0.009101, over 35.00 utterances.], tot_loss[loss=0.3517, over 14440.00 frames; utt_duration=1652 frames, utt_pad_proportion=0.009101, over 35.00 utterances.], batch size: 35, lr: 3.00e-04
|
| 159 |
+
2022-08-12 07:08:15,725 INFO [train.py:737] (2/3) Epoch 18, batch 100, loss[loss=0.3497, over 14014.00 frames; utt_duration=1809 frames, utt_pad_proportion=0.03857, over 31.00 utterances.], tot_loss[loss=0.3659, over 1130965.63 frames; utt_duration=1277 frames, utt_pad_proportion=0.05939, over 3547.06 utterances.], batch size: 31, lr: 3.00e-04
|
| 160 |
+
2022-08-12 07:14:10,520 INFO [train.py:737] (2/3) Epoch 18, batch 200, loss[loss=0.3823, over 12687.00 frames; utt_duration=758.9 frames, utt_pad_proportion=0.1435, over 67.00 utterances.], tot_loss[loss=0.3645, over 1826862.22 frames; utt_duration=1324 frames, utt_pad_proportion=0.03948, over 5523.79 utterances.], batch size: 67, lr: 3.00e-04
|
| 161 |
+
2022-08-12 07:20:03,958 INFO [train.py:737] (2/3) Epoch 18, batch 300, loss[loss=0.3376, over 14433.00 frames; utt_duration=1651 frames, utt_pad_proportion=0.009598, over 35.00 utterances.], tot_loss[loss=0.3612, over 2229834.41 frames; utt_duration=1294 frames, utt_pad_proportion=0.04916, over 6899.30 utterances.], batch size: 35, lr: 3.00e-04
|
| 162 |
+
2022-08-12 07:25:56,870 INFO [train.py:737] (2/3) Epoch 18, batch 400, loss[loss=0.3425, over 14313.00 frames; utt_duration=1082 frames, utt_pad_proportion=0.03928, over 53.00 utterances.], tot_loss[loss=0.3567, over 2481556.07 frames; utt_duration=1302 frames, utt_pad_proportion=0.04896, over 7630.24 utterances.], batch size: 53, lr: 3.00e-04
|
| 163 |
+
2022-08-12 07:31:51,511 INFO [train.py:737] (2/3) Epoch 18, batch 500, loss[loss=0.424, over 12732.00 frames; utt_duration=761.6 frames, utt_pad_proportion=0.1404, over 67.00 utterances.], tot_loss[loss=0.356, over 2625080.63 frames; utt_duration=1277 frames, utt_pad_proportion=0.05553, over 8232.70 utterances.], batch size: 67, lr: 2.99e-04
|
| 164 |
+
2022-08-12 07:37:46,287 INFO [train.py:737] (2/3) Epoch 18, batch 600, loss[loss=0.3463, over 14579.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.005317, over 40.00 utterances.], tot_loss[loss=0.3565, over 2718493.36 frames; utt_duration=1283 frames, utt_pad_proportion=0.05316, over 8487.84 utterances.], batch size: 40, lr: 2.99e-04
|
| 165 |
+
2022-08-12 07:43:44,663 INFO [train.py:737] (2/3) Epoch 18, batch 700, loss[loss=0.317, over 14096.00 frames; utt_duration=1820 frames, utt_pad_proportion=0.04994, over 31.00 utterances.], tot_loss[loss=0.3579, over 2771806.83 frames; utt_duration=1292 frames, utt_pad_proportion=0.05393, over 8589.55 utterances.], batch size: 31, lr: 2.99e-04
|
| 166 |
+
2022-08-12 07:49:44,657 INFO [train.py:737] (2/3) Epoch 18, batch 800, loss[loss=0.3925, over 14524.00 frames; utt_duration=1212 frames, utt_pad_proportion=0.01645, over 48.00 utterances.], tot_loss[loss=0.3589, over 2797692.34 frames; utt_duration=1269 frames, utt_pad_proportion=0.05792, over 8827.93 utterances.], batch size: 48, lr: 2.99e-04
|
| 167 |
+
2022-08-12 07:55:42,382 INFO [train.py:737] (2/3) Epoch 18, batch 900, loss[loss=0.3731, over 14588.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.007055, over 38.00 utterances.], tot_loss[loss=0.3613, over 2828633.65 frames; utt_duration=1282 frames, utt_pad_proportion=0.05201, over 8835.74 utterances.], batch size: 38, lr: 2.99e-04
|
| 168 |
+
2022-08-12 08:01:36,734 INFO [train.py:737] (2/3) Epoch 18, batch 1000, loss[loss=0.3465, over 14570.00 frames; utt_duration=1496 frames, utt_pad_proportion=0.006082, over 39.00 utterances.], tot_loss[loss=0.3612, over 2837279.75 frames; utt_duration=1274 frames, utt_pad_proportion=0.05336, over 8919.05 utterances.], batch size: 39, lr: 2.99e-04
|
| 169 |
+
2022-08-12 08:07:31,600 INFO [train.py:737] (2/3) Epoch 18, batch 1100, loss[loss=0.4695, over 9939.00 frames; utt_duration=433.3 frames, utt_pad_proportion=0.3333, over 92.00 utterances.], tot_loss[loss=0.3664, over 2840404.97 frames; utt_duration=1253 frames, utt_pad_proportion=0.05688, over 9076.72 utterances.], batch size: 92, lr: 2.99e-04
|
| 170 |
+
2022-08-12 08:13:30,145 INFO [train.py:737] (2/3) Epoch 18, batch 1200, loss[loss=0.3254, over 14751.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.006977, over 40.00 utterances.], tot_loss[loss=0.3648, over 2842385.87 frames; utt_duration=1246 frames, utt_pad_proportion=0.05709, over 9135.60 utterances.], batch size: 40, lr: 2.99e-04
|
| 171 |
+
2022-08-12 08:19:25,569 INFO [train.py:737] (2/3) Epoch 18, batch 1300, loss[loss=0.2925, over 14848.00 frames; utt_duration=1651 frames, utt_pad_proportion=0.008887, over 36.00 utterances.], tot_loss[loss=0.3625, over 2863880.86 frames; utt_duration=1285 frames, utt_pad_proportion=0.04688, over 8925.88 utterances.], batch size: 36, lr: 2.99e-04
|
| 172 |
+
2022-08-12 08:25:25,693 INFO [train.py:737] (2/3) Epoch 18, batch 1400, loss[loss=0.3523, over 14763.00 frames; utt_duration=1478 frames, utt_pad_proportion=0.005569, over 40.00 utterances.], tot_loss[loss=0.3599, over 2865845.08 frames; utt_duration=1287 frames, utt_pad_proportion=0.04656, over 8915.55 utterances.], batch size: 40, lr: 2.99e-04
|
| 173 |
+
2022-08-12 08:31:23,720 INFO [train.py:737] (2/3) Epoch 18, batch 1500, loss[loss=0.3426, over 14691.00 frames; utt_duration=1307 frames, utt_pad_proportion=0.009562, over 45.00 utterances.], tot_loss[loss=0.3591, over 2877437.97 frames; utt_duration=1309 frames, utt_pad_proportion=0.03984, over 8803.69 utterances.], batch size: 45, lr: 2.99e-04
|
| 174 |
+
2022-08-12 08:37:25,240 INFO [train.py:737] (2/3) Epoch 18, batch 1600, loss[loss=0.3112, over 14427.00 frames; utt_duration=1650 frames, utt_pad_proportion=0.009958, over 35.00 utterances.], tot_loss[loss=0.359, over 2883610.89 frames; utt_duration=1320 frames, utt_pad_proportion=0.03484, over 8746.36 utterances.], batch size: 35, lr: 2.98e-04
|
| 175 |
+
2022-08-12 08:43:19,483 INFO [train.py:737] (2/3) Epoch 18, batch 1700, loss[loss=0.3071, over 14470.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01054, over 33.00 utterances.], tot_loss[loss=0.3593, over 2875627.94 frames; utt_duration=1303 frames, utt_pad_proportion=0.03997, over 8839.82 utterances.], batch size: 33, lr: 2.98e-04
|
| 176 |
+
2022-08-12 08:49:17,870 INFO [train.py:737] (2/3) Epoch 18, batch 1800, loss[loss=0.3556, over 14452.00 frames; utt_duration=1653 frames, utt_pad_proportion=0.008381, over 35.00 utterances.], tot_loss[loss=0.3616, over 2861427.11 frames; utt_duration=1264 frames, utt_pad_proportion=0.04967, over 9065.74 utterances.], batch size: 35, lr: 2.98e-04
|
| 177 |
+
2022-08-12 08:55:13,602 INFO [train.py:737] (2/3) Epoch 18, batch 1900, loss[loss=0.3114, over 14599.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01026, over 34.00 utterances.], tot_loss[loss=0.3625, over 2851746.91 frames; utt_duration=1254 frames, utt_pad_proportion=0.05645, over 9105.03 utterances.], batch size: 34, lr: 2.98e-04
|
| 178 |
+
2022-08-12 08:59:37,386 INFO [train.py:737] (2/3) Epoch 18, batch 2000, loss[loss=0.4419, over 10247.00 frames; utt_duration=452 frames, utt_pad_proportion=0.3089, over 91.00 utterances.], tot_loss[loss=0.3619, over 2861266.07 frames; utt_duration=1277 frames, utt_pad_proportion=0.05034, over 8972.27 utterances.], batch size: 91, lr: 2.98e-04
|
| 179 |
+
2022-08-12 09:02:23,575 INFO [train.py:737] (2/3) Epoch 18, batch 2100, loss[loss=0.3272, over 14577.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006131, over 37.00 utterances.], tot_loss[loss=0.3604, over 2859232.36 frames; utt_duration=1281 frames, utt_pad_proportion=0.05242, over 8937.37 utterances.], batch size: 37, lr: 2.98e-04
|
| 180 |
+
2022-08-12 09:05:26,495 INFO [train.py:737] (2/3) Epoch 18, batch 2200, loss[loss=0.395, over 14756.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005673, over 41.00 utterances.], tot_loss[loss=0.3604, over 2858168.09 frames; utt_duration=1279 frames, utt_pad_proportion=0.05342, over 8951.91 utterances.], batch size: 41, lr: 2.98e-04
|
| 181 |
+
2022-08-12 09:07:43,370 INFO [train.py:737] (2/3) Epoch 18, batch 2300, loss[loss=0.3229, over 14572.00 frames; utt_duration=1496 frames, utt_pad_proportion=0.005946, over 39.00 utterances.], tot_loss[loss=0.3588, over 2858837.31 frames; utt_duration=1277 frames, utt_pad_proportion=0.05236, over 8967.77 utterances.], batch size: 39, lr: 2.98e-04
|
| 182 |
+
2022-08-12 09:10:16,146 INFO [train.py:737] (2/3) Epoch 18, batch 2400, loss[loss=0.3418, over 14600.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.005597, over 38.00 utterances.], tot_loss[loss=0.3591, over 2866009.06 frames; utt_duration=1285 frames, utt_pad_proportion=0.04694, over 8934.15 utterances.], batch size: 38, lr: 2.98e-04
|
| 183 |
+
2022-08-12 09:12:54,437 INFO [train.py:737] (2/3) Epoch 18, batch 2500, loss[loss=0.3299, over 14770.00 frames; utt_duration=1443 frames, utt_pad_proportion=0.004477, over 41.00 utterances.], tot_loss[loss=0.3616, over 2849821.42 frames; utt_duration=1252 frames, utt_pad_proportion=0.05723, over 9118.61 utterances.], batch size: 41, lr: 2.98e-04
|
| 184 |
+
2022-08-12 09:16:00,773 INFO [train.py:737] (2/3) Epoch 18, batch 2600, loss[loss=0.3895, over 14749.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.00579, over 41.00 utterances.], tot_loss[loss=0.3639, over 2857487.83 frames; utt_duration=1267 frames, utt_pad_proportion=0.05322, over 9032.39 utterances.], batch size: 41, lr: 2.98e-04
|
| 185 |
+
2022-08-12 09:18:22,665 INFO [train.py:737] (2/3) Epoch 18, batch 2700, loss[loss=0.326, over 14587.00 frames; utt_duration=1498 frames, utt_pad_proportion=0.004753, over 39.00 utterances.], tot_loss[loss=0.3638, over 2861208.12 frames; utt_duration=1274 frames, utt_pad_proportion=0.05024, over 8992.10 utterances.], batch size: 39, lr: 2.97e-04
|
| 186 |
+
2022-08-12 09:21:19,620 INFO [train.py:737] (2/3) Epoch 18, batch 2800, loss[loss=0.4414, over 9824.00 frames; utt_duration=428.6 frames, utt_pad_proportion=0.3406, over 92.00 utterances.], tot_loss[loss=0.3643, over 2842892.70 frames; utt_duration=1240 frames, utt_pad_proportion=0.06125, over 9182.81 utterances.], batch size: 92, lr: 2.97e-04
|
| 187 |
+
2022-08-12 09:24:12,307 INFO [train.py:737] (2/3) Epoch 18, batch 2900, loss[loss=0.4558, over 14772.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005532, over 42.00 utterances.], tot_loss[loss=0.3757, over 2835177.08 frames; utt_duration=1223 frames, utt_pad_proportion=0.06733, over 9286.53 utterances.], batch size: 42, lr: 2.97e-04
|
| 188 |
+
2022-08-12 09:26:27,703 INFO [train.py:737] (2/3) Epoch 18, batch 3000, loss[loss=0.4602, over 14596.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005276, over 41.00 utterances.], tot_loss[loss=0.3958, over 2848700.69 frames; utt_duration=1246 frames, utt_pad_proportion=0.05836, over 9156.00 utterances.], batch size: 41, lr: 2.97e-04
|
| 189 |
+
2022-08-12 09:26:27,703 INFO [train.py:761] (2/3) Computing validation loss
|
| 190 |
+
2022-08-12 09:27:08,101 INFO [train.py:770] (2/3) Epoch 18, validation: loss=0.2876, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 191 |
+
2022-08-12 09:29:25,968 INFO [train.py:737] (2/3) Epoch 18, batch 3100, loss[loss=0.4074, over 14782.00 frames; utt_duration=1479 frames, utt_pad_proportion=0.004441, over 40.00 utterances.], tot_loss[loss=0.4074, over 2833488.31 frames; utt_duration=1226 frames, utt_pad_proportion=0.06849, over 9258.68 utterances.], batch size: 40, lr: 2.97e-04
|
| 192 |
+
2022-08-12 09:32:24,406 INFO [train.py:737] (2/3) Epoch 18, batch 3200, loss[loss=0.4729, over 13143.00 frames; utt_duration=774.6 frames, utt_pad_proportion=0.1207, over 68.00 utterances.], tot_loss[loss=0.4189, over 2827478.39 frames; utt_duration=1210 frames, utt_pad_proportion=0.07111, over 9360.28 utterances.], batch size: 68, lr: 2.97e-04
|
| 193 |
+
2022-08-12 09:34:49,898 INFO [train.py:737] (2/3) Epoch 18, batch 3300, loss[loss=0.4336, over 14863.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.008314, over 44.00 utterances.], tot_loss[loss=0.4269, over 2831615.55 frames; utt_duration=1223 frames, utt_pad_proportion=0.0685, over 9274.05 utterances.], batch size: 44, lr: 2.97e-04
|
| 194 |
+
2022-08-12 09:37:01,563 INFO [train.py:737] (2/3) Epoch 18, batch 3400, loss[loss=0.4359, over 14130.00 frames; utt_duration=1825 frames, utt_pad_proportion=0.0467, over 31.00 utterances.], tot_loss[loss=0.4358, over 2817119.66 frames; utt_duration=1190 frames, utt_pad_proportion=0.07728, over 9480.03 utterances.], batch size: 31, lr: 2.97e-04
|
| 195 |
+
2022-08-12 09:39:45,278 INFO [train.py:737] (2/3) Epoch 18, batch 3500, loss[loss=0.4202, over 14576.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.005859, over 42.00 utterances.], tot_loss[loss=0.4352, over 2819780.48 frames; utt_duration=1192 frames, utt_pad_proportion=0.07606, over 9472.80 utterances.], batch size: 42, lr: 2.97e-04
|
| 196 |
+
2022-08-12 09:41:58,169 INFO [train.py:737] (2/3) Epoch 18, batch 3600, loss[loss=0.361, over 14702.00 frames; utt_duration=1682 frames, utt_pad_proportion=0.01187, over 35.00 utterances.], tot_loss[loss=0.4382, over 2829761.95 frames; utt_duration=1201 frames, utt_pad_proportion=0.06952, over 9434.54 utterances.], batch size: 35, lr: 2.97e-04
|
| 197 |
+
2022-08-12 09:44:05,888 INFO [train.py:737] (2/3) Epoch 18, batch 3700, loss[loss=0.3931, over 14569.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.006454, over 37.00 utterances.], tot_loss[loss=0.4398, over 2839654.79 frames; utt_duration=1229 frames, utt_pad_proportion=0.0626, over 9253.65 utterances.], batch size: 37, lr: 2.97e-04
|
| 198 |
+
2022-08-12 09:46:59,242 INFO [train.py:737] (2/3) Epoch 18, batch 3800, loss[loss=0.4727, over 14766.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.005801, over 42.00 utterances.], tot_loss[loss=0.4414, over 2825779.66 frames; utt_duration=1202 frames, utt_pad_proportion=0.07062, over 9414.17 utterances.], batch size: 42, lr: 2.97e-04
|
| 199 |
+
2022-08-12 09:49:10,332 INFO [train.py:737] (2/3) Epoch 18, batch 3900, loss[loss=0.4411, over 14583.00 frames; utt_duration=1424 frames, utt_pad_proportion=0.006042, over 41.00 utterances.], tot_loss[loss=0.44, over 2822741.49 frames; utt_duration=1204 frames, utt_pad_proportion=0.07372, over 9386.93 utterances.], batch size: 41, lr: 2.96e-04
|
| 200 |
+
2022-08-12 09:51:49,476 INFO [train.py:737] (2/3) Epoch 18, batch 4000, loss[loss=0.4392, over 14777.00 frames; utt_duration=1557 frames, utt_pad_proportion=0.005982, over 38.00 utterances.], tot_loss[loss=0.4409, over 2837645.94 frames; utt_duration=1228 frames, utt_pad_proportion=0.06445, over 9252.77 utterances.], batch size: 38, lr: 2.96e-04
|
| 201 |
+
2022-08-12 09:54:08,256 INFO [train.py:737] (2/3) Epoch 18, batch 4100, loss[loss=0.4428, over 14832.00 frames; utt_duration=1649 frames, utt_pad_proportion=0.01004, over 36.00 utterances.], tot_loss[loss=0.4401, over 2832365.32 frames; utt_duration=1227 frames, utt_pad_proportion=0.06892, over 9246.59 utterances.], batch size: 36, lr: 2.96e-04
|
| 202 |
+
2022-08-12 09:56:28,400 INFO [train.py:737] (2/3) Epoch 18, batch 4200, loss[loss=0.439, over 14596.00 frames; utt_duration=1537 frames, utt_pad_proportion=0.006817, over 38.00 utterances.], tot_loss[loss=0.4401, over 2831024.65 frames; utt_duration=1224 frames, utt_pad_proportion=0.06961, over 9265.15 utterances.], batch size: 38, lr: 2.96e-04
|
| 203 |
+
2022-08-12 09:58:21,327 INFO [train.py:737] (2/3) Epoch 18, batch 4300, loss[loss=0.4441, over 12877.00 frames; utt_duration=770.3 frames, utt_pad_proportion=0.1286, over 67.00 utterances.], tot_loss[loss=0.4365, over 2840166.59 frames; utt_duration=1240 frames, utt_pad_proportion=0.06453, over 9173.47 utterances.], batch size: 67, lr: 2.96e-04
|
| 204 |
+
2022-08-12 10:01:09,185 INFO [train.py:737] (2/3) Epoch 18, batch 4400, loss[loss=0.4395, over 13820.00 frames; utt_duration=971.2 frames, utt_pad_proportion=0.0653, over 57.00 utterances.], tot_loss[loss=0.4374, over 2837142.66 frames; utt_duration=1244 frames, utt_pad_proportion=0.06701, over 9130.24 utterances.], batch size: 57, lr: 2.96e-04
|
| 205 |
+
2022-08-12 10:04:01,950 INFO [train.py:737] (2/3) Epoch 18, batch 4500, loss[loss=0.5183, over 14503.00 frames; utt_duration=1162 frames, utt_pad_proportion=0.02209, over 50.00 utterances.], tot_loss[loss=0.4366, over 2833770.57 frames; utt_duration=1240 frames, utt_pad_proportion=0.06797, over 9155.04 utterances.], batch size: 50, lr: 2.96e-04
|
| 206 |
+
2022-08-12 10:07:00,420 INFO [train.py:737] (2/3) Epoch 18, batch 4600, loss[loss=0.4635, over 14654.00 frames; utt_duration=1249 frames, utt_pad_proportion=0.0136, over 47.00 utterances.], tot_loss[loss=0.439, over 2813663.42 frames; utt_duration=1199 frames, utt_pad_proportion=0.07946, over 9396.25 utterances.], batch size: 47, lr: 2.96e-04
|
| 207 |
+
2022-08-12 10:09:39,662 INFO [train.py:737] (2/3) Epoch 18, batch 4700, loss[loss=0.4716, over 14866.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.007303, over 44.00 utterances.], tot_loss[loss=0.44, over 2809328.83 frames; utt_duration=1185 frames, utt_pad_proportion=0.08054, over 9493.43 utterances.], batch size: 44, lr: 2.96e-04
|
| 208 |
+
2022-08-12 10:12:23,882 INFO [train.py:737] (2/3) Epoch 18, batch 4800, loss[loss=0.4888, over 13782.00 frames; utt_duration=968.6 frames, utt_pad_proportion=0.06771, over 57.00 utterances.], tot_loss[loss=0.4397, over 2806520.55 frames; utt_duration=1170 frames, utt_pad_proportion=0.08242, over 9605.71 utterances.], batch size: 57, lr: 2.96e-04
|
| 209 |
+
2022-08-12 10:15:16,743 INFO [train.py:737] (2/3) Epoch 18, batch 4900, loss[loss=0.3951, over 14769.00 frames; utt_duration=1478 frames, utt_pad_proportion=0.005733, over 40.00 utterances.], tot_loss[loss=0.4366, over 2820218.76 frames; utt_duration=1204 frames, utt_pad_proportion=0.07432, over 9384.00 utterances.], batch size: 40, lr: 2.96e-04
|
| 210 |
+
2022-08-12 10:17:43,025 INFO [train.py:737] (2/3) Epoch 18, batch 5000, loss[loss=0.429, over 14719.00 frames; utt_duration=1684 frames, utt_pad_proportion=0.00963, over 35.00 utterances.], tot_loss[loss=0.4355, over 2824737.58 frames; utt_duration=1227 frames, utt_pad_proportion=0.07296, over 9217.36 utterances.], batch size: 35, lr: 2.95e-04
|
| 211 |
+
2022-08-12 10:19:42,979 INFO [train.py:737] (2/3) Epoch 18, batch 5100, loss[loss=0.4862, over 14580.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006416, over 42.00 utterances.], tot_loss[loss=0.438, over 2817375.27 frames; utt_duration=1197 frames, utt_pad_proportion=0.07636, over 9430.22 utterances.], batch size: 42, lr: 2.95e-04
|
| 212 |
+
2022-08-12 10:22:03,879 INFO [train.py:737] (2/3) Epoch 18, batch 5200, loss[loss=0.4181, over 14572.00 frames; utt_duration=1577 frames, utt_pad_proportion=0.005794, over 37.00 utterances.], tot_loss[loss=0.4365, over 2820135.39 frames; utt_duration=1217 frames, utt_pad_proportion=0.07482, over 9284.07 utterances.], batch size: 37, lr: 2.95e-04
|
| 213 |
+
2022-08-12 10:24:26,637 INFO [train.py:737] (2/3) Epoch 18, batch 5300, loss[loss=0.4098, over 14570.00 frames; utt_duration=1459 frames, utt_pad_proportion=0.00642, over 40.00 utterances.], tot_loss[loss=0.4408, over 2824075.07 frames; utt_duration=1214 frames, utt_pad_proportion=0.07194, over 9315.75 utterances.], batch size: 40, lr: 2.95e-04
|
| 214 |
+
2022-08-12 10:26:43,151 INFO [train.py:737] (2/3) Epoch 18, batch 5400, loss[loss=0.5086, over 9773.00 frames; utt_duration=426.3 frames, utt_pad_proportion=0.3451, over 92.00 utterances.], tot_loss[loss=0.4404, over 2834276.61 frames; utt_duration=1233 frames, utt_pad_proportion=0.06624, over 9208.99 utterances.], batch size: 92, lr: 2.95e-04
|
| 215 |
+
2022-08-12 10:28:50,808 INFO [train.py:737] (2/3) Epoch 18, batch 5500, loss[loss=0.4713, over 14528.00 frames; utt_duration=1212 frames, utt_pad_proportion=0.01513, over 48.00 utterances.], tot_loss[loss=0.44, over 2839918.38 frames; utt_duration=1239 frames, utt_pad_proportion=0.06325, over 9180.81 utterances.], batch size: 48, lr: 2.95e-04
|
| 216 |
+
2022-08-12 10:30:53,582 INFO [train.py:737] (2/3) Epoch 18, batch 5600, loss[loss=0.4456, over 14703.00 frames; utt_duration=1280 frames, utt_pad_proportion=0.01099, over 46.00 utterances.], tot_loss[loss=0.4399, over 2823285.10 frames; utt_duration=1212 frames, utt_pad_proportion=0.07428, over 9328.50 utterances.], batch size: 46, lr: 2.95e-04
|
| 217 |
+
2022-08-12 10:33:04,874 INFO [train.py:737] (2/3) Epoch 18, batch 5700, loss[loss=0.4166, over 14556.00 frames; utt_duration=1215 frames, utt_pad_proportion=0.01419, over 48.00 utterances.], tot_loss[loss=0.4411, over 2808173.65 frames; utt_duration=1181 frames, utt_pad_proportion=0.0821, over 9524.77 utterances.], batch size: 48, lr: 2.95e-04
|
| 218 |
+
2022-08-12 10:35:25,141 INFO [train.py:737] (2/3) Epoch 18, batch 5800, loss[loss=0.4066, over 14785.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.005361, over 38.00 utterances.], tot_loss[loss=0.4407, over 2821943.00 frames; utt_duration=1200 frames, utt_pad_proportion=0.07367, over 9419.94 utterances.], batch size: 38, lr: 2.95e-04
|
| 219 |
+
2022-08-12 10:37:04,066 INFO [train.py:737] (2/3) Epoch 18, batch 5900, loss[loss=0.4804, over 14567.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006557, over 40.00 utterances.], tot_loss[loss=0.4419, over 2806350.32 frames; utt_duration=1172 frames, utt_pad_proportion=0.08215, over 9588.42 utterances.], batch size: 40, lr: 2.95e-04
|
| 220 |
+
2022-08-12 10:38:07,036 INFO [train.py:737] (2/3) Epoch 18, batch 6000, loss[loss=0.4891, over 12932.00 frames; utt_duration=773.5 frames, utt_pad_proportion=0.125, over 67.00 utterances.], tot_loss[loss=0.4419, over 2803797.47 frames; utt_duration=1162 frames, utt_pad_proportion=0.08375, over 9666.74 utterances.], batch size: 67, lr: 2.95e-04
|
| 221 |
+
2022-08-12 10:38:07,037 INFO [train.py:761] (2/3) Computing validation loss
|
| 222 |
+
2022-08-12 10:38:38,843 INFO [train.py:770] (2/3) Epoch 18, validation: loss=0.3055, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 223 |
+
2022-08-12 10:40:35,062 INFO [train.py:737] (2/3) Epoch 18, batch 6100, loss[loss=0.5153, over 10016.00 frames; utt_duration=437.1 frames, utt_pad_proportion=0.3255, over 92.00 utterances.], tot_loss[loss=0.4464, over 2747984.25 frames; utt_duration=1078 frames, utt_pad_proportion=0.1143, over 10214.52 utterances.], batch size: 92, lr: 2.95e-04
|
| 224 |
+
2022-08-12 10:41:44,280 INFO [train.py:737] (2/3) Epoch 19, batch 0, loss[loss=0.3925, over 14768.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.006352, over 38.00 utterances.], tot_loss[loss=0.3925, over 14768.00 frames; utt_duration=1556 frames, utt_pad_proportion=0.006352, over 38.00 utterances.], batch size: 38, lr: 2.87e-04
|
| 225 |
+
2022-08-12 10:44:11,839 INFO [train.py:737] (2/3) Epoch 19, batch 100, loss[loss=0.3317, over 14601.00 frames; utt_duration=1719 frames, utt_pad_proportion=0.01086, over 34.00 utterances.], tot_loss[loss=0.3642, over 1158372.02 frames; utt_duration=1393 frames, utt_pad_proportion=0.01887, over 3329.21 utterances.], batch size: 34, lr: 2.87e-04
|
| 226 |
+
2022-08-12 10:46:22,385 INFO [train.py:737] (2/3) Epoch 19, batch 200, loss[loss=0.4738, over 14772.00 frames; utt_duration=1443 frames, utt_pad_proportion=0.004444, over 41.00 utterances.], tot_loss[loss=0.3566, over 1828455.19 frames; utt_duration=1325 frames, utt_pad_proportion=0.0415, over 5525.18 utterances.], batch size: 41, lr: 2.87e-04
|
| 227 |
+
2022-08-12 10:48:43,568 INFO [train.py:737] (2/3) Epoch 19, batch 300, loss[loss=0.3213, over 14576.00 frames; utt_duration=1536 frames, utt_pad_proportion=0.007871, over 38.00 utterances.], tot_loss[loss=0.3572, over 2239085.32 frames; utt_duration=1305 frames, utt_pad_proportion=0.04242, over 6870.45 utterances.], batch size: 38, lr: 2.87e-04
|
| 228 |
+
2022-08-12 10:51:23,915 INFO [train.py:737] (2/3) Epoch 19, batch 400, loss[loss=0.401, over 14437.00 frames; utt_duration=1157 frames, utt_pad_proportion=0.02559, over 50.00 utterances.], tot_loss[loss=0.3585, over 2483619.64 frames; utt_duration=1290 frames, utt_pad_proportion=0.04879, over 7712.36 utterances.], batch size: 50, lr: 2.87e-04
|
| 229 |
+
2022-08-12 10:53:02,774 INFO [train.py:737] (2/3) Epoch 19, batch 500, loss[loss=0.3313, over 14872.00 frames; utt_duration=1354 frames, utt_pad_proportion=0.006703, over 44.00 utterances.], tot_loss[loss=0.3548, over 2644752.82 frames; utt_duration=1324 frames, utt_pad_proportion=0.04182, over 8001.18 utterances.], batch size: 44, lr: 2.87e-04
|
| 230 |
+
2022-08-12 10:55:23,803 INFO [train.py:737] (2/3) Epoch 19, batch 600, loss[loss=0.4013, over 12854.00 frames; utt_duration=768.7 frames, utt_pad_proportion=0.1304, over 67.00 utterances.], tot_loss[loss=0.3552, over 2736694.79 frames; utt_duration=1326 frames, utt_pad_proportion=0.04077, over 8266.42 utterances.], batch size: 67, lr: 2.87e-04
|
| 231 |
+
2022-08-12 10:58:04,976 INFO [train.py:737] (2/3) Epoch 19, batch 700, loss[loss=0.3099, over 14468.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01003, over 33.00 utterances.], tot_loss[loss=0.3564, over 2784284.50 frames; utt_duration=1311 frames, utt_pad_proportion=0.04534, over 8508.00 utterances.], batch size: 33, lr: 2.86e-04
|
| 232 |
+
2022-08-12 11:00:28,079 INFO [train.py:737] (2/3) Epoch 19, batch 800, loss[loss=0.3661, over 14582.00 frames; utt_duration=1390 frames, utt_pad_proportion=0.006161, over 42.00 utterances.], tot_loss[loss=0.3591, over 2821304.34 frames; utt_duration=1311 frames, utt_pad_proportion=0.04418, over 8618.15 utterances.], batch size: 42, lr: 2.86e-04
|
| 233 |
+
2022-08-12 11:02:57,819 INFO [train.py:737] (2/3) Epoch 19, batch 900, loss[loss=0.3606, over 14589.00 frames; utt_duration=1425 frames, utt_pad_proportion=0.005753, over 41.00 utterances.], tot_loss[loss=0.3577, over 2844067.73 frames; utt_duration=1314 frames, utt_pad_proportion=0.0414, over 8666.86 utterances.], batch size: 41, lr: 2.86e-04
|
| 234 |
+
2022-08-12 11:05:28,862 INFO [train.py:737] (2/3) Epoch 19, batch 1000, loss[loss=0.3936, over 14756.00 frames; utt_duration=1477 frames, utt_pad_proportion=0.00659, over 40.00 utterances.], tot_loss[loss=0.3567, over 2849462.94 frames; utt_duration=1309 frames, utt_pad_proportion=0.04748, over 8717.99 utterances.], batch size: 40, lr: 2.86e-04
|
| 235 |
+
2022-08-12 11:07:58,551 INFO [train.py:737] (2/3) Epoch 19, batch 1100, loss[loss=0.3914, over 14542.00 frames; utt_duration=1213 frames, utt_pad_proportion=0.0152, over 48.00 utterances.], tot_loss[loss=0.3601, over 2838112.84 frames; utt_duration=1255 frames, utt_pad_proportion=0.05917, over 9057.63 utterances.], batch size: 48, lr: 2.86e-04
|
| 236 |
+
2022-08-12 11:10:10,292 INFO [train.py:737] (2/3) Epoch 19, batch 1200, loss[loss=0.3663, over 14678.00 frames; utt_duration=1278 frames, utt_pad_proportion=0.01005, over 46.00 utterances.], tot_loss[loss=0.3597, over 2832767.21 frames; utt_duration=1239 frames, utt_pad_proportion=0.06532, over 9159.42 utterances.], batch size: 46, lr: 2.86e-04
|
| 237 |
+
2022-08-12 11:12:02,656 INFO [train.py:737] (2/3) Epoch 19, batch 1300, loss[loss=0.4623, over 14566.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.006101, over 40.00 utterances.], tot_loss[loss=0.3611, over 2843295.60 frames; utt_duration=1246 frames, utt_pad_proportion=0.06093, over 9137.13 utterances.], batch size: 40, lr: 2.86e-04
|
| 238 |
+
2022-08-12 11:14:19,755 INFO [train.py:737] (2/3) Epoch 19, batch 1400, loss[loss=0.4048, over 14355.00 frames; utt_duration=1085 frames, utt_pad_proportion=0.03623, over 53.00 utterances.], tot_loss[loss=0.3601, over 2856048.01 frames; utt_duration=1274 frames, utt_pad_proportion=0.05368, over 8981.12 utterances.], batch size: 53, lr: 2.86e-04
|
| 239 |
+
2022-08-12 11:16:17,938 INFO [train.py:737] (2/3) Epoch 19, batch 1500, loss[loss=0.3919, over 13928.00 frames; utt_duration=962 frames, utt_pad_proportion=0.06873, over 58.00 utterances.], tot_loss[loss=0.3586, over 2865728.59 frames; utt_duration=1290 frames, utt_pad_proportion=0.04817, over 8896.80 utterances.], batch size: 58, lr: 2.86e-04
|
| 240 |
+
2022-08-12 11:18:46,498 INFO [train.py:737] (2/3) Epoch 19, batch 1600, loss[loss=0.3978, over 9864.00 frames; utt_duration=430.5 frames, utt_pad_proportion=0.3366, over 92.00 utterances.], tot_loss[loss=0.3591, over 2857049.77 frames; utt_duration=1278 frames, utt_pad_proportion=0.05311, over 8949.46 utterances.], batch size: 92, lr: 2.86e-04
|
| 241 |
+
2022-08-12 11:21:10,268 INFO [train.py:737] (2/3) Epoch 19, batch 1700, loss[loss=0.3418, over 13978.00 frames; utt_duration=965.4 frames, utt_pad_proportion=0.06638, over 58.00 utterances.], tot_loss[loss=0.3594, over 2853005.28 frames; utt_duration=1260 frames, utt_pad_proportion=0.05591, over 9067.86 utterances.], batch size: 58, lr: 2.86e-04
|
| 242 |
+
2022-08-12 11:23:35,420 INFO [train.py:737] (2/3) Epoch 19, batch 1800, loss[loss=0.3214, over 14593.00 frames; utt_duration=1538 frames, utt_pad_proportion=0.005801, over 38.00 utterances.], tot_loss[loss=0.3592, over 2861754.36 frames; utt_duration=1286 frames, utt_pad_proportion=0.05051, over 8912.26 utterances.], batch size: 38, lr: 2.86e-04
|
| 243 |
+
2022-08-12 11:25:45,260 INFO [train.py:737] (2/3) Epoch 19, batch 1900, loss[loss=0.353, over 14772.00 frames; utt_duration=1408 frames, utt_pad_proportion=0.004695, over 42.00 utterances.], tot_loss[loss=0.3604, over 2860892.75 frames; utt_duration=1274 frames, utt_pad_proportion=0.05111, over 8992.66 utterances.], batch size: 42, lr: 2.86e-04
|
| 244 |
+
2022-08-12 11:27:05,703 INFO [train.py:737] (2/3) Epoch 19, batch 2000, loss[loss=0.4746, over 10171.00 frames; utt_duration=443.6 frames, utt_pad_proportion=0.3186, over 92.00 utterances.], tot_loss[loss=0.3601, over 2848624.03 frames; utt_duration=1256 frames, utt_pad_proportion=0.05999, over 9083.57 utterances.], batch size: 92, lr: 2.85e-04
|
| 245 |
+
2022-08-12 11:29:07,396 INFO [train.py:737] (2/3) Epoch 19, batch 2100, loss[loss=0.3458, over 14569.00 frames; utt_duration=1576 frames, utt_pad_proportion=0.006642, over 37.00 utterances.], tot_loss[loss=0.3602, over 2850966.15 frames; utt_duration=1252 frames, utt_pad_proportion=0.05838, over 9119.92 utterances.], batch size: 37, lr: 2.85e-04
|
| 246 |
+
2022-08-12 11:31:17,548 INFO [train.py:737] (2/3) Epoch 19, batch 2200, loss[loss=0.3735, over 14411.00 frames; utt_duration=1089 frames, utt_pad_proportion=0.03197, over 53.00 utterances.], tot_loss[loss=0.3621, over 2839827.20 frames; utt_duration=1229 frames, utt_pad_proportion=0.06423, over 9256.07 utterances.], batch size: 53, lr: 2.85e-04
|
| 247 |
+
2022-08-12 11:33:35,214 INFO [train.py:737] (2/3) Epoch 19, batch 2300, loss[loss=0.3877, over 13781.00 frames; utt_duration=968.5 frames, utt_pad_proportion=0.06696, over 57.00 utterances.], tot_loss[loss=0.3614, over 2832315.93 frames; utt_duration=1222 frames, utt_pad_proportion=0.06884, over 9281.15 utterances.], batch size: 57, lr: 2.85e-04
|
| 248 |
+
2022-08-12 11:35:43,152 INFO [train.py:737] (2/3) Epoch 19, batch 2400, loss[loss=0.3917, over 9635.00 frames; utt_duration=420.3 frames, utt_pad_proportion=0.3554, over 92.00 utterances.], tot_loss[loss=0.3595, over 2838084.26 frames; utt_duration=1242 frames, utt_pad_proportion=0.06469, over 9151.21 utterances.], batch size: 92, lr: 2.85e-04
|
| 249 |
+
2022-08-12 11:38:20,355 INFO [train.py:737] (2/3) Epoch 19, batch 2500, loss[loss=0.3896, over 14301.00 frames; utt_duration=1081 frames, utt_pad_proportion=0.04022, over 53.00 utterances.], tot_loss[loss=0.3578, over 2848573.40 frames; utt_duration=1257 frames, utt_pad_proportion=0.0591, over 9072.68 utterances.], batch size: 53, lr: 2.85e-04
|
| 250 |
+
2022-08-12 11:40:28,213 INFO [train.py:737] (2/3) Epoch 19, batch 2600, loss[loss=0.4393, over 10270.00 frames; utt_duration=448.1 frames, utt_pad_proportion=0.3117, over 92.00 utterances.], tot_loss[loss=0.3592, over 2849651.09 frames; utt_duration=1254 frames, utt_pad_proportion=0.05952, over 9101.95 utterances.], batch size: 92, lr: 2.85e-04
|
| 251 |
+
2022-08-12 11:42:36,347 INFO [train.py:737] (2/3) Epoch 19, batch 2700, loss[loss=0.4358, over 14869.00 frames; utt_duration=1353 frames, utt_pad_proportion=0.007981, over 44.00 utterances.], tot_loss[loss=0.3609, over 2857705.44 frames; utt_duration=1262 frames, utt_pad_proportion=0.05464, over 9065.23 utterances.], batch size: 44, lr: 2.85e-04
|
| 252 |
+
2022-08-12 11:44:20,898 INFO [train.py:737] (2/3) Epoch 19, batch 2800, loss[loss=0.3027, over 14768.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007403, over 37.00 utterances.], tot_loss[loss=0.3621, over 2857246.95 frames; utt_duration=1269 frames, utt_pad_proportion=0.05296, over 9017.76 utterances.], batch size: 37, lr: 2.85e-04
|
| 253 |
+
2022-08-12 11:46:57,783 INFO [train.py:737] (2/3) Epoch 19, batch 2900, loss[loss=0.4357, over 14770.00 frames; utt_duration=1442 frames, utt_pad_proportion=0.004545, over 41.00 utterances.], tot_loss[loss=0.3789, over 2839139.55 frames; utt_duration=1227 frames, utt_pad_proportion=0.06413, over 9270.53 utterances.], batch size: 41, lr: 2.85e-04
|
| 254 |
+
2022-08-12 11:48:38,712 INFO [train.py:737] (2/3) Epoch 19, batch 3000, loss[loss=0.4009, over 14623.00 frames; utt_duration=1331 frames, utt_pad_proportion=0.009071, over 44.00 utterances.], tot_loss[loss=0.3928, over 2845877.58 frames; utt_duration=1239 frames, utt_pad_proportion=0.06137, over 9198.11 utterances.], batch size: 44, lr: 2.85e-04
|
| 255 |
+
2022-08-12 11:48:38,712 INFO [train.py:761] (2/3) Computing validation loss
|
| 256 |
+
2022-08-12 11:49:10,904 INFO [train.py:770] (2/3) Epoch 19, validation: loss=0.2796, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 257 |
+
2022-08-12 11:51:42,711 INFO [train.py:737] (2/3) Epoch 19, batch 3100, loss[loss=0.413, over 13647.00 frames; utt_duration=959.5 frames, utt_pad_proportion=0.07652, over 57.00 utterances.], tot_loss[loss=0.4073, over 2823255.78 frames; utt_duration=1191 frames, utt_pad_proportion=0.07367, over 9496.95 utterances.], batch size: 57, lr: 2.85e-04
|
| 258 |
+
2022-08-12 11:53:58,742 INFO [train.py:737] (2/3) Epoch 19, batch 3200, loss[loss=0.4432, over 14672.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.01166, over 47.00 utterances.], tot_loss[loss=0.4173, over 2831410.50 frames; utt_duration=1212 frames, utt_pad_proportion=0.0695, over 9359.51 utterances.], batch size: 47, lr: 2.84e-04
|
| 259 |
+
2022-08-12 11:56:02,102 INFO [train.py:737] (2/3) Epoch 19, batch 3300, loss[loss=0.3671, over 14460.00 frames; utt_duration=1754 frames, utt_pad_proportion=0.009987, over 33.00 utterances.], tot_loss[loss=0.4241, over 2833226.33 frames; utt_duration=1220 frames, utt_pad_proportion=0.0683, over 9299.95 utterances.], batch size: 33, lr: 2.84e-04
|
| 260 |
+
2022-08-12 11:58:26,092 INFO [train.py:737] (2/3) Epoch 19, batch 3400, loss[loss=0.3934, over 14566.00 frames; utt_duration=1495 frames, utt_pad_proportion=0.006338, over 39.00 utterances.], tot_loss[loss=0.4275, over 2844758.94 frames; utt_duration=1251 frames, utt_pad_proportion=0.06122, over 9107.75 utterances.], batch size: 39, lr: 2.84e-04
|
| 261 |
+
2022-08-12 12:00:33,870 INFO [train.py:737] (2/3) Epoch 19, batch 3500, loss[loss=0.4547, over 14598.00 frames; utt_duration=1392 frames, utt_pad_proportion=0.004496, over 42.00 utterances.], tot_loss[loss=0.4315, over 2845349.40 frames; utt_duration=1249 frames, utt_pad_proportion=0.06194, over 9122.18 utterances.], batch size: 42, lr: 2.84e-04
|
| 262 |
+
2022-08-12 12:02:46,753 INFO [train.py:737] (2/3) Epoch 19, batch 3600, loss[loss=0.4156, over 14585.00 frames; utt_duration=1622 frames, utt_pad_proportion=0.007815, over 36.00 utterances.], tot_loss[loss=0.433, over 2844372.71 frames; utt_duration=1242 frames, utt_pad_proportion=0.06148, over 9172.26 utterances.], batch size: 36, lr: 2.84e-04
|
| 263 |
+
2022-08-12 12:04:15,351 INFO [train.py:737] (2/3) Epoch 19, batch 3700, loss[loss=0.4921, over 9749.00 frames; utt_duration=430.2 frames, utt_pad_proportion=0.3402, over 91.00 utterances.], tot_loss[loss=0.4378, over 2850973.93 frames; utt_duration=1248 frames, utt_pad_proportion=0.05714, over 9145.78 utterances.], batch size: 91, lr: 2.84e-04
|
| 264 |
+
2022-08-12 12:06:22,784 INFO [train.py:737] (2/3) Epoch 19, batch 3800, loss[loss=0.4204, over 13669.00 frames; utt_duration=960.7 frames, utt_pad_proportion=0.0718, over 57.00 utterances.], tot_loss[loss=0.4387, over 2833009.08 frames; utt_duration=1213 frames, utt_pad_proportion=0.06885, over 9355.14 utterances.], batch size: 57, lr: 2.84e-04
|
| 265 |
+
2022-08-12 12:08:47,678 INFO [train.py:737] (2/3) Epoch 19, batch 3900, loss[loss=0.4234, over 14437.00 frames; utt_duration=1651 frames, utt_pad_proportion=0.009564, over 35.00 utterances.], tot_loss[loss=0.4388, over 2822134.75 frames; utt_duration=1198 frames, utt_pad_proportion=0.0744, over 9436.12 utterances.], batch size: 35, lr: 2.84e-04
|
| 266 |
+
2022-08-12 12:10:40,016 INFO [train.py:737] (2/3) Epoch 19, batch 4000, loss[loss=0.4895, over 14760.00 frames; utt_duration=1407 frames, utt_pad_proportion=0.00607, over 42.00 utterances.], tot_loss[loss=0.4416, over 2800238.56 frames; utt_duration=1154 frames, utt_pad_proportion=0.08686, over 9717.51 utterances.], batch size: 42, lr: 2.84e-04
|
| 267 |
+
2022-08-12 12:11:52,487 INFO [train.py:737] (2/3) Epoch 19, batch 4100, loss[loss=0.4693, over 14785.00 frames; utt_duration=1410 frames, utt_pad_proportion=0.004456, over 42.00 utterances.], tot_loss[loss=0.4393, over 2813289.59 frames; utt_duration=1182 frames, utt_pad_proportion=0.07915, over 9529.19 utterances.], batch size: 42, lr: 2.84e-04
|
| 268 |
+
2022-08-12 12:12:51,328 INFO [train.py:737] (2/3) Epoch 19, batch 4200, loss[loss=0.4102, over 14109.00 frames; utt_duration=1822 frames, utt_pad_proportion=0.03185, over 31.00 utterances.], tot_loss[loss=0.4414, over 2815412.33 frames; utt_duration=1192 frames, utt_pad_proportion=0.07954, over 9456.38 utterances.], batch size: 31, lr: 2.84e-04
|
| 269 |
+
2022-08-12 12:14:56,497 INFO [train.py:737] (2/3) Epoch 19, batch 4300, loss[loss=0.4874, over 14672.00 frames; utt_duration=1250 frames, utt_pad_proportion=0.01169, over 47.00 utterances.], tot_loss[loss=0.4404, over 2827880.45 frames; utt_duration=1216 frames, utt_pad_proportion=0.07094, over 9312.94 utterances.], batch size: 47, lr: 2.84e-04
|
| 270 |
+
2022-08-12 12:17:11,264 INFO [train.py:737] (2/3) Epoch 19, batch 4400, loss[loss=0.4538, over 14498.00 frames; utt_duration=1210 frames, utt_pad_proportion=0.01816, over 48.00 utterances.], tot_loss[loss=0.4389, over 2829342.91 frames; utt_duration=1217 frames, utt_pad_proportion=0.0708, over 9311.52 utterances.], batch size: 48, lr: 2.83e-04
|
| 271 |
+
2022-08-12 12:19:20,860 INFO [train.py:737] (2/3) Epoch 19, batch 4500, loss[loss=0.4452, over 14752.00 frames; utt_duration=1441 frames, utt_pad_proportion=0.005538, over 41.00 utterances.], tot_loss[loss=0.4393, over 2830609.16 frames; utt_duration=1223 frames, utt_pad_proportion=0.07021, over 9268.12 utterances.], batch size: 41, lr: 2.83e-04
|
| 272 |
+
2022-08-12 12:21:26,375 INFO [train.py:737] (2/3) Epoch 19, batch 4600, loss[loss=0.4333, over 14568.00 frames; utt_duration=1458 frames, utt_pad_proportion=0.005913, over 40.00 utterances.], tot_loss[loss=0.4386, over 2847506.91 frames; utt_duration=1246 frames, utt_pad_proportion=0.05804, over 9155.18 utterances.], batch size: 40, lr: 2.83e-04
|
| 273 |
+
2022-08-12 12:23:39,330 INFO [train.py:737] (2/3) Epoch 19, batch 4700, loss[loss=0.3596, over 14790.00 frames; utt_duration=1558 frames, utt_pad_proportion=0.004907, over 38.00 utterances.], tot_loss[loss=0.438, over 2832514.06 frames; utt_duration=1218 frames, utt_pad_proportion=0.06824, over 9317.28 utterances.], batch size: 38, lr: 2.83e-04
|
| 274 |
+
2022-08-12 12:26:00,559 INFO [train.py:737] (2/3) Epoch 19, batch 4800, loss[loss=0.404, over 14583.00 frames; utt_duration=1578 frames, utt_pad_proportion=0.005586, over 37.00 utterances.], tot_loss[loss=0.4356, over 2835828.02 frames; utt_duration=1226 frames, utt_pad_proportion=0.06629, over 9261.02 utterances.], batch size: 37, lr: 2.83e-04
|
| 275 |
+
2022-08-12 12:27:27,857 INFO [train.py:737] (2/3) Epoch 19, batch 4900, loss[loss=0.4089, over 14788.00 frames; utt_duration=1600 frames, utt_pad_proportion=0.006727, over 37.00 utterances.], tot_loss[loss=0.44, over 2813565.29 frames; utt_duration=1187 frames, utt_pad_proportion=0.08002, over 9492.12 utterances.], batch size: 37, lr: 2.83e-04
|
| 276 |
+
2022-08-12 12:29:00,608 INFO [train.py:737] (2/3) Epoch 19, batch 5000, loss[loss=0.4013, over 14735.00 frames; utt_duration=1686 frames, utt_pad_proportion=0.009669, over 35.00 utterances.], tot_loss[loss=0.44, over 2828491.80 frames; utt_duration=1212 frames, utt_pad_proportion=0.07123, over 9348.81 utterances.], batch size: 35, lr: 2.83e-04
|
| 277 |
+
2022-08-12 12:30:53,857 INFO [train.py:737] (2/3) Epoch 19, batch 5100, loss[loss=0.4706, over 14639.00 frames; utt_duration=1332 frames, utt_pad_proportion=0.008055, over 44.00 utterances.], tot_loss[loss=0.4385, over 2830828.67 frames; utt_duration=1221 frames, utt_pad_proportion=0.06964, over 9286.40 utterances.], batch size: 44, lr: 2.83e-04
|
| 278 |
+
2022-08-12 12:33:24,376 INFO [train.py:737] (2/3) Epoch 19, batch 5200, loss[loss=0.4824, over 14786.00 frames; utt_duration=1600 frames, utt_pad_proportion=0.006329, over 37.00 utterances.], tot_loss[loss=0.4386, over 2822838.29 frames; utt_duration=1205 frames, utt_pad_proportion=0.07436, over 9381.02 utterances.], batch size: 37, lr: 2.83e-04
|
| 279 |
+
2022-08-12 12:35:41,952 INFO [train.py:737] (2/3) Epoch 19, batch 5300, loss[loss=0.4857, over 12968.00 frames; utt_duration=775.6 frames, utt_pad_proportion=0.1236, over 67.00 utterances.], tot_loss[loss=0.4374, over 2814059.99 frames; utt_duration=1192 frames, utt_pad_proportion=0.08023, over 9455.28 utterances.], batch size: 67, lr: 2.83e-04
|
| 280 |
+
2022-08-12 12:37:57,203 INFO [train.py:737] (2/3) Epoch 19, batch 5400, loss[loss=0.4052, over 14694.00 frames; utt_duration=1308 frames, utt_pad_proportion=0.009158, over 45.00 utterances.], tot_loss[loss=0.4389, over 2821980.83 frames; utt_duration=1201 frames, utt_pad_proportion=0.07529, over 9407.51 utterances.], batch size: 45, lr: 2.83e-04
|
| 281 |
+
2022-08-12 12:40:13,259 INFO [train.py:737] (2/3) Epoch 19, batch 5500, loss[loss=0.3797, over 14468.00 frames; utt_duration=1755 frames, utt_pad_proportion=0.01012, over 33.00 utterances.], tot_loss[loss=0.4372, over 2821390.86 frames; utt_duration=1198 frames, utt_pad_proportion=0.07555, over 9431.39 utterances.], batch size: 33, lr: 2.83e-04
|
| 282 |
+
2022-08-12 12:42:24,007 INFO [train.py:737] (2/3) Epoch 19, batch 5600, loss[loss=0.4528, over 14770.00 frames; utt_duration=1598 frames, utt_pad_proportion=0.007185, over 37.00 utterances.], tot_loss[loss=0.4426, over 2805552.63 frames; utt_duration=1170 frames, utt_pad_proportion=0.08332, over 9600.55 utterances.], batch size: 37, lr: 2.83e-04
|
| 283 |
+
2022-08-12 12:43:43,252 INFO [train.py:737] (2/3) Epoch 19, batch 5700, loss[loss=0.431, over 14771.00 frames; utt_duration=1479 frames, utt_pad_proportion=0.005481, over 40.00 utterances.], tot_loss[loss=0.4402, over 2814925.38 frames; utt_duration=1192 frames, utt_pad_proportion=0.07851, over 9457.75 utterances.], batch size: 40, lr: 2.82e-04
|
| 284 |
+
2022-08-12 12:44:43,219 INFO [train.py:737] (2/3) Epoch 19, batch 5800, loss[loss=0.4647, over 14327.00 frames; utt_duration=1083 frames, utt_pad_proportion=0.03601, over 53.00 utterances.], tot_loss[loss=0.4354, over 2815798.41 frames; utt_duration=1206 frames, utt_pad_proportion=0.07946, over 9347.33 utterances.], batch size: 53, lr: 2.82e-04
|
| 285 |
+
2022-08-12 12:46:17,973 INFO [train.py:737] (2/3) Epoch 19, batch 5900, loss[loss=0.3669, over 14611.00 frames; utt_duration=1721 frames, utt_pad_proportion=0.008895, over 34.00 utterances.], tot_loss[loss=0.4357, over 2800245.90 frames; utt_duration=1176 frames, utt_pad_proportion=0.08794, over 9535.33 utterances.], batch size: 34, lr: 2.82e-04
|
| 286 |
+
2022-08-12 12:48:42,417 INFO [train.py:737] (2/3) Epoch 19, batch 6000, loss[loss=0.3848, over 14748.00 frames; utt_duration=1476 frames, utt_pad_proportion=0.006679, over 40.00 utterances.], tot_loss[loss=0.4394, over 2808473.26 frames; utt_duration=1180 frames, utt_pad_proportion=0.08358, over 9531.05 utterances.], batch size: 40, lr: 2.82e-04
|
| 287 |
+
2022-08-12 12:48:42,417 INFO [train.py:761] (2/3) Computing validation loss
|
| 288 |
+
2022-08-12 12:49:17,792 INFO [train.py:770] (2/3) Epoch 19, validation: loss=0.2887, over 944034.00 frames; utt_duration=679.8 frames, utt_pad_proportion=0.1339, over 5567.00 utterances.
|
| 289 |
+
2022-08-12 12:51:45,152 INFO [train.py:737] (2/3) Epoch 19, batch 6100, loss[loss=0.5061, over 9817.00 frames; utt_duration=419.3 frames, utt_pad_proportion=0.3418, over 94.00 utterances.], tot_loss[loss=0.4436, over 2754667.04 frames; utt_duration=1090 frames, utt_pad_proportion=0.113, over 10122.75 utterances.], batch size: 94, lr: 2.82e-04
|
| 290 |
+
2022-08-12 12:52:43,855 INFO [train.py:737] (2/3) Epoch 20, batch 0, loss[loss=0.3396, over 14420.00 frames; utt_duration=1650 frames, utt_pad_proportion=0.008637, over 35.00 utterances.], tot_loss[loss=0.3396, over 14420.00 frames; utt_duration=1650 frames, utt_pad_proportion=0.008637, over 35.00 utterances.], batch size: 35, lr: 2.75e-04
|
pretrained.pt → exp/pretrained_epoch17_avg2.pt
RENAMED
|
File without changes
|
{tensorboard_log → exp/tensorboard}/events.out.tfevents.1659925381.dgxs-01.cluster.thspzh.com.18736.0
RENAMED
|
File without changes
|
{tensorboard_log → exp/tensorboard}/events.out.tfevents.1659926695.dgxs-01.cluster.thspzh.com.27410.0
RENAMED
|
File without changes
|
{tensorboard_log → exp/tensorboard}/events.out.tfevents.1659927762.dgxs-01.cluster.thspzh.com.34561.0
RENAMED
|
File without changes
|
{tensorboard_log → exp/tensorboard}/events.out.tfevents.1659928526.dgxs-01.cluster.thspzh.com.39902.0
RENAMED
|
File without changes
|
{tensorboard_log → exp/tensorboard}/events.out.tfevents.1659929206.dgxs-01.cluster.thspzh.com.4352.0
RENAMED
|
File without changes
|
{tensorboard_log → exp/tensorboard}/events.out.tfevents.1659930110.dgxs-01.cluster.thspzh.com.10312.0
RENAMED
|
File without changes
|
{tensorboard_log → exp/tensorboard}/events.out.tfevents.1659931044.dgxs-01.cluster.thspzh.com.16831.0
RENAMED
|
File without changes
|
{tensorboard_log → exp/tensorboard}/events.out.tfevents.1659940194.dgxs-01.cluster.thspzh.com.37638.0
RENAMED
|
File without changes
|
{tensorboard_log → exp/tensorboard}/events.out.tfevents.1659941396.dgxs-01.cluster.thspzh.com.5637.0
RENAMED
|
File without changes
|
{tensorboard_log → exp/tensorboard}/events.out.tfevents.1660034714.dgxs-01.cluster.thspzh.com.3516.0
RENAMED
|
File without changes
|
{tensorboard_log → exp/tensorboard}/events.out.tfevents.1660179164.dgxs-01.cluster.thspzh.com.14444.0
RENAMED
|
File without changes
|
{tensorboard_log → exp/tensorboard}/events.out.tfevents.1660215232.dgxs-01.cluster.thspzh.com.3323.0
RENAMED
|
File without changes
|