| 2023-04-06 14:21:46,233 44k INFO {'train': {'log_interval': 200, 'eval_interval': 800, 'seed': 1234, 'epochs': 10000, 'learning_rate': 0.0001, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 6, 'fp16_run': False, 'lr_decay': 0.999875, 'segment_size': 10240, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0, 'use_sr': True, 'max_speclen': 512, 'port': '8001', 'keep_ckpts': 3, 'all_in_mem': False}, 'data': {'training_files': 'filelists/train.txt', 'validation_files': 'filelists/val.txt', 'max_wav_value': 32768.0, 'sampling_rate': 44100, 'filter_length': 2048, 'hop_length': 512, 'win_length': 2048, 'n_mel_channels': 80, 'mel_fmin': 0.0, 'mel_fmax': 22050}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0.1, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [8, 8, 2, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4, 4], 'n_layers_q': 3, 'use_spectral_norm': False, 'gin_channels': 256, 'ssl_dim': 256, 'n_speakers': 2}, 'spk': {'ojou_speech': 0, 'ojou_sing': 1}, 'model_dir': './logs/44k'} |
| 2023-04-06 14:21:53,913 44k INFO emb_g.weight is not in the checkpoint |
| 2023-04-06 14:21:54,028 44k INFO Loaded checkpoint './logs/44k/G_0.pth' (iteration 0) |
| 2023-04-06 14:21:54,937 44k INFO Loaded checkpoint './logs/44k/D_0.pth' (iteration 0) |
| 2023-04-06 14:28:10,074 44k INFO {'train': {'log_interval': 200, 'eval_interval': 800, 'seed': 1234, 'epochs': 10000, 'learning_rate': 0.0001, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 6, 'fp16_run': False, 'lr_decay': 0.999875, 'segment_size': 10240, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0, 'use_sr': True, 'max_speclen': 512, 'port': '8001', 'keep_ckpts': 3, 'all_in_mem': False}, 'data': {'training_files': 'filelists/train.txt', 'validation_files': 'filelists/val.txt', 'max_wav_value': 32768.0, 'sampling_rate': 44100, 'filter_length': 2048, 'hop_length': 512, 'win_length': 2048, 'n_mel_channels': 80, 'mel_fmin': 0.0, 'mel_fmax': 22050}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0.1, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [8, 8, 2, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4, 4], 'n_layers_q': 3, 'use_spectral_norm': False, 'gin_channels': 256, 'ssl_dim': 256, 'n_speakers': 2}, 'spk': {'ojou_speech': 0, 'ojou_sing': 1}, 'model_dir': './logs/44k'} |
| 2023-04-06 14:28:16,416 44k INFO emb_g.weight is not in the checkpoint |
| 2023-04-06 14:28:16,520 44k INFO Loaded checkpoint './logs/44k/G_0.pth' (iteration 0) |
| 2023-04-06 14:28:16,759 44k INFO Loaded checkpoint './logs/44k/D_0.pth' (iteration 0) |
| 2023-04-06 14:28:44,170 44k INFO Train Epoch: 1 [0%] |
| 2023-04-06 14:28:44,171 44k INFO Losses: [2.573824167251587, 2.517317295074463, 12.462947845458984, 39.484214782714844, 4.558784484863281], step: 0, lr: 0.0001 |
| 2023-04-06 14:29:02,964 44k INFO Saving model and optimizer state at iteration 1 to ./logs/44k/G_0.pth |
| 2023-04-06 14:29:06,278 44k INFO Saving model and optimizer state at iteration 1 to ./logs/44k/D_0.pth |
| 2023-04-06 14:29:36,249 44k INFO ====> Epoch: 1, cost 86.18 s |
| 2023-04-06 14:30:10,389 44k INFO ====> Epoch: 2, cost 34.14 s |
| 2023-04-06 14:30:44,375 44k INFO ====> Epoch: 3, cost 33.99 s |
| 2023-04-06 14:31:17,769 44k INFO ====> Epoch: 4, cost 33.39 s |
| 2023-04-06 14:31:51,295 44k INFO ====> Epoch: 5, cost 33.53 s |
| 2023-04-06 14:32:24,800 44k INFO ====> Epoch: 6, cost 33.51 s |
| 2023-04-06 14:32:58,115 44k INFO ====> Epoch: 7, cost 33.31 s |
| 2023-04-06 14:33:31,700 44k INFO ====> Epoch: 8, cost 33.58 s |
| 2023-04-06 14:34:05,014 44k INFO ====> Epoch: 9, cost 33.31 s |
| 2023-04-06 14:34:38,581 44k INFO ====> Epoch: 10, cost 33.57 s |
| 2023-04-06 14:35:12,124 44k INFO ====> Epoch: 11, cost 33.54 s |
| 2023-04-06 14:35:26,942 44k INFO Train Epoch: 12 [11%] |
| 2023-04-06 14:35:26,943 44k INFO Losses: [2.380369186401367, 2.374410629272461, 9.374342918395996, 25.8035888671875, 1.5873091220855713], step: 200, lr: 9.986258590528146e-05 |
| 2023-04-06 14:35:47,245 44k INFO ====> Epoch: 12, cost 35.12 s |
| 2023-04-06 14:36:20,961 44k INFO ====> Epoch: 13, cost 33.72 s |
| 2023-04-06 14:36:55,209 44k INFO ====> Epoch: 14, cost 34.25 s |
| 2023-04-06 14:37:28,824 44k INFO ====> Epoch: 15, cost 33.62 s |
| 2023-04-06 14:38:02,514 44k INFO ====> Epoch: 16, cost 33.69 s |
| 2023-04-06 14:38:36,540 44k INFO ====> Epoch: 17, cost 34.03 s |
| 2023-04-06 14:39:09,516 44k INFO ====> Epoch: 18, cost 32.98 s |
| 2023-04-06 14:39:42,290 44k INFO ====> Epoch: 19, cost 32.77 s |
| 2023-04-06 14:40:15,260 44k INFO ====> Epoch: 20, cost 32.97 s |
| 2023-04-06 14:40:48,429 44k INFO ====> Epoch: 21, cost 33.17 s |
| 2023-04-06 14:41:22,035 44k INFO ====> Epoch: 22, cost 33.61 s |
| 2023-04-06 14:41:38,373 44k INFO Train Epoch: 23 [22%] |
| 2023-04-06 14:41:38,374 44k INFO Losses: [2.5752055644989014, 2.3014121055603027, 8.904624938964844, 23.154579162597656, 1.0447863340377808], step: 400, lr: 9.972536063689719e-05 |
| 2023-04-06 14:41:56,747 44k INFO ====> Epoch: 23, cost 34.71 s |
| 2023-04-06 14:42:30,058 44k INFO ====> Epoch: 24, cost 33.31 s |
| 2023-04-06 14:43:03,038 44k INFO ====> Epoch: 25, cost 32.98 s |
| 2023-04-06 14:43:36,283 44k INFO ====> Epoch: 26, cost 33.25 s |
| 2023-04-06 14:44:09,581 44k INFO ====> Epoch: 27, cost 33.30 s |
| 2023-04-06 14:44:43,372 44k INFO ====> Epoch: 28, cost 33.79 s |
| 2023-04-06 14:45:16,897 44k INFO ====> Epoch: 29, cost 33.53 s |
| 2023-04-06 14:45:50,410 44k INFO ====> Epoch: 30, cost 33.51 s |
| 2023-04-06 14:46:24,140 44k INFO ====> Epoch: 31, cost 33.73 s |
| 2023-04-06 14:46:58,284 44k INFO ====> Epoch: 32, cost 34.14 s |
| 2023-04-06 14:47:31,802 44k INFO ====> Epoch: 33, cost 33.52 s |
| 2023-04-06 14:47:50,868 44k INFO Train Epoch: 34 [33%] |
| 2023-04-06 14:47:50,869 44k INFO Losses: [2.0391736030578613, 2.6325862407684326, 12.453435897827148, 22.571475982666016, 1.300460696220398], step: 600, lr: 9.95883239353732e-05 |
| 2023-04-06 14:48:06,495 44k INFO ====> Epoch: 34, cost 34.69 s |
| 2023-04-06 14:48:40,108 44k INFO ====> Epoch: 35, cost 33.61 s |
| 2023-04-06 14:49:13,550 44k INFO ====> Epoch: 36, cost 33.44 s |
| 2023-04-06 14:49:47,466 44k INFO ====> Epoch: 37, cost 33.92 s |
| 2023-04-06 14:50:20,905 44k INFO ====> Epoch: 38, cost 33.44 s |
| 2023-04-06 14:50:54,439 44k INFO ====> Epoch: 39, cost 33.53 s |
| 2023-04-06 14:51:28,025 44k INFO ====> Epoch: 40, cost 33.59 s |
| 2023-04-06 14:52:01,642 44k INFO ====> Epoch: 41, cost 33.62 s |
| 2023-04-06 14:52:35,218 44k INFO ====> Epoch: 42, cost 33.58 s |
| 2023-04-06 14:53:08,561 44k INFO ====> Epoch: 43, cost 33.34 s |
| 2023-04-06 14:53:42,989 44k INFO ====> Epoch: 44, cost 34.43 s |
| 2023-04-06 14:54:05,177 44k INFO Train Epoch: 45 [44%] |
| 2023-04-06 14:54:05,178 44k INFO Losses: [2.806058168411255, 2.137558698654175, 5.308585166931152, 20.471132278442383, 1.025601863861084], step: 800, lr: 9.945147554159202e-05 |
| 2023-04-06 14:54:13,630 44k INFO Saving model and optimizer state at iteration 45 to ./logs/44k/G_800.pth |
| 2023-04-06 14:54:16,110 44k INFO Saving model and optimizer state at iteration 45 to ./logs/44k/D_800.pth |
| 2023-04-06 14:54:30,153 44k INFO ====> Epoch: 45, cost 47.16 s |
| 2023-04-06 14:55:03,935 44k INFO ====> Epoch: 46, cost 33.78 s |
| 2023-04-06 14:55:37,393 44k INFO ====> Epoch: 47, cost 33.46 s |
| 2023-04-06 14:56:13,219 44k INFO ====> Epoch: 48, cost 35.83 s |
| 2023-04-06 14:56:49,538 44k INFO ====> Epoch: 49, cost 36.32 s |
| 2023-04-06 14:57:23,880 44k INFO ====> Epoch: 50, cost 34.34 s |
| 2023-04-06 14:57:58,045 44k INFO ====> Epoch: 51, cost 34.16 s |
| 2023-04-06 14:58:33,841 44k INFO ====> Epoch: 52, cost 35.80 s |
| 2023-04-06 14:59:08,197 44k INFO ====> Epoch: 53, cost 34.36 s |
| 2023-04-06 14:59:42,422 44k INFO ====> Epoch: 54, cost 34.23 s |
| 2023-04-06 15:00:16,288 44k INFO ====> Epoch: 55, cost 33.87 s |
| 2023-04-06 15:00:40,883 44k INFO Train Epoch: 56 [56%] |
| 2023-04-06 15:00:40,884 44k INFO Losses: [2.6365833282470703, 2.064959764480591, 5.582603931427002, 19.144601821899414, 0.8247841000556946], step: 1000, lr: 9.931481519679228e-05 |
| 2023-04-06 15:00:51,516 44k INFO ====> Epoch: 56, cost 35.23 s |
| 2023-04-06 15:01:24,869 44k INFO ====> Epoch: 57, cost 33.35 s |
| 2023-04-06 15:01:58,466 44k INFO ====> Epoch: 58, cost 33.60 s |
| 2023-04-06 15:02:32,055 44k INFO ====> Epoch: 59, cost 33.59 s |
| 2023-04-06 15:03:05,621 44k INFO ====> Epoch: 60, cost 33.57 s |
| 2023-04-06 15:03:39,247 44k INFO ====> Epoch: 61, cost 33.63 s |
| 2023-04-06 15:04:12,988 44k INFO ====> Epoch: 62, cost 33.74 s |
| 2023-04-06 15:04:46,696 44k INFO ====> Epoch: 63, cost 33.71 s |
| 2023-04-06 15:05:20,560 44k INFO ====> Epoch: 64, cost 33.86 s |
| 2023-04-06 15:05:54,100 44k INFO ====> Epoch: 65, cost 33.54 s |
| 2023-04-06 15:06:28,189 44k INFO ====> Epoch: 66, cost 34.09 s |
| 2023-04-06 15:06:54,565 44k INFO Train Epoch: 67 [67%] |
| 2023-04-06 15:06:54,572 44k INFO Losses: [2.5330240726470947, 2.2095956802368164, 9.094779014587402, 21.0531005859375, 1.01519775390625], step: 1200, lr: 9.917834264256819e-05 |
| 2023-04-06 15:07:02,883 44k INFO ====> Epoch: 67, cost 34.69 s |
| 2023-04-06 15:07:35,938 44k INFO ====> Epoch: 68, cost 33.06 s |
| 2023-04-06 15:08:09,039 44k INFO ====> Epoch: 69, cost 33.10 s |
| 2023-04-06 15:08:42,455 44k INFO ====> Epoch: 70, cost 33.42 s |
| 2023-04-06 15:09:15,650 44k INFO ====> Epoch: 71, cost 33.19 s |
| 2023-04-06 15:09:49,134 44k INFO ====> Epoch: 72, cost 33.48 s |
| 2023-04-06 15:10:22,449 44k INFO ====> Epoch: 73, cost 33.31 s |
| 2023-04-06 15:10:55,919 44k INFO ====> Epoch: 74, cost 33.47 s |
| 2023-04-06 15:11:29,262 44k INFO ====> Epoch: 75, cost 33.34 s |
| 2023-04-06 15:12:02,619 44k INFO ====> Epoch: 76, cost 33.36 s |
| 2023-04-06 15:12:35,887 44k INFO ====> Epoch: 77, cost 33.27 s |
| 2023-04-06 15:13:05,328 44k INFO Train Epoch: 78 [78%] |
| 2023-04-06 15:13:05,330 44k INFO Losses: [2.4559853076934814, 2.5709311962127686, 8.139681816101074, 22.049203872680664, 1.2676440477371216], step: 1400, lr: 9.904205762086905e-05 |
| 2023-04-06 15:13:11,413 44k INFO ====> Epoch: 78, cost 35.53 s |
| 2023-04-06 15:13:44,951 44k INFO ====> Epoch: 79, cost 33.54 s |
| 2023-04-06 15:14:18,839 44k INFO ====> Epoch: 80, cost 33.89 s |
| 2023-04-06 15:14:52,995 44k INFO ====> Epoch: 81, cost 34.16 s |
| 2023-04-06 15:15:25,852 44k INFO ====> Epoch: 82, cost 32.86 s |
| 2023-04-06 15:15:58,487 44k INFO ====> Epoch: 83, cost 32.63 s |
| 2023-04-06 15:16:31,709 44k INFO ====> Epoch: 84, cost 33.22 s |
| 2023-04-06 15:17:04,874 44k INFO ====> Epoch: 85, cost 33.16 s |
| 2023-04-06 15:17:37,914 44k INFO ====> Epoch: 86, cost 33.04 s |
| 2023-04-06 15:18:11,152 44k INFO ====> Epoch: 87, cost 33.24 s |
| 2023-04-06 15:18:44,387 44k INFO ====> Epoch: 88, cost 33.24 s |
| 2023-04-06 15:19:15,242 44k INFO Train Epoch: 89 [89%] |
| 2023-04-06 15:19:15,243 44k INFO Losses: [2.471696376800537, 2.2750179767608643, 7.19001579284668, 20.06005096435547, 1.0517364740371704], step: 1600, lr: 9.89059598739987e-05 |
| 2023-04-06 15:19:23,190 44k INFO Saving model and optimizer state at iteration 89 to ./logs/44k/G_1600.pth |
| 2023-04-06 15:19:25,938 44k INFO Saving model and optimizer state at iteration 89 to ./logs/44k/D_1600.pth |
| 2023-04-06 15:19:30,586 44k INFO ====> Epoch: 89, cost 46.20 s |
| 2023-04-06 15:20:04,206 44k INFO ====> Epoch: 90, cost 33.62 s |
| 2023-04-06 15:20:37,768 44k INFO ====> Epoch: 91, cost 33.56 s |
| 2023-04-06 15:21:11,633 44k INFO ====> Epoch: 92, cost 33.87 s |
| 2023-04-06 15:21:44,606 44k INFO ====> Epoch: 93, cost 32.97 s |
| 2023-04-06 15:22:17,609 44k INFO ====> Epoch: 94, cost 33.00 s |
| 2023-04-06 15:22:50,786 44k INFO ====> Epoch: 95, cost 33.18 s |
| 2023-04-06 15:23:24,245 44k INFO ====> Epoch: 96, cost 33.46 s |
| 2023-04-06 15:23:57,352 44k INFO ====> Epoch: 97, cost 33.11 s |
| 2023-04-06 15:24:30,780 44k INFO ====> Epoch: 98, cost 33.43 s |
| 2023-04-06 15:25:04,040 44k INFO ====> Epoch: 99, cost 33.26 s |
| 2023-04-06 15:25:37,206 44k INFO ====> Epoch: 100, cost 33.17 s |
| 2023-04-06 15:25:48,955 44k INFO Train Epoch: 101 [0%] |
| 2023-04-06 15:25:48,957 44k INFO Losses: [2.5259549617767334, 2.311516046524048, 11.238818168640137, 24.39948081970215, 1.0041320323944092], step: 1800, lr: 9.875770288847208e-05 |
| 2023-04-06 15:26:12,089 44k INFO ====> Epoch: 101, cost 34.88 s |
| 2023-04-06 15:26:45,466 44k INFO ====> Epoch: 102, cost 33.38 s |
| 2023-04-06 15:27:19,094 44k INFO ====> Epoch: 103, cost 33.63 s |
| 2023-04-06 15:27:52,909 44k INFO ====> Epoch: 104, cost 33.82 s |
| 2023-04-06 15:28:26,521 44k INFO ====> Epoch: 105, cost 33.61 s |
| 2023-04-06 15:28:59,968 44k INFO ====> Epoch: 106, cost 33.45 s |
| 2023-04-06 15:29:33,425 44k INFO ====> Epoch: 107, cost 33.46 s |
| 2023-04-06 15:30:07,804 44k INFO ====> Epoch: 108, cost 34.38 s |
| 2023-04-06 15:30:40,689 44k INFO ====> Epoch: 109, cost 32.88 s |
| 2023-04-06 15:31:13,687 44k INFO ====> Epoch: 110, cost 33.00 s |
| 2023-04-06 15:31:46,516 44k INFO ====> Epoch: 111, cost 32.83 s |
| 2023-04-06 15:32:00,629 44k INFO Train Epoch: 112 [11%] |
| 2023-04-06 15:32:00,630 44k INFO Losses: [2.1719627380371094, 2.386096477508545, 12.244598388671875, 26.043445587158203, 0.8842934370040894], step: 2000, lr: 9.862199588508305e-05 |
| 2023-04-06 15:32:21,530 44k INFO ====> Epoch: 112, cost 35.01 s |
| 2023-04-06 15:32:54,826 44k INFO ====> Epoch: 113, cost 33.30 s |
| 2023-04-06 15:33:28,379 44k INFO ====> Epoch: 114, cost 33.55 s |
| 2023-04-06 15:34:01,717 44k INFO ====> Epoch: 115, cost 33.34 s |
| 2023-04-06 15:34:35,036 44k INFO ====> Epoch: 116, cost 33.32 s |
| 2023-04-06 15:35:08,484 44k INFO ====> Epoch: 117, cost 33.45 s |
| 2023-04-06 15:35:41,980 44k INFO ====> Epoch: 118, cost 33.50 s |
| 2023-04-06 15:36:15,514 44k INFO ====> Epoch: 119, cost 33.53 s |
| 2023-04-06 15:36:49,214 44k INFO ====> Epoch: 120, cost 33.70 s |
| 2023-04-06 15:37:23,093 44k INFO ====> Epoch: 121, cost 33.88 s |
| 2023-04-06 15:37:56,873 44k INFO ====> Epoch: 122, cost 33.78 s |
| 2023-04-06 15:38:14,003 44k INFO Train Epoch: 123 [22%] |
| 2023-04-06 15:38:14,004 44k INFO Losses: [2.781219482421875, 2.1242191791534424, 6.261739253997803, 19.073083877563477, 0.9984504580497742], step: 2200, lr: 9.848647536224416e-05 |
| 2023-04-06 15:38:32,518 44k INFO ====> Epoch: 123, cost 35.65 s |
| 2023-04-06 15:39:05,962 44k INFO ====> Epoch: 124, cost 33.44 s |
| 2023-04-06 15:39:39,114 44k INFO ====> Epoch: 125, cost 33.15 s |
| 2023-04-06 15:40:13,048 44k INFO ====> Epoch: 126, cost 33.93 s |
| 2023-04-06 15:40:46,649 44k INFO ====> Epoch: 127, cost 33.60 s |
| 2023-04-06 15:41:20,247 44k INFO ====> Epoch: 128, cost 33.60 s |
| 2023-04-06 15:41:53,939 44k INFO ====> Epoch: 129, cost 33.69 s |
| 2023-04-06 15:42:27,318 44k INFO ====> Epoch: 130, cost 33.38 s |
| 2023-04-06 15:43:00,635 44k INFO ====> Epoch: 131, cost 33.32 s |
| 2023-04-06 15:43:34,280 44k INFO ====> Epoch: 132, cost 33.64 s |
| 2023-04-06 15:44:07,894 44k INFO ====> Epoch: 133, cost 33.61 s |
| 2023-04-06 15:44:27,373 44k INFO Train Epoch: 134 [33%] |
| 2023-04-06 15:44:27,374 44k INFO Losses: [2.6103241443634033, 2.176192045211792, 7.58054780960083, 21.09679412841797, 0.8060694336891174], step: 2400, lr: 9.835114106370493e-05 |
| 2023-04-06 15:44:35,774 44k INFO Saving model and optimizer state at iteration 134 to ./logs/44k/G_2400.pth |
| 2023-04-06 15:44:38,339 44k INFO Saving model and optimizer state at iteration 134 to ./logs/44k/D_2400.pth |
| 2023-04-06 15:44:54,914 44k INFO ====> Epoch: 134, cost 47.02 s |
| 2023-04-06 15:45:28,270 44k INFO ====> Epoch: 135, cost 33.36 s |
| 2023-04-06 15:46:01,378 44k INFO ====> Epoch: 136, cost 33.11 s |
| 2023-04-06 15:46:34,552 44k INFO ====> Epoch: 137, cost 33.17 s |
| 2023-04-06 15:47:07,890 44k INFO ====> Epoch: 138, cost 33.34 s |
| 2023-04-06 15:47:40,920 44k INFO ====> Epoch: 139, cost 33.03 s |
| 2023-04-06 15:48:14,268 44k INFO ====> Epoch: 140, cost 33.35 s |
| 2023-04-06 15:48:47,542 44k INFO ====> Epoch: 141, cost 33.27 s |
| 2023-04-06 15:49:21,476 44k INFO ====> Epoch: 142, cost 33.93 s |
| 2023-04-06 15:49:55,539 44k INFO ====> Epoch: 143, cost 34.06 s |
| 2023-04-06 15:50:28,796 44k INFO ====> Epoch: 144, cost 33.26 s |
| 2023-04-06 15:50:50,534 44k INFO Train Epoch: 145 [44%] |
| 2023-04-06 15:50:50,535 44k INFO Losses: [2.589505434036255, 2.020362377166748, 7.501511096954346, 20.353748321533203, 1.148757815361023], step: 2600, lr: 9.821599273356685e-05 |
| 2023-04-06 15:51:03,447 44k INFO ====> Epoch: 145, cost 34.65 s |
| 2023-04-06 15:51:37,525 44k INFO ====> Epoch: 146, cost 34.08 s |
| 2023-04-06 15:52:11,490 44k INFO ====> Epoch: 147, cost 33.97 s |
| 2023-04-06 15:52:44,558 44k INFO ====> Epoch: 148, cost 33.07 s |
| 2023-04-06 15:53:17,472 44k INFO ====> Epoch: 149, cost 32.91 s |
| 2023-04-06 15:53:50,395 44k INFO ====> Epoch: 150, cost 32.92 s |
| 2023-04-06 15:54:23,621 44k INFO ====> Epoch: 151, cost 33.23 s |
| 2023-04-06 15:54:56,921 44k INFO ====> Epoch: 152, cost 33.30 s |
| 2023-04-06 15:55:30,097 44k INFO ====> Epoch: 153, cost 33.18 s |
| 2023-04-06 15:56:03,351 44k INFO ====> Epoch: 154, cost 33.25 s |
| 2023-04-06 15:56:36,675 44k INFO ====> Epoch: 155, cost 33.32 s |
| 2023-04-06 15:57:01,017 44k INFO Train Epoch: 156 [56%] |
| 2023-04-06 15:57:01,018 44k INFO Losses: [2.5215423107147217, 2.311556339263916, 8.819522857666016, 20.068315505981445, 0.7969138622283936], step: 2800, lr: 9.808103011628319e-05 |
| 2023-04-06 15:57:11,740 44k INFO ====> Epoch: 156, cost 35.06 s |
| 2023-04-06 15:57:44,760 44k INFO ====> Epoch: 157, cost 33.02 s |
| 2023-04-06 15:58:18,955 44k INFO ====> Epoch: 158, cost 34.20 s |
| 2023-04-06 15:58:52,525 44k INFO ====> Epoch: 159, cost 33.57 s |
| 2023-04-06 15:59:26,604 44k INFO ====> Epoch: 160, cost 34.08 s |
| 2023-04-06 16:00:00,222 44k INFO ====> Epoch: 161, cost 33.62 s |
| 2023-04-06 16:00:35,009 44k INFO ====> Epoch: 162, cost 34.79 s |
| 2023-04-06 16:01:08,369 44k INFO ====> Epoch: 163, cost 33.36 s |
| 2023-04-06 16:01:41,802 44k INFO ====> Epoch: 164, cost 33.43 s |
| 2023-04-06 16:02:15,766 44k INFO ====> Epoch: 165, cost 33.96 s |
| 2023-04-06 16:02:49,461 44k INFO ====> Epoch: 166, cost 33.70 s |
| 2023-04-06 16:03:16,473 44k INFO Train Epoch: 167 [67%] |
| 2023-04-06 16:03:16,474 44k INFO Losses: [2.7111127376556396, 2.1096386909484863, 7.442945957183838, 19.06759262084961, 0.842487096786499], step: 3000, lr: 9.794625295665828e-05 |
| 2023-04-06 16:03:24,941 44k INFO ====> Epoch: 167, cost 35.48 s |
| 2023-04-06 16:03:58,458 44k INFO ====> Epoch: 168, cost 33.52 s |
| 2023-04-06 16:04:32,090 44k INFO ====> Epoch: 169, cost 33.63 s |
| 2023-04-06 16:05:05,688 44k INFO ====> Epoch: 170, cost 33.60 s |
| 2023-04-06 16:05:38,792 44k INFO ====> Epoch: 171, cost 33.10 s |
| 2023-04-06 16:06:12,165 44k INFO ====> Epoch: 172, cost 33.37 s |
| 2023-04-06 16:06:45,433 44k INFO ====> Epoch: 173, cost 33.27 s |
| 2023-04-06 16:07:18,826 44k INFO ====> Epoch: 174, cost 33.39 s |
| 2023-04-06 16:07:52,391 44k INFO ====> Epoch: 175, cost 33.57 s |
| 2023-04-06 16:08:25,643 44k INFO ====> Epoch: 176, cost 33.25 s |
| 2023-04-06 16:08:59,249 44k INFO ====> Epoch: 177, cost 33.61 s |
| 2023-04-06 16:09:28,917 44k INFO Train Epoch: 178 [78%] |
| 2023-04-06 16:09:28,918 44k INFO Losses: [2.699871063232422, 2.6492371559143066, 10.786850929260254, 21.075050354003906, 0.9040826559066772], step: 3200, lr: 9.781166099984716e-05 |
| 2023-04-06 16:09:37,252 44k INFO Saving model and optimizer state at iteration 178 to ./logs/44k/G_3200.pth |
| 2023-04-06 16:09:40,424 44k INFO Saving model and optimizer state at iteration 178 to ./logs/44k/D_3200.pth |
| 2023-04-06 16:09:42,678 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_800.pth |
| 2023-04-06 16:09:42,697 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_800.pth |
| 2023-04-06 16:09:47,370 44k INFO ====> Epoch: 178, cost 48.12 s |
| 2023-04-06 16:10:22,255 44k INFO ====> Epoch: 179, cost 34.88 s |
| 2023-04-06 16:10:56,399 44k INFO ====> Epoch: 180, cost 34.14 s |
| 2023-04-06 16:11:29,952 44k INFO ====> Epoch: 181, cost 33.55 s |
| 2023-04-06 16:12:03,708 44k INFO ====> Epoch: 182, cost 33.76 s |
| 2023-04-06 16:12:37,128 44k INFO ====> Epoch: 183, cost 33.42 s |
| 2023-04-06 16:13:10,864 44k INFO ====> Epoch: 184, cost 33.74 s |
| 2023-04-06 16:13:44,783 44k INFO ====> Epoch: 185, cost 33.92 s |
| 2023-04-06 16:14:18,088 44k INFO ====> Epoch: 186, cost 33.30 s |
| 2023-04-06 16:14:51,126 44k INFO ====> Epoch: 187, cost 33.04 s |
| 2023-04-06 16:15:23,846 44k INFO ====> Epoch: 188, cost 32.72 s |
| 2023-04-06 16:15:54,766 44k INFO Train Epoch: 189 [89%] |
| 2023-04-06 16:15:54,767 44k INFO Losses: [2.4710679054260254, 2.1371006965637207, 8.654294967651367, 19.12156105041504, 0.9505603313446045], step: 3400, lr: 9.767725399135504e-05 |
| 2023-04-06 16:15:58,544 44k INFO ====> Epoch: 189, cost 34.70 s |
| 2023-04-06 16:16:31,701 44k INFO ====> Epoch: 190, cost 33.16 s |
| 2023-04-06 16:17:05,161 44k INFO ====> Epoch: 191, cost 33.46 s |
| 2023-04-06 16:17:38,244 44k INFO ====> Epoch: 192, cost 33.08 s |
| 2023-04-06 16:18:11,457 44k INFO ====> Epoch: 193, cost 33.21 s |
| 2023-04-06 16:18:44,985 44k INFO ====> Epoch: 194, cost 33.53 s |
| 2023-04-06 16:19:18,075 44k INFO ====> Epoch: 195, cost 33.09 s |
| 2023-04-06 16:19:51,298 44k INFO ====> Epoch: 196, cost 33.22 s |
| 2023-04-06 16:20:24,415 44k INFO ====> Epoch: 197, cost 33.12 s |
| 2023-04-06 16:20:58,269 44k INFO ====> Epoch: 198, cost 33.85 s |
| 2023-04-06 16:21:31,916 44k INFO ====> Epoch: 199, cost 33.65 s |
| 2023-04-06 16:22:05,610 44k INFO ====> Epoch: 200, cost 33.69 s |
| 2023-04-06 16:22:18,432 44k INFO Train Epoch: 201 [0%] |
| 2023-04-06 16:22:18,433 44k INFO Losses: [2.5765528678894043, 2.317922592163086, 9.504670143127441, 20.353900909423828, 1.0593161582946777], step: 3600, lr: 9.753083879807726e-05 |
| 2023-04-06 16:22:41,495 44k INFO ====> Epoch: 201, cost 35.88 s |
| 2023-04-06 16:23:15,048 44k INFO ====> Epoch: 202, cost 33.55 s |
| 2023-04-06 16:23:47,482 44k INFO ====> Epoch: 203, cost 32.43 s |
| 2023-04-06 16:24:21,199 44k INFO ====> Epoch: 204, cost 33.72 s |
| 2023-04-06 16:24:54,462 44k INFO ====> Epoch: 205, cost 33.26 s |
| 2023-04-06 16:25:28,278 44k INFO ====> Epoch: 206, cost 33.82 s |
| 2023-04-06 16:26:02,137 44k INFO ====> Epoch: 207, cost 33.86 s |
| 2023-04-06 16:26:36,011 44k INFO ====> Epoch: 208, cost 33.87 s |
| 2023-04-06 16:27:09,036 44k INFO ====> Epoch: 209, cost 33.03 s |
| 2023-04-06 16:27:42,014 44k INFO ====> Epoch: 210, cost 32.98 s |
| 2023-04-06 16:28:15,042 44k INFO ====> Epoch: 211, cost 33.03 s |
| 2023-04-06 16:28:29,366 44k INFO Train Epoch: 212 [11%] |
| 2023-04-06 16:28:29,367 44k INFO Losses: [2.4090394973754883, 2.35317063331604, 6.940808296203613, 20.978504180908203, 0.857198178768158], step: 3800, lr: 9.739681767887146e-05 |
| 2023-04-06 16:28:49,455 44k INFO ====> Epoch: 212, cost 34.41 s |
| 2023-04-06 16:29:23,060 44k INFO ====> Epoch: 213, cost 33.60 s |
| 2023-04-06 16:29:55,832 44k INFO ====> Epoch: 214, cost 32.77 s |
| 2023-04-06 16:30:28,804 44k INFO ====> Epoch: 215, cost 32.97 s |
| 2023-04-06 16:31:02,420 44k INFO ====> Epoch: 216, cost 33.62 s |
| 2023-04-06 16:31:35,997 44k INFO ====> Epoch: 217, cost 33.58 s |
| 2023-04-06 16:32:10,810 44k INFO ====> Epoch: 218, cost 34.81 s |
| 2023-04-06 16:32:44,018 44k INFO ====> Epoch: 219, cost 33.21 s |
| 2023-04-06 16:33:17,237 44k INFO ====> Epoch: 220, cost 33.22 s |
| 2023-04-06 16:33:50,289 44k INFO ====> Epoch: 221, cost 33.05 s |
| 2023-04-06 16:34:23,540 44k INFO ====> Epoch: 222, cost 33.25 s |
| 2023-04-06 16:34:40,302 44k INFO Train Epoch: 223 [22%] |
| 2023-04-06 16:34:40,303 44k INFO Losses: [2.5694820880889893, 2.331211805343628, 10.764482498168945, 20.099136352539062, 0.8609465956687927], step: 4000, lr: 9.726298072357337e-05 |
| 2023-04-06 16:34:48,444 44k INFO Saving model and optimizer state at iteration 223 to ./logs/44k/G_4000.pth |
| 2023-04-06 16:34:51,453 44k INFO Saving model and optimizer state at iteration 223 to ./logs/44k/D_4000.pth |
| 2023-04-06 16:34:53,922 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_1600.pth |
| 2023-04-06 16:34:53,960 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_1600.pth |
| 2023-04-06 16:35:10,618 44k INFO ====> Epoch: 223, cost 47.08 s |
| 2023-04-06 16:35:44,170 44k INFO ====> Epoch: 224, cost 33.55 s |
| 2023-04-06 16:36:17,884 44k INFO ====> Epoch: 225, cost 33.71 s |
| 2023-04-06 16:36:51,720 44k INFO ====> Epoch: 226, cost 33.84 s |
| 2023-04-06 16:37:25,079 44k INFO ====> Epoch: 227, cost 33.36 s |
| 2023-04-06 16:37:58,857 44k INFO ====> Epoch: 228, cost 33.78 s |
| 2023-04-06 16:38:33,315 44k INFO ====> Epoch: 229, cost 34.46 s |
| 2023-04-06 16:39:06,279 44k INFO ====> Epoch: 230, cost 32.96 s |
| 2023-04-06 16:39:39,579 44k INFO ====> Epoch: 231, cost 33.30 s |
| 2023-04-06 16:40:13,317 44k INFO ====> Epoch: 232, cost 33.74 s |
| 2023-04-06 16:40:46,510 44k INFO ====> Epoch: 233, cost 33.19 s |
| 2023-04-06 16:41:05,693 44k INFO Train Epoch: 234 [33%] |
| 2023-04-06 16:41:05,694 44k INFO Losses: [2.280306100845337, 2.313676595687866, 12.137205123901367, 21.159605026245117, 0.682966947555542], step: 4200, lr: 9.71293276791158e-05 |
| 2023-04-06 16:41:21,619 44k INFO ====> Epoch: 234, cost 35.11 s |
| 2023-04-06 16:41:54,856 44k INFO ====> Epoch: 235, cost 33.24 s |
| 2023-04-06 16:42:28,116 44k INFO ====> Epoch: 236, cost 33.26 s |
| 2023-04-06 16:43:01,765 44k INFO ====> Epoch: 237, cost 33.65 s |
| 2023-04-06 16:43:35,630 44k INFO ====> Epoch: 238, cost 33.86 s |
| 2023-04-06 16:44:09,514 44k INFO ====> Epoch: 239, cost 33.88 s |
| 2023-04-06 16:44:43,136 44k INFO ====> Epoch: 240, cost 33.62 s |
| 2023-04-06 16:45:17,116 44k INFO ====> Epoch: 241, cost 33.98 s |
| 2023-04-06 16:45:51,264 44k INFO ====> Epoch: 242, cost 34.15 s |
| 2023-04-06 16:46:25,287 44k INFO ====> Epoch: 243, cost 34.02 s |
| 2023-04-06 16:46:58,858 44k INFO ====> Epoch: 244, cost 33.57 s |
| 2023-04-06 16:47:21,003 44k INFO Train Epoch: 245 [44%] |
| 2023-04-06 16:47:21,004 44k INFO Losses: [2.3801774978637695, 2.1010258197784424, 9.494706153869629, 17.849163055419922, 0.9957065582275391], step: 4400, lr: 9.699585829277933e-05 |
| 2023-04-06 16:47:34,246 44k INFO ====> Epoch: 245, cost 35.39 s |
| 2023-04-06 16:48:07,426 44k INFO ====> Epoch: 246, cost 33.18 s |
| 2023-04-06 16:48:40,848 44k INFO ====> Epoch: 247, cost 33.42 s |
| 2023-04-06 16:49:14,451 44k INFO ====> Epoch: 248, cost 33.60 s |
| 2023-04-06 16:49:47,899 44k INFO ====> Epoch: 249, cost 33.45 s |
| 2023-04-06 16:50:21,514 44k INFO ====> Epoch: 250, cost 33.61 s |
| 2023-04-06 16:50:54,961 44k INFO ====> Epoch: 251, cost 33.45 s |
| 2023-04-06 16:51:28,381 44k INFO ====> Epoch: 252, cost 33.42 s |
| 2023-04-06 16:52:01,907 44k INFO ====> Epoch: 253, cost 33.53 s |
| 2023-04-06 16:52:35,442 44k INFO ====> Epoch: 254, cost 33.53 s |
| 2023-04-06 16:53:09,396 44k INFO ====> Epoch: 255, cost 33.95 s |
| 2023-04-06 16:53:34,616 44k INFO Train Epoch: 256 [56%] |
| 2023-04-06 16:53:34,617 44k INFO Losses: [2.5316028594970703, 2.0892834663391113, 10.589302062988281, 18.844057083129883, 0.5367166996002197], step: 4600, lr: 9.68625723121918e-05 |
| 2023-04-06 16:53:45,389 44k INFO ====> Epoch: 256, cost 35.99 s |
| 2023-04-06 16:54:19,338 44k INFO ====> Epoch: 257, cost 33.95 s |
| 2023-04-06 16:54:53,131 44k INFO ====> Epoch: 258, cost 33.79 s |
| 2023-04-06 16:55:26,597 44k INFO ====> Epoch: 259, cost 33.47 s |
| 2023-04-06 16:56:00,065 44k INFO ====> Epoch: 260, cost 33.47 s |
| 2023-04-06 16:56:33,430 44k INFO ====> Epoch: 261, cost 33.37 s |
| 2023-04-06 16:57:06,610 44k INFO ====> Epoch: 262, cost 33.18 s |
| 2023-04-06 16:57:39,917 44k INFO ====> Epoch: 263, cost 33.31 s |
| 2023-04-06 16:58:13,010 44k INFO ====> Epoch: 264, cost 33.09 s |
| 2023-04-06 16:58:46,623 44k INFO ====> Epoch: 265, cost 33.61 s |
| 2023-04-06 16:59:20,183 44k INFO ====> Epoch: 266, cost 33.56 s |
| 2023-04-06 16:59:46,793 44k INFO Train Epoch: 267 [67%] |
| 2023-04-06 16:59:46,794 44k INFO Losses: [2.5026473999023438, 2.2426648139953613, 7.876044273376465, 19.031328201293945, 1.068752408027649], step: 4800, lr: 9.67294694853279e-05 |
| 2023-04-06 16:59:54,870 44k INFO Saving model and optimizer state at iteration 267 to ./logs/44k/G_4800.pth |
| 2023-04-06 16:59:57,509 44k INFO Saving model and optimizer state at iteration 267 to ./logs/44k/D_4800.pth |
| 2023-04-06 16:59:59,774 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_2400.pth |
| 2023-04-06 16:59:59,792 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_2400.pth |
| 2023-04-06 17:00:06,567 44k INFO ====> Epoch: 267, cost 46.38 s |
| 2023-04-06 17:00:41,439 44k INFO ====> Epoch: 268, cost 34.87 s |
| 2023-04-06 17:01:14,983 44k INFO ====> Epoch: 269, cost 33.54 s |
| 2023-04-06 17:01:48,354 44k INFO ====> Epoch: 270, cost 33.37 s |
| 2023-04-06 17:02:22,124 44k INFO ====> Epoch: 271, cost 33.77 s |
| 2023-04-06 17:02:55,709 44k INFO ====> Epoch: 272, cost 33.59 s |
| 2023-04-06 17:03:29,304 44k INFO ====> Epoch: 273, cost 33.59 s |
| 2023-04-06 17:04:03,052 44k INFO ====> Epoch: 274, cost 33.75 s |
| 2023-04-06 17:04:36,755 44k INFO ====> Epoch: 275, cost 33.70 s |
| 2023-04-06 17:05:10,636 44k INFO ====> Epoch: 276, cost 33.88 s |
| 2023-04-06 17:05:44,147 44k INFO ====> Epoch: 277, cost 33.51 s |
| 2023-04-06 17:06:13,645 44k INFO Train Epoch: 278 [78%] |
| 2023-04-06 17:06:13,646 44k INFO Losses: [2.336780309677124, 2.2390074729919434, 11.930177688598633, 20.725872039794922, 0.5927470326423645], step: 5000, lr: 9.659654956050859e-05 |
| 2023-04-06 17:06:19,541 44k INFO ====> Epoch: 278, cost 35.39 s |
| 2023-04-06 17:06:54,250 44k INFO ====> Epoch: 279, cost 34.71 s |
| 2023-04-06 17:07:28,218 44k INFO ====> Epoch: 280, cost 33.97 s |
| 2023-04-06 17:08:02,618 44k INFO ====> Epoch: 281, cost 34.40 s |
| 2023-04-06 17:08:36,031 44k INFO ====> Epoch: 282, cost 33.41 s |
| 2023-04-06 17:09:09,577 44k INFO ====> Epoch: 283, cost 33.55 s |
| 2023-04-06 17:09:43,485 44k INFO ====> Epoch: 284, cost 33.91 s |
| 2023-04-06 17:10:17,361 44k INFO ====> Epoch: 285, cost 33.88 s |
| 2023-04-06 17:10:50,800 44k INFO ====> Epoch: 286, cost 33.44 s |
| 2023-04-06 17:11:24,329 44k INFO ====> Epoch: 287, cost 33.53 s |
| 2023-04-06 17:11:57,541 44k INFO ====> Epoch: 288, cost 33.21 s |
| 2023-04-06 17:12:28,773 44k INFO Train Epoch: 289 [89%] |
| 2023-04-06 17:12:28,774 44k INFO Losses: [2.385878562927246, 2.2772061824798584, 7.589957237243652, 18.116687774658203, 0.925600528717041], step: 5200, lr: 9.646381228640066e-05 |
| 2023-04-06 17:12:32,702 44k INFO ====> Epoch: 289, cost 35.16 s |
| 2023-04-06 17:13:06,900 44k INFO ====> Epoch: 290, cost 34.20 s |
| 2023-04-06 17:13:40,746 44k INFO ====> Epoch: 291, cost 33.85 s |
| 2023-04-06 17:14:14,405 44k INFO ====> Epoch: 292, cost 33.66 s |
| 2023-04-06 17:14:47,901 44k INFO ====> Epoch: 293, cost 33.50 s |
| 2023-04-06 17:15:22,146 44k INFO ====> Epoch: 294, cost 34.25 s |
| 2023-04-06 17:15:56,201 44k INFO ====> Epoch: 295, cost 34.05 s |
| 2023-04-06 17:16:29,525 44k INFO ====> Epoch: 296, cost 33.32 s |
| 2023-04-06 17:17:02,512 44k INFO ====> Epoch: 297, cost 32.99 s |
| 2023-04-06 17:17:36,118 44k INFO ====> Epoch: 298, cost 33.61 s |
| 2023-04-06 17:18:09,488 44k INFO ====> Epoch: 299, cost 33.37 s |
| 2023-04-06 17:18:43,408 44k INFO ====> Epoch: 300, cost 33.92 s |
| 2023-04-06 17:18:55,456 44k INFO Train Epoch: 301 [0%] |
| 2023-04-06 17:18:55,458 44k INFO Losses: [2.4418349266052246, 2.024925470352173, 11.340639114379883, 21.75313949584961, 0.7900910973548889], step: 5400, lr: 9.631921600483981e-05 |
| 2023-04-06 17:19:19,113 44k INFO ====> Epoch: 301, cost 35.71 s |
| 2023-04-06 17:19:52,445 44k INFO ====> Epoch: 302, cost 33.33 s |
| 2023-04-06 17:20:26,150 44k INFO ====> Epoch: 303, cost 33.71 s |
| 2023-04-06 17:20:59,723 44k INFO ====> Epoch: 304, cost 33.57 s |
| 2023-04-06 17:21:33,468 44k INFO ====> Epoch: 305, cost 33.74 s |
| 2023-04-06 17:22:07,157 44k INFO ====> Epoch: 306, cost 33.69 s |
| 2023-04-06 17:22:40,790 44k INFO ====> Epoch: 307, cost 33.63 s |
| 2023-04-06 17:23:15,437 44k INFO ====> Epoch: 308, cost 34.65 s |
| 2023-04-06 17:23:48,982 44k INFO ====> Epoch: 309, cost 33.54 s |
| 2023-04-06 17:24:22,362 44k INFO ====> Epoch: 310, cost 33.38 s |
| 2023-04-06 17:24:55,753 44k INFO ====> Epoch: 311, cost 33.39 s |
| 2023-04-06 17:25:09,900 44k INFO Train Epoch: 312 [11%] |
| 2023-04-06 17:25:09,901 44k INFO Losses: [2.2993826866149902, 2.5729517936706543, 9.609771728515625, 20.48442268371582, 1.0834064483642578], step: 5600, lr: 9.618685982612675e-05 |
| 2023-04-06 17:25:18,064 44k INFO Saving model and optimizer state at iteration 312 to ./logs/44k/G_5600.pth |
| 2023-04-06 17:25:21,357 44k INFO Saving model and optimizer state at iteration 312 to ./logs/44k/D_5600.pth |
| 2023-04-06 17:25:23,632 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_3200.pth |
| 2023-04-06 17:25:23,654 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_3200.pth |
| 2023-04-06 17:25:42,928 44k INFO ====> Epoch: 312, cost 47.17 s |
| 2023-04-06 17:26:17,141 44k INFO ====> Epoch: 313, cost 34.21 s |
| 2023-04-06 17:26:51,938 44k INFO ====> Epoch: 314, cost 34.80 s |
| 2023-04-06 17:27:27,322 44k INFO ====> Epoch: 315, cost 35.38 s |
| 2023-04-06 17:28:00,914 44k INFO ====> Epoch: 316, cost 33.59 s |
| 2023-04-06 17:28:34,276 44k INFO ====> Epoch: 317, cost 33.36 s |
| 2023-04-06 17:29:07,777 44k INFO ====> Epoch: 318, cost 33.50 s |
| 2023-04-06 17:29:41,937 44k INFO ====> Epoch: 319, cost 34.16 s |
| 2023-04-06 17:30:15,474 44k INFO ====> Epoch: 320, cost 33.54 s |
| 2023-04-06 17:30:48,876 44k INFO ====> Epoch: 321, cost 33.40 s |
| 2023-04-06 17:31:22,605 44k INFO ====> Epoch: 322, cost 33.73 s |
| 2023-04-06 17:31:39,703 44k INFO Train Epoch: 323 [22%] |
| 2023-04-06 17:31:39,704 44k INFO Losses: [2.4820761680603027, 2.1193840503692627, 9.61584186553955, 18.75108528137207, 0.8709684014320374], step: 5800, lr: 9.60546855234585e-05 |
| 2023-04-06 17:31:57,655 44k INFO ====> Epoch: 323, cost 35.05 s |
| 2023-04-06 17:32:31,106 44k INFO ====> Epoch: 324, cost 33.45 s |
| 2023-04-06 17:33:04,825 44k INFO ====> Epoch: 325, cost 33.72 s |
| 2023-04-06 17:33:38,728 44k INFO ====> Epoch: 326, cost 33.90 s |
| 2023-04-06 17:34:13,554 44k INFO ====> Epoch: 327, cost 34.83 s |
| 2023-04-06 17:34:47,381 44k INFO ====> Epoch: 328, cost 33.83 s |
| 2023-04-06 17:35:21,672 44k INFO ====> Epoch: 329, cost 34.29 s |
| 2023-04-06 17:35:55,556 44k INFO ====> Epoch: 330, cost 33.88 s |
| 2023-04-06 17:36:29,481 44k INFO ====> Epoch: 331, cost 33.93 s |
| 2023-04-06 17:37:02,840 44k INFO ====> Epoch: 332, cost 33.36 s |
| 2023-04-06 17:37:36,638 44k INFO ====> Epoch: 333, cost 33.80 s |
| 2023-04-06 17:37:55,526 44k INFO Train Epoch: 334 [33%] |
| 2023-04-06 17:37:55,527 44k INFO Losses: [2.4064953327178955, 2.66304349899292, 10.577417373657227, 20.366371154785156, 0.7555525898933411], step: 6000, lr: 9.592269284691169e-05 |
| 2023-04-06 17:38:11,787 44k INFO ====> Epoch: 334, cost 35.15 s |
| 2023-04-06 17:38:45,301 44k INFO ====> Epoch: 335, cost 33.51 s |
| 2023-04-06 17:39:18,540 44k INFO ====> Epoch: 336, cost 33.24 s |
| 2023-04-06 17:39:51,928 44k INFO ====> Epoch: 337, cost 33.39 s |
| 2023-04-06 17:40:25,228 44k INFO ====> Epoch: 338, cost 33.30 s |
| 2023-04-06 17:40:59,031 44k INFO ====> Epoch: 339, cost 33.80 s |
| 2023-04-06 17:41:32,618 44k INFO ====> Epoch: 340, cost 33.59 s |
| 2023-04-06 17:42:06,122 44k INFO ====> Epoch: 341, cost 33.50 s |
| 2023-04-06 17:42:39,641 44k INFO ====> Epoch: 342, cost 33.52 s |
| 2023-04-06 17:43:14,283 44k INFO ====> Epoch: 343, cost 34.64 s |
| 2023-04-06 17:43:47,305 44k INFO ====> Epoch: 344, cost 33.02 s |
| 2023-04-06 17:44:08,818 44k INFO Train Epoch: 345 [44%] |
| 2023-04-06 17:44:08,820 44k INFO Losses: [2.5008935928344727, 2.476442813873291, 8.65200424194336, 19.446016311645508, 0.897552490234375], step: 6200, lr: 9.579088154690645e-05 |
| 2023-04-06 17:44:22,667 44k INFO ====> Epoch: 345, cost 35.36 s |
| 2023-04-06 17:44:56,141 44k INFO ====> Epoch: 346, cost 33.47 s |
| 2023-04-06 17:45:29,619 44k INFO ====> Epoch: 347, cost 33.48 s |
| 2023-04-06 17:46:02,997 44k INFO ====> Epoch: 348, cost 33.38 s |
| 2023-04-06 17:46:36,029 44k INFO ====> Epoch: 349, cost 33.03 s |
| 2023-04-06 17:47:09,417 44k INFO ====> Epoch: 350, cost 33.39 s |
| 2023-04-06 17:47:43,613 44k INFO ====> Epoch: 351, cost 34.20 s |
| 2023-04-06 17:48:17,278 44k INFO ====> Epoch: 352, cost 33.67 s |
| 2023-04-06 17:48:50,891 44k INFO ====> Epoch: 353, cost 33.61 s |
| 2023-04-06 17:49:24,801 44k INFO ====> Epoch: 354, cost 33.91 s |
| 2023-04-06 17:49:58,546 44k INFO ====> Epoch: 355, cost 33.74 s |
| 2023-04-06 17:50:23,180 44k INFO Train Epoch: 356 [56%] |
| 2023-04-06 17:50:23,195 44k INFO Losses: [2.4418582916259766, 2.2405121326446533, 10.872772216796875, 17.84126853942871, 0.5929334759712219], step: 6400, lr: 9.565925137420586e-05 |
| 2023-04-06 17:50:31,313 44k INFO Saving model and optimizer state at iteration 356 to ./logs/44k/G_6400.pth |
| 2023-04-06 17:50:34,706 44k INFO Saving model and optimizer state at iteration 356 to ./logs/44k/D_6400.pth |
| 2023-04-06 17:50:36,723 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_4000.pth |
| 2023-04-06 17:50:36,741 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_4000.pth |
| 2023-04-06 17:50:46,170 44k INFO ====> Epoch: 356, cost 47.62 s |
| 2023-04-06 17:51:19,938 44k INFO ====> Epoch: 357, cost 33.77 s |
| 2023-04-06 17:51:53,407 44k INFO ====> Epoch: 358, cost 33.47 s |
| 2023-04-06 17:52:26,917 44k INFO ====> Epoch: 359, cost 33.51 s |
| 2023-04-06 17:53:00,258 44k INFO ====> Epoch: 360, cost 33.34 s |
| 2023-04-06 17:53:34,014 44k INFO ====> Epoch: 361, cost 33.76 s |
| 2023-04-06 17:54:07,388 44k INFO ====> Epoch: 362, cost 33.37 s |
| 2023-04-06 17:54:40,878 44k INFO ====> Epoch: 363, cost 33.49 s |
| 2023-04-06 17:55:14,530 44k INFO ====> Epoch: 364, cost 33.65 s |
| 2023-04-06 17:55:48,242 44k INFO ====> Epoch: 365, cost 33.71 s |
| 2023-04-06 17:56:21,978 44k INFO ====> Epoch: 366, cost 33.74 s |
| 2023-04-06 17:56:48,837 44k INFO Train Epoch: 367 [67%] |
| 2023-04-06 17:56:48,838 44k INFO Losses: [2.554368495941162, 2.3327033519744873, 9.829940795898438, 18.284847259521484, 0.6705255508422852], step: 6600, lr: 9.552780207991543e-05 |
| 2023-04-06 17:56:57,173 44k INFO ====> Epoch: 367, cost 35.19 s |
| 2023-04-06 17:57:30,583 44k INFO ====> Epoch: 368, cost 33.41 s |
| 2023-04-06 17:58:04,232 44k INFO ====> Epoch: 369, cost 33.65 s |
| 2023-04-06 17:58:37,876 44k INFO ====> Epoch: 370, cost 33.64 s |
| 2023-04-06 17:59:11,271 44k INFO ====> Epoch: 371, cost 33.39 s |
| 2023-04-06 17:59:44,903 44k INFO ====> Epoch: 372, cost 33.63 s |
| 2023-04-06 18:00:18,504 44k INFO ====> Epoch: 373, cost 33.60 s |
| 2023-04-06 18:00:52,016 44k INFO ====> Epoch: 374, cost 33.51 s |
| 2023-04-06 18:01:25,671 44k INFO ====> Epoch: 375, cost 33.66 s |
| 2023-04-06 18:01:59,300 44k INFO ====> Epoch: 376, cost 33.63 s |
| 2023-04-06 18:02:32,972 44k INFO ====> Epoch: 377, cost 33.67 s |
| 2023-04-06 18:03:02,229 44k INFO Train Epoch: 378 [78%] |
| 2023-04-06 18:03:02,231 44k INFO Losses: [2.393739700317383, 2.3625659942626953, 9.48896312713623, 20.4971866607666, 0.6042823195457458], step: 6800, lr: 9.53965334154828e-05 |
| 2023-04-06 18:03:08,140 44k INFO ====> Epoch: 378, cost 35.17 s |
| 2023-04-06 18:03:42,388 44k INFO ====> Epoch: 379, cost 34.25 s |
| 2023-04-06 18:04:15,398 44k INFO ====> Epoch: 380, cost 33.01 s |
| 2023-04-06 18:04:48,620 44k INFO ====> Epoch: 381, cost 33.22 s |
| 2023-04-06 18:05:22,030 44k INFO ====> Epoch: 382, cost 33.41 s |
| 2023-04-06 18:05:55,530 44k INFO ====> Epoch: 383, cost 33.50 s |
| 2023-04-06 18:06:29,132 44k INFO ====> Epoch: 384, cost 33.60 s |
| 2023-04-06 18:07:02,378 44k INFO ====> Epoch: 385, cost 33.25 s |
| 2023-04-06 18:07:35,862 44k INFO ====> Epoch: 386, cost 33.48 s |
| 2023-04-06 18:08:09,492 44k INFO ====> Epoch: 387, cost 33.63 s |
| 2023-04-06 18:08:42,771 44k INFO ====> Epoch: 388, cost 33.28 s |
| 2023-04-06 18:09:13,869 44k INFO Train Epoch: 389 [89%] |
| 2023-04-06 18:09:13,870 44k INFO Losses: [2.431995391845703, 1.9307184219360352, 9.80262279510498, 17.839599609375, 1.0465185642242432], step: 7000, lr: 9.526544513269702e-05 |
| 2023-04-06 18:09:17,762 44k INFO ====> Epoch: 389, cost 34.99 s |
| 2023-04-06 18:09:51,455 44k INFO ====> Epoch: 390, cost 33.69 s |
| 2023-04-06 18:10:25,326 44k INFO ====> Epoch: 391, cost 33.87 s |
| 2023-04-06 18:10:59,206 44k INFO ====> Epoch: 392, cost 33.88 s |
| 2023-04-06 18:11:33,821 44k INFO ====> Epoch: 393, cost 34.61 s |
| 2023-04-06 18:12:06,960 44k INFO ====> Epoch: 394, cost 33.14 s |
| 2023-04-06 18:12:40,180 44k INFO ====> Epoch: 395, cost 33.22 s |
| 2023-04-06 18:13:13,083 44k INFO ====> Epoch: 396, cost 32.90 s |
| 2023-04-06 18:13:46,451 44k INFO ====> Epoch: 397, cost 33.37 s |
| 2023-04-06 18:14:19,552 44k INFO ====> Epoch: 398, cost 33.10 s |
| 2023-04-06 18:14:53,253 44k INFO ====> Epoch: 399, cost 33.70 s |
| 2023-04-06 18:15:26,650 44k INFO ====> Epoch: 400, cost 33.40 s |
| 2023-04-06 18:15:38,441 44k INFO Train Epoch: 401 [0%] |
| 2023-04-06 18:15:38,442 44k INFO Losses: [2.593489646911621, 2.073544979095459, 8.42275619506836, 20.45761489868164, 0.9182995557785034], step: 7200, lr: 9.512264516656537e-05 |
| 2023-04-06 18:15:46,917 44k INFO Saving model and optimizer state at iteration 401 to ./logs/44k/G_7200.pth |
| 2023-04-06 18:15:49,493 44k INFO Saving model and optimizer state at iteration 401 to ./logs/44k/D_7200.pth |
| 2023-04-06 18:15:51,656 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_4800.pth |
| 2023-04-06 18:15:51,692 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_4800.pth |
| 2023-04-06 18:16:12,795 44k INFO ====> Epoch: 401, cost 46.15 s |
| 2023-04-06 18:16:48,449 44k INFO ====> Epoch: 402, cost 35.65 s |
| 2023-04-06 18:17:21,303 44k INFO ====> Epoch: 403, cost 32.85 s |
| 2023-04-06 18:17:54,163 44k INFO ====> Epoch: 404, cost 32.86 s |
| 2023-04-06 18:18:27,499 44k INFO ====> Epoch: 405, cost 33.34 s |
| 2023-04-06 18:19:00,117 44k INFO ====> Epoch: 406, cost 32.62 s |
| 2023-04-06 18:19:32,815 44k INFO ====> Epoch: 407, cost 32.70 s |
| 2023-04-06 18:20:05,338 44k INFO ====> Epoch: 408, cost 32.52 s |
| 2023-04-06 18:20:38,251 44k INFO ====> Epoch: 409, cost 32.91 s |
| 2023-04-06 18:21:11,470 44k INFO ====> Epoch: 410, cost 33.22 s |
| 2023-04-06 18:21:46,000 44k INFO ====> Epoch: 411, cost 34.53 s |
| 2023-04-06 18:22:01,037 44k INFO Train Epoch: 412 [11%] |
| 2023-04-06 18:22:01,038 44k INFO Losses: [2.1447744369506836, 2.547001838684082, 10.99000072479248, 19.626802444458008, 0.7141999006271362], step: 7400, lr: 9.49919332448374e-05 |
| 2023-04-06 18:22:21,928 44k INFO ====> Epoch: 412, cost 35.93 s |
| 2023-04-06 18:22:56,926 44k INFO ====> Epoch: 413, cost 35.00 s |
| 2023-04-06 18:23:31,265 44k INFO ====> Epoch: 414, cost 34.34 s |
| 2023-04-06 18:24:05,002 44k INFO ====> Epoch: 415, cost 33.74 s |
| 2023-04-06 18:24:38,474 44k INFO ====> Epoch: 416, cost 33.47 s |
| 2023-04-06 18:25:12,157 44k INFO ====> Epoch: 417, cost 33.68 s |
| 2023-04-06 18:25:45,599 44k INFO ====> Epoch: 418, cost 33.44 s |
| 2023-04-06 18:26:18,842 44k INFO ====> Epoch: 419, cost 33.24 s |
| 2023-04-06 18:26:52,083 44k INFO ====> Epoch: 420, cost 33.24 s |
| 2023-04-06 18:27:25,208 44k INFO ====> Epoch: 421, cost 33.13 s |
| 2023-04-06 18:27:59,523 44k INFO ====> Epoch: 422, cost 34.31 s |
| 2023-04-06 18:28:15,815 44k INFO Train Epoch: 423 [22%] |
| 2023-04-06 18:28:15,816 44k INFO Losses: [2.537222385406494, 2.3694236278533936, 8.022411346435547, 19.27557373046875, 0.6971426606178284], step: 7600, lr: 9.486140093971337e-05 |
| 2023-04-06 18:28:33,895 44k INFO ====> Epoch: 423, cost 34.37 s |
| 2023-04-06 18:29:07,256 44k INFO ====> Epoch: 424, cost 33.36 s |
| 2023-04-06 18:29:39,938 44k INFO ====> Epoch: 425, cost 32.68 s |
| 2023-04-06 18:30:12,540 44k INFO ====> Epoch: 426, cost 32.60 s |
| 2023-04-06 18:30:45,176 44k INFO ====> Epoch: 427, cost 32.64 s |
| 2023-04-06 18:31:17,932 44k INFO ====> Epoch: 428, cost 32.76 s |
| 2023-04-06 18:31:51,059 44k INFO ====> Epoch: 429, cost 33.13 s |
| 2023-04-06 18:32:24,138 44k INFO ====> Epoch: 430, cost 33.08 s |
| 2023-04-06 18:32:56,916 44k INFO ====> Epoch: 431, cost 32.78 s |
| 2023-04-06 18:33:29,850 44k INFO ====> Epoch: 432, cost 32.93 s |
| 2023-04-06 18:34:02,979 44k INFO ====> Epoch: 433, cost 33.13 s |
| 2023-04-06 18:34:22,044 44k INFO Train Epoch: 434 [33%] |
| 2023-04-06 18:34:22,046 44k INFO Losses: [2.5888988971710205, 2.652815103530884, 12.130846977233887, 19.23379898071289, 0.6897168159484863], step: 7800, lr: 9.473104800437474e-05 |
| 2023-04-06 18:34:37,583 44k INFO ====> Epoch: 434, cost 34.60 s |
| 2023-04-06 18:35:10,466 44k INFO ====> Epoch: 435, cost 32.88 s |
| 2023-04-06 18:35:43,117 44k INFO ====> Epoch: 436, cost 32.65 s |
| 2023-04-06 18:36:16,021 44k INFO ====> Epoch: 437, cost 32.90 s |
| 2023-04-06 18:36:49,001 44k INFO ====> Epoch: 438, cost 32.98 s |
| 2023-04-06 18:37:22,642 44k INFO ====> Epoch: 439, cost 33.64 s |
| 2023-04-06 18:37:55,974 44k INFO ====> Epoch: 440, cost 33.33 s |
| 2023-04-06 18:38:29,213 44k INFO ====> Epoch: 441, cost 33.24 s |
| 2023-04-06 18:39:02,172 44k INFO ====> Epoch: 442, cost 32.96 s |
| 2023-04-06 18:39:35,318 44k INFO ====> Epoch: 443, cost 33.15 s |
| 2023-04-06 18:40:09,009 44k INFO ====> Epoch: 444, cost 33.69 s |
| 2023-04-06 18:40:29,898 44k INFO Train Epoch: 445 [44%] |
| 2023-04-06 18:40:29,906 44k INFO Losses: [2.449326515197754, 2.2474606037139893, 9.135872840881348, 17.043214797973633, 1.1349270343780518], step: 8000, lr: 9.460087419234215e-05 |
| 2023-04-06 18:40:37,738 44k INFO Saving model and optimizer state at iteration 445 to ./logs/44k/G_8000.pth |
| 2023-04-06 18:40:40,799 44k INFO Saving model and optimizer state at iteration 445 to ./logs/44k/D_8000.pth |
| 2023-04-06 18:40:42,913 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_5600.pth |
| 2023-04-06 18:40:42,931 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_5600.pth |
| 2023-04-06 18:40:54,369 44k INFO ====> Epoch: 445, cost 45.36 s |
| 2023-04-06 18:41:27,287 44k INFO ====> Epoch: 446, cost 32.92 s |
| 2023-04-06 18:42:00,322 44k INFO ====> Epoch: 447, cost 33.04 s |
| 2023-04-06 18:42:33,367 44k INFO ====> Epoch: 448, cost 33.05 s |
| 2023-04-06 18:43:06,523 44k INFO ====> Epoch: 449, cost 33.16 s |
| 2023-04-06 18:43:39,260 44k INFO ====> Epoch: 450, cost 32.74 s |
| 2023-04-06 18:44:12,295 44k INFO ====> Epoch: 451, cost 33.04 s |
| 2023-04-06 18:44:45,316 44k INFO ====> Epoch: 452, cost 33.02 s |
| 2023-04-06 18:45:18,870 44k INFO ====> Epoch: 453, cost 33.55 s |
| 2023-04-06 18:45:51,983 44k INFO ====> Epoch: 454, cost 33.11 s |
| 2023-04-06 18:46:25,270 44k INFO ====> Epoch: 455, cost 33.29 s |
| 2023-04-06 18:46:49,250 44k INFO Train Epoch: 456 [56%] |
| 2023-04-06 18:46:49,252 44k INFO Losses: [2.3567802906036377, 2.143897771835327, 9.234848976135254, 17.149524688720703, 0.6894568800926208], step: 8200, lr: 9.44708792574749e-05 |
| 2023-04-06 18:46:59,763 44k INFO ====> Epoch: 456, cost 34.49 s |
| 2023-04-06 18:47:33,704 44k INFO ====> Epoch: 457, cost 33.94 s |
| 2023-04-06 18:48:06,742 44k INFO ====> Epoch: 458, cost 33.04 s |
| 2023-04-06 18:48:39,574 44k INFO ====> Epoch: 459, cost 32.83 s |
| 2023-04-06 18:49:12,269 44k INFO ====> Epoch: 460, cost 32.70 s |
| 2023-04-06 18:49:45,030 44k INFO ====> Epoch: 461, cost 32.76 s |
| 2023-04-06 18:50:17,896 44k INFO ====> Epoch: 462, cost 32.87 s |
| 2023-04-06 18:50:51,156 44k INFO ====> Epoch: 463, cost 33.26 s |
| 2023-04-06 18:51:24,141 44k INFO ====> Epoch: 464, cost 32.98 s |
| 2023-04-06 18:51:57,196 44k INFO ====> Epoch: 465, cost 33.05 s |
| 2023-04-06 18:52:30,213 44k INFO ====> Epoch: 466, cost 33.02 s |
| 2023-04-06 18:52:56,415 44k INFO Train Epoch: 467 [67%] |
| 2023-04-06 18:52:56,416 44k INFO Losses: [2.467308759689331, 2.284391403198242, 8.696948051452637, 17.621639251708984, 0.924575686454773], step: 8400, lr: 9.434106295397058e-05 |
| 2023-04-06 18:53:04,429 44k INFO ====> Epoch: 467, cost 34.22 s |
| 2023-04-06 18:53:37,410 44k INFO ====> Epoch: 468, cost 32.98 s |
| 2023-04-06 18:54:10,326 44k INFO ====> Epoch: 469, cost 32.92 s |
| 2023-04-06 18:54:43,554 44k INFO ====> Epoch: 470, cost 33.23 s |
| 2023-04-06 18:55:16,737 44k INFO ====> Epoch: 471, cost 33.18 s |
| 2023-04-06 18:55:49,956 44k INFO ====> Epoch: 472, cost 33.22 s |
| 2023-04-06 18:56:23,098 44k INFO ====> Epoch: 473, cost 33.14 s |
| 2023-04-06 18:56:56,318 44k INFO ====> Epoch: 474, cost 33.22 s |
| 2023-04-06 18:57:29,722 44k INFO ====> Epoch: 475, cost 33.40 s |
| 2023-04-06 18:58:02,999 44k INFO ====> Epoch: 476, cost 33.28 s |
| 2023-04-06 18:58:36,919 44k INFO ====> Epoch: 477, cost 33.92 s |
| 2023-04-06 18:59:06,807 44k INFO Train Epoch: 478 [78%] |
| 2023-04-06 18:59:06,808 44k INFO Losses: [2.318368434906006, 2.5831010341644287, 10.86165714263916, 22.857358932495117, 0.560059130191803], step: 8600, lr: 9.421142503636453e-05 |
| 2023-04-06 18:59:12,961 44k INFO ====> Epoch: 478, cost 36.04 s |
| 2023-04-06 18:59:45,995 44k INFO ====> Epoch: 479, cost 33.03 s |
| 2023-04-06 19:00:18,982 44k INFO ====> Epoch: 480, cost 32.99 s |
| 2023-04-06 19:00:52,255 44k INFO ====> Epoch: 481, cost 33.27 s |
| 2023-04-06 19:01:25,500 44k INFO ====> Epoch: 482, cost 33.25 s |
| 2023-04-06 19:01:59,954 44k INFO ====> Epoch: 483, cost 34.45 s |
| 2023-04-06 19:02:34,325 44k INFO ====> Epoch: 484, cost 34.37 s |
| 2023-04-06 19:03:08,200 44k INFO ====> Epoch: 485, cost 33.88 s |
| 2023-04-06 19:03:41,648 44k INFO ====> Epoch: 486, cost 33.45 s |
| 2023-04-06 19:04:14,879 44k INFO ====> Epoch: 487, cost 33.23 s |
| 2023-04-06 19:04:48,296 44k INFO ====> Epoch: 488, cost 33.42 s |
| 2023-04-06 19:05:19,394 44k INFO Train Epoch: 489 [89%] |
| 2023-04-06 19:05:19,395 44k INFO Losses: [2.524198532104492, 2.032053232192993, 7.60945987701416, 17.46242332458496, 1.0244165658950806], step: 8800, lr: 9.408196525952938e-05 |
| 2023-04-06 19:05:27,206 44k INFO Saving model and optimizer state at iteration 489 to ./logs/44k/G_8800.pth |
| 2023-04-06 19:05:29,658 44k INFO Saving model and optimizer state at iteration 489 to ./logs/44k/D_8800.pth |
| 2023-04-06 19:05:32,156 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_6400.pth |
| 2023-04-06 19:05:32,176 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_6400.pth |
| 2023-04-06 19:05:34,623 44k INFO ====> Epoch: 489, cost 46.33 s |
| 2023-04-06 19:06:08,364 44k INFO ====> Epoch: 490, cost 33.74 s |
| 2023-04-06 19:06:41,198 44k INFO ====> Epoch: 491, cost 32.83 s |
| 2023-04-06 19:07:14,232 44k INFO ====> Epoch: 492, cost 33.03 s |
| 2023-04-06 19:07:47,558 44k INFO ====> Epoch: 493, cost 33.33 s |
| 2023-04-06 19:08:20,720 44k INFO ====> Epoch: 494, cost 33.16 s |
| 2023-04-06 19:08:54,066 44k INFO ====> Epoch: 495, cost 33.35 s |
| 2023-04-06 19:09:27,097 44k INFO ====> Epoch: 496, cost 33.03 s |
| 2023-04-06 19:10:00,060 44k INFO ====> Epoch: 497, cost 32.96 s |
| 2023-04-06 19:10:33,147 44k INFO ====> Epoch: 498, cost 33.09 s |
| 2023-04-06 19:11:06,269 44k INFO ====> Epoch: 499, cost 33.12 s |
| 2023-04-06 19:11:39,260 44k INFO ====> Epoch: 500, cost 32.99 s |
| 2023-04-06 19:11:50,845 44k INFO Train Epoch: 501 [0%] |
| 2023-04-06 19:11:50,846 44k INFO Losses: [2.3949179649353027, 2.2692530155181885, 8.20638370513916, 20.383365631103516, 1.1488752365112305], step: 9000, lr: 9.394093929325224e-05 |
| 2023-04-06 19:12:13,818 44k INFO ====> Epoch: 501, cost 34.56 s |
| 2023-04-06 19:12:46,826 44k INFO ====> Epoch: 502, cost 33.01 s |
| 2023-04-06 19:13:19,983 44k INFO ====> Epoch: 503, cost 33.16 s |
| 2023-04-06 19:13:53,209 44k INFO ====> Epoch: 504, cost 33.23 s |
| 2023-04-06 19:14:26,572 44k INFO ====> Epoch: 505, cost 33.36 s |
| 2023-04-06 19:14:59,985 44k INFO ====> Epoch: 506, cost 33.41 s |
| 2023-04-06 19:15:33,236 44k INFO ====> Epoch: 507, cost 33.25 s |
| 2023-04-06 19:16:06,443 44k INFO ====> Epoch: 508, cost 33.21 s |
| 2023-04-06 19:16:40,213 44k INFO ====> Epoch: 509, cost 33.77 s |
| 2023-04-06 19:17:13,136 44k INFO ====> Epoch: 510, cost 32.92 s |
| 2023-04-06 19:17:46,253 44k INFO ====> Epoch: 511, cost 33.12 s |
| 2023-04-06 19:18:00,182 44k INFO Train Epoch: 512 [11%] |
| 2023-04-06 19:18:00,183 44k INFO Losses: [2.0607686042785645, 2.637697458267212, 10.518184661865234, 21.01915168762207, 0.816366970539093], step: 9200, lr: 9.381185120195232e-05 |
| 2023-04-06 19:18:20,407 44k INFO ====> Epoch: 512, cost 34.15 s |
| 2023-04-06 19:18:53,534 44k INFO ====> Epoch: 513, cost 33.13 s |
| 2023-04-06 19:19:26,782 44k INFO ====> Epoch: 514, cost 33.25 s |
| 2023-04-06 19:20:00,171 44k INFO ====> Epoch: 515, cost 33.39 s |
| 2023-04-06 19:20:33,348 44k INFO ====> Epoch: 516, cost 33.18 s |
| 2023-04-06 19:21:06,156 44k INFO ====> Epoch: 517, cost 32.81 s |
| 2023-04-06 19:21:39,395 44k INFO ====> Epoch: 518, cost 33.24 s |
| 2023-04-06 19:22:12,433 44k INFO ====> Epoch: 519, cost 33.04 s |
| 2023-04-06 19:22:45,302 44k INFO ====> Epoch: 520, cost 32.87 s |
| 2023-04-06 19:23:18,505 44k INFO ====> Epoch: 521, cost 33.20 s |
| 2023-04-06 19:23:51,504 44k INFO ====> Epoch: 522, cost 33.00 s |
| 2023-04-06 19:24:08,483 44k INFO Train Epoch: 523 [22%] |
| 2023-04-06 19:24:08,485 44k INFO Losses: [2.554436206817627, 2.2548623085021973, 8.05234146118164, 19.58883285522461, 0.7690815329551697], step: 9400, lr: 9.368294049588446e-05 |
| 2023-04-06 19:24:26,224 44k INFO ====> Epoch: 523, cost 34.72 s |
| 2023-04-06 19:24:59,464 44k INFO ====> Epoch: 524, cost 33.24 s |
| 2023-04-06 19:25:32,713 44k INFO ====> Epoch: 525, cost 33.25 s |
| 2023-04-06 19:26:06,142 44k INFO ====> Epoch: 526, cost 33.43 s |
| 2023-04-06 19:26:39,477 44k INFO ====> Epoch: 527, cost 33.33 s |
| 2023-04-06 19:27:13,261 44k INFO ====> Epoch: 528, cost 33.78 s |
| 2023-04-06 19:27:47,514 44k INFO ====> Epoch: 529, cost 34.25 s |
| 2023-04-06 19:28:20,406 44k INFO ====> Epoch: 530, cost 32.89 s |
| 2023-04-06 19:28:52,926 44k INFO ====> Epoch: 531, cost 32.52 s |
| 2023-04-06 19:29:25,812 44k INFO ====> Epoch: 532, cost 32.89 s |
| 2023-04-06 19:29:58,497 44k INFO ====> Epoch: 533, cost 32.69 s |
| 2023-04-06 19:30:17,026 44k INFO Train Epoch: 534 [33%] |
| 2023-04-06 19:30:17,028 44k INFO Losses: [2.5109479427337646, 2.1757607460021973, 9.281797409057617, 17.23943328857422, 0.5131280422210693], step: 9600, lr: 9.355420693129632e-05 |
| 2023-04-06 19:30:25,640 44k INFO Saving model and optimizer state at iteration 534 to ./logs/44k/G_9600.pth |
| 2023-04-06 19:30:28,296 44k INFO Saving model and optimizer state at iteration 534 to ./logs/44k/D_9600.pth |
| 2023-04-06 19:30:30,428 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_7200.pth |
| 2023-04-06 19:30:30,446 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_7200.pth |
| 2023-04-06 19:30:44,380 44k INFO ====> Epoch: 534, cost 45.88 s |
| 2023-04-06 19:31:17,622 44k INFO ====> Epoch: 535, cost 33.24 s |
| 2023-04-06 19:31:50,594 44k INFO ====> Epoch: 536, cost 32.97 s |
| 2023-04-06 19:32:23,705 44k INFO ====> Epoch: 537, cost 33.11 s |
| 2023-04-06 19:32:56,923 44k INFO ====> Epoch: 538, cost 33.22 s |
| 2023-04-06 19:33:30,263 44k INFO ====> Epoch: 539, cost 33.34 s |
| 2023-04-06 19:34:03,426 44k INFO ====> Epoch: 540, cost 33.16 s |
| 2023-04-06 19:34:36,678 44k INFO ====> Epoch: 541, cost 33.25 s |
| 2023-04-06 19:35:10,191 44k INFO ====> Epoch: 542, cost 33.51 s |
| 2023-04-06 19:35:44,037 44k INFO ====> Epoch: 543, cost 33.85 s |
| 2023-04-06 19:36:16,782 44k INFO ====> Epoch: 544, cost 32.74 s |
| 2023-04-06 19:36:38,044 44k INFO Train Epoch: 545 [44%] |
| 2023-04-06 19:36:38,045 44k INFO Losses: [2.445751667022705, 2.247299909591675, 7.489833831787109, 16.84793472290039, 1.0898230075836182], step: 9800, lr: 9.342565026477056e-05 |
| 2023-04-06 19:36:51,067 44k INFO ====> Epoch: 545, cost 34.28 s |
| 2023-04-06 19:37:23,942 44k INFO ====> Epoch: 546, cost 32.88 s |
| 2023-04-06 19:37:56,989 44k INFO ====> Epoch: 547, cost 33.05 s |
| 2023-04-06 19:38:30,180 44k INFO ====> Epoch: 548, cost 33.19 s |
| 2023-04-06 19:39:03,442 44k INFO ====> Epoch: 549, cost 33.26 s |
| 2023-04-06 19:39:36,796 44k INFO ====> Epoch: 550, cost 33.35 s |
| 2023-04-06 19:40:09,944 44k INFO ====> Epoch: 551, cost 33.15 s |
| 2023-04-06 19:40:43,433 44k INFO ====> Epoch: 552, cost 33.49 s |
| 2023-04-06 19:41:16,969 44k INFO ====> Epoch: 553, cost 33.54 s |
| 2023-04-06 19:41:49,900 44k INFO ====> Epoch: 554, cost 32.93 s |
| 2023-04-06 19:42:23,093 44k INFO ====> Epoch: 555, cost 33.19 s |
| 2023-04-06 19:42:46,977 44k INFO Train Epoch: 556 [56%] |
| 2023-04-06 19:42:46,978 44k INFO Losses: [2.439335346221924, 2.1651785373687744, 8.26251220703125, 17.951847076416016, 0.6958508491516113], step: 10000, lr: 9.32972702532243e-05 |
| 2023-04-06 19:42:57,652 44k INFO ====> Epoch: 556, cost 34.56 s |
| 2023-04-06 19:43:31,192 44k INFO ====> Epoch: 557, cost 33.54 s |
| 2023-04-06 19:44:04,793 44k INFO ====> Epoch: 558, cost 33.60 s |
| 2023-04-06 19:44:38,080 44k INFO ====> Epoch: 559, cost 33.29 s |
| 2023-04-06 19:45:11,594 44k INFO ====> Epoch: 560, cost 33.51 s |
| 2023-04-06 19:45:44,852 44k INFO ====> Epoch: 561, cost 33.26 s |
| 2023-04-06 19:46:18,336 44k INFO ====> Epoch: 562, cost 33.48 s |
| 2023-04-06 19:46:52,163 44k INFO ====> Epoch: 563, cost 33.83 s |
| 2023-04-06 19:47:24,969 44k INFO ====> Epoch: 564, cost 32.81 s |
| 2023-04-06 19:47:57,678 44k INFO ====> Epoch: 565, cost 32.71 s |
| 2023-04-06 19:48:30,506 44k INFO ====> Epoch: 566, cost 32.83 s |
| 2023-04-06 19:48:57,147 44k INFO Train Epoch: 567 [67%] |
| 2023-04-06 19:48:57,148 44k INFO Losses: [2.339824676513672, 2.326724052429199, 7.90087890625, 18.2314395904541, 0.9818569421768188], step: 10200, lr: 9.316906665390869e-05 |
| 2023-04-06 19:49:05,821 44k INFO ====> Epoch: 567, cost 35.31 s |
| 2023-04-06 19:49:38,932 44k INFO ====> Epoch: 568, cost 33.11 s |
| 2023-04-06 19:50:12,102 44k INFO ====> Epoch: 569, cost 33.17 s |
| 2023-04-06 19:50:45,048 44k INFO ====> Epoch: 570, cost 32.95 s |
| 2023-04-06 19:51:17,974 44k INFO ====> Epoch: 571, cost 32.93 s |
| 2023-04-06 19:51:51,777 44k INFO ====> Epoch: 572, cost 33.80 s |
| 2023-04-06 19:52:24,545 44k INFO ====> Epoch: 573, cost 32.77 s |
| 2023-04-06 19:52:57,507 44k INFO ====> Epoch: 574, cost 32.96 s |
| 2023-04-06 19:53:30,310 44k INFO ====> Epoch: 575, cost 32.80 s |
| 2023-04-06 19:54:03,654 44k INFO ====> Epoch: 576, cost 33.34 s |
| 2023-04-06 19:54:37,066 44k INFO ====> Epoch: 577, cost 33.41 s |
| 2023-04-06 19:55:05,884 44k INFO Train Epoch: 578 [78%] |
| 2023-04-06 19:55:05,886 44k INFO Losses: [2.246778964996338, 2.5583128929138184, 11.40976333618164, 19.14501190185547, 0.7166233062744141], step: 10400, lr: 9.304103922440849e-05 |
| 2023-04-06 19:55:13,589 44k INFO Saving model and optimizer state at iteration 578 to ./logs/44k/G_10400.pth |
| 2023-04-06 19:55:16,183 44k INFO Saving model and optimizer state at iteration 578 to ./logs/44k/D_10400.pth |
| 2023-04-06 19:55:18,590 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_8000.pth |
| 2023-04-06 19:55:18,619 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_8000.pth |
| 2023-04-06 19:55:23,170 44k INFO ====> Epoch: 578, cost 46.10 s |
| 2023-04-06 19:55:56,235 44k INFO ====> Epoch: 579, cost 33.07 s |
| 2023-04-06 19:56:29,363 44k INFO ====> Epoch: 580, cost 33.13 s |
| 2023-04-06 19:57:02,635 44k INFO ====> Epoch: 581, cost 33.27 s |
| 2023-04-06 19:57:35,978 44k INFO ====> Epoch: 582, cost 33.34 s |
| 2023-04-06 19:58:09,070 44k INFO ====> Epoch: 583, cost 33.09 s |
| 2023-04-06 19:58:41,826 44k INFO ====> Epoch: 584, cost 32.76 s |
| 2023-04-06 19:59:14,828 44k INFO ====> Epoch: 585, cost 33.00 s |
| 2023-04-06 19:59:47,711 44k INFO ====> Epoch: 586, cost 32.88 s |
| 2023-04-06 20:00:20,712 44k INFO ====> Epoch: 587, cost 33.00 s |
| 2023-04-06 20:00:54,072 44k INFO ====> Epoch: 588, cost 33.36 s |
| 2023-04-06 20:01:24,857 44k INFO Train Epoch: 589 [89%] |
| 2023-04-06 20:01:24,858 44k INFO Losses: [2.548102378845215, 2.1721510887145996, 8.618353843688965, 16.937570571899414, 1.0304343700408936], step: 10600, lr: 9.291318772264153e-05 |
| 2023-04-06 20:01:28,575 44k INFO ====> Epoch: 589, cost 34.50 s |
| 2023-04-06 20:02:01,884 44k INFO ====> Epoch: 590, cost 33.31 s |
| 2023-04-06 20:02:35,471 44k INFO ====> Epoch: 591, cost 33.59 s |
| 2023-04-06 20:03:08,929 44k INFO ====> Epoch: 592, cost 33.46 s |
| 2023-04-06 20:03:42,055 44k INFO ====> Epoch: 593, cost 33.13 s |
| 2023-04-06 20:04:15,337 44k INFO ====> Epoch: 594, cost 33.28 s |
| 2023-04-06 20:04:48,504 44k INFO ====> Epoch: 595, cost 33.17 s |
| 2023-04-06 20:05:21,948 44k INFO ====> Epoch: 596, cost 33.44 s |
| 2023-04-06 20:05:55,381 44k INFO ====> Epoch: 597, cost 33.43 s |
| 2023-04-06 20:06:28,869 44k INFO ====> Epoch: 598, cost 33.49 s |
| 2023-04-06 20:07:01,659 44k INFO ====> Epoch: 599, cost 32.79 s |
| 2023-04-06 20:07:34,713 44k INFO ====> Epoch: 600, cost 33.05 s |
| 2023-04-06 20:07:46,226 44k INFO Train Epoch: 601 [0%] |
| 2023-04-06 20:07:46,228 44k INFO Losses: [2.4384801387786865, 2.300462484359741, 8.600714683532715, 19.138277053833008, 1.3364354372024536], step: 10800, lr: 9.277391371786995e-05 |
| 2023-04-06 20:08:09,338 44k INFO ====> Epoch: 601, cost 34.62 s |
| 2023-04-06 20:08:42,493 44k INFO ====> Epoch: 602, cost 33.16 s |
| 2023-04-06 20:09:15,870 44k INFO ====> Epoch: 603, cost 33.38 s |
| 2023-04-06 20:09:48,696 44k INFO ====> Epoch: 604, cost 32.83 s |
| 2023-04-06 20:10:21,782 44k INFO ====> Epoch: 605, cost 33.09 s |
| 2023-04-06 20:10:54,820 44k INFO ====> Epoch: 606, cost 33.04 s |
| 2023-04-06 20:11:27,910 44k INFO ====> Epoch: 607, cost 33.09 s |
| 2023-04-06 20:12:00,820 44k INFO ====> Epoch: 608, cost 32.91 s |
| 2023-04-06 20:12:33,803 44k INFO ====> Epoch: 609, cost 32.98 s |
| 2023-04-06 20:13:06,825 44k INFO ====> Epoch: 610, cost 33.02 s |
| 2023-04-06 20:13:39,997 44k INFO ====> Epoch: 611, cost 33.17 s |
| 2023-04-06 20:13:54,499 44k INFO Train Epoch: 612 [11%] |
| 2023-04-06 20:13:54,500 44k INFO Losses: [2.083113193511963, 2.611279010772705, 12.885259628295898, 19.54749870300293, 0.6030045747756958], step: 11000, lr: 9.264642928419956e-05 |
| 2023-04-06 20:14:14,614 44k INFO ====> Epoch: 612, cost 34.62 s |
| 2023-04-06 20:14:47,860 44k INFO ====> Epoch: 613, cost 33.25 s |
|
|
