Instructions to use azazen/train-dump with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use azazen/train-dump with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("azazen/train-dump", dtype="auto") - Notebooks
- Google Colab
- Kaggle
File size: 59,917 Bytes
c5850da | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 | 2023-04-06 14:21:46,233 44k INFO {'train': {'log_interval': 200, 'eval_interval': 800, 'seed': 1234, 'epochs': 10000, 'learning_rate': 0.0001, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 6, 'fp16_run': False, 'lr_decay': 0.999875, 'segment_size': 10240, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0, 'use_sr': True, 'max_speclen': 512, 'port': '8001', 'keep_ckpts': 3, 'all_in_mem': False}, 'data': {'training_files': 'filelists/train.txt', 'validation_files': 'filelists/val.txt', 'max_wav_value': 32768.0, 'sampling_rate': 44100, 'filter_length': 2048, 'hop_length': 512, 'win_length': 2048, 'n_mel_channels': 80, 'mel_fmin': 0.0, 'mel_fmax': 22050}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0.1, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [8, 8, 2, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4, 4], 'n_layers_q': 3, 'use_spectral_norm': False, 'gin_channels': 256, 'ssl_dim': 256, 'n_speakers': 2}, 'spk': {'ojou_speech': 0, 'ojou_sing': 1}, 'model_dir': './logs/44k'}
2023-04-06 14:21:53,913 44k INFO emb_g.weight is not in the checkpoint
2023-04-06 14:21:54,028 44k INFO Loaded checkpoint './logs/44k/G_0.pth' (iteration 0)
2023-04-06 14:21:54,937 44k INFO Loaded checkpoint './logs/44k/D_0.pth' (iteration 0)
2023-04-06 14:28:10,074 44k INFO {'train': {'log_interval': 200, 'eval_interval': 800, 'seed': 1234, 'epochs': 10000, 'learning_rate': 0.0001, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 6, 'fp16_run': False, 'lr_decay': 0.999875, 'segment_size': 10240, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0, 'use_sr': True, 'max_speclen': 512, 'port': '8001', 'keep_ckpts': 3, 'all_in_mem': False}, 'data': {'training_files': 'filelists/train.txt', 'validation_files': 'filelists/val.txt', 'max_wav_value': 32768.0, 'sampling_rate': 44100, 'filter_length': 2048, 'hop_length': 512, 'win_length': 2048, 'n_mel_channels': 80, 'mel_fmin': 0.0, 'mel_fmax': 22050}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0.1, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [8, 8, 2, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4, 4], 'n_layers_q': 3, 'use_spectral_norm': False, 'gin_channels': 256, 'ssl_dim': 256, 'n_speakers': 2}, 'spk': {'ojou_speech': 0, 'ojou_sing': 1}, 'model_dir': './logs/44k'}
2023-04-06 14:28:16,416 44k INFO emb_g.weight is not in the checkpoint
2023-04-06 14:28:16,520 44k INFO Loaded checkpoint './logs/44k/G_0.pth' (iteration 0)
2023-04-06 14:28:16,759 44k INFO Loaded checkpoint './logs/44k/D_0.pth' (iteration 0)
2023-04-06 14:28:44,170 44k INFO Train Epoch: 1 [0%]
2023-04-06 14:28:44,171 44k INFO Losses: [2.573824167251587, 2.517317295074463, 12.462947845458984, 39.484214782714844, 4.558784484863281], step: 0, lr: 0.0001
2023-04-06 14:29:02,964 44k INFO Saving model and optimizer state at iteration 1 to ./logs/44k/G_0.pth
2023-04-06 14:29:06,278 44k INFO Saving model and optimizer state at iteration 1 to ./logs/44k/D_0.pth
2023-04-06 14:29:36,249 44k INFO ====> Epoch: 1, cost 86.18 s
2023-04-06 14:30:10,389 44k INFO ====> Epoch: 2, cost 34.14 s
2023-04-06 14:30:44,375 44k INFO ====> Epoch: 3, cost 33.99 s
2023-04-06 14:31:17,769 44k INFO ====> Epoch: 4, cost 33.39 s
2023-04-06 14:31:51,295 44k INFO ====> Epoch: 5, cost 33.53 s
2023-04-06 14:32:24,800 44k INFO ====> Epoch: 6, cost 33.51 s
2023-04-06 14:32:58,115 44k INFO ====> Epoch: 7, cost 33.31 s
2023-04-06 14:33:31,700 44k INFO ====> Epoch: 8, cost 33.58 s
2023-04-06 14:34:05,014 44k INFO ====> Epoch: 9, cost 33.31 s
2023-04-06 14:34:38,581 44k INFO ====> Epoch: 10, cost 33.57 s
2023-04-06 14:35:12,124 44k INFO ====> Epoch: 11, cost 33.54 s
2023-04-06 14:35:26,942 44k INFO Train Epoch: 12 [11%]
2023-04-06 14:35:26,943 44k INFO Losses: [2.380369186401367, 2.374410629272461, 9.374342918395996, 25.8035888671875, 1.5873091220855713], step: 200, lr: 9.986258590528146e-05
2023-04-06 14:35:47,245 44k INFO ====> Epoch: 12, cost 35.12 s
2023-04-06 14:36:20,961 44k INFO ====> Epoch: 13, cost 33.72 s
2023-04-06 14:36:55,209 44k INFO ====> Epoch: 14, cost 34.25 s
2023-04-06 14:37:28,824 44k INFO ====> Epoch: 15, cost 33.62 s
2023-04-06 14:38:02,514 44k INFO ====> Epoch: 16, cost 33.69 s
2023-04-06 14:38:36,540 44k INFO ====> Epoch: 17, cost 34.03 s
2023-04-06 14:39:09,516 44k INFO ====> Epoch: 18, cost 32.98 s
2023-04-06 14:39:42,290 44k INFO ====> Epoch: 19, cost 32.77 s
2023-04-06 14:40:15,260 44k INFO ====> Epoch: 20, cost 32.97 s
2023-04-06 14:40:48,429 44k INFO ====> Epoch: 21, cost 33.17 s
2023-04-06 14:41:22,035 44k INFO ====> Epoch: 22, cost 33.61 s
2023-04-06 14:41:38,373 44k INFO Train Epoch: 23 [22%]
2023-04-06 14:41:38,374 44k INFO Losses: [2.5752055644989014, 2.3014121055603027, 8.904624938964844, 23.154579162597656, 1.0447863340377808], step: 400, lr: 9.972536063689719e-05
2023-04-06 14:41:56,747 44k INFO ====> Epoch: 23, cost 34.71 s
2023-04-06 14:42:30,058 44k INFO ====> Epoch: 24, cost 33.31 s
2023-04-06 14:43:03,038 44k INFO ====> Epoch: 25, cost 32.98 s
2023-04-06 14:43:36,283 44k INFO ====> Epoch: 26, cost 33.25 s
2023-04-06 14:44:09,581 44k INFO ====> Epoch: 27, cost 33.30 s
2023-04-06 14:44:43,372 44k INFO ====> Epoch: 28, cost 33.79 s
2023-04-06 14:45:16,897 44k INFO ====> Epoch: 29, cost 33.53 s
2023-04-06 14:45:50,410 44k INFO ====> Epoch: 30, cost 33.51 s
2023-04-06 14:46:24,140 44k INFO ====> Epoch: 31, cost 33.73 s
2023-04-06 14:46:58,284 44k INFO ====> Epoch: 32, cost 34.14 s
2023-04-06 14:47:31,802 44k INFO ====> Epoch: 33, cost 33.52 s
2023-04-06 14:47:50,868 44k INFO Train Epoch: 34 [33%]
2023-04-06 14:47:50,869 44k INFO Losses: [2.0391736030578613, 2.6325862407684326, 12.453435897827148, 22.571475982666016, 1.300460696220398], step: 600, lr: 9.95883239353732e-05
2023-04-06 14:48:06,495 44k INFO ====> Epoch: 34, cost 34.69 s
2023-04-06 14:48:40,108 44k INFO ====> Epoch: 35, cost 33.61 s
2023-04-06 14:49:13,550 44k INFO ====> Epoch: 36, cost 33.44 s
2023-04-06 14:49:47,466 44k INFO ====> Epoch: 37, cost 33.92 s
2023-04-06 14:50:20,905 44k INFO ====> Epoch: 38, cost 33.44 s
2023-04-06 14:50:54,439 44k INFO ====> Epoch: 39, cost 33.53 s
2023-04-06 14:51:28,025 44k INFO ====> Epoch: 40, cost 33.59 s
2023-04-06 14:52:01,642 44k INFO ====> Epoch: 41, cost 33.62 s
2023-04-06 14:52:35,218 44k INFO ====> Epoch: 42, cost 33.58 s
2023-04-06 14:53:08,561 44k INFO ====> Epoch: 43, cost 33.34 s
2023-04-06 14:53:42,989 44k INFO ====> Epoch: 44, cost 34.43 s
2023-04-06 14:54:05,177 44k INFO Train Epoch: 45 [44%]
2023-04-06 14:54:05,178 44k INFO Losses: [2.806058168411255, 2.137558698654175, 5.308585166931152, 20.471132278442383, 1.025601863861084], step: 800, lr: 9.945147554159202e-05
2023-04-06 14:54:13,630 44k INFO Saving model and optimizer state at iteration 45 to ./logs/44k/G_800.pth
2023-04-06 14:54:16,110 44k INFO Saving model and optimizer state at iteration 45 to ./logs/44k/D_800.pth
2023-04-06 14:54:30,153 44k INFO ====> Epoch: 45, cost 47.16 s
2023-04-06 14:55:03,935 44k INFO ====> Epoch: 46, cost 33.78 s
2023-04-06 14:55:37,393 44k INFO ====> Epoch: 47, cost 33.46 s
2023-04-06 14:56:13,219 44k INFO ====> Epoch: 48, cost 35.83 s
2023-04-06 14:56:49,538 44k INFO ====> Epoch: 49, cost 36.32 s
2023-04-06 14:57:23,880 44k INFO ====> Epoch: 50, cost 34.34 s
2023-04-06 14:57:58,045 44k INFO ====> Epoch: 51, cost 34.16 s
2023-04-06 14:58:33,841 44k INFO ====> Epoch: 52, cost 35.80 s
2023-04-06 14:59:08,197 44k INFO ====> Epoch: 53, cost 34.36 s
2023-04-06 14:59:42,422 44k INFO ====> Epoch: 54, cost 34.23 s
2023-04-06 15:00:16,288 44k INFO ====> Epoch: 55, cost 33.87 s
2023-04-06 15:00:40,883 44k INFO Train Epoch: 56 [56%]
2023-04-06 15:00:40,884 44k INFO Losses: [2.6365833282470703, 2.064959764480591, 5.582603931427002, 19.144601821899414, 0.8247841000556946], step: 1000, lr: 9.931481519679228e-05
2023-04-06 15:00:51,516 44k INFO ====> Epoch: 56, cost 35.23 s
2023-04-06 15:01:24,869 44k INFO ====> Epoch: 57, cost 33.35 s
2023-04-06 15:01:58,466 44k INFO ====> Epoch: 58, cost 33.60 s
2023-04-06 15:02:32,055 44k INFO ====> Epoch: 59, cost 33.59 s
2023-04-06 15:03:05,621 44k INFO ====> Epoch: 60, cost 33.57 s
2023-04-06 15:03:39,247 44k INFO ====> Epoch: 61, cost 33.63 s
2023-04-06 15:04:12,988 44k INFO ====> Epoch: 62, cost 33.74 s
2023-04-06 15:04:46,696 44k INFO ====> Epoch: 63, cost 33.71 s
2023-04-06 15:05:20,560 44k INFO ====> Epoch: 64, cost 33.86 s
2023-04-06 15:05:54,100 44k INFO ====> Epoch: 65, cost 33.54 s
2023-04-06 15:06:28,189 44k INFO ====> Epoch: 66, cost 34.09 s
2023-04-06 15:06:54,565 44k INFO Train Epoch: 67 [67%]
2023-04-06 15:06:54,572 44k INFO Losses: [2.5330240726470947, 2.2095956802368164, 9.094779014587402, 21.0531005859375, 1.01519775390625], step: 1200, lr: 9.917834264256819e-05
2023-04-06 15:07:02,883 44k INFO ====> Epoch: 67, cost 34.69 s
2023-04-06 15:07:35,938 44k INFO ====> Epoch: 68, cost 33.06 s
2023-04-06 15:08:09,039 44k INFO ====> Epoch: 69, cost 33.10 s
2023-04-06 15:08:42,455 44k INFO ====> Epoch: 70, cost 33.42 s
2023-04-06 15:09:15,650 44k INFO ====> Epoch: 71, cost 33.19 s
2023-04-06 15:09:49,134 44k INFO ====> Epoch: 72, cost 33.48 s
2023-04-06 15:10:22,449 44k INFO ====> Epoch: 73, cost 33.31 s
2023-04-06 15:10:55,919 44k INFO ====> Epoch: 74, cost 33.47 s
2023-04-06 15:11:29,262 44k INFO ====> Epoch: 75, cost 33.34 s
2023-04-06 15:12:02,619 44k INFO ====> Epoch: 76, cost 33.36 s
2023-04-06 15:12:35,887 44k INFO ====> Epoch: 77, cost 33.27 s
2023-04-06 15:13:05,328 44k INFO Train Epoch: 78 [78%]
2023-04-06 15:13:05,330 44k INFO Losses: [2.4559853076934814, 2.5709311962127686, 8.139681816101074, 22.049203872680664, 1.2676440477371216], step: 1400, lr: 9.904205762086905e-05
2023-04-06 15:13:11,413 44k INFO ====> Epoch: 78, cost 35.53 s
2023-04-06 15:13:44,951 44k INFO ====> Epoch: 79, cost 33.54 s
2023-04-06 15:14:18,839 44k INFO ====> Epoch: 80, cost 33.89 s
2023-04-06 15:14:52,995 44k INFO ====> Epoch: 81, cost 34.16 s
2023-04-06 15:15:25,852 44k INFO ====> Epoch: 82, cost 32.86 s
2023-04-06 15:15:58,487 44k INFO ====> Epoch: 83, cost 32.63 s
2023-04-06 15:16:31,709 44k INFO ====> Epoch: 84, cost 33.22 s
2023-04-06 15:17:04,874 44k INFO ====> Epoch: 85, cost 33.16 s
2023-04-06 15:17:37,914 44k INFO ====> Epoch: 86, cost 33.04 s
2023-04-06 15:18:11,152 44k INFO ====> Epoch: 87, cost 33.24 s
2023-04-06 15:18:44,387 44k INFO ====> Epoch: 88, cost 33.24 s
2023-04-06 15:19:15,242 44k INFO Train Epoch: 89 [89%]
2023-04-06 15:19:15,243 44k INFO Losses: [2.471696376800537, 2.2750179767608643, 7.19001579284668, 20.06005096435547, 1.0517364740371704], step: 1600, lr: 9.89059598739987e-05
2023-04-06 15:19:23,190 44k INFO Saving model and optimizer state at iteration 89 to ./logs/44k/G_1600.pth
2023-04-06 15:19:25,938 44k INFO Saving model and optimizer state at iteration 89 to ./logs/44k/D_1600.pth
2023-04-06 15:19:30,586 44k INFO ====> Epoch: 89, cost 46.20 s
2023-04-06 15:20:04,206 44k INFO ====> Epoch: 90, cost 33.62 s
2023-04-06 15:20:37,768 44k INFO ====> Epoch: 91, cost 33.56 s
2023-04-06 15:21:11,633 44k INFO ====> Epoch: 92, cost 33.87 s
2023-04-06 15:21:44,606 44k INFO ====> Epoch: 93, cost 32.97 s
2023-04-06 15:22:17,609 44k INFO ====> Epoch: 94, cost 33.00 s
2023-04-06 15:22:50,786 44k INFO ====> Epoch: 95, cost 33.18 s
2023-04-06 15:23:24,245 44k INFO ====> Epoch: 96, cost 33.46 s
2023-04-06 15:23:57,352 44k INFO ====> Epoch: 97, cost 33.11 s
2023-04-06 15:24:30,780 44k INFO ====> Epoch: 98, cost 33.43 s
2023-04-06 15:25:04,040 44k INFO ====> Epoch: 99, cost 33.26 s
2023-04-06 15:25:37,206 44k INFO ====> Epoch: 100, cost 33.17 s
2023-04-06 15:25:48,955 44k INFO Train Epoch: 101 [0%]
2023-04-06 15:25:48,957 44k INFO Losses: [2.5259549617767334, 2.311516046524048, 11.238818168640137, 24.39948081970215, 1.0041320323944092], step: 1800, lr: 9.875770288847208e-05
2023-04-06 15:26:12,089 44k INFO ====> Epoch: 101, cost 34.88 s
2023-04-06 15:26:45,466 44k INFO ====> Epoch: 102, cost 33.38 s
2023-04-06 15:27:19,094 44k INFO ====> Epoch: 103, cost 33.63 s
2023-04-06 15:27:52,909 44k INFO ====> Epoch: 104, cost 33.82 s
2023-04-06 15:28:26,521 44k INFO ====> Epoch: 105, cost 33.61 s
2023-04-06 15:28:59,968 44k INFO ====> Epoch: 106, cost 33.45 s
2023-04-06 15:29:33,425 44k INFO ====> Epoch: 107, cost 33.46 s
2023-04-06 15:30:07,804 44k INFO ====> Epoch: 108, cost 34.38 s
2023-04-06 15:30:40,689 44k INFO ====> Epoch: 109, cost 32.88 s
2023-04-06 15:31:13,687 44k INFO ====> Epoch: 110, cost 33.00 s
2023-04-06 15:31:46,516 44k INFO ====> Epoch: 111, cost 32.83 s
2023-04-06 15:32:00,629 44k INFO Train Epoch: 112 [11%]
2023-04-06 15:32:00,630 44k INFO Losses: [2.1719627380371094, 2.386096477508545, 12.244598388671875, 26.043445587158203, 0.8842934370040894], step: 2000, lr: 9.862199588508305e-05
2023-04-06 15:32:21,530 44k INFO ====> Epoch: 112, cost 35.01 s
2023-04-06 15:32:54,826 44k INFO ====> Epoch: 113, cost 33.30 s
2023-04-06 15:33:28,379 44k INFO ====> Epoch: 114, cost 33.55 s
2023-04-06 15:34:01,717 44k INFO ====> Epoch: 115, cost 33.34 s
2023-04-06 15:34:35,036 44k INFO ====> Epoch: 116, cost 33.32 s
2023-04-06 15:35:08,484 44k INFO ====> Epoch: 117, cost 33.45 s
2023-04-06 15:35:41,980 44k INFO ====> Epoch: 118, cost 33.50 s
2023-04-06 15:36:15,514 44k INFO ====> Epoch: 119, cost 33.53 s
2023-04-06 15:36:49,214 44k INFO ====> Epoch: 120, cost 33.70 s
2023-04-06 15:37:23,093 44k INFO ====> Epoch: 121, cost 33.88 s
2023-04-06 15:37:56,873 44k INFO ====> Epoch: 122, cost 33.78 s
2023-04-06 15:38:14,003 44k INFO Train Epoch: 123 [22%]
2023-04-06 15:38:14,004 44k INFO Losses: [2.781219482421875, 2.1242191791534424, 6.261739253997803, 19.073083877563477, 0.9984504580497742], step: 2200, lr: 9.848647536224416e-05
2023-04-06 15:38:32,518 44k INFO ====> Epoch: 123, cost 35.65 s
2023-04-06 15:39:05,962 44k INFO ====> Epoch: 124, cost 33.44 s
2023-04-06 15:39:39,114 44k INFO ====> Epoch: 125, cost 33.15 s
2023-04-06 15:40:13,048 44k INFO ====> Epoch: 126, cost 33.93 s
2023-04-06 15:40:46,649 44k INFO ====> Epoch: 127, cost 33.60 s
2023-04-06 15:41:20,247 44k INFO ====> Epoch: 128, cost 33.60 s
2023-04-06 15:41:53,939 44k INFO ====> Epoch: 129, cost 33.69 s
2023-04-06 15:42:27,318 44k INFO ====> Epoch: 130, cost 33.38 s
2023-04-06 15:43:00,635 44k INFO ====> Epoch: 131, cost 33.32 s
2023-04-06 15:43:34,280 44k INFO ====> Epoch: 132, cost 33.64 s
2023-04-06 15:44:07,894 44k INFO ====> Epoch: 133, cost 33.61 s
2023-04-06 15:44:27,373 44k INFO Train Epoch: 134 [33%]
2023-04-06 15:44:27,374 44k INFO Losses: [2.6103241443634033, 2.176192045211792, 7.58054780960083, 21.09679412841797, 0.8060694336891174], step: 2400, lr: 9.835114106370493e-05
2023-04-06 15:44:35,774 44k INFO Saving model and optimizer state at iteration 134 to ./logs/44k/G_2400.pth
2023-04-06 15:44:38,339 44k INFO Saving model and optimizer state at iteration 134 to ./logs/44k/D_2400.pth
2023-04-06 15:44:54,914 44k INFO ====> Epoch: 134, cost 47.02 s
2023-04-06 15:45:28,270 44k INFO ====> Epoch: 135, cost 33.36 s
2023-04-06 15:46:01,378 44k INFO ====> Epoch: 136, cost 33.11 s
2023-04-06 15:46:34,552 44k INFO ====> Epoch: 137, cost 33.17 s
2023-04-06 15:47:07,890 44k INFO ====> Epoch: 138, cost 33.34 s
2023-04-06 15:47:40,920 44k INFO ====> Epoch: 139, cost 33.03 s
2023-04-06 15:48:14,268 44k INFO ====> Epoch: 140, cost 33.35 s
2023-04-06 15:48:47,542 44k INFO ====> Epoch: 141, cost 33.27 s
2023-04-06 15:49:21,476 44k INFO ====> Epoch: 142, cost 33.93 s
2023-04-06 15:49:55,539 44k INFO ====> Epoch: 143, cost 34.06 s
2023-04-06 15:50:28,796 44k INFO ====> Epoch: 144, cost 33.26 s
2023-04-06 15:50:50,534 44k INFO Train Epoch: 145 [44%]
2023-04-06 15:50:50,535 44k INFO Losses: [2.589505434036255, 2.020362377166748, 7.501511096954346, 20.353748321533203, 1.148757815361023], step: 2600, lr: 9.821599273356685e-05
2023-04-06 15:51:03,447 44k INFO ====> Epoch: 145, cost 34.65 s
2023-04-06 15:51:37,525 44k INFO ====> Epoch: 146, cost 34.08 s
2023-04-06 15:52:11,490 44k INFO ====> Epoch: 147, cost 33.97 s
2023-04-06 15:52:44,558 44k INFO ====> Epoch: 148, cost 33.07 s
2023-04-06 15:53:17,472 44k INFO ====> Epoch: 149, cost 32.91 s
2023-04-06 15:53:50,395 44k INFO ====> Epoch: 150, cost 32.92 s
2023-04-06 15:54:23,621 44k INFO ====> Epoch: 151, cost 33.23 s
2023-04-06 15:54:56,921 44k INFO ====> Epoch: 152, cost 33.30 s
2023-04-06 15:55:30,097 44k INFO ====> Epoch: 153, cost 33.18 s
2023-04-06 15:56:03,351 44k INFO ====> Epoch: 154, cost 33.25 s
2023-04-06 15:56:36,675 44k INFO ====> Epoch: 155, cost 33.32 s
2023-04-06 15:57:01,017 44k INFO Train Epoch: 156 [56%]
2023-04-06 15:57:01,018 44k INFO Losses: [2.5215423107147217, 2.311556339263916, 8.819522857666016, 20.068315505981445, 0.7969138622283936], step: 2800, lr: 9.808103011628319e-05
2023-04-06 15:57:11,740 44k INFO ====> Epoch: 156, cost 35.06 s
2023-04-06 15:57:44,760 44k INFO ====> Epoch: 157, cost 33.02 s
2023-04-06 15:58:18,955 44k INFO ====> Epoch: 158, cost 34.20 s
2023-04-06 15:58:52,525 44k INFO ====> Epoch: 159, cost 33.57 s
2023-04-06 15:59:26,604 44k INFO ====> Epoch: 160, cost 34.08 s
2023-04-06 16:00:00,222 44k INFO ====> Epoch: 161, cost 33.62 s
2023-04-06 16:00:35,009 44k INFO ====> Epoch: 162, cost 34.79 s
2023-04-06 16:01:08,369 44k INFO ====> Epoch: 163, cost 33.36 s
2023-04-06 16:01:41,802 44k INFO ====> Epoch: 164, cost 33.43 s
2023-04-06 16:02:15,766 44k INFO ====> Epoch: 165, cost 33.96 s
2023-04-06 16:02:49,461 44k INFO ====> Epoch: 166, cost 33.70 s
2023-04-06 16:03:16,473 44k INFO Train Epoch: 167 [67%]
2023-04-06 16:03:16,474 44k INFO Losses: [2.7111127376556396, 2.1096386909484863, 7.442945957183838, 19.06759262084961, 0.842487096786499], step: 3000, lr: 9.794625295665828e-05
2023-04-06 16:03:24,941 44k INFO ====> Epoch: 167, cost 35.48 s
2023-04-06 16:03:58,458 44k INFO ====> Epoch: 168, cost 33.52 s
2023-04-06 16:04:32,090 44k INFO ====> Epoch: 169, cost 33.63 s
2023-04-06 16:05:05,688 44k INFO ====> Epoch: 170, cost 33.60 s
2023-04-06 16:05:38,792 44k INFO ====> Epoch: 171, cost 33.10 s
2023-04-06 16:06:12,165 44k INFO ====> Epoch: 172, cost 33.37 s
2023-04-06 16:06:45,433 44k INFO ====> Epoch: 173, cost 33.27 s
2023-04-06 16:07:18,826 44k INFO ====> Epoch: 174, cost 33.39 s
2023-04-06 16:07:52,391 44k INFO ====> Epoch: 175, cost 33.57 s
2023-04-06 16:08:25,643 44k INFO ====> Epoch: 176, cost 33.25 s
2023-04-06 16:08:59,249 44k INFO ====> Epoch: 177, cost 33.61 s
2023-04-06 16:09:28,917 44k INFO Train Epoch: 178 [78%]
2023-04-06 16:09:28,918 44k INFO Losses: [2.699871063232422, 2.6492371559143066, 10.786850929260254, 21.075050354003906, 0.9040826559066772], step: 3200, lr: 9.781166099984716e-05
2023-04-06 16:09:37,252 44k INFO Saving model and optimizer state at iteration 178 to ./logs/44k/G_3200.pth
2023-04-06 16:09:40,424 44k INFO Saving model and optimizer state at iteration 178 to ./logs/44k/D_3200.pth
2023-04-06 16:09:42,678 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_800.pth
2023-04-06 16:09:42,697 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_800.pth
2023-04-06 16:09:47,370 44k INFO ====> Epoch: 178, cost 48.12 s
2023-04-06 16:10:22,255 44k INFO ====> Epoch: 179, cost 34.88 s
2023-04-06 16:10:56,399 44k INFO ====> Epoch: 180, cost 34.14 s
2023-04-06 16:11:29,952 44k INFO ====> Epoch: 181, cost 33.55 s
2023-04-06 16:12:03,708 44k INFO ====> Epoch: 182, cost 33.76 s
2023-04-06 16:12:37,128 44k INFO ====> Epoch: 183, cost 33.42 s
2023-04-06 16:13:10,864 44k INFO ====> Epoch: 184, cost 33.74 s
2023-04-06 16:13:44,783 44k INFO ====> Epoch: 185, cost 33.92 s
2023-04-06 16:14:18,088 44k INFO ====> Epoch: 186, cost 33.30 s
2023-04-06 16:14:51,126 44k INFO ====> Epoch: 187, cost 33.04 s
2023-04-06 16:15:23,846 44k INFO ====> Epoch: 188, cost 32.72 s
2023-04-06 16:15:54,766 44k INFO Train Epoch: 189 [89%]
2023-04-06 16:15:54,767 44k INFO Losses: [2.4710679054260254, 2.1371006965637207, 8.654294967651367, 19.12156105041504, 0.9505603313446045], step: 3400, lr: 9.767725399135504e-05
2023-04-06 16:15:58,544 44k INFO ====> Epoch: 189, cost 34.70 s
2023-04-06 16:16:31,701 44k INFO ====> Epoch: 190, cost 33.16 s
2023-04-06 16:17:05,161 44k INFO ====> Epoch: 191, cost 33.46 s
2023-04-06 16:17:38,244 44k INFO ====> Epoch: 192, cost 33.08 s
2023-04-06 16:18:11,457 44k INFO ====> Epoch: 193, cost 33.21 s
2023-04-06 16:18:44,985 44k INFO ====> Epoch: 194, cost 33.53 s
2023-04-06 16:19:18,075 44k INFO ====> Epoch: 195, cost 33.09 s
2023-04-06 16:19:51,298 44k INFO ====> Epoch: 196, cost 33.22 s
2023-04-06 16:20:24,415 44k INFO ====> Epoch: 197, cost 33.12 s
2023-04-06 16:20:58,269 44k INFO ====> Epoch: 198, cost 33.85 s
2023-04-06 16:21:31,916 44k INFO ====> Epoch: 199, cost 33.65 s
2023-04-06 16:22:05,610 44k INFO ====> Epoch: 200, cost 33.69 s
2023-04-06 16:22:18,432 44k INFO Train Epoch: 201 [0%]
2023-04-06 16:22:18,433 44k INFO Losses: [2.5765528678894043, 2.317922592163086, 9.504670143127441, 20.353900909423828, 1.0593161582946777], step: 3600, lr: 9.753083879807726e-05
2023-04-06 16:22:41,495 44k INFO ====> Epoch: 201, cost 35.88 s
2023-04-06 16:23:15,048 44k INFO ====> Epoch: 202, cost 33.55 s
2023-04-06 16:23:47,482 44k INFO ====> Epoch: 203, cost 32.43 s
2023-04-06 16:24:21,199 44k INFO ====> Epoch: 204, cost 33.72 s
2023-04-06 16:24:54,462 44k INFO ====> Epoch: 205, cost 33.26 s
2023-04-06 16:25:28,278 44k INFO ====> Epoch: 206, cost 33.82 s
2023-04-06 16:26:02,137 44k INFO ====> Epoch: 207, cost 33.86 s
2023-04-06 16:26:36,011 44k INFO ====> Epoch: 208, cost 33.87 s
2023-04-06 16:27:09,036 44k INFO ====> Epoch: 209, cost 33.03 s
2023-04-06 16:27:42,014 44k INFO ====> Epoch: 210, cost 32.98 s
2023-04-06 16:28:15,042 44k INFO ====> Epoch: 211, cost 33.03 s
2023-04-06 16:28:29,366 44k INFO Train Epoch: 212 [11%]
2023-04-06 16:28:29,367 44k INFO Losses: [2.4090394973754883, 2.35317063331604, 6.940808296203613, 20.978504180908203, 0.857198178768158], step: 3800, lr: 9.739681767887146e-05
2023-04-06 16:28:49,455 44k INFO ====> Epoch: 212, cost 34.41 s
2023-04-06 16:29:23,060 44k INFO ====> Epoch: 213, cost 33.60 s
2023-04-06 16:29:55,832 44k INFO ====> Epoch: 214, cost 32.77 s
2023-04-06 16:30:28,804 44k INFO ====> Epoch: 215, cost 32.97 s
2023-04-06 16:31:02,420 44k INFO ====> Epoch: 216, cost 33.62 s
2023-04-06 16:31:35,997 44k INFO ====> Epoch: 217, cost 33.58 s
2023-04-06 16:32:10,810 44k INFO ====> Epoch: 218, cost 34.81 s
2023-04-06 16:32:44,018 44k INFO ====> Epoch: 219, cost 33.21 s
2023-04-06 16:33:17,237 44k INFO ====> Epoch: 220, cost 33.22 s
2023-04-06 16:33:50,289 44k INFO ====> Epoch: 221, cost 33.05 s
2023-04-06 16:34:23,540 44k INFO ====> Epoch: 222, cost 33.25 s
2023-04-06 16:34:40,302 44k INFO Train Epoch: 223 [22%]
2023-04-06 16:34:40,303 44k INFO Losses: [2.5694820880889893, 2.331211805343628, 10.764482498168945, 20.099136352539062, 0.8609465956687927], step: 4000, lr: 9.726298072357337e-05
2023-04-06 16:34:48,444 44k INFO Saving model and optimizer state at iteration 223 to ./logs/44k/G_4000.pth
2023-04-06 16:34:51,453 44k INFO Saving model and optimizer state at iteration 223 to ./logs/44k/D_4000.pth
2023-04-06 16:34:53,922 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_1600.pth
2023-04-06 16:34:53,960 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_1600.pth
2023-04-06 16:35:10,618 44k INFO ====> Epoch: 223, cost 47.08 s
2023-04-06 16:35:44,170 44k INFO ====> Epoch: 224, cost 33.55 s
2023-04-06 16:36:17,884 44k INFO ====> Epoch: 225, cost 33.71 s
2023-04-06 16:36:51,720 44k INFO ====> Epoch: 226, cost 33.84 s
2023-04-06 16:37:25,079 44k INFO ====> Epoch: 227, cost 33.36 s
2023-04-06 16:37:58,857 44k INFO ====> Epoch: 228, cost 33.78 s
2023-04-06 16:38:33,315 44k INFO ====> Epoch: 229, cost 34.46 s
2023-04-06 16:39:06,279 44k INFO ====> Epoch: 230, cost 32.96 s
2023-04-06 16:39:39,579 44k INFO ====> Epoch: 231, cost 33.30 s
2023-04-06 16:40:13,317 44k INFO ====> Epoch: 232, cost 33.74 s
2023-04-06 16:40:46,510 44k INFO ====> Epoch: 233, cost 33.19 s
2023-04-06 16:41:05,693 44k INFO Train Epoch: 234 [33%]
2023-04-06 16:41:05,694 44k INFO Losses: [2.280306100845337, 2.313676595687866, 12.137205123901367, 21.159605026245117, 0.682966947555542], step: 4200, lr: 9.71293276791158e-05
2023-04-06 16:41:21,619 44k INFO ====> Epoch: 234, cost 35.11 s
2023-04-06 16:41:54,856 44k INFO ====> Epoch: 235, cost 33.24 s
2023-04-06 16:42:28,116 44k INFO ====> Epoch: 236, cost 33.26 s
2023-04-06 16:43:01,765 44k INFO ====> Epoch: 237, cost 33.65 s
2023-04-06 16:43:35,630 44k INFO ====> Epoch: 238, cost 33.86 s
2023-04-06 16:44:09,514 44k INFO ====> Epoch: 239, cost 33.88 s
2023-04-06 16:44:43,136 44k INFO ====> Epoch: 240, cost 33.62 s
2023-04-06 16:45:17,116 44k INFO ====> Epoch: 241, cost 33.98 s
2023-04-06 16:45:51,264 44k INFO ====> Epoch: 242, cost 34.15 s
2023-04-06 16:46:25,287 44k INFO ====> Epoch: 243, cost 34.02 s
2023-04-06 16:46:58,858 44k INFO ====> Epoch: 244, cost 33.57 s
2023-04-06 16:47:21,003 44k INFO Train Epoch: 245 [44%]
2023-04-06 16:47:21,004 44k INFO Losses: [2.3801774978637695, 2.1010258197784424, 9.494706153869629, 17.849163055419922, 0.9957065582275391], step: 4400, lr: 9.699585829277933e-05
2023-04-06 16:47:34,246 44k INFO ====> Epoch: 245, cost 35.39 s
2023-04-06 16:48:07,426 44k INFO ====> Epoch: 246, cost 33.18 s
2023-04-06 16:48:40,848 44k INFO ====> Epoch: 247, cost 33.42 s
2023-04-06 16:49:14,451 44k INFO ====> Epoch: 248, cost 33.60 s
2023-04-06 16:49:47,899 44k INFO ====> Epoch: 249, cost 33.45 s
2023-04-06 16:50:21,514 44k INFO ====> Epoch: 250, cost 33.61 s
2023-04-06 16:50:54,961 44k INFO ====> Epoch: 251, cost 33.45 s
2023-04-06 16:51:28,381 44k INFO ====> Epoch: 252, cost 33.42 s
2023-04-06 16:52:01,907 44k INFO ====> Epoch: 253, cost 33.53 s
2023-04-06 16:52:35,442 44k INFO ====> Epoch: 254, cost 33.53 s
2023-04-06 16:53:09,396 44k INFO ====> Epoch: 255, cost 33.95 s
2023-04-06 16:53:34,616 44k INFO Train Epoch: 256 [56%]
2023-04-06 16:53:34,617 44k INFO Losses: [2.5316028594970703, 2.0892834663391113, 10.589302062988281, 18.844057083129883, 0.5367166996002197], step: 4600, lr: 9.68625723121918e-05
2023-04-06 16:53:45,389 44k INFO ====> Epoch: 256, cost 35.99 s
2023-04-06 16:54:19,338 44k INFO ====> Epoch: 257, cost 33.95 s
2023-04-06 16:54:53,131 44k INFO ====> Epoch: 258, cost 33.79 s
2023-04-06 16:55:26,597 44k INFO ====> Epoch: 259, cost 33.47 s
2023-04-06 16:56:00,065 44k INFO ====> Epoch: 260, cost 33.47 s
2023-04-06 16:56:33,430 44k INFO ====> Epoch: 261, cost 33.37 s
2023-04-06 16:57:06,610 44k INFO ====> Epoch: 262, cost 33.18 s
2023-04-06 16:57:39,917 44k INFO ====> Epoch: 263, cost 33.31 s
2023-04-06 16:58:13,010 44k INFO ====> Epoch: 264, cost 33.09 s
2023-04-06 16:58:46,623 44k INFO ====> Epoch: 265, cost 33.61 s
2023-04-06 16:59:20,183 44k INFO ====> Epoch: 266, cost 33.56 s
2023-04-06 16:59:46,793 44k INFO Train Epoch: 267 [67%]
2023-04-06 16:59:46,794 44k INFO Losses: [2.5026473999023438, 2.2426648139953613, 7.876044273376465, 19.031328201293945, 1.068752408027649], step: 4800, lr: 9.67294694853279e-05
2023-04-06 16:59:54,870 44k INFO Saving model and optimizer state at iteration 267 to ./logs/44k/G_4800.pth
2023-04-06 16:59:57,509 44k INFO Saving model and optimizer state at iteration 267 to ./logs/44k/D_4800.pth
2023-04-06 16:59:59,774 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_2400.pth
2023-04-06 16:59:59,792 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_2400.pth
2023-04-06 17:00:06,567 44k INFO ====> Epoch: 267, cost 46.38 s
2023-04-06 17:00:41,439 44k INFO ====> Epoch: 268, cost 34.87 s
2023-04-06 17:01:14,983 44k INFO ====> Epoch: 269, cost 33.54 s
2023-04-06 17:01:48,354 44k INFO ====> Epoch: 270, cost 33.37 s
2023-04-06 17:02:22,124 44k INFO ====> Epoch: 271, cost 33.77 s
2023-04-06 17:02:55,709 44k INFO ====> Epoch: 272, cost 33.59 s
2023-04-06 17:03:29,304 44k INFO ====> Epoch: 273, cost 33.59 s
2023-04-06 17:04:03,052 44k INFO ====> Epoch: 274, cost 33.75 s
2023-04-06 17:04:36,755 44k INFO ====> Epoch: 275, cost 33.70 s
2023-04-06 17:05:10,636 44k INFO ====> Epoch: 276, cost 33.88 s
2023-04-06 17:05:44,147 44k INFO ====> Epoch: 277, cost 33.51 s
2023-04-06 17:06:13,645 44k INFO Train Epoch: 278 [78%]
2023-04-06 17:06:13,646 44k INFO Losses: [2.336780309677124, 2.2390074729919434, 11.930177688598633, 20.725872039794922, 0.5927470326423645], step: 5000, lr: 9.659654956050859e-05
2023-04-06 17:06:19,541 44k INFO ====> Epoch: 278, cost 35.39 s
2023-04-06 17:06:54,250 44k INFO ====> Epoch: 279, cost 34.71 s
2023-04-06 17:07:28,218 44k INFO ====> Epoch: 280, cost 33.97 s
2023-04-06 17:08:02,618 44k INFO ====> Epoch: 281, cost 34.40 s
2023-04-06 17:08:36,031 44k INFO ====> Epoch: 282, cost 33.41 s
2023-04-06 17:09:09,577 44k INFO ====> Epoch: 283, cost 33.55 s
2023-04-06 17:09:43,485 44k INFO ====> Epoch: 284, cost 33.91 s
2023-04-06 17:10:17,361 44k INFO ====> Epoch: 285, cost 33.88 s
2023-04-06 17:10:50,800 44k INFO ====> Epoch: 286, cost 33.44 s
2023-04-06 17:11:24,329 44k INFO ====> Epoch: 287, cost 33.53 s
2023-04-06 17:11:57,541 44k INFO ====> Epoch: 288, cost 33.21 s
2023-04-06 17:12:28,773 44k INFO Train Epoch: 289 [89%]
2023-04-06 17:12:28,774 44k INFO Losses: [2.385878562927246, 2.2772061824798584, 7.589957237243652, 18.116687774658203, 0.925600528717041], step: 5200, lr: 9.646381228640066e-05
2023-04-06 17:12:32,702 44k INFO ====> Epoch: 289, cost 35.16 s
2023-04-06 17:13:06,900 44k INFO ====> Epoch: 290, cost 34.20 s
2023-04-06 17:13:40,746 44k INFO ====> Epoch: 291, cost 33.85 s
2023-04-06 17:14:14,405 44k INFO ====> Epoch: 292, cost 33.66 s
2023-04-06 17:14:47,901 44k INFO ====> Epoch: 293, cost 33.50 s
2023-04-06 17:15:22,146 44k INFO ====> Epoch: 294, cost 34.25 s
2023-04-06 17:15:56,201 44k INFO ====> Epoch: 295, cost 34.05 s
2023-04-06 17:16:29,525 44k INFO ====> Epoch: 296, cost 33.32 s
2023-04-06 17:17:02,512 44k INFO ====> Epoch: 297, cost 32.99 s
2023-04-06 17:17:36,118 44k INFO ====> Epoch: 298, cost 33.61 s
2023-04-06 17:18:09,488 44k INFO ====> Epoch: 299, cost 33.37 s
2023-04-06 17:18:43,408 44k INFO ====> Epoch: 300, cost 33.92 s
2023-04-06 17:18:55,456 44k INFO Train Epoch: 301 [0%]
2023-04-06 17:18:55,458 44k INFO Losses: [2.4418349266052246, 2.024925470352173, 11.340639114379883, 21.75313949584961, 0.7900910973548889], step: 5400, lr: 9.631921600483981e-05
2023-04-06 17:19:19,113 44k INFO ====> Epoch: 301, cost 35.71 s
2023-04-06 17:19:52,445 44k INFO ====> Epoch: 302, cost 33.33 s
2023-04-06 17:20:26,150 44k INFO ====> Epoch: 303, cost 33.71 s
2023-04-06 17:20:59,723 44k INFO ====> Epoch: 304, cost 33.57 s
2023-04-06 17:21:33,468 44k INFO ====> Epoch: 305, cost 33.74 s
2023-04-06 17:22:07,157 44k INFO ====> Epoch: 306, cost 33.69 s
2023-04-06 17:22:40,790 44k INFO ====> Epoch: 307, cost 33.63 s
2023-04-06 17:23:15,437 44k INFO ====> Epoch: 308, cost 34.65 s
2023-04-06 17:23:48,982 44k INFO ====> Epoch: 309, cost 33.54 s
2023-04-06 17:24:22,362 44k INFO ====> Epoch: 310, cost 33.38 s
2023-04-06 17:24:55,753 44k INFO ====> Epoch: 311, cost 33.39 s
2023-04-06 17:25:09,900 44k INFO Train Epoch: 312 [11%]
2023-04-06 17:25:09,901 44k INFO Losses: [2.2993826866149902, 2.5729517936706543, 9.609771728515625, 20.48442268371582, 1.0834064483642578], step: 5600, lr: 9.618685982612675e-05
2023-04-06 17:25:18,064 44k INFO Saving model and optimizer state at iteration 312 to ./logs/44k/G_5600.pth
2023-04-06 17:25:21,357 44k INFO Saving model and optimizer state at iteration 312 to ./logs/44k/D_5600.pth
2023-04-06 17:25:23,632 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_3200.pth
2023-04-06 17:25:23,654 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_3200.pth
2023-04-06 17:25:42,928 44k INFO ====> Epoch: 312, cost 47.17 s
2023-04-06 17:26:17,141 44k INFO ====> Epoch: 313, cost 34.21 s
2023-04-06 17:26:51,938 44k INFO ====> Epoch: 314, cost 34.80 s
2023-04-06 17:27:27,322 44k INFO ====> Epoch: 315, cost 35.38 s
2023-04-06 17:28:00,914 44k INFO ====> Epoch: 316, cost 33.59 s
2023-04-06 17:28:34,276 44k INFO ====> Epoch: 317, cost 33.36 s
2023-04-06 17:29:07,777 44k INFO ====> Epoch: 318, cost 33.50 s
2023-04-06 17:29:41,937 44k INFO ====> Epoch: 319, cost 34.16 s
2023-04-06 17:30:15,474 44k INFO ====> Epoch: 320, cost 33.54 s
2023-04-06 17:30:48,876 44k INFO ====> Epoch: 321, cost 33.40 s
2023-04-06 17:31:22,605 44k INFO ====> Epoch: 322, cost 33.73 s
2023-04-06 17:31:39,703 44k INFO Train Epoch: 323 [22%]
2023-04-06 17:31:39,704 44k INFO Losses: [2.4820761680603027, 2.1193840503692627, 9.61584186553955, 18.75108528137207, 0.8709684014320374], step: 5800, lr: 9.60546855234585e-05
2023-04-06 17:31:57,655 44k INFO ====> Epoch: 323, cost 35.05 s
2023-04-06 17:32:31,106 44k INFO ====> Epoch: 324, cost 33.45 s
2023-04-06 17:33:04,825 44k INFO ====> Epoch: 325, cost 33.72 s
2023-04-06 17:33:38,728 44k INFO ====> Epoch: 326, cost 33.90 s
2023-04-06 17:34:13,554 44k INFO ====> Epoch: 327, cost 34.83 s
2023-04-06 17:34:47,381 44k INFO ====> Epoch: 328, cost 33.83 s
2023-04-06 17:35:21,672 44k INFO ====> Epoch: 329, cost 34.29 s
2023-04-06 17:35:55,556 44k INFO ====> Epoch: 330, cost 33.88 s
2023-04-06 17:36:29,481 44k INFO ====> Epoch: 331, cost 33.93 s
2023-04-06 17:37:02,840 44k INFO ====> Epoch: 332, cost 33.36 s
2023-04-06 17:37:36,638 44k INFO ====> Epoch: 333, cost 33.80 s
2023-04-06 17:37:55,526 44k INFO Train Epoch: 334 [33%]
2023-04-06 17:37:55,527 44k INFO Losses: [2.4064953327178955, 2.66304349899292, 10.577417373657227, 20.366371154785156, 0.7555525898933411], step: 6000, lr: 9.592269284691169e-05
2023-04-06 17:38:11,787 44k INFO ====> Epoch: 334, cost 35.15 s
2023-04-06 17:38:45,301 44k INFO ====> Epoch: 335, cost 33.51 s
2023-04-06 17:39:18,540 44k INFO ====> Epoch: 336, cost 33.24 s
2023-04-06 17:39:51,928 44k INFO ====> Epoch: 337, cost 33.39 s
2023-04-06 17:40:25,228 44k INFO ====> Epoch: 338, cost 33.30 s
2023-04-06 17:40:59,031 44k INFO ====> Epoch: 339, cost 33.80 s
2023-04-06 17:41:32,618 44k INFO ====> Epoch: 340, cost 33.59 s
2023-04-06 17:42:06,122 44k INFO ====> Epoch: 341, cost 33.50 s
2023-04-06 17:42:39,641 44k INFO ====> Epoch: 342, cost 33.52 s
2023-04-06 17:43:14,283 44k INFO ====> Epoch: 343, cost 34.64 s
2023-04-06 17:43:47,305 44k INFO ====> Epoch: 344, cost 33.02 s
2023-04-06 17:44:08,818 44k INFO Train Epoch: 345 [44%]
2023-04-06 17:44:08,820 44k INFO Losses: [2.5008935928344727, 2.476442813873291, 8.65200424194336, 19.446016311645508, 0.897552490234375], step: 6200, lr: 9.579088154690645e-05
2023-04-06 17:44:22,667 44k INFO ====> Epoch: 345, cost 35.36 s
2023-04-06 17:44:56,141 44k INFO ====> Epoch: 346, cost 33.47 s
2023-04-06 17:45:29,619 44k INFO ====> Epoch: 347, cost 33.48 s
2023-04-06 17:46:02,997 44k INFO ====> Epoch: 348, cost 33.38 s
2023-04-06 17:46:36,029 44k INFO ====> Epoch: 349, cost 33.03 s
2023-04-06 17:47:09,417 44k INFO ====> Epoch: 350, cost 33.39 s
2023-04-06 17:47:43,613 44k INFO ====> Epoch: 351, cost 34.20 s
2023-04-06 17:48:17,278 44k INFO ====> Epoch: 352, cost 33.67 s
2023-04-06 17:48:50,891 44k INFO ====> Epoch: 353, cost 33.61 s
2023-04-06 17:49:24,801 44k INFO ====> Epoch: 354, cost 33.91 s
2023-04-06 17:49:58,546 44k INFO ====> Epoch: 355, cost 33.74 s
2023-04-06 17:50:23,180 44k INFO Train Epoch: 356 [56%]
2023-04-06 17:50:23,195 44k INFO Losses: [2.4418582916259766, 2.2405121326446533, 10.872772216796875, 17.84126853942871, 0.5929334759712219], step: 6400, lr: 9.565925137420586e-05
2023-04-06 17:50:31,313 44k INFO Saving model and optimizer state at iteration 356 to ./logs/44k/G_6400.pth
2023-04-06 17:50:34,706 44k INFO Saving model and optimizer state at iteration 356 to ./logs/44k/D_6400.pth
2023-04-06 17:50:36,723 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_4000.pth
2023-04-06 17:50:36,741 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_4000.pth
2023-04-06 17:50:46,170 44k INFO ====> Epoch: 356, cost 47.62 s
2023-04-06 17:51:19,938 44k INFO ====> Epoch: 357, cost 33.77 s
2023-04-06 17:51:53,407 44k INFO ====> Epoch: 358, cost 33.47 s
2023-04-06 17:52:26,917 44k INFO ====> Epoch: 359, cost 33.51 s
2023-04-06 17:53:00,258 44k INFO ====> Epoch: 360, cost 33.34 s
2023-04-06 17:53:34,014 44k INFO ====> Epoch: 361, cost 33.76 s
2023-04-06 17:54:07,388 44k INFO ====> Epoch: 362, cost 33.37 s
2023-04-06 17:54:40,878 44k INFO ====> Epoch: 363, cost 33.49 s
2023-04-06 17:55:14,530 44k INFO ====> Epoch: 364, cost 33.65 s
2023-04-06 17:55:48,242 44k INFO ====> Epoch: 365, cost 33.71 s
2023-04-06 17:56:21,978 44k INFO ====> Epoch: 366, cost 33.74 s
2023-04-06 17:56:48,837 44k INFO Train Epoch: 367 [67%]
2023-04-06 17:56:48,838 44k INFO Losses: [2.554368495941162, 2.3327033519744873, 9.829940795898438, 18.284847259521484, 0.6705255508422852], step: 6600, lr: 9.552780207991543e-05
2023-04-06 17:56:57,173 44k INFO ====> Epoch: 367, cost 35.19 s
2023-04-06 17:57:30,583 44k INFO ====> Epoch: 368, cost 33.41 s
2023-04-06 17:58:04,232 44k INFO ====> Epoch: 369, cost 33.65 s
2023-04-06 17:58:37,876 44k INFO ====> Epoch: 370, cost 33.64 s
2023-04-06 17:59:11,271 44k INFO ====> Epoch: 371, cost 33.39 s
2023-04-06 17:59:44,903 44k INFO ====> Epoch: 372, cost 33.63 s
2023-04-06 18:00:18,504 44k INFO ====> Epoch: 373, cost 33.60 s
2023-04-06 18:00:52,016 44k INFO ====> Epoch: 374, cost 33.51 s
2023-04-06 18:01:25,671 44k INFO ====> Epoch: 375, cost 33.66 s
2023-04-06 18:01:59,300 44k INFO ====> Epoch: 376, cost 33.63 s
2023-04-06 18:02:32,972 44k INFO ====> Epoch: 377, cost 33.67 s
2023-04-06 18:03:02,229 44k INFO Train Epoch: 378 [78%]
2023-04-06 18:03:02,231 44k INFO Losses: [2.393739700317383, 2.3625659942626953, 9.48896312713623, 20.4971866607666, 0.6042823195457458], step: 6800, lr: 9.53965334154828e-05
2023-04-06 18:03:08,140 44k INFO ====> Epoch: 378, cost 35.17 s
2023-04-06 18:03:42,388 44k INFO ====> Epoch: 379, cost 34.25 s
2023-04-06 18:04:15,398 44k INFO ====> Epoch: 380, cost 33.01 s
2023-04-06 18:04:48,620 44k INFO ====> Epoch: 381, cost 33.22 s
2023-04-06 18:05:22,030 44k INFO ====> Epoch: 382, cost 33.41 s
2023-04-06 18:05:55,530 44k INFO ====> Epoch: 383, cost 33.50 s
2023-04-06 18:06:29,132 44k INFO ====> Epoch: 384, cost 33.60 s
2023-04-06 18:07:02,378 44k INFO ====> Epoch: 385, cost 33.25 s
2023-04-06 18:07:35,862 44k INFO ====> Epoch: 386, cost 33.48 s
2023-04-06 18:08:09,492 44k INFO ====> Epoch: 387, cost 33.63 s
2023-04-06 18:08:42,771 44k INFO ====> Epoch: 388, cost 33.28 s
2023-04-06 18:09:13,869 44k INFO Train Epoch: 389 [89%]
2023-04-06 18:09:13,870 44k INFO Losses: [2.431995391845703, 1.9307184219360352, 9.80262279510498, 17.839599609375, 1.0465185642242432], step: 7000, lr: 9.526544513269702e-05
2023-04-06 18:09:17,762 44k INFO ====> Epoch: 389, cost 34.99 s
2023-04-06 18:09:51,455 44k INFO ====> Epoch: 390, cost 33.69 s
2023-04-06 18:10:25,326 44k INFO ====> Epoch: 391, cost 33.87 s
2023-04-06 18:10:59,206 44k INFO ====> Epoch: 392, cost 33.88 s
2023-04-06 18:11:33,821 44k INFO ====> Epoch: 393, cost 34.61 s
2023-04-06 18:12:06,960 44k INFO ====> Epoch: 394, cost 33.14 s
2023-04-06 18:12:40,180 44k INFO ====> Epoch: 395, cost 33.22 s
2023-04-06 18:13:13,083 44k INFO ====> Epoch: 396, cost 32.90 s
2023-04-06 18:13:46,451 44k INFO ====> Epoch: 397, cost 33.37 s
2023-04-06 18:14:19,552 44k INFO ====> Epoch: 398, cost 33.10 s
2023-04-06 18:14:53,253 44k INFO ====> Epoch: 399, cost 33.70 s
2023-04-06 18:15:26,650 44k INFO ====> Epoch: 400, cost 33.40 s
2023-04-06 18:15:38,441 44k INFO Train Epoch: 401 [0%]
2023-04-06 18:15:38,442 44k INFO Losses: [2.593489646911621, 2.073544979095459, 8.42275619506836, 20.45761489868164, 0.9182995557785034], step: 7200, lr: 9.512264516656537e-05
2023-04-06 18:15:46,917 44k INFO Saving model and optimizer state at iteration 401 to ./logs/44k/G_7200.pth
2023-04-06 18:15:49,493 44k INFO Saving model and optimizer state at iteration 401 to ./logs/44k/D_7200.pth
2023-04-06 18:15:51,656 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_4800.pth
2023-04-06 18:15:51,692 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_4800.pth
2023-04-06 18:16:12,795 44k INFO ====> Epoch: 401, cost 46.15 s
2023-04-06 18:16:48,449 44k INFO ====> Epoch: 402, cost 35.65 s
2023-04-06 18:17:21,303 44k INFO ====> Epoch: 403, cost 32.85 s
2023-04-06 18:17:54,163 44k INFO ====> Epoch: 404, cost 32.86 s
2023-04-06 18:18:27,499 44k INFO ====> Epoch: 405, cost 33.34 s
2023-04-06 18:19:00,117 44k INFO ====> Epoch: 406, cost 32.62 s
2023-04-06 18:19:32,815 44k INFO ====> Epoch: 407, cost 32.70 s
2023-04-06 18:20:05,338 44k INFO ====> Epoch: 408, cost 32.52 s
2023-04-06 18:20:38,251 44k INFO ====> Epoch: 409, cost 32.91 s
2023-04-06 18:21:11,470 44k INFO ====> Epoch: 410, cost 33.22 s
2023-04-06 18:21:46,000 44k INFO ====> Epoch: 411, cost 34.53 s
2023-04-06 18:22:01,037 44k INFO Train Epoch: 412 [11%]
2023-04-06 18:22:01,038 44k INFO Losses: [2.1447744369506836, 2.547001838684082, 10.99000072479248, 19.626802444458008, 0.7141999006271362], step: 7400, lr: 9.49919332448374e-05
2023-04-06 18:22:21,928 44k INFO ====> Epoch: 412, cost 35.93 s
2023-04-06 18:22:56,926 44k INFO ====> Epoch: 413, cost 35.00 s
2023-04-06 18:23:31,265 44k INFO ====> Epoch: 414, cost 34.34 s
2023-04-06 18:24:05,002 44k INFO ====> Epoch: 415, cost 33.74 s
2023-04-06 18:24:38,474 44k INFO ====> Epoch: 416, cost 33.47 s
2023-04-06 18:25:12,157 44k INFO ====> Epoch: 417, cost 33.68 s
2023-04-06 18:25:45,599 44k INFO ====> Epoch: 418, cost 33.44 s
2023-04-06 18:26:18,842 44k INFO ====> Epoch: 419, cost 33.24 s
2023-04-06 18:26:52,083 44k INFO ====> Epoch: 420, cost 33.24 s
2023-04-06 18:27:25,208 44k INFO ====> Epoch: 421, cost 33.13 s
2023-04-06 18:27:59,523 44k INFO ====> Epoch: 422, cost 34.31 s
2023-04-06 18:28:15,815 44k INFO Train Epoch: 423 [22%]
2023-04-06 18:28:15,816 44k INFO Losses: [2.537222385406494, 2.3694236278533936, 8.022411346435547, 19.27557373046875, 0.6971426606178284], step: 7600, lr: 9.486140093971337e-05
2023-04-06 18:28:33,895 44k INFO ====> Epoch: 423, cost 34.37 s
2023-04-06 18:29:07,256 44k INFO ====> Epoch: 424, cost 33.36 s
2023-04-06 18:29:39,938 44k INFO ====> Epoch: 425, cost 32.68 s
2023-04-06 18:30:12,540 44k INFO ====> Epoch: 426, cost 32.60 s
2023-04-06 18:30:45,176 44k INFO ====> Epoch: 427, cost 32.64 s
2023-04-06 18:31:17,932 44k INFO ====> Epoch: 428, cost 32.76 s
2023-04-06 18:31:51,059 44k INFO ====> Epoch: 429, cost 33.13 s
2023-04-06 18:32:24,138 44k INFO ====> Epoch: 430, cost 33.08 s
2023-04-06 18:32:56,916 44k INFO ====> Epoch: 431, cost 32.78 s
2023-04-06 18:33:29,850 44k INFO ====> Epoch: 432, cost 32.93 s
2023-04-06 18:34:02,979 44k INFO ====> Epoch: 433, cost 33.13 s
2023-04-06 18:34:22,044 44k INFO Train Epoch: 434 [33%]
2023-04-06 18:34:22,046 44k INFO Losses: [2.5888988971710205, 2.652815103530884, 12.130846977233887, 19.23379898071289, 0.6897168159484863], step: 7800, lr: 9.473104800437474e-05
2023-04-06 18:34:37,583 44k INFO ====> Epoch: 434, cost 34.60 s
2023-04-06 18:35:10,466 44k INFO ====> Epoch: 435, cost 32.88 s
2023-04-06 18:35:43,117 44k INFO ====> Epoch: 436, cost 32.65 s
2023-04-06 18:36:16,021 44k INFO ====> Epoch: 437, cost 32.90 s
2023-04-06 18:36:49,001 44k INFO ====> Epoch: 438, cost 32.98 s
2023-04-06 18:37:22,642 44k INFO ====> Epoch: 439, cost 33.64 s
2023-04-06 18:37:55,974 44k INFO ====> Epoch: 440, cost 33.33 s
2023-04-06 18:38:29,213 44k INFO ====> Epoch: 441, cost 33.24 s
2023-04-06 18:39:02,172 44k INFO ====> Epoch: 442, cost 32.96 s
2023-04-06 18:39:35,318 44k INFO ====> Epoch: 443, cost 33.15 s
2023-04-06 18:40:09,009 44k INFO ====> Epoch: 444, cost 33.69 s
2023-04-06 18:40:29,898 44k INFO Train Epoch: 445 [44%]
2023-04-06 18:40:29,906 44k INFO Losses: [2.449326515197754, 2.2474606037139893, 9.135872840881348, 17.043214797973633, 1.1349270343780518], step: 8000, lr: 9.460087419234215e-05
2023-04-06 18:40:37,738 44k INFO Saving model and optimizer state at iteration 445 to ./logs/44k/G_8000.pth
2023-04-06 18:40:40,799 44k INFO Saving model and optimizer state at iteration 445 to ./logs/44k/D_8000.pth
2023-04-06 18:40:42,913 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_5600.pth
2023-04-06 18:40:42,931 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_5600.pth
2023-04-06 18:40:54,369 44k INFO ====> Epoch: 445, cost 45.36 s
2023-04-06 18:41:27,287 44k INFO ====> Epoch: 446, cost 32.92 s
2023-04-06 18:42:00,322 44k INFO ====> Epoch: 447, cost 33.04 s
2023-04-06 18:42:33,367 44k INFO ====> Epoch: 448, cost 33.05 s
2023-04-06 18:43:06,523 44k INFO ====> Epoch: 449, cost 33.16 s
2023-04-06 18:43:39,260 44k INFO ====> Epoch: 450, cost 32.74 s
2023-04-06 18:44:12,295 44k INFO ====> Epoch: 451, cost 33.04 s
2023-04-06 18:44:45,316 44k INFO ====> Epoch: 452, cost 33.02 s
2023-04-06 18:45:18,870 44k INFO ====> Epoch: 453, cost 33.55 s
2023-04-06 18:45:51,983 44k INFO ====> Epoch: 454, cost 33.11 s
2023-04-06 18:46:25,270 44k INFO ====> Epoch: 455, cost 33.29 s
2023-04-06 18:46:49,250 44k INFO Train Epoch: 456 [56%]
2023-04-06 18:46:49,252 44k INFO Losses: [2.3567802906036377, 2.143897771835327, 9.234848976135254, 17.149524688720703, 0.6894568800926208], step: 8200, lr: 9.44708792574749e-05
2023-04-06 18:46:59,763 44k INFO ====> Epoch: 456, cost 34.49 s
2023-04-06 18:47:33,704 44k INFO ====> Epoch: 457, cost 33.94 s
2023-04-06 18:48:06,742 44k INFO ====> Epoch: 458, cost 33.04 s
2023-04-06 18:48:39,574 44k INFO ====> Epoch: 459, cost 32.83 s
2023-04-06 18:49:12,269 44k INFO ====> Epoch: 460, cost 32.70 s
2023-04-06 18:49:45,030 44k INFO ====> Epoch: 461, cost 32.76 s
2023-04-06 18:50:17,896 44k INFO ====> Epoch: 462, cost 32.87 s
2023-04-06 18:50:51,156 44k INFO ====> Epoch: 463, cost 33.26 s
2023-04-06 18:51:24,141 44k INFO ====> Epoch: 464, cost 32.98 s
2023-04-06 18:51:57,196 44k INFO ====> Epoch: 465, cost 33.05 s
2023-04-06 18:52:30,213 44k INFO ====> Epoch: 466, cost 33.02 s
2023-04-06 18:52:56,415 44k INFO Train Epoch: 467 [67%]
2023-04-06 18:52:56,416 44k INFO Losses: [2.467308759689331, 2.284391403198242, 8.696948051452637, 17.621639251708984, 0.924575686454773], step: 8400, lr: 9.434106295397058e-05
2023-04-06 18:53:04,429 44k INFO ====> Epoch: 467, cost 34.22 s
2023-04-06 18:53:37,410 44k INFO ====> Epoch: 468, cost 32.98 s
2023-04-06 18:54:10,326 44k INFO ====> Epoch: 469, cost 32.92 s
2023-04-06 18:54:43,554 44k INFO ====> Epoch: 470, cost 33.23 s
2023-04-06 18:55:16,737 44k INFO ====> Epoch: 471, cost 33.18 s
2023-04-06 18:55:49,956 44k INFO ====> Epoch: 472, cost 33.22 s
2023-04-06 18:56:23,098 44k INFO ====> Epoch: 473, cost 33.14 s
2023-04-06 18:56:56,318 44k INFO ====> Epoch: 474, cost 33.22 s
2023-04-06 18:57:29,722 44k INFO ====> Epoch: 475, cost 33.40 s
2023-04-06 18:58:02,999 44k INFO ====> Epoch: 476, cost 33.28 s
2023-04-06 18:58:36,919 44k INFO ====> Epoch: 477, cost 33.92 s
2023-04-06 18:59:06,807 44k INFO Train Epoch: 478 [78%]
2023-04-06 18:59:06,808 44k INFO Losses: [2.318368434906006, 2.5831010341644287, 10.86165714263916, 22.857358932495117, 0.560059130191803], step: 8600, lr: 9.421142503636453e-05
2023-04-06 18:59:12,961 44k INFO ====> Epoch: 478, cost 36.04 s
2023-04-06 18:59:45,995 44k INFO ====> Epoch: 479, cost 33.03 s
2023-04-06 19:00:18,982 44k INFO ====> Epoch: 480, cost 32.99 s
2023-04-06 19:00:52,255 44k INFO ====> Epoch: 481, cost 33.27 s
2023-04-06 19:01:25,500 44k INFO ====> Epoch: 482, cost 33.25 s
2023-04-06 19:01:59,954 44k INFO ====> Epoch: 483, cost 34.45 s
2023-04-06 19:02:34,325 44k INFO ====> Epoch: 484, cost 34.37 s
2023-04-06 19:03:08,200 44k INFO ====> Epoch: 485, cost 33.88 s
2023-04-06 19:03:41,648 44k INFO ====> Epoch: 486, cost 33.45 s
2023-04-06 19:04:14,879 44k INFO ====> Epoch: 487, cost 33.23 s
2023-04-06 19:04:48,296 44k INFO ====> Epoch: 488, cost 33.42 s
2023-04-06 19:05:19,394 44k INFO Train Epoch: 489 [89%]
2023-04-06 19:05:19,395 44k INFO Losses: [2.524198532104492, 2.032053232192993, 7.60945987701416, 17.46242332458496, 1.0244165658950806], step: 8800, lr: 9.408196525952938e-05
2023-04-06 19:05:27,206 44k INFO Saving model and optimizer state at iteration 489 to ./logs/44k/G_8800.pth
2023-04-06 19:05:29,658 44k INFO Saving model and optimizer state at iteration 489 to ./logs/44k/D_8800.pth
2023-04-06 19:05:32,156 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_6400.pth
2023-04-06 19:05:32,176 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_6400.pth
2023-04-06 19:05:34,623 44k INFO ====> Epoch: 489, cost 46.33 s
2023-04-06 19:06:08,364 44k INFO ====> Epoch: 490, cost 33.74 s
2023-04-06 19:06:41,198 44k INFO ====> Epoch: 491, cost 32.83 s
2023-04-06 19:07:14,232 44k INFO ====> Epoch: 492, cost 33.03 s
2023-04-06 19:07:47,558 44k INFO ====> Epoch: 493, cost 33.33 s
2023-04-06 19:08:20,720 44k INFO ====> Epoch: 494, cost 33.16 s
2023-04-06 19:08:54,066 44k INFO ====> Epoch: 495, cost 33.35 s
2023-04-06 19:09:27,097 44k INFO ====> Epoch: 496, cost 33.03 s
2023-04-06 19:10:00,060 44k INFO ====> Epoch: 497, cost 32.96 s
2023-04-06 19:10:33,147 44k INFO ====> Epoch: 498, cost 33.09 s
2023-04-06 19:11:06,269 44k INFO ====> Epoch: 499, cost 33.12 s
2023-04-06 19:11:39,260 44k INFO ====> Epoch: 500, cost 32.99 s
2023-04-06 19:11:50,845 44k INFO Train Epoch: 501 [0%]
2023-04-06 19:11:50,846 44k INFO Losses: [2.3949179649353027, 2.2692530155181885, 8.20638370513916, 20.383365631103516, 1.1488752365112305], step: 9000, lr: 9.394093929325224e-05
2023-04-06 19:12:13,818 44k INFO ====> Epoch: 501, cost 34.56 s
2023-04-06 19:12:46,826 44k INFO ====> Epoch: 502, cost 33.01 s
2023-04-06 19:13:19,983 44k INFO ====> Epoch: 503, cost 33.16 s
2023-04-06 19:13:53,209 44k INFO ====> Epoch: 504, cost 33.23 s
2023-04-06 19:14:26,572 44k INFO ====> Epoch: 505, cost 33.36 s
2023-04-06 19:14:59,985 44k INFO ====> Epoch: 506, cost 33.41 s
2023-04-06 19:15:33,236 44k INFO ====> Epoch: 507, cost 33.25 s
2023-04-06 19:16:06,443 44k INFO ====> Epoch: 508, cost 33.21 s
2023-04-06 19:16:40,213 44k INFO ====> Epoch: 509, cost 33.77 s
2023-04-06 19:17:13,136 44k INFO ====> Epoch: 510, cost 32.92 s
2023-04-06 19:17:46,253 44k INFO ====> Epoch: 511, cost 33.12 s
2023-04-06 19:18:00,182 44k INFO Train Epoch: 512 [11%]
2023-04-06 19:18:00,183 44k INFO Losses: [2.0607686042785645, 2.637697458267212, 10.518184661865234, 21.01915168762207, 0.816366970539093], step: 9200, lr: 9.381185120195232e-05
2023-04-06 19:18:20,407 44k INFO ====> Epoch: 512, cost 34.15 s
2023-04-06 19:18:53,534 44k INFO ====> Epoch: 513, cost 33.13 s
2023-04-06 19:19:26,782 44k INFO ====> Epoch: 514, cost 33.25 s
2023-04-06 19:20:00,171 44k INFO ====> Epoch: 515, cost 33.39 s
2023-04-06 19:20:33,348 44k INFO ====> Epoch: 516, cost 33.18 s
2023-04-06 19:21:06,156 44k INFO ====> Epoch: 517, cost 32.81 s
2023-04-06 19:21:39,395 44k INFO ====> Epoch: 518, cost 33.24 s
2023-04-06 19:22:12,433 44k INFO ====> Epoch: 519, cost 33.04 s
2023-04-06 19:22:45,302 44k INFO ====> Epoch: 520, cost 32.87 s
2023-04-06 19:23:18,505 44k INFO ====> Epoch: 521, cost 33.20 s
2023-04-06 19:23:51,504 44k INFO ====> Epoch: 522, cost 33.00 s
2023-04-06 19:24:08,483 44k INFO Train Epoch: 523 [22%]
2023-04-06 19:24:08,485 44k INFO Losses: [2.554436206817627, 2.2548623085021973, 8.05234146118164, 19.58883285522461, 0.7690815329551697], step: 9400, lr: 9.368294049588446e-05
2023-04-06 19:24:26,224 44k INFO ====> Epoch: 523, cost 34.72 s
2023-04-06 19:24:59,464 44k INFO ====> Epoch: 524, cost 33.24 s
2023-04-06 19:25:32,713 44k INFO ====> Epoch: 525, cost 33.25 s
2023-04-06 19:26:06,142 44k INFO ====> Epoch: 526, cost 33.43 s
2023-04-06 19:26:39,477 44k INFO ====> Epoch: 527, cost 33.33 s
2023-04-06 19:27:13,261 44k INFO ====> Epoch: 528, cost 33.78 s
2023-04-06 19:27:47,514 44k INFO ====> Epoch: 529, cost 34.25 s
2023-04-06 19:28:20,406 44k INFO ====> Epoch: 530, cost 32.89 s
2023-04-06 19:28:52,926 44k INFO ====> Epoch: 531, cost 32.52 s
2023-04-06 19:29:25,812 44k INFO ====> Epoch: 532, cost 32.89 s
2023-04-06 19:29:58,497 44k INFO ====> Epoch: 533, cost 32.69 s
2023-04-06 19:30:17,026 44k INFO Train Epoch: 534 [33%]
2023-04-06 19:30:17,028 44k INFO Losses: [2.5109479427337646, 2.1757607460021973, 9.281797409057617, 17.23943328857422, 0.5131280422210693], step: 9600, lr: 9.355420693129632e-05
2023-04-06 19:30:25,640 44k INFO Saving model and optimizer state at iteration 534 to ./logs/44k/G_9600.pth
2023-04-06 19:30:28,296 44k INFO Saving model and optimizer state at iteration 534 to ./logs/44k/D_9600.pth
2023-04-06 19:30:30,428 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_7200.pth
2023-04-06 19:30:30,446 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_7200.pth
2023-04-06 19:30:44,380 44k INFO ====> Epoch: 534, cost 45.88 s
2023-04-06 19:31:17,622 44k INFO ====> Epoch: 535, cost 33.24 s
2023-04-06 19:31:50,594 44k INFO ====> Epoch: 536, cost 32.97 s
2023-04-06 19:32:23,705 44k INFO ====> Epoch: 537, cost 33.11 s
2023-04-06 19:32:56,923 44k INFO ====> Epoch: 538, cost 33.22 s
2023-04-06 19:33:30,263 44k INFO ====> Epoch: 539, cost 33.34 s
2023-04-06 19:34:03,426 44k INFO ====> Epoch: 540, cost 33.16 s
2023-04-06 19:34:36,678 44k INFO ====> Epoch: 541, cost 33.25 s
2023-04-06 19:35:10,191 44k INFO ====> Epoch: 542, cost 33.51 s
2023-04-06 19:35:44,037 44k INFO ====> Epoch: 543, cost 33.85 s
2023-04-06 19:36:16,782 44k INFO ====> Epoch: 544, cost 32.74 s
2023-04-06 19:36:38,044 44k INFO Train Epoch: 545 [44%]
2023-04-06 19:36:38,045 44k INFO Losses: [2.445751667022705, 2.247299909591675, 7.489833831787109, 16.84793472290039, 1.0898230075836182], step: 9800, lr: 9.342565026477056e-05
2023-04-06 19:36:51,067 44k INFO ====> Epoch: 545, cost 34.28 s
2023-04-06 19:37:23,942 44k INFO ====> Epoch: 546, cost 32.88 s
2023-04-06 19:37:56,989 44k INFO ====> Epoch: 547, cost 33.05 s
2023-04-06 19:38:30,180 44k INFO ====> Epoch: 548, cost 33.19 s
2023-04-06 19:39:03,442 44k INFO ====> Epoch: 549, cost 33.26 s
2023-04-06 19:39:36,796 44k INFO ====> Epoch: 550, cost 33.35 s
2023-04-06 19:40:09,944 44k INFO ====> Epoch: 551, cost 33.15 s
2023-04-06 19:40:43,433 44k INFO ====> Epoch: 552, cost 33.49 s
2023-04-06 19:41:16,969 44k INFO ====> Epoch: 553, cost 33.54 s
2023-04-06 19:41:49,900 44k INFO ====> Epoch: 554, cost 32.93 s
2023-04-06 19:42:23,093 44k INFO ====> Epoch: 555, cost 33.19 s
2023-04-06 19:42:46,977 44k INFO Train Epoch: 556 [56%]
2023-04-06 19:42:46,978 44k INFO Losses: [2.439335346221924, 2.1651785373687744, 8.26251220703125, 17.951847076416016, 0.6958508491516113], step: 10000, lr: 9.32972702532243e-05
2023-04-06 19:42:57,652 44k INFO ====> Epoch: 556, cost 34.56 s
2023-04-06 19:43:31,192 44k INFO ====> Epoch: 557, cost 33.54 s
2023-04-06 19:44:04,793 44k INFO ====> Epoch: 558, cost 33.60 s
2023-04-06 19:44:38,080 44k INFO ====> Epoch: 559, cost 33.29 s
2023-04-06 19:45:11,594 44k INFO ====> Epoch: 560, cost 33.51 s
2023-04-06 19:45:44,852 44k INFO ====> Epoch: 561, cost 33.26 s
2023-04-06 19:46:18,336 44k INFO ====> Epoch: 562, cost 33.48 s
2023-04-06 19:46:52,163 44k INFO ====> Epoch: 563, cost 33.83 s
2023-04-06 19:47:24,969 44k INFO ====> Epoch: 564, cost 32.81 s
2023-04-06 19:47:57,678 44k INFO ====> Epoch: 565, cost 32.71 s
2023-04-06 19:48:30,506 44k INFO ====> Epoch: 566, cost 32.83 s
2023-04-06 19:48:57,147 44k INFO Train Epoch: 567 [67%]
2023-04-06 19:48:57,148 44k INFO Losses: [2.339824676513672, 2.326724052429199, 7.90087890625, 18.2314395904541, 0.9818569421768188], step: 10200, lr: 9.316906665390869e-05
2023-04-06 19:49:05,821 44k INFO ====> Epoch: 567, cost 35.31 s
2023-04-06 19:49:38,932 44k INFO ====> Epoch: 568, cost 33.11 s
2023-04-06 19:50:12,102 44k INFO ====> Epoch: 569, cost 33.17 s
2023-04-06 19:50:45,048 44k INFO ====> Epoch: 570, cost 32.95 s
2023-04-06 19:51:17,974 44k INFO ====> Epoch: 571, cost 32.93 s
2023-04-06 19:51:51,777 44k INFO ====> Epoch: 572, cost 33.80 s
2023-04-06 19:52:24,545 44k INFO ====> Epoch: 573, cost 32.77 s
2023-04-06 19:52:57,507 44k INFO ====> Epoch: 574, cost 32.96 s
2023-04-06 19:53:30,310 44k INFO ====> Epoch: 575, cost 32.80 s
2023-04-06 19:54:03,654 44k INFO ====> Epoch: 576, cost 33.34 s
2023-04-06 19:54:37,066 44k INFO ====> Epoch: 577, cost 33.41 s
2023-04-06 19:55:05,884 44k INFO Train Epoch: 578 [78%]
2023-04-06 19:55:05,886 44k INFO Losses: [2.246778964996338, 2.5583128929138184, 11.40976333618164, 19.14501190185547, 0.7166233062744141], step: 10400, lr: 9.304103922440849e-05
2023-04-06 19:55:13,589 44k INFO Saving model and optimizer state at iteration 578 to ./logs/44k/G_10400.pth
2023-04-06 19:55:16,183 44k INFO Saving model and optimizer state at iteration 578 to ./logs/44k/D_10400.pth
2023-04-06 19:55:18,590 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_8000.pth
2023-04-06 19:55:18,619 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_8000.pth
2023-04-06 19:55:23,170 44k INFO ====> Epoch: 578, cost 46.10 s
2023-04-06 19:55:56,235 44k INFO ====> Epoch: 579, cost 33.07 s
2023-04-06 19:56:29,363 44k INFO ====> Epoch: 580, cost 33.13 s
2023-04-06 19:57:02,635 44k INFO ====> Epoch: 581, cost 33.27 s
2023-04-06 19:57:35,978 44k INFO ====> Epoch: 582, cost 33.34 s
2023-04-06 19:58:09,070 44k INFO ====> Epoch: 583, cost 33.09 s
2023-04-06 19:58:41,826 44k INFO ====> Epoch: 584, cost 32.76 s
2023-04-06 19:59:14,828 44k INFO ====> Epoch: 585, cost 33.00 s
2023-04-06 19:59:47,711 44k INFO ====> Epoch: 586, cost 32.88 s
2023-04-06 20:00:20,712 44k INFO ====> Epoch: 587, cost 33.00 s
2023-04-06 20:00:54,072 44k INFO ====> Epoch: 588, cost 33.36 s
2023-04-06 20:01:24,857 44k INFO Train Epoch: 589 [89%]
2023-04-06 20:01:24,858 44k INFO Losses: [2.548102378845215, 2.1721510887145996, 8.618353843688965, 16.937570571899414, 1.0304343700408936], step: 10600, lr: 9.291318772264153e-05
2023-04-06 20:01:28,575 44k INFO ====> Epoch: 589, cost 34.50 s
2023-04-06 20:02:01,884 44k INFO ====> Epoch: 590, cost 33.31 s
2023-04-06 20:02:35,471 44k INFO ====> Epoch: 591, cost 33.59 s
2023-04-06 20:03:08,929 44k INFO ====> Epoch: 592, cost 33.46 s
2023-04-06 20:03:42,055 44k INFO ====> Epoch: 593, cost 33.13 s
2023-04-06 20:04:15,337 44k INFO ====> Epoch: 594, cost 33.28 s
2023-04-06 20:04:48,504 44k INFO ====> Epoch: 595, cost 33.17 s
2023-04-06 20:05:21,948 44k INFO ====> Epoch: 596, cost 33.44 s
2023-04-06 20:05:55,381 44k INFO ====> Epoch: 597, cost 33.43 s
2023-04-06 20:06:28,869 44k INFO ====> Epoch: 598, cost 33.49 s
2023-04-06 20:07:01,659 44k INFO ====> Epoch: 599, cost 32.79 s
2023-04-06 20:07:34,713 44k INFO ====> Epoch: 600, cost 33.05 s
2023-04-06 20:07:46,226 44k INFO Train Epoch: 601 [0%]
2023-04-06 20:07:46,228 44k INFO Losses: [2.4384801387786865, 2.300462484359741, 8.600714683532715, 19.138277053833008, 1.3364354372024536], step: 10800, lr: 9.277391371786995e-05
2023-04-06 20:08:09,338 44k INFO ====> Epoch: 601, cost 34.62 s
2023-04-06 20:08:42,493 44k INFO ====> Epoch: 602, cost 33.16 s
2023-04-06 20:09:15,870 44k INFO ====> Epoch: 603, cost 33.38 s
2023-04-06 20:09:48,696 44k INFO ====> Epoch: 604, cost 32.83 s
2023-04-06 20:10:21,782 44k INFO ====> Epoch: 605, cost 33.09 s
2023-04-06 20:10:54,820 44k INFO ====> Epoch: 606, cost 33.04 s
2023-04-06 20:11:27,910 44k INFO ====> Epoch: 607, cost 33.09 s
2023-04-06 20:12:00,820 44k INFO ====> Epoch: 608, cost 32.91 s
2023-04-06 20:12:33,803 44k INFO ====> Epoch: 609, cost 32.98 s
2023-04-06 20:13:06,825 44k INFO ====> Epoch: 610, cost 33.02 s
2023-04-06 20:13:39,997 44k INFO ====> Epoch: 611, cost 33.17 s
2023-04-06 20:13:54,499 44k INFO Train Epoch: 612 [11%]
2023-04-06 20:13:54,500 44k INFO Losses: [2.083113193511963, 2.611279010772705, 12.885259628295898, 19.54749870300293, 0.6030045747756958], step: 11000, lr: 9.264642928419956e-05
2023-04-06 20:14:14,614 44k INFO ====> Epoch: 612, cost 34.62 s
2023-04-06 20:14:47,860 44k INFO ====> Epoch: 613, cost 33.25 s
|