| Model pushed to 1 GPU(s), type NVIDIA GeForce RTX 3090. |
| Loaded pretrained model from /data/shared/luojun/runs/night1/weights/checkpoint.pth.tar |
| --- Training --- |
| Epoch 0 train loss: 0.4977, acc: 0.9129 |
| --- Validation - Source --- |
| * Acc 0.932 |
| --- Validation - Target --- |
| * Acc 0.620 |
| mIoU improved from 0.0000 to 0.2443. |
| mIoU improved from 0.0000 to 0.6431. |
| --- Training --- |
| Epoch 1 train loss: 0.4929, acc: 0.9141 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.625 |
| mIoU improved from 0.2443 to 0.2500. |
| --- Training --- |
| Epoch 2 train loss: 0.4872, acc: 0.9153 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.621 |
| mIoU improved from 0.6431 to 0.6458. |
| --- Training --- |
| Epoch 3 train loss: 0.4823, acc: 0.9160 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.616 |
| mIoU improved from 0.2500 to 0.2534. |
| mIoU improved from 0.6458 to 0.6473. |
| --- Training --- |
| Epoch 4 train loss: 0.4765, acc: 0.9175 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.621 |
| mIoU improved from 0.6473 to 0.6517. |
| --- Training --- |
| Epoch 5 train loss: 0.4760, acc: 0.9175 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.627 |
| mIoU improved from 0.2534 to 0.2534. |
| --- Training --- |
| Epoch 6 train loss: 0.4716, acc: 0.9185 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.626 |
| --- Training --- |
| Epoch 7 train loss: 0.4676, acc: 0.9190 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.626 |
| --- Training --- |
| Epoch 8 train loss: 0.4672, acc: 0.9188 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.630 |
| mIoU improved from 0.2534 to 0.2561. |
| --- Training --- |
| Epoch 9 train loss: 0.4661, acc: 0.9194 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.628 |
| mIoU improved from 0.6517 to 0.6554. |
| --- Training --- |
| Epoch 10 train loss: 0.4601, acc: 0.9203 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.626 |
| --- Training --- |
| Epoch 11 train loss: 0.4619, acc: 0.9200 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.629 |
| mIoU improved from 0.2561 to 0.2593. |
| --- Training --- |
| Epoch 12 train loss: 0.4569, acc: 0.9211 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.618 |
| --- Training --- |
| Epoch 13 train loss: 0.4560, acc: 0.9210 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.627 |
| --- Training --- |
| Epoch 14 train loss: 0.4535, acc: 0.9214 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.630 |
| --- Training --- |
| Epoch 15 train loss: 0.4515, acc: 0.9220 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.632 |
| mIoU improved from 0.2593 to 0.2626. |
| mIoU improved from 0.6554 to 0.6568. |
| --- Training --- |
| Epoch 16 train loss: 0.4533, acc: 0.9219 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.632 |
| --- Training --- |
| Epoch 17 train loss: 0.4507, acc: 0.9219 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.628 |
| --- Training --- |
| Epoch 18 train loss: 0.4465, acc: 0.9227 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.631 |
| --- Training --- |
| Epoch 19 train loss: 0.4448, acc: 0.9231 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.636 |
| mIoU improved from 0.2626 to 0.2644. |
| mIoU improved from 0.6568 to 0.6581. |
| --- Training --- |
| Epoch 20 train loss: 0.4461, acc: 0.9227 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.629 |
| mIoU improved from 0.6581 to 0.6609. |
| --- Training --- |
| Epoch 21 train loss: 0.4420, acc: 0.9236 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.637 |
| mIoU improved from 0.6609 to 0.6610. |
| --- Training --- |
| Epoch 22 train loss: 0.4416, acc: 0.9235 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.626 |
| mIoU improved from 0.6610 to 0.6613. |
| --- Training --- |
| Epoch 23 train loss: 0.4395, acc: 0.9239 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.628 |
| --- Training --- |
| Epoch 24 train loss: 0.4380, acc: 0.9243 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.624 |
| --- Training --- |
| Epoch 25 train loss: 0.4389, acc: 0.9241 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.626 |
| --- Training --- |
| Epoch 26 train loss: 0.4375, acc: 0.9243 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.636 |
| --- Training --- |
| Epoch 27 train loss: 0.4344, acc: 0.9248 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.628 |
| --- Training --- |
| Epoch 28 train loss: 0.4341, acc: 0.9250 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.624 |
| --- Training --- |
| Epoch 29 train loss: 0.4324, acc: 0.9251 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.629 |
| mIoU improved from 0.6613 to 0.6613. |
| --- Training --- |
| Epoch 30 train loss: 0.4303, acc: 0.9253 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.623 |
| --- Training --- |
| Epoch 31 train loss: 0.4294, acc: 0.9257 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.635 |
| mIoU improved from 0.2644 to 0.2654. |
| mIoU improved from 0.6613 to 0.6628. |
| --- Training --- |
| Epoch 32 train loss: 0.4301, acc: 0.9255 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.613 |
| --- Training --- |
| Epoch 33 train loss: 0.4274, acc: 0.9258 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.641 |
| mIoU improved from 0.2654 to 0.2680. |
| mIoU improved from 0.6628 to 0.6637. |
| --- Training --- |
| Epoch 34 train loss: 0.4278, acc: 0.9259 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.632 |
| --- Training --- |
| Epoch 35 train loss: 0.4247, acc: 0.9267 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.632 |
| --- Training --- |
| Epoch 36 train loss: 0.4247, acc: 0.9267 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.626 |
| --- Training --- |
| Epoch 37 train loss: 0.4270, acc: 0.9262 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.628 |
| --- Training --- |
| Epoch 38 train loss: 0.4221, acc: 0.9267 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.634 |
| --- Training --- |
| Epoch 39 train loss: 0.4222, acc: 0.9269 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.626 |
| mIoU improved from 0.6637 to 0.6650. |
| --- Training --- |
| Epoch 40 train loss: 0.4224, acc: 0.9267 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.640 |
| mIoU improved from 0.2680 to 0.2696. |
| --- Training --- |
| Epoch 41 train loss: 0.4195, acc: 0.9271 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.631 |
| --- Training --- |
| Epoch 42 train loss: 0.4201, acc: 0.9274 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.637 |
| --- Training --- |
| Epoch 43 train loss: 0.4178, acc: 0.9277 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.627 |
| --- Training --- |
| Epoch 44 train loss: 0.4170, acc: 0.9278 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.630 |
| --- Training --- |
| Epoch 45 train loss: 0.4153, acc: 0.9282 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.624 |
| mIoU improved from 0.6650 to 0.6675. |
| --- Training --- |
| Epoch 46 train loss: 0.4186, acc: 0.9275 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.641 |
| --- Training --- |
| Epoch 47 train loss: 0.4161, acc: 0.9281 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.629 |
| --- Training --- |
| Epoch 48 train loss: 0.4149, acc: 0.9282 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.631 |
| --- Training --- |
| Epoch 49 train loss: 0.4128, acc: 0.9283 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.626 |
| --- Training --- |
| Epoch 50 train loss: 0.4123, acc: 0.9283 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.624 |
| --- Training --- |
| Epoch 51 train loss: 0.4111, acc: 0.9287 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.641 |
| --- Training --- |
| Epoch 52 train loss: 0.4111, acc: 0.9289 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.634 |
| --- Training --- |
| Epoch 53 train loss: 0.4114, acc: 0.9287 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.633 |
| --- Training --- |
| Epoch 54 train loss: 0.4094, acc: 0.9289 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.629 |
| --- Training --- |
| Epoch 55 train loss: 0.4090, acc: 0.9292 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.639 |
| --- Training --- |
| Epoch 56 train loss: 0.4099, acc: 0.9289 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.631 |
| mIoU improved from 0.2696 to 0.2716. |
| --- Training --- |
| Epoch 57 train loss: 0.4085, acc: 0.9292 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.637 |
| --- Training --- |
| Epoch 58 train loss: 0.4081, acc: 0.9291 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.631 |
| mIoU improved from 0.6675 to 0.6712. |
| --- Training --- |
| Epoch 59 train loss: 0.4050, acc: 0.9296 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.626 |
| --- Training --- |
| Epoch 60 train loss: 0.4053, acc: 0.9298 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.640 |
| --- Training --- |
| Epoch 61 train loss: 0.4073, acc: 0.9294 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.631 |
| --- Training --- |
| Epoch 62 train loss: 0.4060, acc: 0.9298 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.639 |
| --- Training --- |
| Epoch 63 train loss: 0.4048, acc: 0.9298 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.635 |
| --- Training --- |
| Epoch 64 train loss: 0.4051, acc: 0.9298 |
| --- Validation - Source --- |
| * Acc 0.939 |
| --- Validation - Target --- |
| * Acc 0.640 |
| mIoU improved from 0.2716 to 0.2763. |
| mIoU improved from 0.6712 to 0.6718. |
| --- Training --- |
| Epoch 65 train loss: 0.4042, acc: 0.9300 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.641 |
| --- Training --- |
| Epoch 66 train loss: 0.4019, acc: 0.9302 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.628 |
| --- Training --- |
| Epoch 67 train loss: 0.4022, acc: 0.9303 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.632 |
| --- Training --- |
| Epoch 68 train loss: 0.4026, acc: 0.9301 |
| --- Validation - Source --- |
| * Acc 0.939 |
| --- Validation - Target --- |
| * Acc 0.628 |
| --- Training --- |
| Epoch 69 train loss: 0.4023, acc: 0.9303 |
| --- Validation - Source --- |
| * Acc 0.939 |
| --- Validation - Target --- |
| * Acc 0.638 |
| mIoU improved from 0.6718 to 0.6739. |
| --- Training --- |
| Epoch 70 train loss: 0.4017, acc: 0.9304 |
| --- Validation - Source --- |
| * Acc 0.939 |
| --- Validation - Target --- |
| * Acc 0.630 |
| --- Training --- |
| Epoch 71 train loss: 0.4021, acc: 0.9303 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.633 |
| --- Training --- |
| Epoch 72 train loss: 0.3998, acc: 0.9306 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.631 |
| --- Training --- |
| Epoch 73 train loss: 0.3996, acc: 0.9305 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.618 |
| --- Training --- |
| Epoch 74 train loss: 0.4006, acc: 0.9304 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.637 |
| --- Training --- |
| Epoch 75 train loss: 0.3997, acc: 0.9308 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.632 |
| --- Training --- |
| Epoch 76 train loss: 0.4005, acc: 0.9307 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.635 |
| --- Training --- |
| Epoch 77 train loss: 0.3961, acc: 0.9313 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.637 |
| --- Training --- |
| Epoch 78 train loss: 0.3975, acc: 0.9310 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.640 |
| --- Training --- |
| Epoch 79 train loss: 0.3996, acc: 0.9309 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.639 |
| --- Training --- |
| Epoch 80 train loss: 0.3977, acc: 0.9310 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.645 |
| mIoU improved from 0.2763 to 0.2789. |
| --- Training --- |
| Epoch 81 train loss: 0.3948, acc: 0.9316 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.635 |
| --- Training --- |
| Epoch 82 train loss: 0.3992, acc: 0.9311 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.643 |
| --- Training --- |
| Epoch 83 train loss: 0.3957, acc: 0.9315 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.629 |
| --- Training --- |
| Epoch 84 train loss: 0.3975, acc: 0.9309 |
| --- Validation - Source --- |
| * Acc 0.939 |
| --- Validation - Target --- |
| * Acc 0.636 |
| --- Training --- |
| Epoch 85 train loss: 0.3967, acc: 0.9312 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.643 |
| --- Training --- |
| Epoch 86 train loss: 0.3962, acc: 0.9310 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.633 |
| --- Training --- |
| Epoch 87 train loss: 0.3955, acc: 0.9314 |
| --- Validation - Source --- |
| * Acc 0.939 |
| --- Validation - Target --- |
| * Acc 0.634 |
| --- Training --- |
| Epoch 88 train loss: 0.3955, acc: 0.9313 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.636 |
| --- Training --- |
| Epoch 89 train loss: 0.3954, acc: 0.9315 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.633 |
| --- Training --- |
| Epoch 90 train loss: 0.3956, acc: 0.9315 |
| --- Validation - Source --- |
| * Acc 0.939 |
| --- Validation - Target --- |
| * Acc 0.623 |
| --- Training --- |
| Epoch 91 train loss: 0.3930, acc: 0.9317 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.631 |
| --- Training --- |
| Epoch 92 train loss: 0.3950, acc: 0.9314 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.631 |
| --- Training --- |
| Epoch 93 train loss: 0.3958, acc: 0.9313 |
| --- Validation - Source --- |
| * Acc 0.939 |
| --- Validation - Target --- |
| * Acc 0.633 |
| --- Training --- |
| Epoch 94 train loss: 0.3939, acc: 0.9317 |
| --- Validation - Source --- |
| * Acc 0.939 |
| --- Validation - Target --- |
| * Acc 0.629 |
| --- Training --- |
| Epoch 95 train loss: 0.3953, acc: 0.9315 |
| --- Validation - Source --- |
| * Acc 0.939 |
| --- Validation - Target --- |
| * Acc 0.633 |
| mIoU improved from 0.6739 to 0.6748. |
| --- Training --- |
| Epoch 96 train loss: 0.3936, acc: 0.9318 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.627 |
| --- Training --- |
| Epoch 97 train loss: 0.3969, acc: 0.9312 |
| --- Validation - Source --- |
| * Acc 0.939 |
| --- Validation - Target --- |
| * Acc 0.625 |
| --- Training --- |
| Epoch 98 train loss: 0.3978, acc: 0.9311 |
| --- Validation - Source --- |
| * Acc 0.938 |
| --- Validation - Target --- |
| * Acc 0.639 |
| --- Training --- |
| Epoch 99 train loss: 0.3942, acc: 0.9318 |
| --- Validation - Source --- |
| * Acc 0.939 |
| --- Validation - Target --- |
| * Acc 0.632 |
| Training complete in 309m 33s |
| Best mIoU target: 0.278944 |
|
|
