| Model pushed to 1 GPU(s), type NVIDIA GeForce RTX 3090. |
| Loaded pretrained model from /data/shared/luojun/runs/snow1/weights/checkpoint.pth.tar |
| --- Training --- |
| Epoch 0 train loss: 0.6085, acc: 0.8926 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.829 |
| mIoU improved from 0.0000 to 0.4715. |
| mIoU improved from 0.0000 to 0.6457. |
| --- Training --- |
| Epoch 1 train loss: 0.5992, acc: 0.8947 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.854 |
| mIoU improved from 0.4715 to 0.4840. |
| mIoU improved from 0.6457 to 0.6482. |
| --- Training --- |
| Epoch 2 train loss: 0.5943, acc: 0.8953 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.839 |
| --- Training --- |
| Epoch 3 train loss: 0.5837, acc: 0.8977 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.828 |
| --- Training --- |
| Epoch 4 train loss: 0.5794, acc: 0.8984 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.847 |
| mIoU improved from 0.6482 to 0.6508. |
| --- Training --- |
| Epoch 5 train loss: 0.5752, acc: 0.8990 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.816 |
| --- Training --- |
| Epoch 6 train loss: 0.5719, acc: 0.8999 |
| --- Validation - Source --- |
| * Acc 0.932 |
| --- Validation - Target --- |
| * Acc 0.833 |
| --- Training --- |
| Epoch 7 train loss: 0.5682, acc: 0.9004 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.843 |
| --- Training --- |
| Epoch 8 train loss: 0.5659, acc: 0.9010 |
| --- Validation - Source --- |
| * Acc 0.931 |
| --- Validation - Target --- |
| * Acc 0.847 |
| --- Training --- |
| Epoch 9 train loss: 0.5675, acc: 0.9005 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.826 |
| --- Training --- |
| Epoch 10 train loss: 0.5579, acc: 0.9027 |
| --- Validation - Source --- |
| * Acc 0.932 |
| --- Validation - Target --- |
| * Acc 0.838 |
| --- Training --- |
| Epoch 11 train loss: 0.5569, acc: 0.9024 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.836 |
| --- Training --- |
| Epoch 12 train loss: 0.5530, acc: 0.9033 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.832 |
| --- Training --- |
| Epoch 13 train loss: 0.5535, acc: 0.9032 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.857 |
| --- Training --- |
| Epoch 14 train loss: 0.5509, acc: 0.9035 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.825 |
| --- Training --- |
| Epoch 15 train loss: 0.5475, acc: 0.9043 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.835 |
| --- Training --- |
| Epoch 16 train loss: 0.5471, acc: 0.9041 |
| --- Validation - Source --- |
| * Acc 0.932 |
| --- Validation - Target --- |
| * Acc 0.820 |
| --- Training --- |
| Epoch 17 train loss: 0.5439, acc: 0.9052 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.833 |
| --- Training --- |
| Epoch 18 train loss: 0.5415, acc: 0.9053 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.845 |
| --- Training --- |
| Epoch 19 train loss: 0.5382, acc: 0.9059 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.837 |
| mIoU improved from 0.6508 to 0.6520. |
| --- Training --- |
| Epoch 20 train loss: 0.5375, acc: 0.9062 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.831 |
| mIoU improved from 0.6520 to 0.6532. |
| --- Training --- |
| Epoch 21 train loss: 0.5339, acc: 0.9067 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.835 |
| mIoU improved from 0.6532 to 0.6549. |
| --- Training --- |
| Epoch 22 train loss: 0.5356, acc: 0.9058 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.818 |
| --- Training --- |
| Epoch 23 train loss: 0.5318, acc: 0.9071 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.841 |
| --- Training --- |
| Epoch 24 train loss: 0.5310, acc: 0.9072 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.830 |
| --- Training --- |
| Epoch 25 train loss: 0.5306, acc: 0.9070 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.834 |
| --- Training --- |
| Epoch 26 train loss: 0.5282, acc: 0.9079 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.836 |
| --- Training --- |
| Epoch 27 train loss: 0.5270, acc: 0.9080 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.848 |
| --- Training --- |
| Epoch 28 train loss: 0.5243, acc: 0.9084 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.848 |
| --- Training --- |
| Epoch 29 train loss: 0.5216, acc: 0.9087 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.841 |
| --- Training --- |
| Epoch 30 train loss: 0.5198, acc: 0.9094 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.847 |
| --- Training --- |
| Epoch 31 train loss: 0.5196, acc: 0.9094 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.860 |
| --- Training --- |
| Epoch 32 train loss: 0.5171, acc: 0.9096 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.840 |
| --- Training --- |
| Epoch 33 train loss: 0.5165, acc: 0.9098 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.840 |
| --- Training --- |
| Epoch 34 train loss: 0.5156, acc: 0.9099 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.826 |
| mIoU improved from 0.6549 to 0.6551. |
| --- Training --- |
| Epoch 35 train loss: 0.5144, acc: 0.9103 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.838 |
| --- Training --- |
| Epoch 36 train loss: 0.5123, acc: 0.9107 |
| --- Validation - Source --- |
| * Acc 0.933 |
| --- Validation - Target --- |
| * Acc 0.832 |
| --- Training --- |
| Epoch 37 train loss: 0.5160, acc: 0.9102 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.833 |
| mIoU improved from 0.6551 to 0.6556. |
| --- Training --- |
| Epoch 38 train loss: 0.5113, acc: 0.9106 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.824 |
| --- Training --- |
| Epoch 39 train loss: 0.5128, acc: 0.9100 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.826 |
| --- Training --- |
| Epoch 40 train loss: 0.5097, acc: 0.9114 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.847 |
| --- Training --- |
| Epoch 41 train loss: 0.5076, acc: 0.9112 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.839 |
| --- Training --- |
| Epoch 42 train loss: 0.5074, acc: 0.9114 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.842 |
| mIoU improved from 0.6556 to 0.6569. |
| --- Training --- |
| Epoch 43 train loss: 0.5054, acc: 0.9119 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.840 |
| mIoU improved from 0.6569 to 0.6578. |
| --- Training --- |
| Epoch 44 train loss: 0.5049, acc: 0.9118 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.838 |
| --- Training --- |
| Epoch 45 train loss: 0.5026, acc: 0.9125 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.837 |
| --- Training --- |
| Epoch 46 train loss: 0.5046, acc: 0.9120 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.839 |
| mIoU improved from 0.6578 to 0.6579. |
| --- Training --- |
| Epoch 47 train loss: 0.5023, acc: 0.9125 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.822 |
| --- Training --- |
| Epoch 48 train loss: 0.4998, acc: 0.9128 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.826 |
| mIoU improved from 0.6579 to 0.6581. |
| --- Training --- |
| Epoch 49 train loss: 0.4978, acc: 0.9131 |
| --- Validation - Source --- |
| * Acc 0.934 |
| --- Validation - Target --- |
| * Acc 0.805 |
| --- Training --- |
| Epoch 50 train loss: 0.4982, acc: 0.9132 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.793 |
| --- Training --- |
| Epoch 51 train loss: 0.4959, acc: 0.9136 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.839 |
| --- Training --- |
| Epoch 52 train loss: 0.4948, acc: 0.9138 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.828 |
| --- Training --- |
| Epoch 53 train loss: 0.4965, acc: 0.9132 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.824 |
| --- Training --- |
| Epoch 54 train loss: 0.4954, acc: 0.9134 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.849 |
| mIoU improved from 0.6581 to 0.6589. |
| --- Training --- |
| Epoch 55 train loss: 0.4943, acc: 0.9138 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.857 |
| --- Training --- |
| Epoch 56 train loss: 0.4943, acc: 0.9137 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.819 |
| --- Training --- |
| Epoch 57 train loss: 0.4944, acc: 0.9137 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.845 |
| --- Training --- |
| Epoch 58 train loss: 0.4924, acc: 0.9141 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.837 |
| --- Training --- |
| Epoch 59 train loss: 0.4903, acc: 0.9143 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.823 |
| --- Training --- |
| Epoch 60 train loss: 0.4906, acc: 0.9139 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.847 |
| --- Training --- |
| Epoch 61 train loss: 0.4900, acc: 0.9145 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.834 |
| --- Training --- |
| Epoch 62 train loss: 0.4918, acc: 0.9141 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.847 |
| mIoU improved from 0.6589 to 0.6596. |
| --- Training --- |
| Epoch 63 train loss: 0.4899, acc: 0.9145 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.841 |
| --- Training --- |
| Epoch 64 train loss: 0.4886, acc: 0.9148 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.836 |
| --- Training --- |
| Epoch 65 train loss: 0.4874, acc: 0.9151 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.840 |
| --- Training --- |
| Epoch 66 train loss: 0.4842, acc: 0.9153 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.833 |
| --- Training --- |
| Epoch 67 train loss: 0.4847, acc: 0.9153 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.850 |
| --- Training --- |
| Epoch 68 train loss: 0.4836, acc: 0.9158 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.849 |
| mIoU improved from 0.6596 to 0.6599. |
| --- Training --- |
| Epoch 69 train loss: 0.4842, acc: 0.9156 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.835 |
| mIoU improved from 0.6599 to 0.6629. |
| --- Training --- |
| Epoch 70 train loss: 0.4835, acc: 0.9158 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.848 |
| --- Training --- |
| Epoch 71 train loss: 0.4861, acc: 0.9152 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.845 |
| --- Training --- |
| Epoch 72 train loss: 0.4815, acc: 0.9159 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.839 |
| --- Training --- |
| Epoch 73 train loss: 0.4819, acc: 0.9159 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.826 |
| --- Training --- |
| Epoch 74 train loss: 0.4832, acc: 0.9157 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.844 |
| --- Training --- |
| Epoch 75 train loss: 0.4829, acc: 0.9160 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.845 |
| --- Training --- |
| Epoch 76 train loss: 0.4840, acc: 0.9157 |
| --- Validation - Source --- |
| * Acc 0.935 |
| --- Validation - Target --- |
| * Acc 0.833 |
| --- Training --- |
| Epoch 77 train loss: 0.4781, acc: 0.9165 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.825 |
| --- Training --- |
| Epoch 78 train loss: 0.4819, acc: 0.9158 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.809 |
| --- Training --- |
| Epoch 79 train loss: 0.4801, acc: 0.9163 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.834 |
| --- Training --- |
| Epoch 80 train loss: 0.4822, acc: 0.9161 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.851 |
| --- Training --- |
| Epoch 81 train loss: 0.4770, acc: 0.9166 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.822 |
| --- Training --- |
| Epoch 82 train loss: 0.4809, acc: 0.9163 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.830 |
| --- Training --- |
| Epoch 83 train loss: 0.4776, acc: 0.9166 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.815 |
| mIoU improved from 0.6629 to 0.6652. |
| --- Training --- |
| Epoch 84 train loss: 0.4795, acc: 0.9163 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.833 |
| --- Training --- |
| Epoch 85 train loss: 0.4802, acc: 0.9163 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.851 |
| --- Training --- |
| Epoch 86 train loss: 0.4788, acc: 0.9162 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.847 |
| --- Training --- |
| Epoch 87 train loss: 0.4777, acc: 0.9167 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.832 |
| --- Training --- |
| Epoch 88 train loss: 0.4757, acc: 0.9173 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.832 |
| --- Training --- |
| Epoch 89 train loss: 0.4772, acc: 0.9169 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.838 |
| --- Training --- |
| Epoch 90 train loss: 0.4767, acc: 0.9169 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.823 |
| --- Training --- |
| Epoch 91 train loss: 0.4745, acc: 0.9174 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.832 |
| --- Training --- |
| Epoch 92 train loss: 0.4764, acc: 0.9171 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.844 |
| --- Training --- |
| Epoch 93 train loss: 0.4765, acc: 0.9168 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.842 |
| --- Training --- |
| Epoch 94 train loss: 0.4763, acc: 0.9169 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.841 |
| --- Training --- |
| Epoch 95 train loss: 0.4769, acc: 0.9170 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.821 |
| --- Training --- |
| Epoch 96 train loss: 0.4754, acc: 0.9170 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.840 |
| --- Training --- |
| Epoch 97 train loss: 0.4773, acc: 0.9169 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.831 |
| --- Training --- |
| Epoch 98 train loss: 0.4775, acc: 0.9168 |
| --- Validation - Source --- |
| * Acc 0.936 |
| --- Validation - Target --- |
| * Acc 0.840 |
| --- Training --- |
| Epoch 99 train loss: 0.4746, acc: 0.9171 |
| --- Validation - Source --- |
| * Acc 0.937 |
| --- Validation - Target --- |
| * Acc 0.840 |
| Training complete in 333m 51s |
| Best mIoU target: 0.484036 |
|
|
