ilessio-aiflowlab commited on
Commit
b66cac9
·
verified ·
1 Parent(s): a86d77f

Upload logs/train_pst900_20260404_1832.log with huggingface_hub

Browse files
Files changed (1) hide show
  1. logs/train_pst900_20260404_1832.log +272 -0
logs/train_pst900_20260404_1832.log ADDED
@@ -0,0 +1,272 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-04-04 18:32:49 [INFO] Loaded 664 pretrained weights into both streams. Missing backbone keys: 0, Unexpected: 0
2
+ 2026-04-04 18:32:49 [INFO] [MODEL] mit_b2 | 50.4M params | 5 classes
3
+ 2026-04-04 18:32:49 [INFO] RGBTDataset [data/ade_pst/training]: 597 samples (skipped 0 without annotation)
4
+ 2026-04-04 18:32:49 [INFO] RGBTDataset [data/ade_pst/validation]: 288 samples (skipped 0 without annotation)
5
+ 2026-04-04 18:32:49 [INFO] [DATA] pst900 | train=597 val=288 | crop=(512, 512)
6
+ 2026-04-04 18:32:49 [INFO] [BATCH] Config batch_size=4
7
+ 2026-04-04 18:32:49 [INFO] [CONFIG] configs/pst900_b2.toml
8
+ 2026-04-04 18:32:49 [INFO] [GPU] NVIDIA L4 (22563MB)
9
+ 2026-04-04 18:32:49 [INFO] [TRAIN] 300 epochs, lr=3.00e-05, optimizer=adamw
10
+ 2026-04-04 18:32:49 [INFO] [CKPT] save every 10 epochs, keep best 2
11
+ 2026-04-04 18:34:10 [INFO] [TRAIN] Epoch 0 | loss=5.3551 | lr=1.00e-05 | 7.4 samples/sec | 80.2s
12
+ 2026-04-04 18:35:22 [INFO] [TRAIN] Epoch 1 | loss=3.1578 | lr=1.00e-05 | 8.3 samples/sec | 72.1s
13
+ 2026-04-04 18:36:34 [INFO] [TRAIN] Epoch 2 | loss=1.7337 | lr=1.00e-05 | 8.3 samples/sec | 72.1s
14
+ 2026-04-04 18:37:47 [INFO] [TRAIN] Epoch 3 | loss=1.0491 | lr=1.00e-05 | 8.2 samples/sec | 73.0s
15
+ 2026-04-04 18:39:00 [INFO] [TRAIN] Epoch 4 | loss=0.6940 | lr=1.00e-05 | 8.2 samples/sec | 72.8s
16
+ 2026-04-04 18:40:01 [INFO] [VAL] loss=0.4213 | mIoU=0.3351 | acc=0.9800
17
+ 2026-04-04 18:40:02 [INFO] [BEST] mIoU=0.3351 at epoch 4
18
+ 2026-04-04 18:41:13 [INFO] [TRAIN] Epoch 5 | loss=0.6394 | lr=1.20e-05 | 8.4 samples/sec | 71.0s
19
+ 2026-04-04 18:42:24 [INFO] [TRAIN] Epoch 6 | loss=0.5789 | lr=1.40e-05 | 8.5 samples/sec | 70.3s
20
+ 2026-04-04 18:43:34 [INFO] [TRAIN] Epoch 7 | loss=0.5886 | lr=1.60e-05 | 8.5 samples/sec | 70.4s
21
+ 2026-04-04 18:44:45 [INFO] [TRAIN] Epoch 8 | loss=0.5356 | lr=1.80e-05 | 8.5 samples/sec | 70.5s
22
+ 2026-04-04 18:45:56 [INFO] [TRAIN] Epoch 9 | loss=0.4590 | lr=2.00e-05 | 8.4 samples/sec | 71.2s
23
+ 2026-04-04 18:46:57 [INFO] [VAL] loss=0.3578 | mIoU=0.6778 | acc=0.9900
24
+ 2026-04-04 18:46:59 [INFO] [BEST] mIoU=0.6778 at epoch 9
25
+ 2026-04-04 18:48:11 [INFO] [TRAIN] Epoch 10 | loss=0.4880 | lr=2.20e-05 | 8.3 samples/sec | 71.6s
26
+ 2026-04-04 18:49:22 [INFO] [TRAIN] Epoch 11 | loss=0.4892 | lr=2.40e-05 | 8.4 samples/sec | 70.9s
27
+ 2026-04-04 18:50:32 [INFO] [TRAIN] Epoch 12 | loss=0.4921 | lr=2.60e-05 | 8.5 samples/sec | 70.3s
28
+ 2026-04-04 18:51:43 [INFO] [TRAIN] Epoch 13 | loss=0.4563 | lr=2.80e-05 | 8.5 samples/sec | 70.5s
29
+ 2026-04-04 18:52:54 [INFO] [TRAIN] Epoch 14 | loss=0.4382 | lr=3.00e-05 | 8.3 samples/sec | 71.6s
30
+ 2026-04-04 18:53:57 [INFO] [VAL] loss=0.2866 | mIoU=0.7671 | acc=0.9929
31
+ 2026-04-04 18:53:59 [INFO] [BEST] mIoU=0.7671 at epoch 14
32
+ 2026-04-04 18:55:11 [INFO] [TRAIN] Epoch 15 | loss=0.4453 | lr=3.00e-05 | 8.3 samples/sec | 72.3s
33
+ 2026-04-04 18:56:22 [INFO] [TRAIN] Epoch 16 | loss=0.4273 | lr=3.00e-05 | 8.4 samples/sec | 70.9s
34
+ 2026-04-04 18:57:33 [INFO] [TRAIN] Epoch 17 | loss=0.3862 | lr=3.00e-05 | 8.3 samples/sec | 71.6s
35
+ 2026-04-04 18:58:44 [INFO] [TRAIN] Epoch 18 | loss=0.3998 | lr=3.00e-05 | 8.4 samples/sec | 71.0s
36
+ 2026-04-04 18:59:54 [INFO] [TRAIN] Epoch 19 | loss=0.3959 | lr=3.00e-05 | 8.5 samples/sec | 70.2s
37
+ 2026-04-04 19:00:54 [INFO] [VAL] loss=0.2255 | mIoU=0.7516 | acc=0.9931
38
+ 2026-04-04 19:02:08 [INFO] [TRAIN] Epoch 20 | loss=0.3832 | lr=3.00e-05 | 8.3 samples/sec | 72.2s
39
+ 2026-04-04 19:03:21 [INFO] [TRAIN] Epoch 21 | loss=0.3644 | lr=3.00e-05 | 8.3 samples/sec | 72.2s
40
+ 2026-04-04 19:04:32 [INFO] [TRAIN] Epoch 22 | loss=0.3977 | lr=2.99e-05 | 8.3 samples/sec | 71.8s
41
+ 2026-04-04 19:05:44 [INFO] [TRAIN] Epoch 23 | loss=0.3888 | lr=2.99e-05 | 8.3 samples/sec | 71.9s
42
+ 2026-04-04 19:06:56 [INFO] [TRAIN] Epoch 24 | loss=0.3934 | lr=2.99e-05 | 8.3 samples/sec | 71.9s
43
+ 2026-04-04 19:07:56 [INFO] [VAL] loss=0.2472 | mIoU=0.7780 | acc=0.9939
44
+ 2026-04-04 19:07:58 [INFO] [BEST] mIoU=0.7780 at epoch 24
45
+ 2026-04-04 19:09:10 [INFO] [TRAIN] Epoch 25 | loss=0.3388 | lr=2.99e-05 | 8.3 samples/sec | 71.9s
46
+ 2026-04-04 19:10:22 [INFO] [TRAIN] Epoch 26 | loss=0.3420 | lr=2.99e-05 | 8.3 samples/sec | 72.1s
47
+ 2026-04-04 19:11:34 [INFO] [TRAIN] Epoch 27 | loss=0.3531 | lr=2.98e-05 | 8.4 samples/sec | 71.5s
48
+ 2026-04-04 19:13:04 [INFO] [TRAIN] Epoch 28 | loss=0.3432 | lr=2.98e-05 | 6.6 samples/sec | 90.5s
49
+ 2026-04-04 19:14:52 [INFO] [TRAIN] Epoch 29 | loss=0.3590 | lr=2.98e-05 | 5.5 samples/sec | 108.0s
50
+ 2026-04-04 19:15:55 [INFO] [VAL] loss=0.2368 | mIoU=0.8109 | acc=0.9946
51
+ 2026-04-04 19:15:57 [INFO] [BEST] mIoU=0.8109 at epoch 29
52
+ 2026-04-04 19:17:45 [INFO] [TRAIN] Epoch 30 | loss=0.4190 | lr=2.98e-05 | 5.5 samples/sec | 108.0s
53
+ 2026-04-04 19:19:28 [INFO] [TRAIN] Epoch 31 | loss=0.3623 | lr=2.97e-05 | 5.8 samples/sec | 103.8s
54
+ 2026-04-04 19:20:39 [INFO] [TRAIN] Epoch 32 | loss=0.3394 | lr=2.97e-05 | 8.5 samples/sec | 70.4s
55
+ 2026-04-04 19:21:53 [INFO] [TRAIN] Epoch 33 | loss=0.3660 | lr=2.97e-05 | 8.1 samples/sec | 74.0s
56
+ 2026-04-04 19:23:41 [INFO] [TRAIN] Epoch 34 | loss=0.3426 | lr=2.96e-05 | 5.5 samples/sec | 108.0s
57
+ 2026-04-04 19:24:45 [INFO] [VAL] loss=0.2173 | mIoU=0.8053 | acc=0.9942
58
+ 2026-04-04 19:26:33 [INFO] [TRAIN] Epoch 35 | loss=0.3474 | lr=2.96e-05 | 5.5 samples/sec | 108.4s
59
+ 2026-04-04 19:28:22 [INFO] [TRAIN] Epoch 36 | loss=0.3130 | lr=2.96e-05 | 5.5 samples/sec | 109.0s
60
+ 2026-04-04 19:30:07 [INFO] [TRAIN] Epoch 37 | loss=0.3448 | lr=2.95e-05 | 5.7 samples/sec | 104.7s
61
+ 2026-04-04 19:31:54 [INFO] [TRAIN] Epoch 38 | loss=0.3533 | lr=2.95e-05 | 5.6 samples/sec | 107.4s
62
+ 2026-04-04 19:33:42 [INFO] [TRAIN] Epoch 39 | loss=0.3522 | lr=2.94e-05 | 5.6 samples/sec | 107.6s
63
+ 2026-04-04 19:34:41 [INFO] [VAL] loss=0.2636 | mIoU=0.8052 | acc=0.9945
64
+ 2026-04-04 19:36:30 [INFO] [TRAIN] Epoch 40 | loss=0.3627 | lr=2.94e-05 | 5.6 samples/sec | 105.9s
65
+ 2026-04-04 19:38:17 [INFO] [TRAIN] Epoch 41 | loss=0.3775 | lr=2.93e-05 | 5.5 samples/sec | 107.7s
66
+ 2026-04-04 19:40:05 [INFO] [TRAIN] Epoch 42 | loss=0.3258 | lr=2.93e-05 | 5.5 samples/sec | 108.0s
67
+ 2026-04-04 19:41:53 [INFO] [TRAIN] Epoch 43 | loss=0.3545 | lr=2.92e-05 | 5.5 samples/sec | 107.9s
68
+ 2026-04-04 19:43:41 [INFO] [TRAIN] Epoch 44 | loss=0.3339 | lr=2.92e-05 | 5.6 samples/sec | 107.6s
69
+ 2026-04-04 19:44:42 [INFO] [VAL] loss=0.2703 | mIoU=0.7963 | acc=0.9939
70
+ 2026-04-04 19:46:29 [INFO] [TRAIN] Epoch 45 | loss=0.3502 | lr=2.91e-05 | 5.5 samples/sec | 107.8s
71
+ 2026-04-04 19:48:17 [INFO] [TRAIN] Epoch 46 | loss=0.3323 | lr=2.91e-05 | 5.5 samples/sec | 107.8s
72
+ 2026-04-04 19:50:05 [INFO] [TRAIN] Epoch 47 | loss=0.3360 | lr=2.90e-05 | 5.5 samples/sec | 107.9s
73
+ 2026-04-04 19:51:51 [INFO] [TRAIN] Epoch 48 | loss=0.3183 | lr=2.90e-05 | 5.6 samples/sec | 105.7s
74
+ 2026-04-04 19:53:39 [INFO] [TRAIN] Epoch 49 | loss=0.3922 | lr=2.89e-05 | 5.5 samples/sec | 107.8s
75
+ 2026-04-04 19:54:39 [INFO] [VAL] loss=0.3928 | mIoU=0.8023 | acc=0.9944
76
+ 2026-04-04 19:56:28 [INFO] [TRAIN] Epoch 50 | loss=0.3587 | lr=2.88e-05 | 5.5 samples/sec | 107.8s
77
+ 2026-04-04 19:58:16 [INFO] [TRAIN] Epoch 51 | loss=0.3560 | lr=2.88e-05 | 5.5 samples/sec | 107.9s
78
+ 2026-04-04 20:00:02 [INFO] [TRAIN] Epoch 52 | loss=0.3374 | lr=2.87e-05 | 5.7 samples/sec | 105.6s
79
+ 2026-04-04 20:01:50 [INFO] [TRAIN] Epoch 53 | loss=0.3240 | lr=2.86e-05 | 5.6 samples/sec | 107.5s
80
+ 2026-04-04 20:03:37 [INFO] [TRAIN] Epoch 54 | loss=0.3343 | lr=2.86e-05 | 5.5 samples/sec | 107.7s
81
+ 2026-04-04 20:04:37 [INFO] [VAL] loss=0.3267 | mIoU=0.8116 | acc=0.9945
82
+ 2026-04-04 20:04:39 [INFO] [BEST] mIoU=0.8116 at epoch 54
83
+ 2026-04-04 20:06:26 [INFO] [TRAIN] Epoch 55 | loss=0.3933 | lr=2.85e-05 | 5.6 samples/sec | 106.8s
84
+ 2026-04-04 20:08:12 [INFO] [TRAIN] Epoch 56 | loss=0.3313 | lr=2.84e-05 | 5.6 samples/sec | 106.1s
85
+ 2026-04-04 20:10:00 [INFO] [TRAIN] Epoch 57 | loss=0.3560 | lr=2.83e-05 | 5.5 samples/sec | 107.8s
86
+ 2026-04-04 20:11:47 [INFO] [TRAIN] Epoch 58 | loss=0.3557 | lr=2.83e-05 | 5.5 samples/sec | 107.6s
87
+ 2026-04-04 20:13:35 [INFO] [TRAIN] Epoch 59 | loss=0.3098 | lr=2.82e-05 | 5.6 samples/sec | 107.5s
88
+ 2026-04-04 20:14:35 [INFO] [VAL] loss=0.2601 | mIoU=0.8358 | acc=0.9952
89
+ 2026-04-04 20:14:37 [INFO] [BEST] mIoU=0.8358 at epoch 59
90
+ 2026-04-04 20:16:24 [INFO] [TRAIN] Epoch 60 | loss=0.3298 | lr=2.81e-05 | 5.5 samples/sec | 107.6s
91
+ 2026-04-04 20:18:12 [INFO] [TRAIN] Epoch 61 | loss=0.3598 | lr=2.80e-05 | 5.5 samples/sec | 107.7s
92
+ 2026-04-04 20:19:59 [INFO] [TRAIN] Epoch 62 | loss=0.3254 | lr=2.79e-05 | 5.6 samples/sec | 107.3s
93
+ 2026-04-04 20:21:47 [INFO] [TRAIN] Epoch 63 | loss=0.3475 | lr=2.79e-05 | 5.5 samples/sec | 107.7s
94
+ 2026-04-04 20:23:33 [INFO] [TRAIN] Epoch 64 | loss=0.3128 | lr=2.78e-05 | 5.6 samples/sec | 106.0s
95
+ 2026-04-04 20:24:34 [INFO] [VAL] loss=0.2368 | mIoU=0.7981 | acc=0.9945
96
+ 2026-04-04 20:26:22 [INFO] [TRAIN] Epoch 65 | loss=0.3212 | lr=2.77e-05 | 5.5 samples/sec | 107.8s
97
+ 2026-04-04 20:28:10 [INFO] [TRAIN] Epoch 66 | loss=0.3275 | lr=2.76e-05 | 5.5 samples/sec | 107.7s
98
+ 2026-04-04 20:29:55 [INFO] [TRAIN] Epoch 67 | loss=0.3452 | lr=2.75e-05 | 5.7 samples/sec | 105.6s
99
+ 2026-04-04 20:31:43 [INFO] [TRAIN] Epoch 68 | loss=0.3268 | lr=2.74e-05 | 5.6 samples/sec | 107.5s
100
+ 2026-04-04 20:33:30 [INFO] [TRAIN] Epoch 69 | loss=0.3329 | lr=2.73e-05 | 5.6 samples/sec | 107.5s
101
+ 2026-04-04 20:34:29 [INFO] [VAL] loss=0.2931 | mIoU=0.7884 | acc=0.9940
102
+ 2026-04-04 20:36:19 [INFO] [TRAIN] Epoch 70 | loss=0.3654 | lr=2.72e-05 | 5.5 samples/sec | 107.8s
103
+ 2026-04-04 20:38:05 [INFO] [TRAIN] Epoch 71 | loss=0.3395 | lr=2.71e-05 | 5.6 samples/sec | 105.9s
104
+ 2026-04-04 20:39:53 [INFO] [TRAIN] Epoch 72 | loss=0.3399 | lr=2.70e-05 | 5.6 samples/sec | 107.4s
105
+ 2026-04-04 20:41:40 [INFO] [TRAIN] Epoch 73 | loss=0.3297 | lr=2.69e-05 | 5.5 samples/sec | 107.7s
106
+ 2026-04-04 20:43:28 [INFO] [TRAIN] Epoch 74 | loss=0.3394 | lr=2.68e-05 | 5.6 samples/sec | 107.3s
107
+ 2026-04-04 20:44:27 [INFO] [VAL] loss=0.3123 | mIoU=0.8017 | acc=0.9944
108
+ 2026-04-04 20:46:15 [INFO] [TRAIN] Epoch 75 | loss=0.3382 | lr=2.67e-05 | 5.6 samples/sec | 107.5s
109
+ 2026-04-04 20:48:02 [INFO] [TRAIN] Epoch 76 | loss=0.3580 | lr=2.66e-05 | 5.6 samples/sec | 107.4s
110
+ 2026-04-04 20:49:49 [INFO] [TRAIN] Epoch 77 | loss=0.3495 | lr=2.65e-05 | 5.6 samples/sec | 107.4s
111
+ 2026-04-04 20:51:37 [INFO] [TRAIN] Epoch 78 | loss=0.3324 | lr=2.64e-05 | 5.6 samples/sec | 107.4s
112
+ 2026-04-04 20:53:23 [INFO] [TRAIN] Epoch 79 | loss=0.3520 | lr=2.63e-05 | 5.6 samples/sec | 106.0s
113
+ 2026-04-04 20:54:23 [INFO] [VAL] loss=0.2920 | mIoU=0.8213 | acc=0.9947
114
+ 2026-04-04 20:56:12 [INFO] [TRAIN] Epoch 80 | loss=0.3440 | lr=2.62e-05 | 5.6 samples/sec | 107.2s
115
+ 2026-04-04 20:57:59 [INFO] [TRAIN] Epoch 81 | loss=0.3260 | lr=2.61e-05 | 5.6 samples/sec | 107.4s
116
+ 2026-04-04 20:59:45 [INFO] [TRAIN] Epoch 82 | loss=0.3456 | lr=2.60e-05 | 5.6 samples/sec | 105.8s
117
+ 2026-04-04 21:01:32 [INFO] [TRAIN] Epoch 83 | loss=0.3306 | lr=2.59e-05 | 5.6 samples/sec | 107.3s
118
+ 2026-04-04 21:03:20 [INFO] [TRAIN] Epoch 84 | loss=0.3474 | lr=2.58e-05 | 5.6 samples/sec | 107.4s
119
+ 2026-04-04 21:04:20 [INFO] [VAL] loss=0.3302 | mIoU=0.8107 | acc=0.9945
120
+ 2026-04-04 21:06:07 [INFO] [TRAIN] Epoch 85 | loss=0.3694 | lr=2.56e-05 | 5.6 samples/sec | 107.5s
121
+ 2026-04-04 21:07:53 [INFO] [TRAIN] Epoch 86 | loss=0.3284 | lr=2.55e-05 | 5.6 samples/sec | 105.7s
122
+ 2026-04-04 21:09:41 [INFO] [TRAIN] Epoch 87 | loss=0.3368 | lr=2.54e-05 | 5.6 samples/sec | 107.5s
123
+ 2026-04-04 21:11:28 [INFO] [TRAIN] Epoch 88 | loss=0.3731 | lr=2.53e-05 | 5.6 samples/sec | 107.4s
124
+ 2026-04-04 21:13:15 [INFO] [TRAIN] Epoch 89 | loss=0.3472 | lr=2.52e-05 | 5.6 samples/sec | 107.3s
125
+ 2026-04-04 21:14:15 [INFO] [VAL] loss=0.3009 | mIoU=0.8175 | acc=0.9947
126
+ 2026-04-04 21:16:04 [INFO] [TRAIN] Epoch 90 | loss=0.3592 | lr=2.50e-05 | 5.6 samples/sec | 107.4s
127
+ 2026-04-04 21:17:51 [INFO] [TRAIN] Epoch 91 | loss=0.3440 | lr=2.49e-05 | 5.6 samples/sec | 107.5s
128
+ 2026-04-04 21:19:39 [INFO] [TRAIN] Epoch 92 | loss=0.3402 | lr=2.48e-05 | 5.6 samples/sec | 107.1s
129
+ 2026-04-04 21:21:26 [INFO] [TRAIN] Epoch 93 | loss=0.3360 | lr=2.47e-05 | 5.6 samples/sec | 107.5s
130
+ 2026-04-04 21:23:11 [INFO] [TRAIN] Epoch 94 | loss=0.3366 | lr=2.45e-05 | 5.7 samples/sec | 105.2s
131
+ 2026-04-04 21:24:11 [INFO] [VAL] loss=0.3015 | mIoU=0.8186 | acc=0.9949
132
+ 2026-04-04 21:25:59 [INFO] [TRAIN] Epoch 95 | loss=0.3541 | lr=2.44e-05 | 5.5 samples/sec | 107.7s
133
+ 2026-04-04 21:27:46 [INFO] [TRAIN] Epoch 96 | loss=0.3269 | lr=2.43e-05 | 5.6 samples/sec | 107.4s
134
+ 2026-04-04 21:29:33 [INFO] [TRAIN] Epoch 97 | loss=0.3382 | lr=2.41e-05 | 5.6 samples/sec | 106.8s
135
+ 2026-04-04 21:31:19 [INFO] [TRAIN] Epoch 98 | loss=0.3739 | lr=2.40e-05 | 5.6 samples/sec | 106.2s
136
+ 2026-04-04 21:33:07 [INFO] [TRAIN] Epoch 99 | loss=0.3224 | lr=2.39e-05 | 5.6 samples/sec | 107.5s
137
+ 2026-04-04 21:34:07 [INFO] [VAL] loss=0.3512 | mIoU=0.8211 | acc=0.9951
138
+ 2026-04-04 21:35:56 [INFO] [TRAIN] Epoch 100 | loss=0.3686 | lr=2.37e-05 | 5.6 samples/sec | 107.3s
139
+ 2026-04-04 21:37:42 [INFO] [TRAIN] Epoch 101 | loss=0.3476 | lr=2.36e-05 | 5.6 samples/sec | 105.7s
140
+ 2026-04-04 21:39:29 [INFO] [TRAIN] Epoch 102 | loss=0.3576 | lr=2.35e-05 | 5.5 samples/sec | 107.6s
141
+ 2026-04-04 21:41:16 [INFO] [TRAIN] Epoch 103 | loss=0.3690 | lr=2.33e-05 | 5.6 samples/sec | 107.0s
142
+ 2026-04-04 21:43:04 [INFO] [TRAIN] Epoch 104 | loss=0.3231 | lr=2.32e-05 | 5.5 samples/sec | 108.2s
143
+ 2026-04-04 21:44:05 [INFO] [VAL] loss=0.3054 | mIoU=0.8199 | acc=0.9949
144
+ 2026-04-04 21:45:18 [INFO] [TRAIN] Epoch 105 | loss=0.3606 | lr=2.31e-05 | 8.2 samples/sec | 73.2s
145
+ 2026-04-04 21:46:24 [INFO] [TRAIN] Epoch 106 | loss=0.3279 | lr=2.29e-05 | 9.1 samples/sec | 65.9s
146
+ 2026-04-04 21:47:30 [INFO] [TRAIN] Epoch 107 | loss=0.3499 | lr=2.28e-05 | 9.0 samples/sec | 66.6s
147
+ 2026-04-04 21:48:37 [INFO] [TRAIN] Epoch 108 | loss=0.3284 | lr=2.26e-05 | 9.0 samples/sec | 66.6s
148
+ 2026-04-04 21:49:43 [INFO] [TRAIN] Epoch 109 | loss=0.3740 | lr=2.25e-05 | 9.1 samples/sec | 65.9s
149
+ 2026-04-04 21:50:40 [INFO] [VAL] loss=0.4609 | mIoU=0.8229 | acc=0.9951
150
+ 2026-04-04 21:51:47 [INFO] [TRAIN] Epoch 110 | loss=0.3357 | lr=2.24e-05 | 9.1 samples/sec | 65.7s
151
+ 2026-04-04 21:52:53 [INFO] [TRAIN] Epoch 111 | loss=0.3862 | lr=2.22e-05 | 9.1 samples/sec | 65.5s
152
+ 2026-04-04 21:53:58 [INFO] [TRAIN] Epoch 112 | loss=0.3450 | lr=2.21e-05 | 9.1 samples/sec | 65.4s
153
+ 2026-04-04 21:55:04 [INFO] [TRAIN] Epoch 113 | loss=0.3304 | lr=2.19e-05 | 9.1 samples/sec | 65.7s
154
+ 2026-04-04 21:56:09 [INFO] [TRAIN] Epoch 114 | loss=0.3299 | lr=2.18e-05 | 9.1 samples/sec | 65.3s
155
+ 2026-04-04 21:57:07 [INFO] [VAL] loss=0.3259 | mIoU=0.7914 | acc=0.9942
156
+ 2026-04-04 21:58:12 [INFO] [TRAIN] Epoch 115 | loss=0.3735 | lr=2.16e-05 | 9.1 samples/sec | 65.7s
157
+ 2026-04-04 21:59:18 [INFO] [TRAIN] Epoch 116 | loss=0.3723 | lr=2.15e-05 | 9.0 samples/sec | 66.0s
158
+ 2026-04-04 22:00:24 [INFO] [TRAIN] Epoch 117 | loss=0.3577 | lr=2.13e-05 | 9.1 samples/sec | 65.5s
159
+ 2026-04-04 22:01:29 [INFO] [TRAIN] Epoch 118 | loss=0.3528 | lr=2.12e-05 | 9.1 samples/sec | 65.4s
160
+ 2026-04-04 22:02:35 [INFO] [TRAIN] Epoch 119 | loss=0.3899 | lr=2.10e-05 | 9.1 samples/sec | 65.3s
161
+ 2026-04-04 22:03:32 [INFO] [VAL] loss=0.3588 | mIoU=0.8040 | acc=0.9944
162
+ 2026-04-04 22:04:39 [INFO] [TRAIN] Epoch 120 | loss=0.3598 | lr=2.09e-05 | 9.1 samples/sec | 65.5s
163
+ 2026-04-04 22:05:45 [INFO] [TRAIN] Epoch 121 | loss=0.3647 | lr=2.07e-05 | 9.1 samples/sec | 65.8s
164
+ 2026-04-04 22:06:52 [INFO] [TRAIN] Epoch 122 | loss=0.3707 | lr=2.06e-05 | 8.9 samples/sec | 66.8s
165
+ 2026-04-04 22:07:57 [INFO] [TRAIN] Epoch 123 | loss=0.3537 | lr=2.04e-05 | 9.1 samples/sec | 65.3s
166
+ 2026-04-04 22:09:02 [INFO] [TRAIN] Epoch 124 | loss=0.3776 | lr=2.03e-05 | 9.2 samples/sec | 64.8s
167
+ 2026-04-04 22:09:58 [INFO] [VAL] loss=0.3780 | mIoU=0.8239 | acc=0.9950
168
+ 2026-04-04 22:11:03 [INFO] [TRAIN] Epoch 125 | loss=0.3493 | lr=2.01e-05 | 9.1 samples/sec | 65.5s
169
+ 2026-04-04 22:12:09 [INFO] [TRAIN] Epoch 126 | loss=0.3686 | lr=1.99e-05 | 9.1 samples/sec | 65.9s
170
+ 2026-04-04 22:13:15 [INFO] [TRAIN] Epoch 127 | loss=0.3469 | lr=1.98e-05 | 9.1 samples/sec | 65.4s
171
+ 2026-04-04 22:14:19 [INFO] [TRAIN] Epoch 128 | loss=0.3737 | lr=1.96e-05 | 9.2 samples/sec | 64.8s
172
+ 2026-04-04 22:15:24 [INFO] [TRAIN] Epoch 129 | loss=0.3530 | lr=1.95e-05 | 9.2 samples/sec | 65.0s
173
+ 2026-04-04 22:16:21 [INFO] [VAL] loss=0.3213 | mIoU=0.8091 | acc=0.9945
174
+ 2026-04-04 22:17:29 [INFO] [TRAIN] Epoch 130 | loss=0.3784 | lr=1.93e-05 | 9.1 samples/sec | 65.9s
175
+ 2026-04-04 22:18:35 [INFO] [TRAIN] Epoch 131 | loss=0.3803 | lr=1.92e-05 | 9.1 samples/sec | 65.7s
176
+ 2026-04-04 22:19:41 [INFO] [TRAIN] Epoch 132 | loss=0.3454 | lr=1.90e-05 | 9.0 samples/sec | 66.2s
177
+ 2026-04-04 22:20:47 [INFO] [TRAIN] Epoch 133 | loss=0.3781 | lr=1.88e-05 | 9.1 samples/sec | 65.7s
178
+ 2026-04-04 22:21:52 [INFO] [TRAIN] Epoch 134 | loss=0.3620 | lr=1.87e-05 | 9.2 samples/sec | 65.1s
179
+ 2026-04-04 22:22:50 [INFO] [VAL] loss=0.4052 | mIoU=0.8058 | acc=0.9941
180
+ 2026-04-04 22:23:55 [INFO] [TRAIN] Epoch 135 | loss=0.3648 | lr=1.85e-05 | 9.2 samples/sec | 65.2s
181
+ 2026-04-04 22:25:00 [INFO] [TRAIN] Epoch 136 | loss=0.3550 | lr=1.84e-05 | 9.1 samples/sec | 65.7s
182
+ 2026-04-04 22:26:06 [INFO] [TRAIN] Epoch 137 | loss=0.3603 | lr=1.82e-05 | 9.1 samples/sec | 65.7s
183
+ 2026-04-04 22:27:12 [INFO] [TRAIN] Epoch 138 | loss=0.3597 | lr=1.80e-05 | 9.1 samples/sec | 65.4s
184
+ 2026-04-04 22:28:18 [INFO] [TRAIN] Epoch 139 | loss=0.3683 | lr=1.79e-05 | 9.0 samples/sec | 66.6s
185
+ 2026-04-04 22:29:16 [INFO] [VAL] loss=0.4628 | mIoU=0.8207 | acc=0.9948
186
+ 2026-04-04 22:30:24 [INFO] [TRAIN] Epoch 140 | loss=0.4030 | lr=1.77e-05 | 9.0 samples/sec | 66.0s
187
+ 2026-04-04 22:31:30 [INFO] [TRAIN] Epoch 141 | loss=0.3558 | lr=1.76e-05 | 9.0 samples/sec | 66.2s
188
+ 2026-04-04 22:32:36 [INFO] [TRAIN] Epoch 142 | loss=0.3527 | lr=1.74e-05 | 9.2 samples/sec | 65.2s
189
+ 2026-04-04 22:33:41 [INFO] [TRAIN] Epoch 143 | loss=0.3628 | lr=1.72e-05 | 9.2 samples/sec | 65.1s
190
+ 2026-04-04 22:34:46 [INFO] [TRAIN] Epoch 144 | loss=0.3438 | lr=1.71e-05 | 9.1 samples/sec | 65.4s
191
+ 2026-04-04 22:35:42 [INFO] [VAL] loss=0.3600 | mIoU=0.8209 | acc=0.9948
192
+ 2026-04-04 22:36:48 [INFO] [TRAIN] Epoch 145 | loss=0.3425 | lr=1.69e-05 | 9.1 samples/sec | 65.9s
193
+ 2026-04-04 22:37:55 [INFO] [TRAIN] Epoch 146 | loss=0.3770 | lr=1.67e-05 | 9.0 samples/sec | 66.3s
194
+ 2026-04-04 22:39:01 [INFO] [TRAIN] Epoch 147 | loss=0.3700 | lr=1.66e-05 | 8.9 samples/sec | 66.7s
195
+ 2026-04-04 22:40:06 [INFO] [TRAIN] Epoch 148 | loss=0.3540 | lr=1.64e-05 | 9.2 samples/sec | 64.8s
196
+ 2026-04-04 22:41:12 [INFO] [TRAIN] Epoch 149 | loss=0.3555 | lr=1.62e-05 | 9.1 samples/sec | 65.5s
197
+ 2026-04-04 22:42:08 [INFO] [VAL] loss=0.7071 | mIoU=0.8193 | acc=0.9949
198
+ 2026-04-04 22:43:16 [INFO] [TRAIN] Epoch 150 | loss=0.3751 | lr=1.61e-05 | 9.1 samples/sec | 65.5s
199
+ 2026-04-04 22:44:21 [INFO] [TRAIN] Epoch 151 | loss=0.3566 | lr=1.59e-05 | 9.1 samples/sec | 65.6s
200
+ 2026-04-04 22:45:27 [INFO] [TRAIN] Epoch 152 | loss=0.3595 | lr=1.57e-05 | 9.0 samples/sec | 66.1s
201
+ 2026-04-04 22:46:34 [INFO] [TRAIN] Epoch 153 | loss=0.3747 | lr=1.56e-05 | 9.0 samples/sec | 66.4s
202
+ 2026-04-04 22:47:40 [INFO] [TRAIN] Epoch 154 | loss=0.3629 | lr=1.54e-05 | 9.0 samples/sec | 66.2s
203
+ 2026-04-04 22:48:39 [INFO] [VAL] loss=0.4464 | mIoU=0.8283 | acc=0.9949
204
+ 2026-04-04 22:49:45 [INFO] [TRAIN] Epoch 155 | loss=0.3522 | lr=1.52e-05 | 9.1 samples/sec | 65.6s
205
+ 2026-04-04 22:50:50 [INFO] [TRAIN] Epoch 156 | loss=0.3489 | lr=1.51e-05 | 9.2 samples/sec | 65.2s
206
+ 2026-04-04 22:51:55 [INFO] [TRAIN] Epoch 157 | loss=0.3512 | lr=1.49e-05 | 9.3 samples/sec | 64.5s
207
+ 2026-04-04 22:52:59 [INFO] [TRAIN] Epoch 158 | loss=0.3461 | lr=1.48e-05 | 9.2 samples/sec | 64.6s
208
+ 2026-04-04 22:54:04 [INFO] [TRAIN] Epoch 159 | loss=0.3459 | lr=1.46e-05 | 9.2 samples/sec | 65.0s
209
+ 2026-04-04 22:55:02 [INFO] [VAL] loss=0.3438 | mIoU=0.8179 | acc=0.9947
210
+ 2026-04-04 22:56:10 [INFO] [TRAIN] Epoch 160 | loss=0.3343 | lr=1.44e-05 | 9.1 samples/sec | 65.5s
211
+ 2026-04-04 22:57:15 [INFO] [TRAIN] Epoch 161 | loss=0.3739 | lr=1.43e-05 | 9.1 samples/sec | 65.3s
212
+ 2026-04-04 22:58:21 [INFO] [TRAIN] Epoch 162 | loss=0.3540 | lr=1.41e-05 | 9.0 samples/sec | 66.2s
213
+ 2026-04-04 22:59:27 [INFO] [TRAIN] Epoch 163 | loss=0.3388 | lr=1.39e-05 | 9.1 samples/sec | 65.6s
214
+ 2026-04-04 23:00:33 [INFO] [TRAIN] Epoch 164 | loss=0.3346 | lr=1.38e-05 | 9.1 samples/sec | 65.9s
215
+ 2026-04-04 23:01:31 [INFO] [VAL] loss=0.3047 | mIoU=0.8165 | acc=0.9948
216
+ 2026-04-04 23:02:36 [INFO] [TRAIN] Epoch 165 | loss=0.3467 | lr=1.36e-05 | 9.1 samples/sec | 65.3s
217
+ 2026-04-04 23:03:42 [INFO] [TRAIN] Epoch 166 | loss=0.3733 | lr=1.34e-05 | 9.1 samples/sec | 65.5s
218
+ 2026-04-04 23:04:47 [INFO] [TRAIN] Epoch 167 | loss=0.3588 | lr=1.33e-05 | 9.2 samples/sec | 65.0s
219
+ 2026-04-04 23:05:52 [INFO] [TRAIN] Epoch 168 | loss=0.3984 | lr=1.31e-05 | 9.2 samples/sec | 65.2s
220
+ 2026-04-04 23:06:57 [INFO] [TRAIN] Epoch 169 | loss=0.3456 | lr=1.29e-05 | 9.1 samples/sec | 65.4s
221
+ 2026-04-04 23:07:54 [INFO] [VAL] loss=0.3806 | mIoU=0.8216 | acc=0.9949
222
+ 2026-04-04 23:09:01 [INFO] [TRAIN] Epoch 170 | loss=0.3422 | lr=1.28e-05 | 9.1 samples/sec | 65.5s
223
+ 2026-04-04 23:10:07 [INFO] [TRAIN] Epoch 171 | loss=0.3503 | lr=1.26e-05 | 9.1 samples/sec | 65.3s
224
+ 2026-04-04 23:11:12 [INFO] [TRAIN] Epoch 172 | loss=0.3293 | lr=1.24e-05 | 9.2 samples/sec | 65.1s
225
+ 2026-04-04 23:12:17 [INFO] [TRAIN] Epoch 173 | loss=0.3340 | lr=1.23e-05 | 9.1 samples/sec | 65.4s
226
+ 2026-04-04 23:13:23 [INFO] [TRAIN] Epoch 174 | loss=0.3549 | lr=1.21e-05 | 9.1 samples/sec | 65.6s
227
+ 2026-04-04 23:14:20 [INFO] [VAL] loss=0.3406 | mIoU=0.8156 | acc=0.9947
228
+ 2026-04-04 23:15:25 [INFO] [TRAIN] Epoch 175 | loss=0.3565 | lr=1.20e-05 | 9.1 samples/sec | 65.3s
229
+ 2026-04-04 23:16:31 [INFO] [TRAIN] Epoch 176 | loss=0.3544 | lr=1.18e-05 | 9.1 samples/sec | 65.8s
230
+ 2026-04-04 23:17:37 [INFO] [TRAIN] Epoch 177 | loss=0.3347 | lr=1.16e-05 | 9.1 samples/sec | 65.6s
231
+ 2026-04-04 23:18:42 [INFO] [TRAIN] Epoch 178 | loss=0.3474 | lr=1.15e-05 | 9.1 samples/sec | 65.7s
232
+ 2026-04-04 23:19:48 [INFO] [TRAIN] Epoch 179 | loss=0.3521 | lr=1.13e-05 | 9.1 samples/sec | 65.5s
233
+ 2026-04-04 23:20:45 [INFO] [VAL] loss=0.3669 | mIoU=0.8145 | acc=0.9948
234
+ 2026-04-04 23:21:53 [INFO] [TRAIN] Epoch 180 | loss=0.3570 | lr=1.12e-05 | 9.1 samples/sec | 65.6s
235
+ 2026-04-04 23:22:58 [INFO] [TRAIN] Epoch 181 | loss=0.3603 | lr=1.10e-05 | 9.1 samples/sec | 65.6s
236
+ 2026-04-04 23:24:04 [INFO] [TRAIN] Epoch 182 | loss=0.3703 | lr=1.08e-05 | 9.1 samples/sec | 65.5s
237
+ 2026-04-04 23:25:10 [INFO] [TRAIN] Epoch 183 | loss=0.3690 | lr=1.07e-05 | 9.1 samples/sec | 65.9s
238
+ 2026-04-04 23:26:15 [INFO] [TRAIN] Epoch 184 | loss=0.3588 | lr=1.05e-05 | 9.2 samples/sec | 65.1s
239
+ 2026-04-04 23:27:11 [INFO] [VAL] loss=0.4010 | mIoU=0.8222 | acc=0.9949
240
+ 2026-04-04 23:28:17 [INFO] [TRAIN] Epoch 185 | loss=0.3585 | lr=1.04e-05 | 9.1 samples/sec | 65.9s
241
+ 2026-04-04 23:29:23 [INFO] [TRAIN] Epoch 186 | loss=0.3601 | lr=1.02e-05 | 9.1 samples/sec | 65.9s
242
+ 2026-04-04 23:30:29 [INFO] [TRAIN] Epoch 187 | loss=0.3405 | lr=1.01e-05 | 9.0 samples/sec | 66.2s
243
+ 2026-04-04 23:31:35 [INFO] [TRAIN] Epoch 188 | loss=0.3463 | lr=1.00e-05 | 9.1 samples/sec | 65.6s
244
+ 2026-04-04 23:32:40 [INFO] [TRAIN] Epoch 189 | loss=0.3534 | lr=1.00e-05 | 9.1 samples/sec | 65.4s
245
+ 2026-04-04 23:33:39 [INFO] [VAL] loss=0.3704 | mIoU=0.8107 | acc=0.9947
246
+ 2026-04-04 23:34:46 [INFO] [TRAIN] Epoch 190 | loss=0.3579 | lr=1.00e-05 | 9.2 samples/sec | 65.1s
247
+ 2026-04-04 23:35:52 [INFO] [TRAIN] Epoch 191 | loss=0.3678 | lr=1.00e-05 | 9.1 samples/sec | 65.9s
248
+ 2026-04-04 23:36:57 [INFO] [TRAIN] Epoch 192 | loss=0.3588 | lr=1.00e-05 | 9.1 samples/sec | 65.4s
249
+ 2026-04-04 23:38:02 [INFO] [TRAIN] Epoch 193 | loss=0.3825 | lr=1.00e-05 | 9.2 samples/sec | 65.1s
250
+ 2026-04-04 23:39:08 [INFO] [TRAIN] Epoch 194 | loss=0.3904 | lr=1.00e-05 | 9.1 samples/sec | 65.3s
251
+ 2026-04-04 23:40:05 [INFO] [VAL] loss=0.5035 | mIoU=0.8233 | acc=0.9949
252
+ 2026-04-04 23:41:10 [INFO] [TRAIN] Epoch 195 | loss=0.3946 | lr=1.00e-05 | 9.1 samples/sec | 65.3s
253
+ 2026-04-04 23:42:15 [INFO] [TRAIN] Epoch 196 | loss=0.3832 | lr=1.00e-05 | 9.1 samples/sec | 65.3s
254
+ 2026-04-04 23:43:21 [INFO] [TRAIN] Epoch 197 | loss=0.3531 | lr=1.00e-05 | 9.1 samples/sec | 65.5s
255
+ 2026-04-04 23:44:25 [INFO] [TRAIN] Epoch 198 | loss=0.3744 | lr=1.00e-05 | 9.2 samples/sec | 64.8s
256
+ 2026-04-04 23:45:30 [INFO] [TRAIN] Epoch 199 | loss=0.4001 | lr=1.00e-05 | 9.2 samples/sec | 64.7s
257
+ 2026-04-04 23:46:27 [INFO] [VAL] loss=0.4437 | mIoU=0.8110 | acc=0.9947
258
+ 2026-04-04 23:47:35 [INFO] [TRAIN] Epoch 200 | loss=0.4056 | lr=1.00e-05 | 9.1 samples/sec | 65.5s
259
+ 2026-04-04 23:48:40 [INFO] [TRAIN] Epoch 201 | loss=0.3712 | lr=1.00e-05 | 9.2 samples/sec | 65.0s
260
+ 2026-04-04 23:49:46 [INFO] [TRAIN] Epoch 202 | loss=0.3711 | lr=1.00e-05 | 9.1 samples/sec | 65.8s
261
+ 2026-04-04 23:50:51 [INFO] [TRAIN] Epoch 203 | loss=0.3867 | lr=1.00e-05 | 9.2 samples/sec | 65.1s
262
+ 2026-04-04 23:51:56 [INFO] [TRAIN] Epoch 204 | loss=0.3729 | lr=1.00e-05 | 9.2 samples/sec | 65.1s
263
+ 2026-04-04 23:52:52 [INFO] [VAL] loss=0.4379 | mIoU=0.8274 | acc=0.9951
264
+ 2026-04-04 23:53:58 [INFO] [TRAIN] Epoch 205 | loss=0.3837 | lr=1.00e-05 | 9.2 samples/sec | 65.1s
265
+ 2026-04-04 23:55:02 [INFO] [TRAIN] Epoch 206 | loss=0.3769 | lr=1.00e-05 | 9.2 samples/sec | 64.9s
266
+ 2026-04-04 23:56:08 [INFO] [TRAIN] Epoch 207 | loss=0.4149 | lr=1.00e-05 | 9.2 samples/sec | 65.2s
267
+ 2026-04-04 23:57:13 [INFO] [TRAIN] Epoch 208 | loss=0.3849 | lr=1.00e-05 | 9.1 samples/sec | 65.4s
268
+ 2026-04-04 23:58:19 [INFO] [TRAIN] Epoch 209 | loss=0.3821 | lr=1.00e-05 | 9.1 samples/sec | 65.7s
269
+ 2026-04-04 23:59:15 [INFO] [VAL] loss=0.4603 | mIoU=0.8248 | acc=0.9950
270
+ 2026-04-04 23:59:17 [INFO] [EARLY STOP] No improvement for 30 epochs
271
+ 2026-04-04 23:59:17 [INFO] [DONE] Best mIoU=0.8358
272
+ 2026-04-04 23:59:17 [INFO] [CKPT] Best: /mnt/artifacts-datai/checkpoints/DEF-rtfdnet/best.pth