Dhruv1000's picture
Organize complete final models, all ablations, logs and checkpoints with visual guides
954bb80 verified
Raw History Blame Contribute Delete
28.5 kB
/usr/lib/python3/dist-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.4
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
[FT-v2] train=76 patches (72 unique) val=76 patches (71 unique)
[FT-v2] world=1 batch=2 patches x 2 crops -> 76 steps/epoch/rank
[FT-v2] crop_size=32 (fixed diagonal) aug=dihedral+temporal+spectral
[FT-v2] base_lr=1e-05 enc_mult=0.5 wd=0.02 ls=0.05 ema=0.999 warmup_ep=0.0 min_lr_ratio=0.01 patience=0
[FT-v2] seed=3407 epochs=20 checkpoints -> /home/ubuntu/tsr/research_round/checkpoints/sam
[FT-v2] loading encoder: /home/ubuntu/tsr/latest.utae.tar
[INIT] fine-tuned weights from /home/ubuntu/tsr/checkpoints/finetune_v2/32X32_exp3_low_lr_refinement/model_best_raw.tar; optimizer/schedule start fresh
/home/ubuntu/tsr/finetune_research.py:465: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
scaler = GradScaler()
[FT-v2] params=6.02M (encoder 4.00M)
[FT-v2] starting fresh (--resume=none)
[HYPOTHESIS] temporal_aug=True mask_ce_invalid=True boundary_weight=0.5 ema_decay=0.999
/home/ubuntu/tsr/finetune_research.py:280: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with autocast():
[INITIAL] mIoU=0.493545
[ENCODER] frozen=False
/home/ubuntu/tsr/finetune_research.py:475: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with autocast():
ep 0 batch 0/38 step 2/76 loss=1.2565 lr=1.00e-05 t=3s
ep 0 batch 10/38 step 22/76 loss=0.9800 lr=1.00e-05 t=21s
ep 0 batch 20/38 step 42/76 loss=0.9509 lr=9.98e-06 t=39s
ep 0 batch 30/38 step 62/76 loss=1.0833 lr=9.96e-06 t=55s
[ep 0 DONE] loss=0.9818 raw_mIoU=0.4943 ema_mIoU=0.4875 best_raw=0.4943@0 best_ema=0.4875@0 patience=0/0 t=85s
[ep 0 RAW] mIoU=0.4943 OA=0.7704 mF1=0.6113 Kappa=0.7448
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7563 0.8612
2 Soft winter wheat 0.8067 0.8930
3 Corn 0.7593 0.8632
4 Winter barley 0.4519 0.6225
5 Winter rapeseed 0.9092 0.9525
6 Spring barley 0.5049 0.6710
7 Sunflower 0.5603 0.7182
8 Grapevine 0.7170 0.8352
9 Beet 0.3533 0.5221
10 Winter triticale 0.1512 0.2627
11 Winter durum wheat 0.8476 0.9175
12 Fruits, vegetables, flowers 0.1781 0.3023
13 Potatoes 0.0971 0.1771 *rare*
14 Leguminous fodder 0.3353 0.5022
15 Soybeans 0.7456 0.8543
16 Orchard 0.4978 0.6647
17 Mixed cereal 0.0278 0.0541 *rare*
18 Sorghum 0.1974 0.3297
[ep 0 EMA] mIoU=0.4875 OA=0.7641 mF1=0.6063 Kappa=0.7383
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7653 0.8671
2 Soft winter wheat 0.7984 0.8879
3 Corn 0.7464 0.8548
4 Winter barley 0.4418 0.6128
5 Winter rapeseed 0.9051 0.9502
6 Spring barley 0.4855 0.6536
7 Sunflower 0.5178 0.6823
8 Grapevine 0.7200 0.8372
9 Beet 0.3578 0.5270
10 Winter triticale 0.1546 0.2678
11 Winter durum wheat 0.8438 0.9153
12 Fruits, vegetables, flowers 0.2002 0.3337
13 Potatoes 0.1038 0.1881
14 Leguminous fodder 0.3298 0.4960
15 Soybeans 0.7171 0.8353
16 Orchard 0.4830 0.6513
17 Mixed cereal 0.0216 0.0422 *rare*
18 Sorghum 0.1837 0.3104
[ENCODER] frozen=False
ep 1 batch 0/38 step 2/76 loss=1.0369 lr=9.94e-06 t=2s
ep 1 batch 10/38 step 22/76 loss=1.1736 lr=9.90e-06 t=20s
[SAM AMP] skipped overflowing first pass; scale= 16384.0
ep 1 batch 20/38 step 41/76 loss=1.3125 lr=9.86e-06 t=36s
ep 1 batch 30/38 step 61/76 loss=0.9776 lr=9.81e-06 t=54s
[ep 1 DONE] loss=0.9925 raw_mIoU=0.4915 ema_mIoU=0.4886 best_raw=0.4943@0 best_ema=0.4886@1 patience=1/0 t=83s
[ep 1 RAW] mIoU=0.4915 OA=0.7655 mF1=0.6081 Kappa=0.7399
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7715 0.8710
2 Soft winter wheat 0.7955 0.8861
3 Corn 0.7386 0.8496
4 Winter barley 0.4159 0.5875
5 Winter rapeseed 0.9064 0.9509
6 Spring barley 0.4726 0.6418
7 Sunflower 0.5691 0.7254
8 Grapevine 0.7361 0.8480
9 Beet 0.3544 0.5234
10 Winter triticale 0.1434 0.2508
11 Winter durum wheat 0.8546 0.9216
12 Fruits, vegetables, flowers 0.1941 0.3251
13 Potatoes 0.0970 0.1769 *rare*
14 Leguminous fodder 0.3170 0.4813
15 Soybeans 0.7290 0.8433
16 Orchard 0.5450 0.7055
17 Mixed cereal 0.0271 0.0528 *rare*
18 Sorghum 0.1798 0.3048
[ep 1 EMA] mIoU=0.4886 OA=0.7598 mF1=0.6058 Kappa=0.7339
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7629 0.8655
2 Soft winter wheat 0.7994 0.8885
3 Corn 0.7165 0.8349
4 Winter barley 0.4477 0.6185
5 Winter rapeseed 0.9052 0.9502
6 Spring barley 0.5338 0.6960
7 Sunflower 0.5746 0.7298
8 Grapevine 0.6959 0.8207
9 Beet 0.3546 0.5235
10 Winter triticale 0.1527 0.2650
11 Winter durum wheat 0.8385 0.9121
12 Fruits, vegetables, flowers 0.1745 0.2971
13 Potatoes 0.0998 0.1815 *rare*
14 Leguminous fodder 0.3014 0.4631
15 Soybeans 0.7533 0.8593
16 Orchard 0.4846 0.6528
17 Mixed cereal 0.0241 0.0470 *rare*
18 Sorghum 0.1755 0.2986
[ENCODER] frozen=False
ep 2 batch 0/38 step 2/76 loss=1.0892 lr=9.76e-06 t=2s
ep 2 batch 10/38 step 22/76 loss=0.8419 lr=9.69e-06 t=21s
ep 2 batch 20/38 step 42/76 loss=1.0137 lr=9.62e-06 t=40s
ep 2 batch 30/38 step 62/76 loss=0.8333 lr=9.53e-06 t=55s
[ep 2 DONE] loss=1.0698 raw_mIoU=0.4857 ema_mIoU=0.4930 best_raw=0.4943@0 best_ema=0.4930@2 patience=2/0 t=83s
[ep 2 RAW] mIoU=0.4857 OA=0.7617 mF1=0.6049 Kappa=0.7367
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7704 0.8703
2 Soft winter wheat 0.8081 0.8938
3 Corn 0.7414 0.8515
4 Winter barley 0.4145 0.5861
5 Winter rapeseed 0.9001 0.9474
6 Spring barley 0.4441 0.6151
7 Sunflower 0.6002 0.7502
8 Grapevine 0.6830 0.8117
9 Beet 0.3561 0.5251
10 Winter triticale 0.1465 0.2555
11 Winter durum wheat 0.8466 0.9169
12 Fruits, vegetables, flowers 0.2058 0.3413
13 Potatoes 0.1022 0.1854
14 Leguminous fodder 0.3236 0.4889
15 Soybeans 0.7231 0.8393
16 Orchard 0.4610 0.6310
17 Mixed cereal 0.0436 0.0836 *rare*
18 Sorghum 0.1731 0.2952
[ep 2 EMA] mIoU=0.4930 OA=0.7655 mF1=0.6109 Kappa=0.7403
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7692 0.8695
2 Soft winter wheat 0.8070 0.8932
3 Corn 0.7498 0.8570
4 Winter barley 0.4293 0.6007
5 Winter rapeseed 0.9021 0.9485
6 Spring barley 0.5272 0.6904
7 Sunflower 0.5670 0.7237
8 Grapevine 0.6903 0.8168
9 Beet 0.3524 0.5211
10 Winter triticale 0.1564 0.2704
11 Winter durum wheat 0.8358 0.9105
12 Fruits, vegetables, flowers 0.1943 0.3253
13 Potatoes 0.1016 0.1844
14 Leguminous fodder 0.3159 0.4801
15 Soybeans 0.7651 0.8669
16 Orchard 0.4879 0.6558
17 Mixed cereal 0.0279 0.0543 *rare*
18 Sorghum 0.1957 0.3274
[ENCODER] frozen=False
ep 3 batch 0/38 step 2/76 loss=0.8394 lr=9.46e-06 t=2s
ep 3 batch 10/38 step 22/76 loss=0.8932 lr=9.36e-06 t=20s
ep 3 batch 20/38 step 42/76 loss=0.8633 lr=9.26e-06 t=38s
ep 3 batch 30/38 step 62/76 loss=1.3055 lr=9.15e-06 t=56s
[ep 3 DONE] loss=0.9701 raw_mIoU=0.4807 ema_mIoU=0.4899 best_raw=0.4943@0 best_ema=0.4930@2 patience=3/0 t=85s
[ENCODER] frozen=False
ep 4 batch 0/38 step 2/76 loss=0.9411 lr=9.05e-06 t=2s
ep 4 batch 10/38 step 22/76 loss=1.1435 lr=8.93e-06 t=20s
ep 4 batch 20/38 step 42/76 loss=0.8485 lr=8.80e-06 t=38s
ep 4 batch 30/38 step 62/76 loss=1.0160 lr=8.66e-06 t=55s
[ep 4 DONE] loss=1.0175 raw_mIoU=0.4897 ema_mIoU=0.4923 best_raw=0.4943@0 best_ema=0.4930@2 patience=4/0 t=85s
[ENCODER] frozen=False
ep 5 batch 0/38 step 2/76 loss=0.9980 lr=8.55e-06 t=1s
ep 5 batch 10/38 step 22/76 loss=1.6623 lr=8.40e-06 t=19s
ep 5 batch 20/38 step 42/76 loss=0.8671 lr=8.25e-06 t=38s
ep 5 batch 30/38 step 62/76 loss=1.8051 lr=8.09e-06 t=56s
[ep 5 DONE] loss=0.9885 raw_mIoU=0.4724 ema_mIoU=0.4928 best_raw=0.4943@0 best_ema=0.4930@2 patience=5/0 t=85s
[ep 5 RAW] mIoU=0.4724 OA=0.7592 mF1=0.5887 Kappa=0.7326
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7747 0.8731
2 Soft winter wheat 0.7711 0.8707
3 Corn 0.7406 0.8510
4 Winter barley 0.3901 0.5613
5 Winter rapeseed 0.9024 0.9487
6 Spring barley 0.4282 0.5997
7 Sunflower 0.5747 0.7299
8 Grapevine 0.6580 0.7937
9 Beet 0.3596 0.5290
10 Winter triticale 0.0563 0.1065 *rare*
11 Winter durum wheat 0.8237 0.9033
12 Fruits, vegetables, flowers 0.1822 0.3082
13 Potatoes 0.1054 0.1907
14 Leguminous fodder 0.3215 0.4865
15 Soybeans 0.7485 0.8561
16 Orchard 0.4539 0.6244
17 Mixed cereal 0.0266 0.0519 *rare*
18 Sorghum 0.1853 0.3127
[ep 5 EMA] mIoU=0.4928 OA=0.7660 mF1=0.6111 Kappa=0.7407
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7706 0.8704
2 Soft winter wheat 0.8116 0.8960
3 Corn 0.7503 0.8573
4 Winter barley 0.4551 0.6255
5 Winter rapeseed 0.9000 0.9473
6 Spring barley 0.5112 0.6765
7 Sunflower 0.5739 0.7293
8 Grapevine 0.6821 0.8110
9 Beet 0.3581 0.5273
10 Winter triticale 0.1612 0.2777
11 Winter durum wheat 0.8309 0.9076
12 Fruits, vegetables, flowers 0.1866 0.3145
13 Potatoes 0.1096 0.1975
14 Leguminous fodder 0.3044 0.4667
15 Soybeans 0.7720 0.8713
16 Orchard 0.4671 0.6368
17 Mixed cereal 0.0336 0.0649 *rare*
18 Sorghum 0.1918 0.3219
[ENCODER] frozen=False
ep 6 batch 0/38 step 2/76 loss=1.0674 lr=7.96e-06 t=2s
ep 6 batch 10/38 step 22/76 loss=0.9435 lr=7.79e-06 t=18s
ep 6 batch 20/38 step 42/76 loss=1.1685 lr=7.62e-06 t=36s
ep 6 batch 30/38 step 62/76 loss=0.9837 lr=7.44e-06 t=54s
[ep 6 DONE] loss=1.1088 raw_mIoU=0.4922 ema_mIoU=0.4923 best_raw=0.4943@0 best_ema=0.4930@2 patience=6/0 t=84s
[ENCODER] frozen=False
ep 7 batch 0/38 step 2/76 loss=1.0351 lr=7.30e-06 t=2s
ep 7 batch 10/38 step 22/76 loss=1.1396 lr=7.11e-06 t=19s
ep 7 batch 20/38 step 42/76 loss=0.8706 lr=6.93e-06 t=37s
ep 7 batch 30/38 step 62/76 loss=0.9462 lr=6.73e-06 t=56s
[ep 7 DONE] loss=1.0131 raw_mIoU=0.4783 ema_mIoU=0.4928 best_raw=0.4943@0 best_ema=0.4930@2 patience=7/0 t=84s
[ENCODER] frozen=False
ep 8 batch 0/38 step 2/76 loss=1.1579 lr=6.58e-06 t=2s
ep 8 batch 10/38 step 22/76 loss=0.8332 lr=6.38e-06 t=19s
ep 8 batch 20/38 step 42/76 loss=0.8346 lr=6.19e-06 t=37s
ep 8 batch 30/38 step 62/76 loss=1.3588 lr=5.99e-06 t=54s
[ep 8 DONE] loss=1.0577 raw_mIoU=0.4911 ema_mIoU=0.4935 best_raw=0.4943@0 best_ema=0.4935@8 patience=8/0 t=84s
[ep 8 RAW] mIoU=0.4911 OA=0.7653 mF1=0.6092 Kappa=0.7397
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7693 0.8696
2 Soft winter wheat 0.8022 0.8902
3 Corn 0.7526 0.8588
4 Winter barley 0.4376 0.6088
5 Winter rapeseed 0.8908 0.9423
6 Spring barley 0.5370 0.6987
7 Sunflower 0.5621 0.7197
8 Grapevine 0.6866 0.8141
9 Beet 0.3614 0.5309
10 Winter triticale 0.1500 0.2608
11 Winter durum wheat 0.8429 0.9147
12 Fruits, vegetables, flowers 0.1984 0.3311
13 Potatoes 0.0992 0.1804 *rare*
14 Leguminous fodder 0.2874 0.4465
15 Soybeans 0.7640 0.8662
16 Orchard 0.4634 0.6333
17 Mixed cereal 0.0345 0.0667 *rare*
18 Sorghum 0.1999 0.3332
[ep 8 EMA] mIoU=0.4935 OA=0.7667 mF1=0.6107 Kappa=0.7412
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7647 0.8666
2 Soft winter wheat 0.8079 0.8937
3 Corn 0.7566 0.8615
4 Winter barley 0.4562 0.6266
5 Winter rapeseed 0.9029 0.9490
6 Spring barley 0.5286 0.6916
7 Sunflower 0.5721 0.7279
8 Grapevine 0.6793 0.8091
9 Beet 0.3578 0.5270
10 Winter triticale 0.1560 0.2699
11 Winter durum wheat 0.8357 0.9105
12 Fruits, vegetables, flowers 0.1858 0.3134
13 Potatoes 0.1056 0.1911
14 Leguminous fodder 0.2935 0.4538
15 Soybeans 0.7798 0.8763
16 Orchard 0.4776 0.6464
17 Mixed cereal 0.0248 0.0484 *rare*
18 Sorghum 0.1973 0.3296
[ENCODER] frozen=False
ep 9 batch 0/38 step 2/76 loss=0.7937 lr=5.82e-06 t=2s
ep 9 batch 10/38 step 22/76 loss=1.2420 lr=5.62e-06 t=20s
ep 9 batch 20/38 step 42/76 loss=1.2476 lr=5.42e-06 t=37s
ep 9 batch 30/38 step 62/76 loss=1.0501 lr=5.21e-06 t=55s
[ep 9 DONE] loss=1.1418 raw_mIoU=0.4859 ema_mIoU=0.4933 best_raw=0.4943@0 best_ema=0.4935@8 patience=9/0 t=84s
[ENCODER] frozen=False
ep 10 batch 0/38 step 2/76 loss=0.9952 lr=5.05e-06 t=1s
ep 10 batch 10/38 step 22/76 loss=0.8996 lr=4.85e-06 t=20s
ep 10 batch 20/38 step 42/76 loss=0.9801 lr=4.64e-06 t=38s
ep 10 batch 30/38 step 62/76 loss=1.2108 lr=4.44e-06 t=55s
[ep 10 DONE] loss=1.1027 raw_mIoU=0.5001 ema_mIoU=0.4947 best_raw=0.5001@10 best_ema=0.4947@10 patience=0/0 t=83s
[ep 10 RAW] mIoU=0.5001 OA=0.7696 mF1=0.6175 Kappa=0.7447
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7700 0.8700
2 Soft winter wheat 0.8006 0.8893
3 Corn 0.7660 0.8675
4 Winter barley 0.4817 0.6502
5 Winter rapeseed 0.9033 0.9492
6 Spring barley 0.5560 0.7147
7 Sunflower 0.5724 0.7280
8 Grapevine 0.6786 0.8085
9 Beet 0.3556 0.5247
10 Winter triticale 0.1601 0.2760
11 Winter durum wheat 0.8452 0.9161
12 Fruits, vegetables, flowers 0.2045 0.3396
13 Potatoes 0.0969 0.1767 *rare*
14 Leguminous fodder 0.3058 0.4684
15 Soybeans 0.7742 0.8727
16 Orchard 0.4986 0.6654
17 Mixed cereal 0.0367 0.0708 *rare*
18 Sorghum 0.1950 0.3264
[ep 10 EMA] mIoU=0.4947 OA=0.7673 mF1=0.6122 Kappa=0.7421
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7677 0.8686
2 Soft winter wheat 0.8106 0.8954
3 Corn 0.7594 0.8632
4 Winter barley 0.4567 0.6270
5 Winter rapeseed 0.8951 0.9447
6 Spring barley 0.5318 0.6943
7 Sunflower 0.5777 0.7324
8 Grapevine 0.6815 0.8106
9 Beet 0.3584 0.5277
10 Winter triticale 0.1620 0.2788
11 Winter durum wheat 0.8343 0.9096
12 Fruits, vegetables, flowers 0.1853 0.3126
13 Potatoes 0.1054 0.1907
14 Leguminous fodder 0.2918 0.4518
15 Soybeans 0.7772 0.8746
16 Orchard 0.4827 0.6511
17 Mixed cereal 0.0294 0.0572 *rare*
18 Sorghum 0.1973 0.3296
[ENCODER] frozen=False
ep 11 batch 0/38 step 2/76 loss=1.0654 lr=4.28e-06 t=2s
ep 11 batch 10/38 step 22/76 loss=1.2464 lr=4.07e-06 t=20s
ep 11 batch 20/38 step 42/76 loss=0.8273 lr=3.87e-06 t=37s
ep 11 batch 30/38 step 62/76 loss=0.8437 lr=3.68e-06 t=54s
[ep 11 DONE] loss=1.0302 raw_mIoU=0.4928 ema_mIoU=0.4951 best_raw=0.5001@10 best_ema=0.4951@11 patience=1/0 t=83s
[ep 11 RAW] mIoU=0.4928 OA=0.7636 mF1=0.6098 Kappa=0.7380
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7638 0.8661
2 Soft winter wheat 0.8109 0.8956
3 Corn 0.7530 0.8591
4 Winter barley 0.4260 0.5975
5 Winter rapeseed 0.8933 0.9436
6 Spring barley 0.4764 0.6453
7 Sunflower 0.5668 0.7235
8 Grapevine 0.6945 0.8197
9 Beet 0.3555 0.5245
10 Winter triticale 0.1682 0.2880
11 Winter durum wheat 0.8266 0.9051
12 Fruits, vegetables, flowers 0.1944 0.3255
13 Potatoes 0.1096 0.1975
14 Leguminous fodder 0.2889 0.4483
15 Soybeans 0.8166 0.8990
16 Orchard 0.5181 0.6825
17 Mixed cereal 0.0196 0.0384 *rare*
18 Sorghum 0.1880 0.3165
[ep 11 EMA] mIoU=0.4951 OA=0.7676 mF1=0.6124 Kappa=0.7424
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7670 0.8681
2 Soft winter wheat 0.8107 0.8955
3 Corn 0.7576 0.8621
4 Winter barley 0.4598 0.6300
5 Winter rapeseed 0.8910 0.9423
6 Spring barley 0.5348 0.6969
7 Sunflower 0.5733 0.7288
8 Grapevine 0.6896 0.8163
9 Beet 0.3592 0.5286
10 Winter triticale 0.1624 0.2794
11 Winter durum wheat 0.8370 0.9113
12 Fruits, vegetables, flowers 0.1855 0.3130
13 Potatoes 0.1054 0.1906
14 Leguminous fodder 0.2923 0.4524
15 Soybeans 0.7697 0.8698
16 Orchard 0.4964 0.6635
17 Mixed cereal 0.0245 0.0479 *rare*
18 Sorghum 0.1947 0.3260
[ENCODER] frozen=False
ep 12 batch 0/38 step 2/76 loss=0.9112 lr=3.52e-06 t=2s
ep 12 batch 10/38 step 22/76 loss=0.8932 lr=3.33e-06 t=20s
ep 12 batch 20/38 step 42/76 loss=1.0576 lr=3.14e-06 t=38s
ep 12 batch 30/38 step 62/76 loss=1.0116 lr=2.95e-06 t=55s
[ep 12 DONE] loss=1.0529 raw_mIoU=0.4896 ema_mIoU=0.4921 best_raw=0.5001@10 best_ema=0.4951@11 patience=2/0 t=85s
[ENCODER] frozen=False
ep 13 batch 0/38 step 2/76 loss=0.8882 lr=2.80e-06 t=2s
ep 13 batch 10/38 step 22/76 loss=1.2548 lr=2.62e-06 t=20s
ep 13 batch 20/38 step 42/76 loss=0.8904 lr=2.45e-06 t=38s
ep 13 batch 30/38 step 62/76 loss=0.7522 lr=2.27e-06 t=56s
[ep 13 DONE] loss=1.1104 raw_mIoU=0.4891 ema_mIoU=0.4933 best_raw=0.5001@10 best_ema=0.4951@11 patience=3/0 t=85s
[ENCODER] frozen=False
ep 14 batch 0/38 step 2/76 loss=1.3615 lr=2.14e-06 t=2s
ep 14 batch 10/38 step 22/76 loss=1.2865 lr=1.98e-06 t=18s
ep 14 batch 20/38 step 42/76 loss=1.0458 lr=1.82e-06 t=36s
ep 14 batch 30/38 step 62/76 loss=0.8706 lr=1.67e-06 t=54s
[ep 14 DONE] loss=1.0259 raw_mIoU=0.4857 ema_mIoU=0.4922 best_raw=0.5001@10 best_ema=0.4951@11 patience=4/0 t=83s
[ENCODER] frozen=False
ep 15 batch 0/38 step 2/76 loss=1.1097 lr=1.55e-06 t=2s
ep 15 batch 10/38 step 22/76 loss=0.8428 lr=1.41e-06 t=19s
ep 15 batch 20/38 step 42/76 loss=0.8653 lr=1.27e-06 t=37s
ep 15 batch 30/38 step 62/76 loss=0.9819 lr=1.14e-06 t=55s
[ep 15 DONE] loss=1.0581 raw_mIoU=0.4951 ema_mIoU=0.4914 best_raw=0.5001@10 best_ema=0.4951@11 patience=5/0 t=85s
[ep 15 RAW] mIoU=0.4951 OA=0.7660 mF1=0.6097 Kappa=0.7405
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7662 0.8677
2 Soft winter wheat 0.7989 0.8882
3 Corn 0.7608 0.8642
4 Winter barley 0.4230 0.5945
5 Winter rapeseed 0.9041 0.9497
6 Spring barley 0.5747 0.7299
7 Sunflower 0.5679 0.7244
8 Grapevine 0.7039 0.8262
9 Beet 0.3539 0.5228
10 Winter triticale 0.1331 0.2350
11 Winter durum wheat 0.8158 0.8985
12 Fruits, vegetables, flowers 0.1900 0.3193
13 Potatoes 0.0913 0.1674 *rare*
14 Leguminous fodder 0.2914 0.4513
15 Soybeans 0.8091 0.8945
16 Orchard 0.5093 0.6749
17 Mixed cereal 0.0113 0.0223 *rare*
18 Sorghum 0.2074 0.3436
[ep 15 EMA] mIoU=0.4914 OA=0.7669 mF1=0.6076 Kappa=0.7413
Cls Name IoU F1
--------------------------------------------
1 Meadow 0.7655 0.8672
2 Soft winter wheat 0.7998 0.8888
3 Corn 0.7588 0.8629
4 Winter barley 0.4628 0.6328
5 Winter rapeseed 0.8988 0.9467
6 Spring barley 0.5112 0.6766
7 Sunflower 0.5753 0.7304
8 Grapevine 0.6890 0.8159
9 Beet 0.3597 0.5291
10 Winter triticale 0.1282 0.2273
11 Winter durum wheat 0.8348 0.9100
12 Fruits, vegetables, flowers 0.1848 0.3120
13 Potatoes 0.1050 0.1901
14 Leguminous fodder 0.2845 0.4430
15 Soybeans 0.7790 0.8758
16 Orchard 0.4908 0.6584
17 Mixed cereal 0.0223 0.0436 *rare*
18 Sorghum 0.1945 0.3257
[ENCODER] frozen=False
ep 16 batch 0/38 step 2/76 loss=0.8404 lr=1.05e-06 t=2s
ep 16 batch 10/38 step 22/76 loss=1.0777 lr=9.29e-07 t=21s
ep 16 batch 20/38 step 42/76 loss=0.8808 lr=8.19e-07 t=39s
ep 16 batch 30/38 step 62/76 loss=1.0732 lr=7.16e-07 t=57s
[ep 16 DONE] loss=1.1745 raw_mIoU=0.4932 ema_mIoU=0.4918 best_raw=0.5001@10 best_ema=0.4951@11 patience=6/0 t=85s
[ENCODER] frozen=False
ep 17 batch 0/38 step 2/76 loss=1.2017 lr=6.40e-07 t=2s
ep 17 batch 10/38 step 22/76 loss=0.9884 lr=5.50e-07 t=21s
ep 17 batch 20/38 step 42/76 loss=0.9091 lr=4.69e-07 t=40s
ep 17 batch 30/38 step 62/76 loss=0.8634 lr=3.95e-07 t=56s
[ep 17 DONE] loss=1.0234 raw_mIoU=0.4877 ema_mIoU=0.4921 best_raw=0.5001@10 best_ema=0.4951@11 patience=7/0 t=86s
[ENCODER] frozen=False
ep 18 batch 0/38 step 2/76 loss=1.3783 lr=3.42e-07 t=2s
ep 18 batch 10/38 step 22/76 loss=1.1151 lr=2.83e-07 t=20s
ep 18 batch 20/38 step 42/76 loss=0.8103 lr=2.32e-07 t=37s
ep 18 batch 30/38 step 62/76 loss=1.1393 lr=1.89e-07 t=56s
[ep 18 DONE] loss=1.0740 raw_mIoU=0.4827 ema_mIoU=0.4930 best_raw=0.5001@10 best_ema=0.4951@11 patience=8/0 t=85s
[ENCODER] frozen=False
ep 19 batch 0/38 step 2/76 loss=1.2257 lr=1.61e-07 t=2s
ep 19 batch 10/38 step 22/76 loss=1.1855 lr=1.33e-07 t=20s
ep 19 batch 20/38 step 42/76 loss=2.5882 lr=1.14e-07 t=36s
ep 19 batch 30/38 step 62/76 loss=0.9655 lr=1.03e-07 t=53s
[ep 19 DONE] loss=1.1689 raw_mIoU=0.4937 ema_mIoU=0.4925 best_raw=0.5001@10 best_ema=0.4951@11 patience=9/0 t=83s
[FT-v2 COMPLETE] best_raw={'miou': 0.5000673383500418, 'epoch': 10} best_ema={'miou': 0.49505459886180897, 'epoch': 11}
[FT-v2] checkpoints in /home/ubuntu/tsr/research_round/checkpoints/sam