Upload folder using huggingface_hub
Browse files
stage2/lightningdit-xl-dinov3-vit-s16-bf16/log.txt
ADDED
|
@@ -0,0 +1,357 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[[34m2025-10-29 05:36:56[0m] Experiment directory created at results/stage2/hfdata/lightningdit-xl-dinov3-vit-s16-bf16
|
| 2 |
+
[[34m2025-10-29 05:36:58[0m] using base=100 for rope new
|
| 3 |
+
[[34m2025-10-29 05:36:58[0m] using min_period=None for rope new
|
| 4 |
+
[[34m2025-10-29 05:36:58[0m] using max_period=None for rope new
|
| 5 |
+
[[34m2025-10-29 05:36:58[0m] using normalize_coords=separate for rope new
|
| 6 |
+
[[34m2025-10-29 05:36:58[0m] using shift_coords=None for rope new
|
| 7 |
+
[[34m2025-10-29 05:36:58[0m] using rescale_coords=2 for rope new
|
| 8 |
+
[[34m2025-10-29 05:36:58[0m] using jitter_coords=None for rope new
|
| 9 |
+
[[34m2025-10-29 05:36:58[0m] using dtype=fp32 for rope new
|
| 10 |
+
[[34m2025-10-29 05:36:58[0m] using mlp layer as FFN
|
| 11 |
+
[[34m2025-10-29 05:37:13[0m] Model Parameters: 1200.86M
|
| 12 |
+
[[34m2025-10-29 05:37:17[0m] Dataset contains 1,281,167 images (/scratch/xingjian.leng/data/train)
|
| 13 |
+
[[34m2025-10-29 05:37:17[0m] Gradient accumulation: steps=1, micro batch=128, per-GPU batch=128, global batch=1024
|
| 14 |
+
[[34m2025-10-29 05:37:17[0m] Precision mode: bf16
|
| 15 |
+
[[34m2025-10-29 05:37:17[0m] Training configured for 80 epochs, 1251 steps per epoch.
|
| 16 |
+
[[34m2025-10-29 05:37:17[0m] Optimizer: ADAMW with lr=0.0002, betas=(0.9, 0.95), weight_decay=0.0, eps=1e-08
|
| 17 |
+
Scheduler: linear with warmup_steps=0, decay_end_steps=0, final_lr=0.0002
|
| 18 |
+
[[34m2025-10-29 05:37:17[0m] Training for 80 epochs...
|
| 19 |
+
[[34m2025-10-29 05:37:17[0m] Beginning epoch 0...
|
| 20 |
+
[[34m2025-10-29 05:37:22[0m] Generating EMA samples...
|
| 21 |
+
[[34m2025-10-29 05:37:50[0m] Generating EMA samples done.
|
| 22 |
+
[[34m2025-10-29 05:39:07[0m] (step=0000100) Train Loss: 1.5261, Train Steps/Sec: 0.91
|
| 23 |
+
[[34m2025-10-29 05:40:26[0m] (step=0000200) Train Loss: 0.8134, Train Steps/Sec: 1.27
|
| 24 |
+
[[34m2025-10-29 05:41:46[0m] (step=0000300) Train Loss: 0.6489, Train Steps/Sec: 1.27
|
| 25 |
+
[[34m2025-10-29 05:43:05[0m] (step=0000400) Train Loss: 0.5884, Train Steps/Sec: 1.27
|
| 26 |
+
[[34m2025-10-29 05:44:24[0m] (step=0000500) Train Loss: 0.5466, Train Steps/Sec: 1.27
|
| 27 |
+
[[34m2025-10-29 05:45:43[0m] (step=0000600) Train Loss: 0.5207, Train Steps/Sec: 1.27
|
| 28 |
+
[[34m2025-10-29 05:47:02[0m] (step=0000700) Train Loss: 0.5021, Train Steps/Sec: 1.27
|
| 29 |
+
[[34m2025-10-29 05:48:21[0m] (step=0000800) Train Loss: 0.4870, Train Steps/Sec: 1.27
|
| 30 |
+
[[34m2025-10-29 05:49:40[0m] (step=0000900) Train Loss: 0.4769, Train Steps/Sec: 1.26
|
| 31 |
+
[[34m2025-10-29 05:50:59[0m] (step=0001000) Train Loss: 0.4682, Train Steps/Sec: 1.26
|
| 32 |
+
[[34m2025-10-29 05:52:18[0m] (step=0001100) Train Loss: 0.4592, Train Steps/Sec: 1.27
|
| 33 |
+
[[34m2025-10-29 05:53:37[0m] (step=0001200) Train Loss: 0.4531, Train Steps/Sec: 1.27
|
| 34 |
+
[[34m2025-10-29 05:54:18[0m] Beginning epoch 1...
|
| 35 |
+
[[34m2025-10-29 05:54:59[0m] (step=0001300) Train Loss: 0.4488, Train Steps/Sec: 1.22
|
| 36 |
+
[[34m2025-10-29 05:56:18[0m] (step=0001400) Train Loss: 0.4414, Train Steps/Sec: 1.27
|
| 37 |
+
[[34m2025-10-29 05:57:37[0m] (step=0001500) Train Loss: 0.4381, Train Steps/Sec: 1.27
|
| 38 |
+
[[34m2025-10-29 05:58:56[0m] (step=0001600) Train Loss: 0.4339, Train Steps/Sec: 1.27
|
| 39 |
+
[[34m2025-10-29 06:00:15[0m] (step=0001700) Train Loss: 0.4297, Train Steps/Sec: 1.27
|
| 40 |
+
[[34m2025-10-29 06:01:34[0m] (step=0001800) Train Loss: 0.4283, Train Steps/Sec: 1.27
|
| 41 |
+
[[34m2025-10-29 06:02:53[0m] (step=0001900) Train Loss: 0.4250, Train Steps/Sec: 1.27
|
| 42 |
+
[[34m2025-10-29 06:04:12[0m] (step=0002000) Train Loss: 0.4210, Train Steps/Sec: 1.27
|
| 43 |
+
[[34m2025-10-29 06:05:31[0m] (step=0002100) Train Loss: 0.4205, Train Steps/Sec: 1.26
|
| 44 |
+
[[34m2025-10-29 06:06:50[0m] (step=0002200) Train Loss: 0.4180, Train Steps/Sec: 1.27
|
| 45 |
+
[[34m2025-10-29 06:08:10[0m] (step=0002300) Train Loss: 0.4139, Train Steps/Sec: 1.26
|
| 46 |
+
[[34m2025-10-29 06:09:29[0m] (step=0002400) Train Loss: 0.4129, Train Steps/Sec: 1.26
|
| 47 |
+
[[34m2025-10-29 06:10:48[0m] (step=0002500) Train Loss: 0.4119, Train Steps/Sec: 1.27
|
| 48 |
+
[[34m2025-10-29 06:10:50[0m] Beginning epoch 2...
|
| 49 |
+
[[34m2025-10-29 06:12:10[0m] (step=0002600) Train Loss: 0.4087, Train Steps/Sec: 1.21
|
| 50 |
+
[[34m2025-10-29 06:13:29[0m] (step=0002700) Train Loss: 0.4066, Train Steps/Sec: 1.26
|
| 51 |
+
[[34m2025-10-29 06:14:48[0m] (step=0002800) Train Loss: 0.4051, Train Steps/Sec: 1.26
|
| 52 |
+
[[34m2025-10-29 06:16:07[0m] (step=0002900) Train Loss: 0.4051, Train Steps/Sec: 1.26
|
| 53 |
+
[[34m2025-10-29 06:17:26[0m] (step=0003000) Train Loss: 0.4023, Train Steps/Sec: 1.27
|
| 54 |
+
[[34m2025-10-29 06:18:45[0m] (step=0003100) Train Loss: 0.4009, Train Steps/Sec: 1.27
|
| 55 |
+
[[34m2025-10-29 06:20:04[0m] (step=0003200) Train Loss: 0.4005, Train Steps/Sec: 1.27
|
| 56 |
+
[[34m2025-10-29 06:21:23[0m] (step=0003300) Train Loss: 0.3994, Train Steps/Sec: 1.27
|
| 57 |
+
[[34m2025-10-29 06:22:42[0m] (step=0003400) Train Loss: 0.3972, Train Steps/Sec: 1.27
|
| 58 |
+
[[34m2025-10-29 06:24:01[0m] (step=0003500) Train Loss: 0.3970, Train Steps/Sec: 1.27
|
| 59 |
+
[[34m2025-10-29 06:25:20[0m] (step=0003600) Train Loss: 0.3948, Train Steps/Sec: 1.26
|
| 60 |
+
[[34m2025-10-29 06:26:40[0m] (step=0003700) Train Loss: 0.3932, Train Steps/Sec: 1.26
|
| 61 |
+
[[34m2025-10-29 06:27:22[0m] Beginning epoch 3...
|
| 62 |
+
[[34m2025-10-29 06:28:01[0m] (step=0003800) Train Loss: 0.3936, Train Steps/Sec: 1.22
|
| 63 |
+
[[34m2025-10-29 06:29:21[0m] (step=0003900) Train Loss: 0.3916, Train Steps/Sec: 1.26
|
| 64 |
+
[[34m2025-10-29 06:30:40[0m] (step=0004000) Train Loss: 0.3909, Train Steps/Sec: 1.27
|
| 65 |
+
[[34m2025-10-29 06:31:59[0m] (step=0004100) Train Loss: 0.3898, Train Steps/Sec: 1.27
|
| 66 |
+
[[34m2025-10-29 06:33:18[0m] (step=0004200) Train Loss: 0.3899, Train Steps/Sec: 1.27
|
| 67 |
+
[[34m2025-10-29 06:34:38[0m] (step=0004300) Train Loss: 0.3879, Train Steps/Sec: 1.25
|
| 68 |
+
[[34m2025-10-29 06:35:57[0m] (step=0004400) Train Loss: 0.3872, Train Steps/Sec: 1.26
|
| 69 |
+
[[34m2025-10-29 06:37:16[0m] (step=0004500) Train Loss: 0.3870, Train Steps/Sec: 1.26
|
| 70 |
+
[[34m2025-10-29 06:38:35[0m] (step=0004600) Train Loss: 0.3851, Train Steps/Sec: 1.26
|
| 71 |
+
[[34m2025-10-29 06:39:54[0m] (step=0004700) Train Loss: 0.3843, Train Steps/Sec: 1.26
|
| 72 |
+
[[34m2025-10-29 06:41:13[0m] (step=0004800) Train Loss: 0.3834, Train Steps/Sec: 1.27
|
| 73 |
+
[[34m2025-10-29 06:42:32[0m] (step=0004900) Train Loss: 0.3831, Train Steps/Sec: 1.26
|
| 74 |
+
[[34m2025-10-29 06:43:51[0m] (step=0005000) Train Loss: 0.3824, Train Steps/Sec: 1.27
|
| 75 |
+
[[34m2025-10-29 06:43:55[0m] Beginning epoch 4...
|
| 76 |
+
[[34m2025-10-29 06:45:13[0m] (step=0005100) Train Loss: 0.3830, Train Steps/Sec: 1.22
|
| 77 |
+
[[34m2025-10-29 06:46:32[0m] (step=0005200) Train Loss: 0.3811, Train Steps/Sec: 1.27
|
| 78 |
+
[[34m2025-10-29 06:47:51[0m] (step=0005300) Train Loss: 0.3813, Train Steps/Sec: 1.26
|
| 79 |
+
[[34m2025-10-29 06:49:10[0m] (step=0005400) Train Loss: 0.3807, Train Steps/Sec: 1.27
|
| 80 |
+
[[34m2025-10-29 06:50:29[0m] (step=0005500) Train Loss: 0.3799, Train Steps/Sec: 1.26
|
| 81 |
+
[[34m2025-10-29 06:51:48[0m] (step=0005600) Train Loss: 0.3792, Train Steps/Sec: 1.26
|
| 82 |
+
[[34m2025-10-29 06:53:07[0m] (step=0005700) Train Loss: 0.3780, Train Steps/Sec: 1.27
|
| 83 |
+
[[34m2025-10-29 06:54:26[0m] (step=0005800) Train Loss: 0.3787, Train Steps/Sec: 1.26
|
| 84 |
+
[[34m2025-10-29 06:55:46[0m] (step=0005900) Train Loss: 0.3774, Train Steps/Sec: 1.26
|
| 85 |
+
[[34m2025-10-29 06:57:05[0m] (step=0006000) Train Loss: 0.3771, Train Steps/Sec: 1.26
|
| 86 |
+
[[34m2025-10-29 06:58:24[0m] (step=0006100) Train Loss: 0.3760, Train Steps/Sec: 1.27
|
| 87 |
+
[[34m2025-10-29 06:59:43[0m] (step=0006200) Train Loss: 0.3753, Train Steps/Sec: 1.27
|
| 88 |
+
[[34m2025-10-29 07:00:27[0m] Beginning epoch 5...
|
| 89 |
+
[[34m2025-10-29 07:01:05[0m] (step=0006300) Train Loss: 0.3758, Train Steps/Sec: 1.22
|
| 90 |
+
[[34m2025-10-29 07:02:24[0m] (step=0006400) Train Loss: 0.3735, Train Steps/Sec: 1.27
|
| 91 |
+
[[34m2025-10-29 07:03:43[0m] (step=0006500) Train Loss: 0.3735, Train Steps/Sec: 1.26
|
| 92 |
+
[[34m2025-10-29 07:05:02[0m] (step=0006600) Train Loss: 0.3748, Train Steps/Sec: 1.27
|
| 93 |
+
[[34m2025-10-29 07:06:21[0m] (step=0006700) Train Loss: 0.3721, Train Steps/Sec: 1.27
|
| 94 |
+
[[34m2025-10-29 07:07:41[0m] (step=0006800) Train Loss: 0.3721, Train Steps/Sec: 1.27
|
| 95 |
+
[[34m2025-10-29 07:09:00[0m] (step=0006900) Train Loss: 0.3727, Train Steps/Sec: 1.27
|
| 96 |
+
[[34m2025-10-29 07:10:19[0m] (step=0007000) Train Loss: 0.3736, Train Steps/Sec: 1.26
|
| 97 |
+
[[34m2025-10-29 07:11:38[0m] (step=0007100) Train Loss: 0.3714, Train Steps/Sec: 1.27
|
| 98 |
+
[[34m2025-10-29 07:12:57[0m] (step=0007200) Train Loss: 0.3714, Train Steps/Sec: 1.26
|
| 99 |
+
[[34m2025-10-29 07:14:16[0m] (step=0007300) Train Loss: 0.3717, Train Steps/Sec: 1.26
|
| 100 |
+
[[34m2025-10-29 07:15:35[0m] (step=0007400) Train Loss: 0.3700, Train Steps/Sec: 1.26
|
| 101 |
+
[[34m2025-10-29 07:16:54[0m] (step=0007500) Train Loss: 0.3699, Train Steps/Sec: 1.26
|
| 102 |
+
[[34m2025-10-29 07:16:59[0m] Beginning epoch 6...
|
| 103 |
+
[[34m2025-10-29 07:18:16[0m] (step=0007600) Train Loss: 0.3703, Train Steps/Sec: 1.22
|
| 104 |
+
[[34m2025-10-29 07:19:35[0m] (step=0007700) Train Loss: 0.3705, Train Steps/Sec: 1.26
|
| 105 |
+
[[34m2025-10-29 07:20:54[0m] (step=0007800) Train Loss: 0.3697, Train Steps/Sec: 1.27
|
| 106 |
+
[[34m2025-10-29 07:22:13[0m] (step=0007900) Train Loss: 0.3684, Train Steps/Sec: 1.26
|
| 107 |
+
[[34m2025-10-29 07:23:32[0m] (step=0008000) Train Loss: 0.3688, Train Steps/Sec: 1.27
|
| 108 |
+
[[34m2025-10-29 07:24:51[0m] (step=0008100) Train Loss: 0.3672, Train Steps/Sec: 1.27
|
| 109 |
+
[[34m2025-10-29 07:26:10[0m] (step=0008200) Train Loss: 0.3674, Train Steps/Sec: 1.27
|
| 110 |
+
[[34m2025-10-29 07:27:30[0m] (step=0008300) Train Loss: 0.3659, Train Steps/Sec: 1.27
|
| 111 |
+
[[34m2025-10-29 07:28:49[0m] (step=0008400) Train Loss: 0.3665, Train Steps/Sec: 1.27
|
| 112 |
+
[[34m2025-10-29 07:30:08[0m] (step=0008500) Train Loss: 0.3671, Train Steps/Sec: 1.27
|
| 113 |
+
[[34m2025-10-29 07:31:27[0m] (step=0008600) Train Loss: 0.3655, Train Steps/Sec: 1.27
|
| 114 |
+
[[34m2025-10-29 07:32:46[0m] (step=0008700) Train Loss: 0.3662, Train Steps/Sec: 1.27
|
| 115 |
+
[[34m2025-10-29 07:33:31[0m] Beginning epoch 7...
|
| 116 |
+
[[34m2025-10-29 07:34:07[0m] (step=0008800) Train Loss: 0.3652, Train Steps/Sec: 1.23
|
| 117 |
+
[[34m2025-10-29 07:35:26[0m] (step=0008900) Train Loss: 0.3637, Train Steps/Sec: 1.27
|
| 118 |
+
[[34m2025-10-29 07:36:45[0m] (step=0009000) Train Loss: 0.3653, Train Steps/Sec: 1.27
|
| 119 |
+
[[34m2025-10-29 07:38:04[0m] (step=0009100) Train Loss: 0.3652, Train Steps/Sec: 1.27
|
| 120 |
+
[[34m2025-10-29 07:39:23[0m] (step=0009200) Train Loss: 0.3654, Train Steps/Sec: 1.27
|
| 121 |
+
[[34m2025-10-29 07:40:43[0m] (step=0009300) Train Loss: 0.3643, Train Steps/Sec: 1.25
|
| 122 |
+
[[34m2025-10-29 07:42:02[0m] (step=0009400) Train Loss: 0.3640, Train Steps/Sec: 1.26
|
| 123 |
+
[[34m2025-10-29 07:43:21[0m] (step=0009500) Train Loss: 0.3639, Train Steps/Sec: 1.27
|
| 124 |
+
[[34m2025-10-29 07:44:40[0m] (step=0009600) Train Loss: 0.3615, Train Steps/Sec: 1.26
|
| 125 |
+
[[34m2025-10-29 07:45:59[0m] (step=0009700) Train Loss: 0.3625, Train Steps/Sec: 1.26
|
| 126 |
+
[[34m2025-10-29 07:47:18[0m] (step=0009800) Train Loss: 0.3622, Train Steps/Sec: 1.27
|
| 127 |
+
[[34m2025-10-29 07:48:38[0m] (step=0009900) Train Loss: 0.3617, Train Steps/Sec: 1.27
|
| 128 |
+
[[34m2025-10-29 07:49:57[0m] (step=0010000) Train Loss: 0.3631, Train Steps/Sec: 1.27
|
| 129 |
+
[[34m2025-10-29 07:50:03[0m] Beginning epoch 8...
|
| 130 |
+
[[34m2025-10-29 07:51:18[0m] (step=0010100) Train Loss: 0.3631, Train Steps/Sec: 1.23
|
| 131 |
+
[[34m2025-10-29 07:52:37[0m] (step=0010200) Train Loss: 0.3622, Train Steps/Sec: 1.27
|
| 132 |
+
[[34m2025-10-29 07:53:56[0m] (step=0010300) Train Loss: 0.3620, Train Steps/Sec: 1.27
|
| 133 |
+
[[34m2025-10-29 07:55:15[0m] (step=0010400) Train Loss: 0.3606, Train Steps/Sec: 1.27
|
| 134 |
+
[[34m2025-10-29 07:56:34[0m] (step=0010500) Train Loss: 0.3616, Train Steps/Sec: 1.27
|
| 135 |
+
[[34m2025-10-29 07:57:53[0m] (step=0010600) Train Loss: 0.3610, Train Steps/Sec: 1.26
|
| 136 |
+
[[34m2025-10-29 07:59:12[0m] (step=0010700) Train Loss: 0.3610, Train Steps/Sec: 1.27
|
| 137 |
+
[[34m2025-10-29 08:00:31[0m] (step=0010800) Train Loss: 0.3613, Train Steps/Sec: 1.26
|
| 138 |
+
[[34m2025-10-29 08:01:51[0m] (step=0010900) Train Loss: 0.3599, Train Steps/Sec: 1.26
|
| 139 |
+
[[34m2025-10-29 08:03:10[0m] (step=0011000) Train Loss: 0.3606, Train Steps/Sec: 1.25
|
| 140 |
+
[[34m2025-10-29 08:04:29[0m] (step=0011100) Train Loss: 0.3602, Train Steps/Sec: 1.27
|
| 141 |
+
[[34m2025-10-29 08:05:48[0m] (step=0011200) Train Loss: 0.3608, Train Steps/Sec: 1.27
|
| 142 |
+
[[34m2025-10-29 08:06:36[0m] Beginning epoch 9...
|
| 143 |
+
[[34m2025-10-29 08:07:10[0m] (step=0011300) Train Loss: 0.3583, Train Steps/Sec: 1.23
|
| 144 |
+
[[34m2025-10-29 08:08:29[0m] (step=0011400) Train Loss: 0.3591, Train Steps/Sec: 1.27
|
| 145 |
+
[[34m2025-10-29 08:09:48[0m] (step=0011500) Train Loss: 0.3589, Train Steps/Sec: 1.27
|
| 146 |
+
[[34m2025-10-29 08:11:07[0m] (step=0011600) Train Loss: 0.3579, Train Steps/Sec: 1.27
|
| 147 |
+
[[34m2025-10-29 08:12:26[0m] (step=0011700) Train Loss: 0.3588, Train Steps/Sec: 1.27
|
| 148 |
+
[[34m2025-10-29 08:13:45[0m] (step=0011800) Train Loss: 0.3581, Train Steps/Sec: 1.27
|
| 149 |
+
[[34m2025-10-29 08:15:04[0m] (step=0011900) Train Loss: 0.3589, Train Steps/Sec: 1.27
|
| 150 |
+
[[34m2025-10-29 08:16:23[0m] (step=0012000) Train Loss: 0.3587, Train Steps/Sec: 1.27
|
| 151 |
+
[[34m2025-10-29 08:17:42[0m] (step=0012100) Train Loss: 0.3567, Train Steps/Sec: 1.27
|
| 152 |
+
[[34m2025-10-29 08:19:01[0m] (step=0012200) Train Loss: 0.3575, Train Steps/Sec: 1.26
|
| 153 |
+
[[34m2025-10-29 08:20:20[0m] (step=0012300) Train Loss: 0.3571, Train Steps/Sec: 1.27
|
| 154 |
+
[[34m2025-10-29 08:21:39[0m] (step=0012400) Train Loss: 0.3566, Train Steps/Sec: 1.27
|
| 155 |
+
[[34m2025-10-29 08:22:58[0m] (step=0012500) Train Loss: 0.3559, Train Steps/Sec: 1.27
|
| 156 |
+
[[34m2025-10-29 08:23:07[0m] Beginning epoch 10...
|
| 157 |
+
[[34m2025-10-29 08:24:20[0m] (step=0012600) Train Loss: 0.3557, Train Steps/Sec: 1.23
|
| 158 |
+
[[34m2025-10-29 08:25:40[0m] (step=0012700) Train Loss: 0.3553, Train Steps/Sec: 1.26
|
| 159 |
+
[[34m2025-10-29 08:26:59[0m] (step=0012800) Train Loss: 0.3564, Train Steps/Sec: 1.27
|
| 160 |
+
[[34m2025-10-29 08:28:18[0m] (step=0012900) Train Loss: 0.3562, Train Steps/Sec: 1.27
|
| 161 |
+
[[34m2025-10-29 08:29:37[0m] (step=0013000) Train Loss: 0.3553, Train Steps/Sec: 1.26
|
| 162 |
+
[[34m2025-10-29 08:30:56[0m] (step=0013100) Train Loss: 0.3547, Train Steps/Sec: 1.26
|
| 163 |
+
[[34m2025-10-29 08:32:15[0m] (step=0013200) Train Loss: 0.3549, Train Steps/Sec: 1.27
|
| 164 |
+
[[34m2025-10-29 08:33:34[0m] (step=0013300) Train Loss: 0.3551, Train Steps/Sec: 1.27
|
| 165 |
+
[[34m2025-10-29 08:34:53[0m] (step=0013400) Train Loss: 0.3537, Train Steps/Sec: 1.27
|
| 166 |
+
[[34m2025-10-29 08:36:12[0m] (step=0013500) Train Loss: 0.3561, Train Steps/Sec: 1.27
|
| 167 |
+
[[34m2025-10-29 08:37:31[0m] (step=0013600) Train Loss: 0.3551, Train Steps/Sec: 1.26
|
| 168 |
+
[[34m2025-10-29 08:38:50[0m] (step=0013700) Train Loss: 0.3546, Train Steps/Sec: 1.27
|
| 169 |
+
[[34m2025-10-29 08:39:39[0m] Beginning epoch 11...
|
| 170 |
+
[[34m2025-10-29 08:40:11[0m] (step=0013800) Train Loss: 0.3541, Train Steps/Sec: 1.23
|
| 171 |
+
[[34m2025-10-29 08:41:30[0m] (step=0013900) Train Loss: 0.3529, Train Steps/Sec: 1.27
|
| 172 |
+
[[34m2025-10-29 08:42:49[0m] (step=0014000) Train Loss: 0.3540, Train Steps/Sec: 1.27
|
| 173 |
+
[[34m2025-10-29 08:44:08[0m] (step=0014100) Train Loss: 0.3509, Train Steps/Sec: 1.27
|
| 174 |
+
[[34m2025-10-29 08:45:27[0m] (step=0014200) Train Loss: 0.3527, Train Steps/Sec: 1.27
|
| 175 |
+
[[34m2025-10-29 08:46:47[0m] (step=0014300) Train Loss: 0.3525, Train Steps/Sec: 1.25
|
| 176 |
+
[[34m2025-10-29 08:48:06[0m] (step=0014400) Train Loss: 0.3544, Train Steps/Sec: 1.26
|
| 177 |
+
[[34m2025-10-29 08:49:26[0m] (step=0014500) Train Loss: 0.3530, Train Steps/Sec: 1.26
|
| 178 |
+
[[34m2025-10-29 08:50:45[0m] (step=0014600) Train Loss: 0.3531, Train Steps/Sec: 1.27
|
| 179 |
+
[[34m2025-10-29 08:52:04[0m] (step=0014700) Train Loss: 0.3533, Train Steps/Sec: 1.27
|
| 180 |
+
[[34m2025-10-29 08:53:23[0m] (step=0014800) Train Loss: 0.3524, Train Steps/Sec: 1.27
|
| 181 |
+
[[34m2025-10-29 08:54:42[0m] (step=0014900) Train Loss: 0.3524, Train Steps/Sec: 1.27
|
| 182 |
+
[[34m2025-10-29 08:56:01[0m] (step=0015000) Train Loss: 0.3540, Train Steps/Sec: 1.27
|
| 183 |
+
[[34m2025-10-29 08:56:11[0m] Beginning epoch 12...
|
| 184 |
+
[[34m2025-10-29 08:57:22[0m] (step=0015100) Train Loss: 0.3517, Train Steps/Sec: 1.23
|
| 185 |
+
[[34m2025-10-29 08:58:41[0m] (step=0015200) Train Loss: 0.3506, Train Steps/Sec: 1.27
|
| 186 |
+
[[34m2025-10-29 09:00:00[0m] (step=0015300) Train Loss: 0.3528, Train Steps/Sec: 1.27
|
| 187 |
+
[[34m2025-10-29 09:01:19[0m] (step=0015400) Train Loss: 0.3516, Train Steps/Sec: 1.27
|
| 188 |
+
[[34m2025-10-29 09:02:38[0m] (step=0015500) Train Loss: 0.3524, Train Steps/Sec: 1.27
|
| 189 |
+
[[34m2025-10-29 09:03:57[0m] (step=0015600) Train Loss: 0.3522, Train Steps/Sec: 1.27
|
| 190 |
+
[[34m2025-10-29 09:05:16[0m] (step=0015700) Train Loss: 0.3526, Train Steps/Sec: 1.27
|
| 191 |
+
[[34m2025-10-29 09:06:35[0m] (step=0015800) Train Loss: 0.3503, Train Steps/Sec: 1.27
|
| 192 |
+
[[34m2025-10-29 09:07:54[0m] (step=0015900) Train Loss: 0.3501, Train Steps/Sec: 1.27
|
| 193 |
+
[[34m2025-10-29 09:09:14[0m] (step=0016000) Train Loss: 0.3513, Train Steps/Sec: 1.26
|
| 194 |
+
[[34m2025-10-29 09:10:33[0m] (step=0016100) Train Loss: 0.3509, Train Steps/Sec: 1.26
|
| 195 |
+
[[34m2025-10-29 09:11:52[0m] (step=0016200) Train Loss: 0.3505, Train Steps/Sec: 1.27
|
| 196 |
+
[[34m2025-10-29 09:12:42[0m] Beginning epoch 13...
|
| 197 |
+
[[34m2025-10-29 09:13:13[0m] (step=0016300) Train Loss: 0.3503, Train Steps/Sec: 1.23
|
| 198 |
+
[[34m2025-10-29 09:14:32[0m] (step=0016400) Train Loss: 0.3512, Train Steps/Sec: 1.27
|
| 199 |
+
[[34m2025-10-29 09:15:51[0m] (step=0016500) Train Loss: 0.3494, Train Steps/Sec: 1.26
|
| 200 |
+
[[34m2025-10-29 09:17:10[0m] (step=0016600) Train Loss: 0.3494, Train Steps/Sec: 1.27
|
| 201 |
+
[[34m2025-10-29 09:18:29[0m] (step=0016700) Train Loss: 0.3498, Train Steps/Sec: 1.27
|
| 202 |
+
[[34m2025-10-29 09:19:48[0m] (step=0016800) Train Loss: 0.3497, Train Steps/Sec: 1.27
|
| 203 |
+
[[34m2025-10-29 09:21:07[0m] (step=0016900) Train Loss: 0.3496, Train Steps/Sec: 1.27
|
| 204 |
+
[[34m2025-10-29 09:22:26[0m] (step=0017000) Train Loss: 0.3498, Train Steps/Sec: 1.27
|
| 205 |
+
[[34m2025-10-29 09:23:45[0m] (step=0017100) Train Loss: 0.3504, Train Steps/Sec: 1.27
|
| 206 |
+
[[34m2025-10-29 09:25:04[0m] (step=0017200) Train Loss: 0.3497, Train Steps/Sec: 1.27
|
| 207 |
+
[[34m2025-10-29 09:26:23[0m] (step=0017300) Train Loss: 0.3496, Train Steps/Sec: 1.27
|
| 208 |
+
[[34m2025-10-29 09:27:42[0m] (step=0017400) Train Loss: 0.3477, Train Steps/Sec: 1.27
|
| 209 |
+
[[34m2025-10-29 09:29:01[0m] (step=0017500) Train Loss: 0.3491, Train Steps/Sec: 1.27
|
| 210 |
+
[[34m2025-10-29 09:29:13[0m] Beginning epoch 14...
|
| 211 |
+
[[34m2025-10-29 09:30:23[0m] (step=0017600) Train Loss: 0.3483, Train Steps/Sec: 1.23
|
| 212 |
+
[[34m2025-10-29 09:31:42[0m] (step=0017700) Train Loss: 0.3479, Train Steps/Sec: 1.26
|
| 213 |
+
[[34m2025-10-29 09:33:02[0m] (step=0017800) Train Loss: 0.3479, Train Steps/Sec: 1.26
|
| 214 |
+
[[34m2025-10-29 09:34:21[0m] (step=0017900) Train Loss: 0.3481, Train Steps/Sec: 1.26
|
| 215 |
+
[[34m2025-10-29 09:35:40[0m] (step=0018000) Train Loss: 0.3490, Train Steps/Sec: 1.27
|
| 216 |
+
[[34m2025-10-29 09:36:59[0m] (step=0018100) Train Loss: 0.3487, Train Steps/Sec: 1.27
|
| 217 |
+
[[34m2025-10-29 09:38:18[0m] (step=0018200) Train Loss: 0.3477, Train Steps/Sec: 1.27
|
| 218 |
+
[[34m2025-10-29 09:39:37[0m] (step=0018300) Train Loss: 0.3478, Train Steps/Sec: 1.27
|
| 219 |
+
[[34m2025-10-29 09:40:56[0m] (step=0018400) Train Loss: 0.3485, Train Steps/Sec: 1.27
|
| 220 |
+
[[34m2025-10-29 09:42:15[0m] (step=0018500) Train Loss: 0.3477, Train Steps/Sec: 1.27
|
| 221 |
+
[[34m2025-10-29 09:43:34[0m] (step=0018600) Train Loss: 0.3486, Train Steps/Sec: 1.27
|
| 222 |
+
[[34m2025-10-29 09:44:53[0m] (step=0018700) Train Loss: 0.3481, Train Steps/Sec: 1.27
|
| 223 |
+
[[34m2025-10-29 09:45:45[0m] Beginning epoch 15...
|
| 224 |
+
[[34m2025-10-29 09:46:14[0m] (step=0018800) Train Loss: 0.3473, Train Steps/Sec: 1.23
|
| 225 |
+
[[34m2025-10-29 09:47:33[0m] (step=0018900) Train Loss: 0.3456, Train Steps/Sec: 1.27
|
| 226 |
+
[[34m2025-10-29 09:48:52[0m] (step=0019000) Train Loss: 0.3469, Train Steps/Sec: 1.27
|
| 227 |
+
[[34m2025-10-29 09:50:11[0m] (step=0019100) Train Loss: 0.3477, Train Steps/Sec: 1.27
|
| 228 |
+
[[34m2025-10-29 09:51:30[0m] (step=0019200) Train Loss: 0.3458, Train Steps/Sec: 1.27
|
| 229 |
+
[[34m2025-10-29 09:52:49[0m] (step=0019300) Train Loss: 0.3454, Train Steps/Sec: 1.26
|
| 230 |
+
[[34m2025-10-29 09:54:09[0m] (step=0019400) Train Loss: 0.3454, Train Steps/Sec: 1.26
|
| 231 |
+
[[34m2025-10-29 09:55:28[0m] (step=0019500) Train Loss: 0.3473, Train Steps/Sec: 1.26
|
| 232 |
+
[[34m2025-10-29 09:56:47[0m] (step=0019600) Train Loss: 0.3476, Train Steps/Sec: 1.27
|
| 233 |
+
[[34m2025-10-29 09:58:06[0m] (step=0019700) Train Loss: 0.3462, Train Steps/Sec: 1.27
|
| 234 |
+
[[34m2025-10-29 09:59:25[0m] (step=0019800) Train Loss: 0.3459, Train Steps/Sec: 1.27
|
| 235 |
+
[[34m2025-10-29 10:00:44[0m] (step=0019900) Train Loss: 0.3452, Train Steps/Sec: 1.27
|
| 236 |
+
[[34m2025-10-29 10:02:03[0m] (step=0020000) Train Loss: 0.3459, Train Steps/Sec: 1.27
|
| 237 |
+
[[34m2025-10-29 10:02:16[0m] Beginning epoch 16...
|
| 238 |
+
[[34m2025-10-29 10:03:25[0m] (step=0020100) Train Loss: 0.3461, Train Steps/Sec: 1.22
|
| 239 |
+
[[34m2025-10-29 10:04:44[0m] (step=0020200) Train Loss: 0.3453, Train Steps/Sec: 1.27
|
| 240 |
+
[[34m2025-10-29 10:06:03[0m] (step=0020300) Train Loss: 0.3463, Train Steps/Sec: 1.27
|
| 241 |
+
[[34m2025-10-29 10:07:22[0m] (step=0020400) Train Loss: 0.3453, Train Steps/Sec: 1.27
|
| 242 |
+
[[34m2025-10-29 10:08:41[0m] (step=0020500) Train Loss: 0.3462, Train Steps/Sec: 1.27
|
| 243 |
+
[[34m2025-10-29 10:10:00[0m] (step=0020600) Train Loss: 0.3454, Train Steps/Sec: 1.27
|
| 244 |
+
[[34m2025-10-29 10:11:19[0m] (step=0020700) Train Loss: 0.3461, Train Steps/Sec: 1.26
|
| 245 |
+
[[34m2025-10-29 10:12:38[0m] (step=0020800) Train Loss: 0.3454, Train Steps/Sec: 1.27
|
| 246 |
+
[[34m2025-10-29 10:13:57[0m] (step=0020900) Train Loss: 0.3440, Train Steps/Sec: 1.27
|
| 247 |
+
[[34m2025-10-29 10:15:16[0m] (step=0021000) Train Loss: 0.3442, Train Steps/Sec: 1.26
|
| 248 |
+
[[34m2025-10-29 10:16:36[0m] (step=0021100) Train Loss: 0.3452, Train Steps/Sec: 1.26
|
| 249 |
+
[[34m2025-10-29 10:17:55[0m] (step=0021200) Train Loss: 0.3452, Train Steps/Sec: 1.26
|
| 250 |
+
[[34m2025-10-29 10:18:48[0m] Beginning epoch 17...
|
| 251 |
+
[[34m2025-10-29 10:19:17[0m] (step=0021300) Train Loss: 0.3449, Train Steps/Sec: 1.22
|
| 252 |
+
[[34m2025-10-29 10:20:36[0m] (step=0021400) Train Loss: 0.3453, Train Steps/Sec: 1.27
|
| 253 |
+
[[34m2025-10-29 10:21:55[0m] (step=0021500) Train Loss: 0.3449, Train Steps/Sec: 1.27
|
| 254 |
+
[[34m2025-10-29 10:23:14[0m] (step=0021600) Train Loss: 0.3435, Train Steps/Sec: 1.27
|
| 255 |
+
[[34m2025-10-29 10:24:33[0m] (step=0021700) Train Loss: 0.3436, Train Steps/Sec: 1.27
|
| 256 |
+
[[34m2025-10-29 10:25:52[0m] (step=0021800) Train Loss: 0.3438, Train Steps/Sec: 1.27
|
| 257 |
+
[[34m2025-10-29 10:27:11[0m] (step=0021900) Train Loss: 0.3432, Train Steps/Sec: 1.27
|
| 258 |
+
[[34m2025-10-29 10:28:30[0m] (step=0022000) Train Loss: 0.3434, Train Steps/Sec: 1.27
|
| 259 |
+
[[34m2025-10-29 10:29:49[0m] (step=0022100) Train Loss: 0.3441, Train Steps/Sec: 1.27
|
| 260 |
+
[[34m2025-10-29 10:31:08[0m] (step=0022200) Train Loss: 0.3434, Train Steps/Sec: 1.26
|
| 261 |
+
[[34m2025-10-29 10:32:27[0m] (step=0022300) Train Loss: 0.3428, Train Steps/Sec: 1.27
|
| 262 |
+
[[34m2025-10-29 10:33:46[0m] (step=0022400) Train Loss: 0.3430, Train Steps/Sec: 1.27
|
| 263 |
+
[[34m2025-10-29 10:35:05[0m] (step=0022500) Train Loss: 0.3439, Train Steps/Sec: 1.27
|
| 264 |
+
[[34m2025-10-29 10:35:19[0m] Beginning epoch 18...
|
| 265 |
+
[[34m2025-10-29 10:36:26[0m] (step=0022600) Train Loss: 0.3429, Train Steps/Sec: 1.23
|
| 266 |
+
[[34m2025-10-29 10:37:46[0m] (step=0022700) Train Loss: 0.3462, Train Steps/Sec: 1.26
|
| 267 |
+
[[34m2025-10-29 10:39:05[0m] (step=0022800) Train Loss: 0.3756, Train Steps/Sec: 1.26
|
| 268 |
+
[[34m2025-10-29 10:40:24[0m] (step=0022900) Train Loss: 0.9067, Train Steps/Sec: 1.27
|
| 269 |
+
[[34m2025-10-29 10:41:43[0m] (step=0023000) Train Loss: 0.6716, Train Steps/Sec: 1.27
|
| 270 |
+
[[34m2025-10-29 10:43:02[0m] (step=0023100) Train Loss: 0.7666, Train Steps/Sec: 1.26
|
| 271 |
+
[[34m2025-10-29 10:44:21[0m] (step=0023200) Train Loss: 0.7574, Train Steps/Sec: 1.26
|
| 272 |
+
[[34m2025-10-29 10:45:40[0m] (step=0023300) Train Loss: 0.9820, Train Steps/Sec: 1.26
|
| 273 |
+
[[34m2025-10-29 10:46:59[0m] (step=0023400) Train Loss: 0.6974, Train Steps/Sec: 1.26
|
| 274 |
+
[[34m2025-10-29 10:48:18[0m] (step=0023500) Train Loss: 0.7070, Train Steps/Sec: 1.26
|
| 275 |
+
[[34m2025-10-29 10:49:38[0m] (step=0023600) Train Loss: 0.8931, Train Steps/Sec: 1.26
|
| 276 |
+
[[34m2025-10-29 10:50:57[0m] (step=0023700) Train Loss: 0.5795, Train Steps/Sec: 1.26
|
| 277 |
+
[[34m2025-10-29 10:51:50[0m] Beginning epoch 19...
|
| 278 |
+
[[34m2025-10-29 10:52:16[0m] (step=0023800) Train Loss: nan, Train Steps/Sec: 1.27
|
| 279 |
+
[[34m2025-10-29 10:53:32[0m] (step=0023900) Train Loss: nan, Train Steps/Sec: 1.31
|
| 280 |
+
[[34m2025-10-29 10:54:48[0m] (step=0024000) Train Loss: nan, Train Steps/Sec: 1.31
|
| 281 |
+
[[34m2025-10-29 10:56:04[0m] (step=0024100) Train Loss: nan, Train Steps/Sec: 1.31
|
| 282 |
+
[[34m2025-10-29 10:57:21[0m] (step=0024200) Train Loss: nan, Train Steps/Sec: 1.31
|
| 283 |
+
[[34m2025-10-29 10:58:37[0m] (step=0024300) Train Loss: nan, Train Steps/Sec: 1.31
|
| 284 |
+
[[34m2025-10-29 10:59:54[0m] (step=0024400) Train Loss: nan, Train Steps/Sec: 1.30
|
| 285 |
+
[[34m2025-10-29 11:01:11[0m] (step=0024500) Train Loss: nan, Train Steps/Sec: 1.31
|
| 286 |
+
[[34m2025-10-29 11:02:27[0m] (step=0024600) Train Loss: nan, Train Steps/Sec: 1.31
|
| 287 |
+
[[34m2025-10-29 11:03:43[0m] (step=0024700) Train Loss: nan, Train Steps/Sec: 1.31
|
| 288 |
+
[[34m2025-10-29 11:05:00[0m] (step=0024800) Train Loss: nan, Train Steps/Sec: 1.31
|
| 289 |
+
[[34m2025-10-29 11:06:16[0m] (step=0024900) Train Loss: nan, Train Steps/Sec: 1.31
|
| 290 |
+
[[34m2025-10-29 11:07:32[0m] (step=0025000) Train Loss: nan, Train Steps/Sec: 1.31
|
| 291 |
+
[[34m2025-10-29 11:08:29[0m] Saved checkpoint to results/stage2/hfdata/lightningdit-xl-dinov3-vit-s16-bf16/checkpoints/0025000.pt
|
| 292 |
+
[[34m2025-10-29 11:08:29[0m] Generating EMA samples...
|
| 293 |
+
[[34m2025-10-29 11:08:42[0m] Generating EMA samples done.
|
| 294 |
+
[[34m2025-10-29 11:08:58[0m] Beginning epoch 20...
|
| 295 |
+
[[34m2025-10-29 11:10:01[0m] (step=0025100) Train Loss: nan, Train Steps/Sec: 0.67
|
| 296 |
+
[[34m2025-10-29 11:11:18[0m] (step=0025200) Train Loss: nan, Train Steps/Sec: 1.31
|
| 297 |
+
[[34m2025-10-29 11:12:34[0m] (step=0025300) Train Loss: nan, Train Steps/Sec: 1.31
|
| 298 |
+
[[34m2025-10-29 11:13:50[0m] (step=0025400) Train Loss: nan, Train Steps/Sec: 1.31
|
| 299 |
+
[[34m2025-10-29 11:15:07[0m] (step=0025500) Train Loss: nan, Train Steps/Sec: 1.31
|
| 300 |
+
[[34m2025-10-29 11:16:23[0m] (step=0025600) Train Loss: nan, Train Steps/Sec: 1.31
|
| 301 |
+
[[34m2025-10-29 11:17:40[0m] (step=0025700) Train Loss: nan, Train Steps/Sec: 1.31
|
| 302 |
+
[[34m2025-10-29 11:18:56[0m] (step=0025800) Train Loss: nan, Train Steps/Sec: 1.31
|
| 303 |
+
[[34m2025-10-29 11:20:12[0m] (step=0025900) Train Loss: nan, Train Steps/Sec: 1.31
|
| 304 |
+
[[34m2025-10-29 11:21:29[0m] (step=0026000) Train Loss: nan, Train Steps/Sec: 1.30
|
| 305 |
+
[[34m2025-10-29 11:22:46[0m] (step=0026100) Train Loss: nan, Train Steps/Sec: 1.30
|
| 306 |
+
[[34m2025-10-29 11:24:02[0m] (step=0026200) Train Loss: nan, Train Steps/Sec: 1.31
|
| 307 |
+
[[34m2025-10-29 11:24:57[0m] Beginning epoch 21...
|
| 308 |
+
[[34m2025-10-29 11:25:21[0m] (step=0026300) Train Loss: nan, Train Steps/Sec: 1.27
|
| 309 |
+
[[34m2025-10-29 11:26:38[0m] (step=0026400) Train Loss: nan, Train Steps/Sec: 1.31
|
| 310 |
+
[[34m2025-10-29 11:27:54[0m] (step=0026500) Train Loss: nan, Train Steps/Sec: 1.31
|
| 311 |
+
[[34m2025-10-29 11:29:10[0m] (step=0026600) Train Loss: nan, Train Steps/Sec: 1.31
|
| 312 |
+
[[34m2025-10-29 11:30:27[0m] (step=0026700) Train Loss: nan, Train Steps/Sec: 1.31
|
| 313 |
+
[[34m2025-10-29 11:31:43[0m] (step=0026800) Train Loss: nan, Train Steps/Sec: 1.31
|
| 314 |
+
[[34m2025-10-29 11:33:00[0m] (step=0026900) Train Loss: nan, Train Steps/Sec: 1.31
|
| 315 |
+
[[34m2025-10-29 11:34:16[0m] (step=0027000) Train Loss: nan, Train Steps/Sec: 1.31
|
| 316 |
+
[[34m2025-10-29 11:35:32[0m] (step=0027100) Train Loss: nan, Train Steps/Sec: 1.31
|
| 317 |
+
[[34m2025-10-29 11:36:49[0m] (step=0027200) Train Loss: nan, Train Steps/Sec: 1.31
|
| 318 |
+
[[34m2025-10-29 11:38:05[0m] (step=0027300) Train Loss: nan, Train Steps/Sec: 1.31
|
| 319 |
+
[[34m2025-10-29 11:39:21[0m] (step=0027400) Train Loss: nan, Train Steps/Sec: 1.31
|
| 320 |
+
[[34m2025-10-29 11:40:38[0m] (step=0027500) Train Loss: nan, Train Steps/Sec: 1.31
|
| 321 |
+
[[34m2025-10-29 11:40:55[0m] Beginning epoch 22...
|
| 322 |
+
[[34m2025-10-29 11:41:57[0m] (step=0027600) Train Loss: nan, Train Steps/Sec: 1.26
|
| 323 |
+
[[34m2025-10-29 11:43:14[0m] (step=0027700) Train Loss: nan, Train Steps/Sec: 1.30
|
| 324 |
+
[[34m2025-10-29 11:44:31[0m] (step=0027800) Train Loss: nan, Train Steps/Sec: 1.30
|
| 325 |
+
[[34m2025-10-29 11:45:47[0m] (step=0027900) Train Loss: nan, Train Steps/Sec: 1.31
|
| 326 |
+
[[34m2025-10-29 11:47:03[0m] (step=0028000) Train Loss: nan, Train Steps/Sec: 1.31
|
| 327 |
+
[[34m2025-10-29 11:48:20[0m] (step=0028100) Train Loss: nan, Train Steps/Sec: 1.31
|
| 328 |
+
[[34m2025-10-29 11:49:36[0m] (step=0028200) Train Loss: nan, Train Steps/Sec: 1.31
|
| 329 |
+
[[34m2025-10-29 11:50:53[0m] (step=0028300) Train Loss: nan, Train Steps/Sec: 1.31
|
| 330 |
+
[[34m2025-10-29 11:52:09[0m] (step=0028400) Train Loss: nan, Train Steps/Sec: 1.31
|
| 331 |
+
[[34m2025-10-29 11:53:25[0m] (step=0028500) Train Loss: nan, Train Steps/Sec: 1.31
|
| 332 |
+
[[34m2025-10-29 11:54:42[0m] (step=0028600) Train Loss: nan, Train Steps/Sec: 1.31
|
| 333 |
+
[[34m2025-10-29 11:55:58[0m] (step=0028700) Train Loss: nan, Train Steps/Sec: 1.31
|
| 334 |
+
[[34m2025-10-29 11:56:54[0m] Beginning epoch 23...
|
| 335 |
+
[[34m2025-10-29 11:57:17[0m] (step=0028800) Train Loss: nan, Train Steps/Sec: 1.27
|
| 336 |
+
[[34m2025-10-29 11:58:33[0m] (step=0028900) Train Loss: nan, Train Steps/Sec: 1.31
|
| 337 |
+
[[34m2025-10-29 11:59:50[0m] (step=0029000) Train Loss: nan, Train Steps/Sec: 1.31
|
| 338 |
+
[[34m2025-10-29 12:01:06[0m] (step=0029100) Train Loss: nan, Train Steps/Sec: 1.31
|
| 339 |
+
[[34m2025-10-29 12:02:22[0m] (step=0029200) Train Loss: nan, Train Steps/Sec: 1.31
|
| 340 |
+
[[34m2025-10-29 12:03:39[0m] (step=0029300) Train Loss: nan, Train Steps/Sec: 1.31
|
| 341 |
+
[[34m2025-10-29 12:04:56[0m] (step=0029400) Train Loss: nan, Train Steps/Sec: 1.30
|
| 342 |
+
[[34m2025-10-29 12:06:13[0m] (step=0029500) Train Loss: nan, Train Steps/Sec: 1.30
|
| 343 |
+
[[34m2025-10-29 12:07:29[0m] (step=0029600) Train Loss: nan, Train Steps/Sec: 1.31
|
| 344 |
+
[[34m2025-10-29 12:08:45[0m] (step=0029700) Train Loss: nan, Train Steps/Sec: 1.31
|
| 345 |
+
[[34m2025-10-29 12:10:02[0m] (step=0029800) Train Loss: nan, Train Steps/Sec: 1.31
|
| 346 |
+
[[34m2025-10-29 12:11:19[0m] (step=0029900) Train Loss: nan, Train Steps/Sec: 1.31
|
| 347 |
+
[[34m2025-10-29 12:12:35[0m] (step=0030000) Train Loss: nan, Train Steps/Sec: 1.31
|
| 348 |
+
[[34m2025-10-29 12:12:54[0m] Beginning epoch 24...
|
| 349 |
+
[[34m2025-10-29 12:13:54[0m] (step=0030100) Train Loss: nan, Train Steps/Sec: 1.27
|
| 350 |
+
[[34m2025-10-29 12:15:10[0m] (step=0030200) Train Loss: nan, Train Steps/Sec: 1.31
|
| 351 |
+
[[34m2025-10-29 12:16:26[0m] (step=0030300) Train Loss: nan, Train Steps/Sec: 1.31
|
| 352 |
+
[[34m2025-10-29 12:17:43[0m] (step=0030400) Train Loss: nan, Train Steps/Sec: 1.31
|
| 353 |
+
[[34m2025-10-29 12:18:59[0m] (step=0030500) Train Loss: nan, Train Steps/Sec: 1.31
|
| 354 |
+
[[34m2025-10-29 12:20:16[0m] (step=0030600) Train Loss: nan, Train Steps/Sec: 1.31
|
| 355 |
+
[[34m2025-10-29 12:21:32[0m] (step=0030700) Train Loss: nan, Train Steps/Sec: 1.31
|
| 356 |
+
[[34m2025-10-29 12:22:48[0m] (step=0030800) Train Loss: nan, Train Steps/Sec: 1.31
|
| 357 |
+
[[34m2025-10-29 12:24:05[0m] (step=0030900) Train Loss: nan, Train Steps/Sec: 1.31
|