Upload folder using huggingface_hub
Browse files
stage2/lightningdit-xl-dinov2-vit-s-spnorm-bf16/checkpoints/0025000.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b504dc19ea9884d9c86787c3d0f97de7c6f54d1e597bbeadb0d607e6034fa6de
|
| 3 |
+
size 19211551090
|
stage2/lightningdit-xl-dinov2-vit-s-spnorm-bf16/checkpoints/0050000.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a796517cfcb1d864c362b34fe649aeb07f2146ffcd8ec3724cfad595dcf1f569
|
| 3 |
+
size 19211551090
|
stage2/lightningdit-xl-dinov2-vit-s-spnorm-bf16/log.txt
ADDED
|
@@ -0,0 +1,667 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[[34m2025-10-29 12:51:05[0m] Experiment directory created at results/stage2/hfdata/lightningdit-xl-dinov2-vit-s-spnorm-bf16
|
| 2 |
+
[[34m2025-10-29 12:51:08[0m] using MLP layer as FFN
|
| 3 |
+
[[34m2025-10-29 12:51:23[0m] Model Parameters: 1200.86M
|
| 4 |
+
[[34m2025-10-29 12:51:28[0m] Dataset contains 1,281,167 images (/scratch/xingjian.leng/data/train)
|
| 5 |
+
[[34m2025-10-29 12:51:28[0m] Gradient accumulation: steps=1, micro batch=128, per-GPU batch=128, global batch=1024
|
| 6 |
+
[[34m2025-10-29 12:51:28[0m] Precision mode: bf16
|
| 7 |
+
[[34m2025-10-29 12:51:28[0m] Training configured for 80 epochs, 1251 steps per epoch.
|
| 8 |
+
[[34m2025-10-29 12:51:28[0m] Optimizer: ADAMW with lr=0.0002, betas=(0.9, 0.95), weight_decay=0.0, eps=1e-08
|
| 9 |
+
Scheduler: linear with warmup_steps=0, decay_end_steps=0, final_lr=0.0002
|
| 10 |
+
[[34m2025-10-29 12:51:28[0m] Training for 80 epochs...
|
| 11 |
+
[[34m2025-10-29 12:51:28[0m] Beginning epoch 0...
|
| 12 |
+
[[34m2025-10-29 13:28:24[0m] Experiment directory created at results/stage2/hfdata/lightningdit-xl-dinov2-vit-s-spnorm-bf16
|
| 13 |
+
[[34m2025-10-29 13:28:26[0m] using MLP layer as FFN
|
| 14 |
+
[[34m2025-10-29 13:28:41[0m] Model Parameters: 1200.86M
|
| 15 |
+
[[34m2025-10-29 13:28:45[0m] Dataset contains 1,281,167 images (/scratch/xingjian.leng/data/train)
|
| 16 |
+
[[34m2025-10-29 13:28:45[0m] Gradient accumulation: steps=1, micro batch=128, per-GPU batch=128, global batch=1024
|
| 17 |
+
[[34m2025-10-29 13:28:45[0m] Precision mode: bf16
|
| 18 |
+
[[34m2025-10-29 13:28:45[0m] Training configured for 80 epochs, 1251 steps per epoch.
|
| 19 |
+
[[34m2025-10-29 13:28:45[0m] Optimizer: ADAMW with lr=0.0002, betas=(0.9, 0.95), weight_decay=0.0, eps=1e-08
|
| 20 |
+
Scheduler: linear with warmup_steps=0, decay_end_steps=0, final_lr=0.0002
|
| 21 |
+
[[34m2025-10-29 13:28:45[0m] Training for 80 epochs...
|
| 22 |
+
[[34m2025-10-29 13:28:45[0m] Beginning epoch 0...
|
| 23 |
+
[[34m2025-10-29 13:28:50[0m] Generating EMA samples...
|
| 24 |
+
[[34m2025-10-29 13:29:20[0m] Generating EMA samples done.
|
| 25 |
+
[[34m2025-10-29 13:30:37[0m] (step=0000100) Train Loss: 1.5675, Train Steps/Sec: 0.89
|
| 26 |
+
[[34m2025-10-29 13:31:56[0m] (step=0000200) Train Loss: 1.2364, Train Steps/Sec: 1.27
|
| 27 |
+
[[34m2025-10-29 13:33:14[0m] (step=0000300) Train Loss: 1.0025, Train Steps/Sec: 1.27
|
| 28 |
+
[[34m2025-10-29 13:34:33[0m] (step=0000400) Train Loss: 0.9085, Train Steps/Sec: 1.27
|
| 29 |
+
[[34m2025-10-29 13:35:51[0m] (step=0000500) Train Loss: 0.8605, Train Steps/Sec: 1.27
|
| 30 |
+
[[34m2025-10-29 13:37:10[0m] (step=0000600) Train Loss: 0.8292, Train Steps/Sec: 1.27
|
| 31 |
+
[[34m2025-10-29 13:38:29[0m] (step=0000700) Train Loss: 0.8044, Train Steps/Sec: 1.27
|
| 32 |
+
[[34m2025-10-29 13:39:47[0m] (step=0000800) Train Loss: 0.7864, Train Steps/Sec: 1.27
|
| 33 |
+
[[34m2025-10-29 13:41:06[0m] (step=0000900) Train Loss: 0.7697, Train Steps/Sec: 1.27
|
| 34 |
+
[[34m2025-10-29 13:42:25[0m] (step=0001000) Train Loss: 0.7574, Train Steps/Sec: 1.27
|
| 35 |
+
[[34m2025-10-29 13:43:43[0m] (step=0001100) Train Loss: 0.7472, Train Steps/Sec: 1.27
|
| 36 |
+
[[34m2025-10-29 13:45:02[0m] (step=0001200) Train Loss: 0.7379, Train Steps/Sec: 1.27
|
| 37 |
+
[[34m2025-10-29 13:45:42[0m] Beginning epoch 1...
|
| 38 |
+
[[34m2025-10-29 13:46:23[0m] (step=0001300) Train Loss: 0.7302, Train Steps/Sec: 1.23
|
| 39 |
+
[[34m2025-10-29 13:47:41[0m] (step=0001400) Train Loss: 0.7233, Train Steps/Sec: 1.27
|
| 40 |
+
[[34m2025-10-29 13:49:00[0m] (step=0001500) Train Loss: 0.7171, Train Steps/Sec: 1.27
|
| 41 |
+
[[34m2025-10-29 13:50:19[0m] (step=0001600) Train Loss: 0.7139, Train Steps/Sec: 1.27
|
| 42 |
+
[[34m2025-10-29 13:51:37[0m] (step=0001700) Train Loss: 0.7080, Train Steps/Sec: 1.27
|
| 43 |
+
[[34m2025-10-29 13:52:56[0m] (step=0001800) Train Loss: 0.7049, Train Steps/Sec: 1.27
|
| 44 |
+
[[34m2025-10-29 13:54:14[0m] (step=0001900) Train Loss: 0.6997, Train Steps/Sec: 1.27
|
| 45 |
+
[[34m2025-10-29 13:55:33[0m] (step=0002000) Train Loss: 0.6957, Train Steps/Sec: 1.27
|
| 46 |
+
[[34m2025-10-29 13:56:52[0m] (step=0002100) Train Loss: 0.6924, Train Steps/Sec: 1.27
|
| 47 |
+
[[34m2025-10-29 13:58:10[0m] (step=0002200) Train Loss: 0.6906, Train Steps/Sec: 1.27
|
| 48 |
+
[[34m2025-10-29 13:59:29[0m] (step=0002300) Train Loss: 0.6877, Train Steps/Sec: 1.27
|
| 49 |
+
[[34m2025-10-29 14:00:47[0m] (step=0002400) Train Loss: 0.6842, Train Steps/Sec: 1.27
|
| 50 |
+
[[34m2025-10-29 14:02:06[0m] (step=0002500) Train Loss: 0.6800, Train Steps/Sec: 1.27
|
| 51 |
+
[[34m2025-10-29 14:02:08[0m] Beginning epoch 2...
|
| 52 |
+
[[34m2025-10-29 14:03:28[0m] (step=0002600) Train Loss: 0.6775, Train Steps/Sec: 1.22
|
| 53 |
+
[[34m2025-10-29 14:04:46[0m] (step=0002700) Train Loss: 0.6772, Train Steps/Sec: 1.27
|
| 54 |
+
[[34m2025-10-29 14:06:05[0m] (step=0002800) Train Loss: 0.6746, Train Steps/Sec: 1.27
|
| 55 |
+
[[34m2025-10-29 14:07:23[0m] (step=0002900) Train Loss: 0.6732, Train Steps/Sec: 1.27
|
| 56 |
+
[[34m2025-10-29 14:08:42[0m] (step=0003000) Train Loss: 0.6712, Train Steps/Sec: 1.27
|
| 57 |
+
[[34m2025-10-29 14:10:00[0m] (step=0003100) Train Loss: 0.6688, Train Steps/Sec: 1.27
|
| 58 |
+
[[34m2025-10-29 14:11:19[0m] (step=0003200) Train Loss: 0.6676, Train Steps/Sec: 1.27
|
| 59 |
+
[[34m2025-10-29 14:12:38[0m] (step=0003300) Train Loss: 0.6662, Train Steps/Sec: 1.27
|
| 60 |
+
[[34m2025-10-29 14:13:56[0m] (step=0003400) Train Loss: 0.6648, Train Steps/Sec: 1.27
|
| 61 |
+
[[34m2025-10-29 14:15:15[0m] (step=0003500) Train Loss: 0.6628, Train Steps/Sec: 1.27
|
| 62 |
+
[[34m2025-10-29 14:16:34[0m] (step=0003600) Train Loss: 0.6611, Train Steps/Sec: 1.27
|
| 63 |
+
[[34m2025-10-29 14:17:52[0m] (step=0003700) Train Loss: 0.6596, Train Steps/Sec: 1.27
|
| 64 |
+
[[34m2025-10-29 14:18:34[0m] Beginning epoch 3...
|
| 65 |
+
[[34m2025-10-29 14:19:13[0m] (step=0003800) Train Loss: 0.6582, Train Steps/Sec: 1.23
|
| 66 |
+
[[34m2025-10-29 14:20:32[0m] (step=0003900) Train Loss: 0.6571, Train Steps/Sec: 1.27
|
| 67 |
+
[[34m2025-10-29 14:21:50[0m] (step=0004000) Train Loss: 0.6551, Train Steps/Sec: 1.27
|
| 68 |
+
[[34m2025-10-29 14:23:09[0m] (step=0004100) Train Loss: 0.6547, Train Steps/Sec: 1.27
|
| 69 |
+
[[34m2025-10-29 14:24:28[0m] (step=0004200) Train Loss: 0.6545, Train Steps/Sec: 1.27
|
| 70 |
+
[[34m2025-10-29 14:25:47[0m] (step=0004300) Train Loss: 0.6521, Train Steps/Sec: 1.26
|
| 71 |
+
[[34m2025-10-29 14:27:05[0m] (step=0004400) Train Loss: 0.6504, Train Steps/Sec: 1.27
|
| 72 |
+
[[34m2025-10-29 14:28:24[0m] (step=0004500) Train Loss: 0.6495, Train Steps/Sec: 1.27
|
| 73 |
+
[[34m2025-10-29 14:29:43[0m] (step=0004600) Train Loss: 0.6495, Train Steps/Sec: 1.27
|
| 74 |
+
[[34m2025-10-29 14:31:01[0m] (step=0004700) Train Loss: 0.6487, Train Steps/Sec: 1.27
|
| 75 |
+
[[34m2025-10-29 14:32:20[0m] (step=0004800) Train Loss: 0.6474, Train Steps/Sec: 1.27
|
| 76 |
+
[[34m2025-10-29 14:33:38[0m] (step=0004900) Train Loss: 0.6462, Train Steps/Sec: 1.27
|
| 77 |
+
[[34m2025-10-29 14:34:57[0m] (step=0005000) Train Loss: 0.6439, Train Steps/Sec: 1.27
|
| 78 |
+
[[34m2025-10-29 14:35:01[0m] Beginning epoch 4...
|
| 79 |
+
[[34m2025-10-29 14:36:18[0m] (step=0005100) Train Loss: 0.6434, Train Steps/Sec: 1.23
|
| 80 |
+
[[34m2025-10-29 14:37:36[0m] (step=0005200) Train Loss: 0.6424, Train Steps/Sec: 1.27
|
| 81 |
+
[[34m2025-10-29 14:38:55[0m] (step=0005300) Train Loss: 0.6429, Train Steps/Sec: 1.27
|
| 82 |
+
[[34m2025-10-29 14:40:14[0m] (step=0005400) Train Loss: 0.6420, Train Steps/Sec: 1.27
|
| 83 |
+
[[34m2025-10-29 14:41:32[0m] (step=0005500) Train Loss: 0.6408, Train Steps/Sec: 1.27
|
| 84 |
+
[[34m2025-10-29 14:42:51[0m] (step=0005600) Train Loss: 0.6394, Train Steps/Sec: 1.27
|
| 85 |
+
[[34m2025-10-29 14:44:09[0m] (step=0005700) Train Loss: 0.6383, Train Steps/Sec: 1.27
|
| 86 |
+
[[34m2025-10-29 14:45:28[0m] (step=0005800) Train Loss: 0.6397, Train Steps/Sec: 1.27
|
| 87 |
+
[[34m2025-10-29 14:46:46[0m] (step=0005900) Train Loss: 0.6375, Train Steps/Sec: 1.27
|
| 88 |
+
[[34m2025-10-29 14:48:06[0m] (step=0006000) Train Loss: 0.6384, Train Steps/Sec: 1.26
|
| 89 |
+
[[34m2025-10-29 14:49:24[0m] (step=0006100) Train Loss: 0.6361, Train Steps/Sec: 1.27
|
| 90 |
+
[[34m2025-10-29 14:50:43[0m] (step=0006200) Train Loss: 0.6369, Train Steps/Sec: 1.27
|
| 91 |
+
[[34m2025-10-29 14:51:26[0m] Beginning epoch 5...
|
| 92 |
+
[[34m2025-10-29 14:52:04[0m] (step=0006300) Train Loss: 0.6342, Train Steps/Sec: 1.23
|
| 93 |
+
[[34m2025-10-29 14:53:22[0m] (step=0006400) Train Loss: 0.6330, Train Steps/Sec: 1.27
|
| 94 |
+
[[34m2025-10-29 14:54:41[0m] (step=0006500) Train Loss: 0.6340, Train Steps/Sec: 1.27
|
| 95 |
+
[[34m2025-10-29 14:55:59[0m] (step=0006600) Train Loss: 0.6332, Train Steps/Sec: 1.27
|
| 96 |
+
[[34m2025-10-29 14:57:18[0m] (step=0006700) Train Loss: 0.6329, Train Steps/Sec: 1.27
|
| 97 |
+
[[34m2025-10-29 14:58:36[0m] (step=0006800) Train Loss: 0.6319, Train Steps/Sec: 1.27
|
| 98 |
+
[[34m2025-10-29 14:59:55[0m] (step=0006900) Train Loss: 0.6317, Train Steps/Sec: 1.27
|
| 99 |
+
[[34m2025-10-29 15:01:14[0m] (step=0007000) Train Loss: 0.6300, Train Steps/Sec: 1.27
|
| 100 |
+
[[34m2025-10-29 15:02:32[0m] (step=0007100) Train Loss: 0.6297, Train Steps/Sec: 1.27
|
| 101 |
+
[[34m2025-10-29 15:03:51[0m] (step=0007200) Train Loss: 0.6284, Train Steps/Sec: 1.27
|
| 102 |
+
[[34m2025-10-29 15:05:09[0m] (step=0007300) Train Loss: 0.6310, Train Steps/Sec: 1.27
|
| 103 |
+
[[34m2025-10-29 15:06:28[0m] (step=0007400) Train Loss: 0.6286, Train Steps/Sec: 1.27
|
| 104 |
+
[[34m2025-10-29 15:07:46[0m] (step=0007500) Train Loss: 0.6279, Train Steps/Sec: 1.27
|
| 105 |
+
[[34m2025-10-29 15:07:51[0m] Beginning epoch 6...
|
| 106 |
+
[[34m2025-10-29 15:09:07[0m] (step=0007600) Train Loss: 0.6269, Train Steps/Sec: 1.23
|
| 107 |
+
[[34m2025-10-29 15:10:27[0m] (step=0007700) Train Loss: 0.6267, Train Steps/Sec: 1.26
|
| 108 |
+
[[34m2025-10-29 15:11:45[0m] (step=0007800) Train Loss: 0.6272, Train Steps/Sec: 1.27
|
| 109 |
+
[[34m2025-10-29 15:13:04[0m] (step=0007900) Train Loss: 0.6241, Train Steps/Sec: 1.27
|
| 110 |
+
[[34m2025-10-29 15:14:22[0m] (step=0008000) Train Loss: 0.6258, Train Steps/Sec: 1.27
|
| 111 |
+
[[34m2025-10-29 15:15:41[0m] (step=0008100) Train Loss: 0.6245, Train Steps/Sec: 1.27
|
| 112 |
+
[[34m2025-10-29 15:16:59[0m] (step=0008200) Train Loss: 0.6232, Train Steps/Sec: 1.27
|
| 113 |
+
[[34m2025-10-29 15:18:18[0m] (step=0008300) Train Loss: 0.6245, Train Steps/Sec: 1.27
|
| 114 |
+
[[34m2025-10-29 15:19:37[0m] (step=0008400) Train Loss: 0.6217, Train Steps/Sec: 1.27
|
| 115 |
+
[[34m2025-10-29 15:20:55[0m] (step=0008500) Train Loss: 0.6223, Train Steps/Sec: 1.27
|
| 116 |
+
[[34m2025-10-29 15:22:14[0m] (step=0008600) Train Loss: 0.6222, Train Steps/Sec: 1.27
|
| 117 |
+
[[34m2025-10-29 15:23:32[0m] (step=0008700) Train Loss: 0.6232, Train Steps/Sec: 1.27
|
| 118 |
+
[[34m2025-10-29 15:24:17[0m] Beginning epoch 7...
|
| 119 |
+
[[34m2025-10-29 15:24:53[0m] (step=0008800) Train Loss: 0.6205, Train Steps/Sec: 1.23
|
| 120 |
+
[[34m2025-10-29 15:26:12[0m] (step=0008900) Train Loss: 0.6207, Train Steps/Sec: 1.27
|
| 121 |
+
[[34m2025-10-29 15:27:30[0m] (step=0009000) Train Loss: 0.6211, Train Steps/Sec: 1.27
|
| 122 |
+
[[34m2025-10-29 15:28:49[0m] (step=0009100) Train Loss: 0.6188, Train Steps/Sec: 1.27
|
| 123 |
+
[[34m2025-10-29 15:30:07[0m] (step=0009200) Train Loss: 0.6197, Train Steps/Sec: 1.27
|
| 124 |
+
[[34m2025-10-29 15:31:27[0m] (step=0009300) Train Loss: 0.6191, Train Steps/Sec: 1.26
|
| 125 |
+
[[34m2025-10-29 15:32:45[0m] (step=0009400) Train Loss: 0.6190, Train Steps/Sec: 1.27
|
| 126 |
+
[[34m2025-10-29 15:34:04[0m] (step=0009500) Train Loss: 0.6179, Train Steps/Sec: 1.27
|
| 127 |
+
[[34m2025-10-29 15:35:22[0m] (step=0009600) Train Loss: 0.6173, Train Steps/Sec: 1.27
|
| 128 |
+
[[34m2025-10-29 15:36:41[0m] (step=0009700) Train Loss: 0.6168, Train Steps/Sec: 1.27
|
| 129 |
+
[[34m2025-10-29 15:37:59[0m] (step=0009800) Train Loss: 0.6157, Train Steps/Sec: 1.27
|
| 130 |
+
[[34m2025-10-29 15:39:18[0m] (step=0009900) Train Loss: 0.6162, Train Steps/Sec: 1.27
|
| 131 |
+
[[34m2025-10-29 15:40:37[0m] (step=0010000) Train Loss: 0.6178, Train Steps/Sec: 1.27
|
| 132 |
+
[[34m2025-10-29 15:40:43[0m] Beginning epoch 8...
|
| 133 |
+
[[34m2025-10-29 15:41:58[0m] (step=0010100) Train Loss: 0.6159, Train Steps/Sec: 1.23
|
| 134 |
+
[[34m2025-10-29 15:43:16[0m] (step=0010200) Train Loss: 0.6154, Train Steps/Sec: 1.27
|
| 135 |
+
[[34m2025-10-29 15:44:35[0m] (step=0010300) Train Loss: 0.6147, Train Steps/Sec: 1.27
|
| 136 |
+
[[34m2025-10-29 15:45:54[0m] (step=0010400) Train Loss: 0.6143, Train Steps/Sec: 1.27
|
| 137 |
+
[[34m2025-10-29 15:47:12[0m] (step=0010500) Train Loss: 0.6148, Train Steps/Sec: 1.27
|
| 138 |
+
[[34m2025-10-29 15:48:31[0m] (step=0010600) Train Loss: 0.6129, Train Steps/Sec: 1.27
|
| 139 |
+
[[34m2025-10-29 15:49:49[0m] (step=0010700) Train Loss: 0.6141, Train Steps/Sec: 1.27
|
| 140 |
+
[[34m2025-10-29 15:51:08[0m] (step=0010800) Train Loss: 0.6131, Train Steps/Sec: 1.27
|
| 141 |
+
[[34m2025-10-29 15:52:27[0m] (step=0010900) Train Loss: 0.6135, Train Steps/Sec: 1.27
|
| 142 |
+
[[34m2025-10-29 15:53:46[0m] (step=0011000) Train Loss: 0.6112, Train Steps/Sec: 1.26
|
| 143 |
+
[[34m2025-10-29 15:55:04[0m] (step=0011100) Train Loss: 0.6135, Train Steps/Sec: 1.27
|
| 144 |
+
[[34m2025-10-29 15:56:23[0m] (step=0011200) Train Loss: 0.6129, Train Steps/Sec: 1.27
|
| 145 |
+
[[34m2025-10-29 15:57:10[0m] Beginning epoch 9...
|
| 146 |
+
[[34m2025-10-29 15:57:44[0m] (step=0011300) Train Loss: 0.6111, Train Steps/Sec: 1.23
|
| 147 |
+
[[34m2025-10-29 15:59:03[0m] (step=0011400) Train Loss: 0.6105, Train Steps/Sec: 1.27
|
| 148 |
+
[[34m2025-10-29 16:00:21[0m] (step=0011500) Train Loss: 0.6104, Train Steps/Sec: 1.27
|
| 149 |
+
[[34m2025-10-29 16:01:40[0m] (step=0011600) Train Loss: 0.6104, Train Steps/Sec: 1.27
|
| 150 |
+
[[34m2025-10-29 16:02:58[0m] (step=0011700) Train Loss: 0.6099, Train Steps/Sec: 1.27
|
| 151 |
+
[[34m2025-10-29 16:04:17[0m] (step=0011800) Train Loss: 0.6098, Train Steps/Sec: 1.27
|
| 152 |
+
[[34m2025-10-29 16:05:36[0m] (step=0011900) Train Loss: 0.6076, Train Steps/Sec: 1.27
|
| 153 |
+
[[34m2025-10-29 16:06:54[0m] (step=0012000) Train Loss: 0.6097, Train Steps/Sec: 1.27
|
| 154 |
+
[[34m2025-10-29 16:08:13[0m] (step=0012100) Train Loss: 0.6084, Train Steps/Sec: 1.27
|
| 155 |
+
[[34m2025-10-29 16:09:31[0m] (step=0012200) Train Loss: 0.6072, Train Steps/Sec: 1.27
|
| 156 |
+
[[34m2025-10-29 16:10:50[0m] (step=0012300) Train Loss: 0.6080, Train Steps/Sec: 1.27
|
| 157 |
+
[[34m2025-10-29 16:12:09[0m] (step=0012400) Train Loss: 0.6071, Train Steps/Sec: 1.27
|
| 158 |
+
[[34m2025-10-29 16:13:27[0m] (step=0012500) Train Loss: 0.6081, Train Steps/Sec: 1.27
|
| 159 |
+
[[34m2025-10-29 16:13:35[0m] Beginning epoch 10...
|
| 160 |
+
[[34m2025-10-29 16:14:48[0m] (step=0012600) Train Loss: 0.6061, Train Steps/Sec: 1.23
|
| 161 |
+
[[34m2025-10-29 16:16:07[0m] (step=0012700) Train Loss: 0.6068, Train Steps/Sec: 1.26
|
| 162 |
+
[[34m2025-10-29 16:17:26[0m] (step=0012800) Train Loss: 0.6073, Train Steps/Sec: 1.27
|
| 163 |
+
[[34m2025-10-29 16:18:44[0m] (step=0012900) Train Loss: 0.6070, Train Steps/Sec: 1.27
|
| 164 |
+
[[34m2025-10-29 16:20:03[0m] (step=0013000) Train Loss: 0.6052, Train Steps/Sec: 1.27
|
| 165 |
+
[[34m2025-10-29 16:21:22[0m] (step=0013100) Train Loss: 0.6048, Train Steps/Sec: 1.27
|
| 166 |
+
[[34m2025-10-29 16:22:40[0m] (step=0013200) Train Loss: 0.6060, Train Steps/Sec: 1.27
|
| 167 |
+
[[34m2025-10-29 16:23:59[0m] (step=0013300) Train Loss: 0.6056, Train Steps/Sec: 1.27
|
| 168 |
+
[[34m2025-10-29 16:25:17[0m] (step=0013400) Train Loss: 0.6040, Train Steps/Sec: 1.27
|
| 169 |
+
[[34m2025-10-29 16:26:36[0m] (step=0013500) Train Loss: 0.6044, Train Steps/Sec: 1.27
|
| 170 |
+
[[34m2025-10-29 16:27:55[0m] (step=0013600) Train Loss: 0.6055, Train Steps/Sec: 1.27
|
| 171 |
+
[[34m2025-10-29 16:29:13[0m] (step=0013700) Train Loss: 0.6049, Train Steps/Sec: 1.27
|
| 172 |
+
[[34m2025-10-29 16:30:02[0m] Beginning epoch 11...
|
| 173 |
+
[[34m2025-10-29 16:30:35[0m] (step=0013800) Train Loss: 0.6048, Train Steps/Sec: 1.22
|
| 174 |
+
[[34m2025-10-29 16:31:53[0m] (step=0013900) Train Loss: 0.6031, Train Steps/Sec: 1.27
|
| 175 |
+
[[34m2025-10-29 16:33:12[0m] (step=0014000) Train Loss: 0.6026, Train Steps/Sec: 1.27
|
| 176 |
+
[[34m2025-10-29 16:34:31[0m] (step=0014100) Train Loss: 0.6009, Train Steps/Sec: 1.27
|
| 177 |
+
[[34m2025-10-29 16:35:49[0m] (step=0014200) Train Loss: 0.6026, Train Steps/Sec: 1.27
|
| 178 |
+
[[34m2025-10-29 16:37:08[0m] (step=0014300) Train Loss: 0.6014, Train Steps/Sec: 1.27
|
| 179 |
+
[[34m2025-10-29 16:38:27[0m] (step=0014400) Train Loss: 0.6020, Train Steps/Sec: 1.26
|
| 180 |
+
[[34m2025-10-29 16:39:46[0m] (step=0014500) Train Loss: 0.6008, Train Steps/Sec: 1.27
|
| 181 |
+
[[34m2025-10-29 16:41:04[0m] (step=0014600) Train Loss: 0.5996, Train Steps/Sec: 1.27
|
| 182 |
+
[[34m2025-10-29 16:42:23[0m] (step=0014700) Train Loss: 0.6019, Train Steps/Sec: 1.27
|
| 183 |
+
[[34m2025-10-29 16:43:41[0m] (step=0014800) Train Loss: 0.6006, Train Steps/Sec: 1.27
|
| 184 |
+
[[34m2025-10-29 16:45:00[0m] (step=0014900) Train Loss: 0.6013, Train Steps/Sec: 1.27
|
| 185 |
+
[[34m2025-10-29 16:46:19[0m] (step=0015000) Train Loss: 0.5996, Train Steps/Sec: 1.27
|
| 186 |
+
[[34m2025-10-29 16:46:28[0m] Beginning epoch 12...
|
| 187 |
+
[[34m2025-10-29 16:47:40[0m] (step=0015100) Train Loss: 0.5984, Train Steps/Sec: 1.23
|
| 188 |
+
[[34m2025-10-29 16:48:58[0m] (step=0015200) Train Loss: 0.5993, Train Steps/Sec: 1.27
|
| 189 |
+
[[34m2025-10-29 16:50:17[0m] (step=0015300) Train Loss: 0.5985, Train Steps/Sec: 1.27
|
| 190 |
+
[[34m2025-10-29 16:51:35[0m] (step=0015400) Train Loss: 0.5994, Train Steps/Sec: 1.27
|
| 191 |
+
[[34m2025-10-29 16:52:54[0m] (step=0015500) Train Loss: 0.5982, Train Steps/Sec: 1.27
|
| 192 |
+
[[34m2025-10-29 16:54:12[0m] (step=0015600) Train Loss: 0.5984, Train Steps/Sec: 1.27
|
| 193 |
+
[[34m2025-10-29 16:55:31[0m] (step=0015700) Train Loss: 0.5976, Train Steps/Sec: 1.27
|
| 194 |
+
[[34m2025-10-29 16:56:49[0m] (step=0015800) Train Loss: 0.5979, Train Steps/Sec: 1.27
|
| 195 |
+
[[34m2025-10-29 16:58:08[0m] (step=0015900) Train Loss: 0.5988, Train Steps/Sec: 1.27
|
| 196 |
+
[[34m2025-10-29 16:59:27[0m] (step=0016000) Train Loss: 0.5968, Train Steps/Sec: 1.27
|
| 197 |
+
[[34m2025-10-29 17:00:46[0m] (step=0016100) Train Loss: 0.5966, Train Steps/Sec: 1.27
|
| 198 |
+
[[34m2025-10-29 17:02:04[0m] (step=0016200) Train Loss: 0.5964, Train Steps/Sec: 1.27
|
| 199 |
+
[[34m2025-10-29 17:02:54[0m] Beginning epoch 13...
|
| 200 |
+
[[34m2025-10-29 17:03:26[0m] (step=0016300) Train Loss: 0.5974, Train Steps/Sec: 1.23
|
| 201 |
+
[[34m2025-10-29 17:04:44[0m] (step=0016400) Train Loss: 0.5959, Train Steps/Sec: 1.27
|
| 202 |
+
[[34m2025-10-29 17:06:03[0m] (step=0016500) Train Loss: 0.5935, Train Steps/Sec: 1.27
|
| 203 |
+
[[34m2025-10-29 17:07:21[0m] (step=0016600) Train Loss: 0.5954, Train Steps/Sec: 1.27
|
| 204 |
+
[[34m2025-10-29 17:08:40[0m] (step=0016700) Train Loss: 0.5960, Train Steps/Sec: 1.27
|
| 205 |
+
[[34m2025-10-29 17:09:58[0m] (step=0016800) Train Loss: 0.5954, Train Steps/Sec: 1.27
|
| 206 |
+
[[34m2025-10-29 17:11:17[0m] (step=0016900) Train Loss: 0.5944, Train Steps/Sec: 1.27
|
| 207 |
+
[[34m2025-10-29 17:12:36[0m] (step=0017000) Train Loss: 0.5940, Train Steps/Sec: 1.27
|
| 208 |
+
[[34m2025-10-29 17:13:54[0m] (step=0017100) Train Loss: 0.5943, Train Steps/Sec: 1.27
|
| 209 |
+
[[34m2025-10-29 17:15:13[0m] (step=0017200) Train Loss: 0.5942, Train Steps/Sec: 1.27
|
| 210 |
+
[[34m2025-10-29 17:16:31[0m] (step=0017300) Train Loss: 0.5940, Train Steps/Sec: 1.27
|
| 211 |
+
[[34m2025-10-29 17:17:50[0m] (step=0017400) Train Loss: 0.5941, Train Steps/Sec: 1.27
|
| 212 |
+
[[34m2025-10-29 17:19:08[0m] (step=0017500) Train Loss: 0.5944, Train Steps/Sec: 1.27
|
| 213 |
+
[[34m2025-10-29 17:19:20[0m] Beginning epoch 14...
|
| 214 |
+
[[34m2025-10-29 17:20:30[0m] (step=0017600) Train Loss: 0.5925, Train Steps/Sec: 1.23
|
| 215 |
+
[[34m2025-10-29 17:21:49[0m] (step=0017700) Train Loss: 0.5914, Train Steps/Sec: 1.26
|
| 216 |
+
[[34m2025-10-29 17:23:08[0m] (step=0017800) Train Loss: 0.5922, Train Steps/Sec: 1.27
|
| 217 |
+
[[34m2025-10-29 17:24:27[0m] (step=0017900) Train Loss: 0.5930, Train Steps/Sec: 1.27
|
| 218 |
+
[[34m2025-10-29 17:25:45[0m] (step=0018000) Train Loss: 0.5932, Train Steps/Sec: 1.27
|
| 219 |
+
[[34m2025-10-29 17:27:04[0m] (step=0018100) Train Loss: 0.5917, Train Steps/Sec: 1.27
|
| 220 |
+
[[34m2025-10-29 17:28:22[0m] (step=0018200) Train Loss: 0.5920, Train Steps/Sec: 1.27
|
| 221 |
+
[[34m2025-10-29 17:29:41[0m] (step=0018300) Train Loss: 0.5929, Train Steps/Sec: 1.27
|
| 222 |
+
[[34m2025-10-29 17:30:59[0m] (step=0018400) Train Loss: 0.5919, Train Steps/Sec: 1.27
|
| 223 |
+
[[34m2025-10-29 17:32:18[0m] (step=0018500) Train Loss: 0.5925, Train Steps/Sec: 1.27
|
| 224 |
+
[[34m2025-10-29 17:33:36[0m] (step=0018600) Train Loss: 0.5913, Train Steps/Sec: 1.27
|
| 225 |
+
[[34m2025-10-29 17:34:55[0m] (step=0018700) Train Loss: 0.5900, Train Steps/Sec: 1.27
|
| 226 |
+
[[34m2025-10-29 17:35:47[0m] Beginning epoch 15...
|
| 227 |
+
[[34m2025-10-29 17:36:16[0m] (step=0018800) Train Loss: 0.5916, Train Steps/Sec: 1.23
|
| 228 |
+
[[34m2025-10-29 17:37:35[0m] (step=0018900) Train Loss: 0.5911, Train Steps/Sec: 1.27
|
| 229 |
+
[[34m2025-10-29 17:38:54[0m] (step=0019000) Train Loss: 0.5907, Train Steps/Sec: 1.27
|
| 230 |
+
[[34m2025-10-29 17:40:12[0m] (step=0019100) Train Loss: 0.5902, Train Steps/Sec: 1.27
|
| 231 |
+
[[34m2025-10-29 17:41:31[0m] (step=0019200) Train Loss: 0.5889, Train Steps/Sec: 1.27
|
| 232 |
+
[[34m2025-10-29 17:42:49[0m] (step=0019300) Train Loss: 0.5899, Train Steps/Sec: 1.27
|
| 233 |
+
[[34m2025-10-29 17:44:09[0m] (step=0019400) Train Loss: 0.5902, Train Steps/Sec: 1.26
|
| 234 |
+
[[34m2025-10-29 17:45:27[0m] (step=0019500) Train Loss: 0.5888, Train Steps/Sec: 1.27
|
| 235 |
+
[[34m2025-10-29 17:46:46[0m] (step=0019600) Train Loss: 0.5892, Train Steps/Sec: 1.27
|
| 236 |
+
[[34m2025-10-29 17:48:04[0m] (step=0019700) Train Loss: 0.5881, Train Steps/Sec: 1.27
|
| 237 |
+
[[34m2025-10-29 17:49:23[0m] (step=0019800) Train Loss: 0.5885, Train Steps/Sec: 1.27
|
| 238 |
+
[[34m2025-10-29 17:50:41[0m] (step=0019900) Train Loss: 0.5885, Train Steps/Sec: 1.27
|
| 239 |
+
[[34m2025-10-29 17:52:00[0m] (step=0020000) Train Loss: 0.5879, Train Steps/Sec: 1.27
|
| 240 |
+
[[34m2025-10-29 17:52:13[0m] Beginning epoch 16...
|
| 241 |
+
[[34m2025-10-29 17:53:21[0m] (step=0020100) Train Loss: 0.5885, Train Steps/Sec: 1.23
|
| 242 |
+
[[34m2025-10-29 17:54:40[0m] (step=0020200) Train Loss: 0.5876, Train Steps/Sec: 1.27
|
| 243 |
+
[[34m2025-10-29 17:55:58[0m] (step=0020300) Train Loss: 0.5878, Train Steps/Sec: 1.27
|
| 244 |
+
[[34m2025-10-29 17:57:17[0m] (step=0020400) Train Loss: 0.5862, Train Steps/Sec: 1.27
|
| 245 |
+
[[34m2025-10-29 17:58:35[0m] (step=0020500) Train Loss: 0.5860, Train Steps/Sec: 1.27
|
| 246 |
+
[[34m2025-10-29 17:59:54[0m] (step=0020600) Train Loss: 0.5865, Train Steps/Sec: 1.27
|
| 247 |
+
[[34m2025-10-29 18:01:13[0m] (step=0020700) Train Loss: 0.5864, Train Steps/Sec: 1.27
|
| 248 |
+
[[34m2025-10-29 18:02:31[0m] (step=0020800) Train Loss: 0.5872, Train Steps/Sec: 1.27
|
| 249 |
+
[[34m2025-10-29 18:03:50[0m] (step=0020900) Train Loss: 0.5873, Train Steps/Sec: 1.27
|
| 250 |
+
[[34m2025-10-29 18:05:09[0m] (step=0021000) Train Loss: 0.5844, Train Steps/Sec: 1.26
|
| 251 |
+
[[34m2025-10-29 18:06:28[0m] (step=0021100) Train Loss: 0.5867, Train Steps/Sec: 1.27
|
| 252 |
+
[[34m2025-10-29 18:07:46[0m] (step=0021200) Train Loss: 0.5863, Train Steps/Sec: 1.27
|
| 253 |
+
[[34m2025-10-29 18:08:40[0m] Beginning epoch 17...
|
| 254 |
+
[[34m2025-10-29 18:09:08[0m] (step=0021300) Train Loss: 0.5845, Train Steps/Sec: 1.23
|
| 255 |
+
[[34m2025-10-29 18:10:26[0m] (step=0021400) Train Loss: 0.5843, Train Steps/Sec: 1.27
|
| 256 |
+
[[34m2025-10-29 18:11:45[0m] (step=0021500) Train Loss: 0.5833, Train Steps/Sec: 1.27
|
| 257 |
+
[[34m2025-10-29 18:13:03[0m] (step=0021600) Train Loss: 0.5854, Train Steps/Sec: 1.27
|
| 258 |
+
[[34m2025-10-29 18:14:22[0m] (step=0021700) Train Loss: 0.5853, Train Steps/Sec: 1.27
|
| 259 |
+
[[34m2025-10-29 18:15:41[0m] (step=0021800) Train Loss: 0.5859, Train Steps/Sec: 1.27
|
| 260 |
+
[[34m2025-10-29 18:16:59[0m] (step=0021900) Train Loss: 0.5833, Train Steps/Sec: 1.27
|
| 261 |
+
[[34m2025-10-29 18:18:18[0m] (step=0022000) Train Loss: 0.5844, Train Steps/Sec: 1.27
|
| 262 |
+
[[34m2025-10-29 18:19:36[0m] (step=0022100) Train Loss: 0.5853, Train Steps/Sec: 1.27
|
| 263 |
+
[[34m2025-10-29 18:20:55[0m] (step=0022200) Train Loss: 0.5833, Train Steps/Sec: 1.27
|
| 264 |
+
[[34m2025-10-29 18:22:13[0m] (step=0022300) Train Loss: 0.5854, Train Steps/Sec: 1.27
|
| 265 |
+
[[34m2025-10-29 18:23:32[0m] (step=0022400) Train Loss: 0.5839, Train Steps/Sec: 1.27
|
| 266 |
+
[[34m2025-10-29 18:24:51[0m] (step=0022500) Train Loss: 0.5841, Train Steps/Sec: 1.27
|
| 267 |
+
[[34m2025-10-29 18:25:05[0m] Beginning epoch 18...
|
| 268 |
+
[[34m2025-10-29 18:26:12[0m] (step=0022600) Train Loss: 0.5833, Train Steps/Sec: 1.23
|
| 269 |
+
[[34m2025-10-29 18:27:31[0m] (step=0022700) Train Loss: 0.5830, Train Steps/Sec: 1.26
|
| 270 |
+
[[34m2025-10-29 18:28:50[0m] (step=0022800) Train Loss: 0.5825, Train Steps/Sec: 1.27
|
| 271 |
+
[[34m2025-10-29 18:30:09[0m] (step=0022900) Train Loss: 0.5812, Train Steps/Sec: 1.27
|
| 272 |
+
[[34m2025-10-29 18:31:27[0m] (step=0023000) Train Loss: 0.5832, Train Steps/Sec: 1.27
|
| 273 |
+
[[34m2025-10-29 18:32:46[0m] (step=0023100) Train Loss: 0.5822, Train Steps/Sec: 1.27
|
| 274 |
+
[[34m2025-10-29 18:34:04[0m] (step=0023200) Train Loss: 0.5804, Train Steps/Sec: 1.27
|
| 275 |
+
[[34m2025-10-29 18:35:23[0m] (step=0023300) Train Loss: 0.5815, Train Steps/Sec: 1.27
|
| 276 |
+
[[34m2025-10-29 18:36:41[0m] (step=0023400) Train Loss: 0.5822, Train Steps/Sec: 1.27
|
| 277 |
+
[[34m2025-10-29 18:38:00[0m] (step=0023500) Train Loss: 0.5833, Train Steps/Sec: 1.27
|
| 278 |
+
[[34m2025-10-29 18:39:19[0m] (step=0023600) Train Loss: 0.5820, Train Steps/Sec: 1.27
|
| 279 |
+
[[34m2025-10-29 18:40:37[0m] (step=0023700) Train Loss: 0.5818, Train Steps/Sec: 1.27
|
| 280 |
+
[[34m2025-10-29 18:41:32[0m] Beginning epoch 19...
|
| 281 |
+
[[34m2025-10-29 18:41:58[0m] (step=0023800) Train Loss: 0.5823, Train Steps/Sec: 1.23
|
| 282 |
+
[[34m2025-10-29 18:43:17[0m] (step=0023900) Train Loss: 0.5812, Train Steps/Sec: 1.27
|
| 283 |
+
[[34m2025-10-29 18:44:36[0m] (step=0024000) Train Loss: 0.5816, Train Steps/Sec: 1.27
|
| 284 |
+
[[34m2025-10-29 18:45:54[0m] (step=0024100) Train Loss: 0.5800, Train Steps/Sec: 1.27
|
| 285 |
+
[[34m2025-10-29 18:47:13[0m] (step=0024200) Train Loss: 0.5797, Train Steps/Sec: 1.27
|
| 286 |
+
[[34m2025-10-29 18:48:31[0m] (step=0024300) Train Loss: 0.5801, Train Steps/Sec: 1.27
|
| 287 |
+
[[34m2025-10-29 18:49:51[0m] (step=0024400) Train Loss: 0.5810, Train Steps/Sec: 1.26
|
| 288 |
+
[[34m2025-10-29 18:51:09[0m] (step=0024500) Train Loss: 0.5802, Train Steps/Sec: 1.27
|
| 289 |
+
[[34m2025-10-29 18:52:28[0m] (step=0024600) Train Loss: 0.5798, Train Steps/Sec: 1.27
|
| 290 |
+
[[34m2025-10-29 18:53:47[0m] (step=0024700) Train Loss: 0.5803, Train Steps/Sec: 1.27
|
| 291 |
+
[[34m2025-10-29 18:55:05[0m] (step=0024800) Train Loss: 0.5800, Train Steps/Sec: 1.27
|
| 292 |
+
[[34m2025-10-29 18:56:24[0m] (step=0024900) Train Loss: 0.5788, Train Steps/Sec: 1.27
|
| 293 |
+
[[34m2025-10-29 18:57:43[0m] (step=0025000) Train Loss: 0.5789, Train Steps/Sec: 1.27
|
| 294 |
+
[[34m2025-10-29 18:58:43[0m] Saved checkpoint to results/stage2/hfdata/lightningdit-xl-dinov2-vit-s-spnorm-bf16/checkpoints/0025000.pt
|
| 295 |
+
[[34m2025-10-29 18:58:43[0m] Generating EMA samples...
|
| 296 |
+
[[34m2025-10-29 18:59:11[0m] Generating EMA samples done.
|
| 297 |
+
[[34m2025-10-29 18:59:27[0m] Beginning epoch 20...
|
| 298 |
+
[[34m2025-10-29 19:00:31[0m] (step=0025100) Train Loss: 0.5793, Train Steps/Sec: 0.59
|
| 299 |
+
[[34m2025-10-29 19:01:50[0m] (step=0025200) Train Loss: 0.5766, Train Steps/Sec: 1.27
|
| 300 |
+
[[34m2025-10-29 19:03:08[0m] (step=0025300) Train Loss: 0.5788, Train Steps/Sec: 1.27
|
| 301 |
+
[[34m2025-10-29 19:04:27[0m] (step=0025400) Train Loss: 0.5795, Train Steps/Sec: 1.27
|
| 302 |
+
[[34m2025-10-29 19:05:46[0m] (step=0025500) Train Loss: 0.5779, Train Steps/Sec: 1.27
|
| 303 |
+
[[34m2025-10-29 19:07:04[0m] (step=0025600) Train Loss: 0.5776, Train Steps/Sec: 1.27
|
| 304 |
+
[[34m2025-10-29 19:08:23[0m] (step=0025700) Train Loss: 0.5783, Train Steps/Sec: 1.27
|
| 305 |
+
[[34m2025-10-29 19:09:41[0m] (step=0025800) Train Loss: 0.5790, Train Steps/Sec: 1.27
|
| 306 |
+
[[34m2025-10-29 19:11:00[0m] (step=0025900) Train Loss: 0.5788, Train Steps/Sec: 1.27
|
| 307 |
+
[[34m2025-10-29 19:12:19[0m] (step=0026000) Train Loss: 0.5791, Train Steps/Sec: 1.27
|
| 308 |
+
[[34m2025-10-29 19:13:38[0m] (step=0026100) Train Loss: 0.5794, Train Steps/Sec: 1.26
|
| 309 |
+
[[34m2025-10-29 19:14:57[0m] (step=0026200) Train Loss: 0.5787, Train Steps/Sec: 1.27
|
| 310 |
+
[[34m2025-10-29 19:15:53[0m] Beginning epoch 21...
|
| 311 |
+
[[34m2025-10-29 19:16:18[0m] (step=0026300) Train Loss: 0.5768, Train Steps/Sec: 1.23
|
| 312 |
+
[[34m2025-10-29 19:17:36[0m] (step=0026400) Train Loss: 0.5771, Train Steps/Sec: 1.27
|
| 313 |
+
[[34m2025-10-29 19:18:55[0m] (step=0026500) Train Loss: 0.5755, Train Steps/Sec: 1.27
|
| 314 |
+
[[34m2025-10-29 19:20:13[0m] (step=0026600) Train Loss: 0.5769, Train Steps/Sec: 1.27
|
| 315 |
+
[[34m2025-10-29 19:21:32[0m] (step=0026700) Train Loss: 0.5780, Train Steps/Sec: 1.27
|
| 316 |
+
[[34m2025-10-29 19:22:51[0m] (step=0026800) Train Loss: 0.5764, Train Steps/Sec: 1.27
|
| 317 |
+
[[34m2025-10-29 19:24:09[0m] (step=0026900) Train Loss: 0.5764, Train Steps/Sec: 1.27
|
| 318 |
+
[[34m2025-10-29 19:25:28[0m] (step=0027000) Train Loss: 0.5769, Train Steps/Sec: 1.27
|
| 319 |
+
[[34m2025-10-29 19:26:46[0m] (step=0027100) Train Loss: 0.5775, Train Steps/Sec: 1.27
|
| 320 |
+
[[34m2025-10-29 19:28:05[0m] (step=0027200) Train Loss: 0.5760, Train Steps/Sec: 1.27
|
| 321 |
+
[[34m2025-10-29 19:29:23[0m] (step=0027300) Train Loss: 0.5761, Train Steps/Sec: 1.27
|
| 322 |
+
[[34m2025-10-29 19:30:42[0m] (step=0027400) Train Loss: 0.5759, Train Steps/Sec: 1.27
|
| 323 |
+
[[34m2025-10-29 19:32:00[0m] (step=0027500) Train Loss: 0.5753, Train Steps/Sec: 1.27
|
| 324 |
+
[[34m2025-10-29 19:32:18[0m] Beginning epoch 22...
|
| 325 |
+
[[34m2025-10-29 19:33:22[0m] (step=0027600) Train Loss: 0.5752, Train Steps/Sec: 1.23
|
| 326 |
+
[[34m2025-10-29 19:34:41[0m] (step=0027700) Train Loss: 0.5747, Train Steps/Sec: 1.26
|
| 327 |
+
[[34m2025-10-29 19:36:00[0m] (step=0027800) Train Loss: 0.5764, Train Steps/Sec: 1.27
|
| 328 |
+
[[34m2025-10-29 19:37:18[0m] (step=0027900) Train Loss: 0.5748, Train Steps/Sec: 1.27
|
| 329 |
+
[[34m2025-10-29 19:38:37[0m] (step=0028000) Train Loss: 0.5743, Train Steps/Sec: 1.27
|
| 330 |
+
[[34m2025-10-29 19:39:55[0m] (step=0028100) Train Loss: 0.5769, Train Steps/Sec: 1.27
|
| 331 |
+
[[34m2025-10-29 19:41:14[0m] (step=0028200) Train Loss: 0.5751, Train Steps/Sec: 1.27
|
| 332 |
+
[[34m2025-10-29 19:42:32[0m] (step=0028300) Train Loss: 0.5754, Train Steps/Sec: 1.27
|
| 333 |
+
[[34m2025-10-29 19:43:51[0m] (step=0028400) Train Loss: 0.5752, Train Steps/Sec: 1.27
|
| 334 |
+
[[34m2025-10-29 19:45:09[0m] (step=0028500) Train Loss: 0.5757, Train Steps/Sec: 1.27
|
| 335 |
+
[[34m2025-10-29 19:46:28[0m] (step=0028600) Train Loss: 0.5729, Train Steps/Sec: 1.27
|
| 336 |
+
[[34m2025-10-29 19:47:46[0m] (step=0028700) Train Loss: 0.5738, Train Steps/Sec: 1.27
|
| 337 |
+
[[34m2025-10-29 19:48:44[0m] Beginning epoch 23...
|
| 338 |
+
[[34m2025-10-29 19:49:08[0m] (step=0028800) Train Loss: 0.5750, Train Steps/Sec: 1.23
|
| 339 |
+
[[34m2025-10-29 19:50:26[0m] (step=0028900) Train Loss: 0.5731, Train Steps/Sec: 1.27
|
| 340 |
+
[[34m2025-10-29 19:51:45[0m] (step=0029000) Train Loss: 0.5739, Train Steps/Sec: 1.27
|
| 341 |
+
[[34m2025-10-29 19:53:03[0m] (step=0029100) Train Loss: 0.5726, Train Steps/Sec: 1.27
|
| 342 |
+
[[34m2025-10-29 19:54:22[0m] (step=0029200) Train Loss: 0.5737, Train Steps/Sec: 1.27
|
| 343 |
+
[[34m2025-10-29 19:55:40[0m] (step=0029300) Train Loss: 0.5735, Train Steps/Sec: 1.27
|
| 344 |
+
[[34m2025-10-29 19:57:00[0m] (step=0029400) Train Loss: 0.5738, Train Steps/Sec: 1.26
|
| 345 |
+
[[34m2025-10-29 19:58:19[0m] (step=0029500) Train Loss: 0.5740, Train Steps/Sec: 1.27
|
| 346 |
+
[[34m2025-10-29 19:59:37[0m] (step=0029600) Train Loss: 0.5730, Train Steps/Sec: 1.27
|
| 347 |
+
[[34m2025-10-29 20:00:56[0m] (step=0029700) Train Loss: 0.5737, Train Steps/Sec: 1.27
|
| 348 |
+
[[34m2025-10-29 20:02:14[0m] (step=0029800) Train Loss: 0.5730, Train Steps/Sec: 1.27
|
| 349 |
+
[[34m2025-10-29 20:03:33[0m] (step=0029900) Train Loss: 0.5726, Train Steps/Sec: 1.27
|
| 350 |
+
[[34m2025-10-29 20:04:51[0m] (step=0030000) Train Loss: 0.5730, Train Steps/Sec: 1.27
|
| 351 |
+
[[34m2025-10-29 20:05:11[0m] Beginning epoch 24...
|
| 352 |
+
[[34m2025-10-29 20:06:13[0m] (step=0030100) Train Loss: 0.5729, Train Steps/Sec: 1.23
|
| 353 |
+
[[34m2025-10-29 20:07:31[0m] (step=0030200) Train Loss: 0.5721, Train Steps/Sec: 1.27
|
| 354 |
+
[[34m2025-10-29 20:08:50[0m] (step=0030300) Train Loss: 0.5727, Train Steps/Sec: 1.27
|
| 355 |
+
[[34m2025-10-29 20:10:08[0m] (step=0030400) Train Loss: 0.5718, Train Steps/Sec: 1.27
|
| 356 |
+
[[34m2025-10-29 20:14:43[0m] (step=0030500) Train Loss: 0.5708, Train Steps/Sec: 1.28
|
| 357 |
+
[[34m2025-10-29 20:16:01[0m] (step=0030600) Train Loss: 0.5726, Train Steps/Sec: 1.27
|
| 358 |
+
[[34m2025-10-29 20:17:20[0m] (step=0030700) Train Loss: 0.5718, Train Steps/Sec: 1.27
|
| 359 |
+
[[34m2025-10-29 20:18:38[0m] (step=0030800) Train Loss: 0.5713, Train Steps/Sec: 1.27
|
| 360 |
+
[[34m2025-10-29 20:19:57[0m] (step=0030900) Train Loss: 0.5720, Train Steps/Sec: 1.27
|
| 361 |
+
[[34m2025-10-29 20:21:15[0m] (step=0031000) Train Loss: 0.5709, Train Steps/Sec: 1.27
|
| 362 |
+
[[34m2025-10-29 20:22:35[0m] (step=0031100) Train Loss: 0.5720, Train Steps/Sec: 1.26
|
| 363 |
+
[[34m2025-10-29 20:23:53[0m] (step=0031200) Train Loss: 0.5733, Train Steps/Sec: 1.27
|
| 364 |
+
[[34m2025-10-29 20:24:53[0m] Beginning epoch 25...
|
| 365 |
+
[[34m2025-10-29 20:25:14[0m] (step=0031300) Train Loss: 0.5713, Train Steps/Sec: 1.24
|
| 366 |
+
[[34m2025-10-29 20:26:33[0m] (step=0031400) Train Loss: 0.5715, Train Steps/Sec: 1.27
|
| 367 |
+
[[34m2025-10-29 20:27:51[0m] (step=0031500) Train Loss: 0.5690, Train Steps/Sec: 1.27
|
| 368 |
+
[[34m2025-10-29 20:29:10[0m] (step=0031600) Train Loss: 0.5711, Train Steps/Sec: 1.27
|
| 369 |
+
[[34m2025-10-29 20:30:28[0m] (step=0031700) Train Loss: 0.5704, Train Steps/Sec: 1.27
|
| 370 |
+
[[34m2025-10-29 20:31:47[0m] (step=0031800) Train Loss: 0.5707, Train Steps/Sec: 1.27
|
| 371 |
+
[[34m2025-10-29 20:33:06[0m] (step=0031900) Train Loss: 0.5703, Train Steps/Sec: 1.27
|
| 372 |
+
[[34m2025-10-29 20:34:24[0m] (step=0032000) Train Loss: 0.5691, Train Steps/Sec: 1.27
|
| 373 |
+
[[34m2025-10-29 20:35:43[0m] (step=0032100) Train Loss: 0.5702, Train Steps/Sec: 1.27
|
| 374 |
+
[[34m2025-10-29 20:37:01[0m] (step=0032200) Train Loss: 0.5706, Train Steps/Sec: 1.27
|
| 375 |
+
[[34m2025-10-29 20:38:20[0m] (step=0032300) Train Loss: 0.5687, Train Steps/Sec: 1.27
|
| 376 |
+
[[34m2025-10-29 20:39:38[0m] (step=0032400) Train Loss: 0.5693, Train Steps/Sec: 1.27
|
| 377 |
+
[[34m2025-10-29 20:40:57[0m] (step=0032500) Train Loss: 0.5700, Train Steps/Sec: 1.27
|
| 378 |
+
[[34m2025-10-29 20:41:18[0m] Beginning epoch 26...
|
| 379 |
+
[[34m2025-10-29 20:42:18[0m] (step=0032600) Train Loss: 0.5710, Train Steps/Sec: 1.23
|
| 380 |
+
[[34m2025-10-29 20:43:37[0m] (step=0032700) Train Loss: 0.5708, Train Steps/Sec: 1.26
|
| 381 |
+
[[34m2025-10-29 20:44:56[0m] (step=0032800) Train Loss: 0.5694, Train Steps/Sec: 1.27
|
| 382 |
+
[[34m2025-10-29 20:46:15[0m] (step=0032900) Train Loss: 0.5691, Train Steps/Sec: 1.27
|
| 383 |
+
[[34m2025-10-29 20:47:33[0m] (step=0033000) Train Loss: 0.5678, Train Steps/Sec: 1.27
|
| 384 |
+
[[34m2025-10-29 20:48:52[0m] (step=0033100) Train Loss: 0.5697, Train Steps/Sec: 1.27
|
| 385 |
+
[[34m2025-10-29 20:50:10[0m] (step=0033200) Train Loss: 0.5689, Train Steps/Sec: 1.27
|
| 386 |
+
[[34m2025-10-29 20:51:29[0m] (step=0033300) Train Loss: 0.5720, Train Steps/Sec: 1.27
|
| 387 |
+
[[34m2025-10-29 20:52:48[0m] (step=0033400) Train Loss: 0.5688, Train Steps/Sec: 1.27
|
| 388 |
+
[[34m2025-10-29 20:54:06[0m] (step=0033500) Train Loss: 0.5693, Train Steps/Sec: 1.27
|
| 389 |
+
[[34m2025-10-29 20:55:25[0m] (step=0033600) Train Loss: 0.5699, Train Steps/Sec: 1.27
|
| 390 |
+
[[34m2025-10-29 20:56:43[0m] (step=0033700) Train Loss: 0.5686, Train Steps/Sec: 1.27
|
| 391 |
+
[[34m2025-10-29 20:57:44[0m] Beginning epoch 27...
|
| 392 |
+
[[34m2025-10-29 20:58:05[0m] (step=0033800) Train Loss: 0.5679, Train Steps/Sec: 1.23
|
| 393 |
+
[[34m2025-10-29 20:59:23[0m] (step=0033900) Train Loss: 0.5676, Train Steps/Sec: 1.27
|
| 394 |
+
[[34m2025-10-29 21:00:42[0m] (step=0034000) Train Loss: 0.5683, Train Steps/Sec: 1.27
|
| 395 |
+
[[34m2025-10-29 21:02:00[0m] (step=0034100) Train Loss: 0.5681, Train Steps/Sec: 1.27
|
| 396 |
+
[[34m2025-10-29 21:03:19[0m] (step=0034200) Train Loss: 0.5674, Train Steps/Sec: 1.27
|
| 397 |
+
[[34m2025-10-29 21:04:37[0m] (step=0034300) Train Loss: 0.5681, Train Steps/Sec: 1.27
|
| 398 |
+
[[34m2025-10-29 21:05:57[0m] (step=0034400) Train Loss: 0.5669, Train Steps/Sec: 1.26
|
| 399 |
+
[[34m2025-10-29 21:07:16[0m] (step=0034500) Train Loss: 0.5668, Train Steps/Sec: 1.27
|
| 400 |
+
[[34m2025-10-29 21:08:34[0m] (step=0034600) Train Loss: 0.5661, Train Steps/Sec: 1.27
|
| 401 |
+
[[34m2025-10-29 21:09:53[0m] (step=0034700) Train Loss: 0.5670, Train Steps/Sec: 1.27
|
| 402 |
+
[[34m2025-10-29 21:11:11[0m] (step=0034800) Train Loss: 0.5679, Train Steps/Sec: 1.27
|
| 403 |
+
[[34m2025-10-29 21:12:30[0m] (step=0034900) Train Loss: 0.5666, Train Steps/Sec: 1.27
|
| 404 |
+
[[34m2025-10-29 21:13:48[0m] (step=0035000) Train Loss: 0.5660, Train Steps/Sec: 1.27
|
| 405 |
+
[[34m2025-10-29 21:14:11[0m] Beginning epoch 28...
|
| 406 |
+
[[34m2025-10-29 21:15:09[0m] (step=0035100) Train Loss: 0.5662, Train Steps/Sec: 1.23
|
| 407 |
+
[[34m2025-10-29 21:16:28[0m] (step=0035200) Train Loss: 0.5682, Train Steps/Sec: 1.27
|
| 408 |
+
[[34m2025-10-29 21:17:47[0m] (step=0035300) Train Loss: 0.5681, Train Steps/Sec: 1.27
|
| 409 |
+
[[34m2025-10-29 21:19:05[0m] (step=0035400) Train Loss: 0.5685, Train Steps/Sec: 1.27
|
| 410 |
+
[[34m2025-10-29 21:20:24[0m] (step=0035500) Train Loss: 0.5663, Train Steps/Sec: 1.27
|
| 411 |
+
[[34m2025-10-29 21:21:42[0m] (step=0035600) Train Loss: 0.5653, Train Steps/Sec: 1.27
|
| 412 |
+
[[34m2025-10-29 21:23:01[0m] (step=0035700) Train Loss: 0.5668, Train Steps/Sec: 1.27
|
| 413 |
+
[[34m2025-10-29 21:24:19[0m] (step=0035800) Train Loss: 0.5664, Train Steps/Sec: 1.27
|
| 414 |
+
[[34m2025-10-29 21:25:38[0m] (step=0035900) Train Loss: 0.5675, Train Steps/Sec: 1.27
|
| 415 |
+
[[34m2025-10-29 21:26:56[0m] (step=0036000) Train Loss: 0.5672, Train Steps/Sec: 1.27
|
| 416 |
+
[[34m2025-10-29 21:28:16[0m] (step=0036100) Train Loss: 0.5654, Train Steps/Sec: 1.26
|
| 417 |
+
[[34m2025-10-29 21:29:34[0m] (step=0036200) Train Loss: 0.5663, Train Steps/Sec: 1.27
|
| 418 |
+
[[34m2025-10-29 21:30:37[0m] Beginning epoch 29...
|
| 419 |
+
[[34m2025-10-29 21:30:56[0m] (step=0036300) Train Loss: 0.5656, Train Steps/Sec: 1.23
|
| 420 |
+
[[34m2025-10-29 21:32:14[0m] (step=0036400) Train Loss: 0.5656, Train Steps/Sec: 1.27
|
| 421 |
+
[[34m2025-10-29 21:33:33[0m] (step=0036500) Train Loss: 0.5658, Train Steps/Sec: 1.27
|
| 422 |
+
[[34m2025-10-29 21:34:51[0m] (step=0036600) Train Loss: 0.5646, Train Steps/Sec: 1.27
|
| 423 |
+
[[34m2025-10-29 21:36:10[0m] (step=0036700) Train Loss: 0.5657, Train Steps/Sec: 1.27
|
| 424 |
+
[[34m2025-10-29 21:37:28[0m] (step=0036800) Train Loss: 0.5649, Train Steps/Sec: 1.27
|
| 425 |
+
[[34m2025-10-29 21:38:47[0m] (step=0036900) Train Loss: 0.5652, Train Steps/Sec: 1.27
|
| 426 |
+
[[34m2025-10-29 21:40:05[0m] (step=0037000) Train Loss: 0.5665, Train Steps/Sec: 1.27
|
| 427 |
+
[[34m2025-10-29 21:41:24[0m] (step=0037100) Train Loss: 0.5667, Train Steps/Sec: 1.27
|
| 428 |
+
[[34m2025-10-29 21:42:42[0m] (step=0037200) Train Loss: 0.5650, Train Steps/Sec: 1.27
|
| 429 |
+
[[34m2025-10-29 21:44:01[0m] (step=0037300) Train Loss: 0.5648, Train Steps/Sec: 1.27
|
| 430 |
+
[[34m2025-10-29 21:45:20[0m] (step=0037400) Train Loss: 0.5650, Train Steps/Sec: 1.27
|
| 431 |
+
[[34m2025-10-29 21:46:38[0m] (step=0037500) Train Loss: 0.5648, Train Steps/Sec: 1.27
|
| 432 |
+
[[34m2025-10-29 21:47:02[0m] Beginning epoch 30...
|
| 433 |
+
[[34m2025-10-29 21:47:59[0m] (step=0037600) Train Loss: 0.5632, Train Steps/Sec: 1.23
|
| 434 |
+
[[34m2025-10-29 21:49:18[0m] (step=0037700) Train Loss: 0.5648, Train Steps/Sec: 1.27
|
| 435 |
+
[[34m2025-10-29 21:50:37[0m] (step=0037800) Train Loss: 0.5628, Train Steps/Sec: 1.26
|
| 436 |
+
[[34m2025-10-29 21:51:56[0m] (step=0037900) Train Loss: 0.5638, Train Steps/Sec: 1.27
|
| 437 |
+
[[34m2025-10-29 21:53:14[0m] (step=0038000) Train Loss: 0.5648, Train Steps/Sec: 1.27
|
| 438 |
+
[[34m2025-10-29 21:54:33[0m] (step=0038100) Train Loss: 0.5648, Train Steps/Sec: 1.27
|
| 439 |
+
[[34m2025-10-29 21:55:51[0m] (step=0038200) Train Loss: 0.5647, Train Steps/Sec: 1.27
|
| 440 |
+
[[34m2025-10-29 21:57:10[0m] (step=0038300) Train Loss: 0.5643, Train Steps/Sec: 1.27
|
| 441 |
+
[[34m2025-10-29 21:58:29[0m] (step=0038400) Train Loss: 0.5638, Train Steps/Sec: 1.27
|
| 442 |
+
[[34m2025-10-29 21:59:47[0m] (step=0038500) Train Loss: 0.5635, Train Steps/Sec: 1.27
|
| 443 |
+
[[34m2025-10-29 22:01:06[0m] (step=0038600) Train Loss: 0.5629, Train Steps/Sec: 1.27
|
| 444 |
+
[[34m2025-10-29 22:02:24[0m] (step=0038700) Train Loss: 0.5638, Train Steps/Sec: 1.27
|
| 445 |
+
[[34m2025-10-29 22:03:28[0m] Beginning epoch 31...
|
| 446 |
+
[[34m2025-10-29 22:03:45[0m] (step=0038800) Train Loss: 0.5626, Train Steps/Sec: 1.23
|
| 447 |
+
[[34m2025-10-29 22:05:04[0m] (step=0038900) Train Loss: 0.5634, Train Steps/Sec: 1.27
|
| 448 |
+
[[34m2025-10-29 22:06:22[0m] (step=0039000) Train Loss: 0.5640, Train Steps/Sec: 1.27
|
| 449 |
+
[[34m2025-10-29 22:07:41[0m] (step=0039100) Train Loss: 0.5624, Train Steps/Sec: 1.27
|
| 450 |
+
[[34m2025-10-29 22:08:59[0m] (step=0039200) Train Loss: 0.5638, Train Steps/Sec: 1.27
|
| 451 |
+
[[34m2025-10-29 22:10:18[0m] (step=0039300) Train Loss: 0.5631, Train Steps/Sec: 1.27
|
| 452 |
+
[[34m2025-10-29 22:11:37[0m] (step=0039400) Train Loss: 0.5616, Train Steps/Sec: 1.26
|
| 453 |
+
[[34m2025-10-29 22:12:56[0m] (step=0039500) Train Loss: 0.5623, Train Steps/Sec: 1.27
|
| 454 |
+
[[34m2025-10-29 22:14:14[0m] (step=0039600) Train Loss: 0.5631, Train Steps/Sec: 1.27
|
| 455 |
+
[[34m2025-10-29 22:15:33[0m] (step=0039700) Train Loss: 0.5633, Train Steps/Sec: 1.27
|
| 456 |
+
[[34m2025-10-29 22:16:52[0m] (step=0039800) Train Loss: 0.5619, Train Steps/Sec: 1.27
|
| 457 |
+
[[34m2025-10-29 22:18:10[0m] (step=0039900) Train Loss: 0.5627, Train Steps/Sec: 1.27
|
| 458 |
+
[[34m2025-10-29 22:19:29[0m] (step=0040000) Train Loss: 0.5614, Train Steps/Sec: 1.27
|
| 459 |
+
[[34m2025-10-29 22:19:54[0m] Beginning epoch 32...
|
| 460 |
+
[[34m2025-10-29 22:20:50[0m] (step=0040100) Train Loss: 0.5611, Train Steps/Sec: 1.23
|
| 461 |
+
[[34m2025-10-29 22:22:08[0m] (step=0040200) Train Loss: 0.5612, Train Steps/Sec: 1.27
|
| 462 |
+
[[34m2025-10-29 22:23:27[0m] (step=0040300) Train Loss: 0.5601, Train Steps/Sec: 1.27
|
| 463 |
+
[[34m2025-10-29 22:24:45[0m] (step=0040400) Train Loss: 0.5613, Train Steps/Sec: 1.27
|
| 464 |
+
[[34m2025-10-29 22:26:04[0m] (step=0040500) Train Loss: 0.5617, Train Steps/Sec: 1.27
|
| 465 |
+
[[34m2025-10-29 22:27:23[0m] (step=0040600) Train Loss: 0.5616, Train Steps/Sec: 1.27
|
| 466 |
+
[[34m2025-10-29 22:28:41[0m] (step=0040700) Train Loss: 0.5624, Train Steps/Sec: 1.27
|
| 467 |
+
[[34m2025-10-29 22:30:00[0m] (step=0040800) Train Loss: 0.5605, Train Steps/Sec: 1.27
|
| 468 |
+
[[34m2025-10-29 22:31:18[0m] (step=0040900) Train Loss: 0.5616, Train Steps/Sec: 1.27
|
| 469 |
+
[[34m2025-10-29 22:32:37[0m] (step=0041000) Train Loss: 0.5616, Train Steps/Sec: 1.27
|
| 470 |
+
[[34m2025-10-29 22:33:56[0m] (step=0041100) Train Loss: 0.5604, Train Steps/Sec: 1.26
|
| 471 |
+
[[34m2025-10-29 22:35:15[0m] (step=0041200) Train Loss: 0.5624, Train Steps/Sec: 1.27
|
| 472 |
+
[[34m2025-10-29 22:36:20[0m] Beginning epoch 33...
|
| 473 |
+
[[34m2025-10-29 22:36:36[0m] (step=0041300) Train Loss: 0.5611, Train Steps/Sec: 1.23
|
| 474 |
+
[[34m2025-10-29 22:37:54[0m] (step=0041400) Train Loss: 0.5602, Train Steps/Sec: 1.27
|
| 475 |
+
[[34m2025-10-29 22:39:13[0m] (step=0041500) Train Loss: 0.5621, Train Steps/Sec: 1.27
|
| 476 |
+
[[34m2025-10-29 22:40:32[0m] (step=0041600) Train Loss: 0.5604, Train Steps/Sec: 1.27
|
| 477 |
+
[[34m2025-10-29 22:41:50[0m] (step=0041700) Train Loss: 0.5620, Train Steps/Sec: 1.27
|
| 478 |
+
[[34m2025-10-29 22:43:09[0m] (step=0041800) Train Loss: 0.5616, Train Steps/Sec: 1.27
|
| 479 |
+
[[34m2025-10-29 22:44:27[0m] (step=0041900) Train Loss: 0.5609, Train Steps/Sec: 1.27
|
| 480 |
+
[[34m2025-10-29 22:45:46[0m] (step=0042000) Train Loss: 0.5607, Train Steps/Sec: 1.27
|
| 481 |
+
[[34m2025-10-29 22:47:04[0m] (step=0042100) Train Loss: 0.5609, Train Steps/Sec: 1.27
|
| 482 |
+
[[34m2025-10-29 22:48:23[0m] (step=0042200) Train Loss: 0.5617, Train Steps/Sec: 1.27
|
| 483 |
+
[[34m2025-10-29 22:49:41[0m] (step=0042300) Train Loss: 0.5609, Train Steps/Sec: 1.27
|
| 484 |
+
[[34m2025-10-29 22:51:00[0m] (step=0042400) Train Loss: 0.5610, Train Steps/Sec: 1.27
|
| 485 |
+
[[34m2025-10-29 22:52:19[0m] (step=0042500) Train Loss: 0.5621, Train Steps/Sec: 1.27
|
| 486 |
+
[[34m2025-10-29 22:52:46[0m] Beginning epoch 34...
|
| 487 |
+
[[34m2025-10-29 22:53:40[0m] (step=0042600) Train Loss: 0.5610, Train Steps/Sec: 1.23
|
| 488 |
+
[[34m2025-10-29 22:54:58[0m] (step=0042700) Train Loss: 0.5593, Train Steps/Sec: 1.27
|
| 489 |
+
[[34m2025-10-29 22:56:18[0m] (step=0042800) Train Loss: 0.5603, Train Steps/Sec: 1.26
|
| 490 |
+
[[34m2025-10-29 22:57:36[0m] (step=0042900) Train Loss: 0.5606, Train Steps/Sec: 1.27
|
| 491 |
+
[[34m2025-10-29 22:58:55[0m] (step=0043000) Train Loss: 0.5601, Train Steps/Sec: 1.27
|
| 492 |
+
[[34m2025-10-29 23:00:13[0m] (step=0043100) Train Loss: 0.5600, Train Steps/Sec: 1.27
|
| 493 |
+
[[34m2025-10-29 23:01:32[0m] (step=0043200) Train Loss: 0.5606, Train Steps/Sec: 1.27
|
| 494 |
+
[[34m2025-10-29 23:02:51[0m] (step=0043300) Train Loss: 0.5602, Train Steps/Sec: 1.27
|
| 495 |
+
[[34m2025-10-29 23:04:09[0m] (step=0043400) Train Loss: 0.5611, Train Steps/Sec: 1.27
|
| 496 |
+
[[34m2025-10-29 23:05:28[0m] (step=0043500) Train Loss: 0.5614, Train Steps/Sec: 1.27
|
| 497 |
+
[[34m2025-10-29 23:06:46[0m] (step=0043600) Train Loss: 0.5601, Train Steps/Sec: 1.27
|
| 498 |
+
[[34m2025-10-29 23:08:05[0m] (step=0043700) Train Loss: 0.5592, Train Steps/Sec: 1.27
|
| 499 |
+
[[34m2025-10-29 23:09:12[0m] Beginning epoch 35...
|
| 500 |
+
[[34m2025-10-29 23:09:26[0m] (step=0043800) Train Loss: 0.5602, Train Steps/Sec: 1.23
|
| 501 |
+
[[34m2025-10-29 23:10:45[0m] (step=0043900) Train Loss: 0.5592, Train Steps/Sec: 1.27
|
| 502 |
+
[[34m2025-10-29 23:12:03[0m] (step=0044000) Train Loss: 0.5596, Train Steps/Sec: 1.27
|
| 503 |
+
[[34m2025-10-29 23:13:22[0m] (step=0044100) Train Loss: 0.5598, Train Steps/Sec: 1.27
|
| 504 |
+
[[34m2025-10-29 23:14:40[0m] (step=0044200) Train Loss: 0.5607, Train Steps/Sec: 1.27
|
| 505 |
+
[[34m2025-10-29 23:15:59[0m] (step=0044300) Train Loss: 0.5595, Train Steps/Sec: 1.27
|
| 506 |
+
[[34m2025-10-29 23:17:17[0m] (step=0044400) Train Loss: 0.5590, Train Steps/Sec: 1.27
|
| 507 |
+
[[34m2025-10-29 23:18:37[0m] (step=0044500) Train Loss: 0.5598, Train Steps/Sec: 1.26
|
| 508 |
+
[[34m2025-10-29 23:19:55[0m] (step=0044600) Train Loss: 0.5575, Train Steps/Sec: 1.27
|
| 509 |
+
[[34m2025-10-29 23:21:14[0m] (step=0044700) Train Loss: 0.5601, Train Steps/Sec: 1.27
|
| 510 |
+
[[34m2025-10-29 23:22:33[0m] (step=0044800) Train Loss: 0.5618, Train Steps/Sec: 1.27
|
| 511 |
+
[[34m2025-10-29 23:23:51[0m] (step=0044900) Train Loss: 0.5593, Train Steps/Sec: 1.27
|
| 512 |
+
[[34m2025-10-29 23:25:10[0m] (step=0045000) Train Loss: 0.5609, Train Steps/Sec: 1.27
|
| 513 |
+
[[34m2025-10-29 23:25:38[0m] Beginning epoch 36...
|
| 514 |
+
[[34m2025-10-29 23:26:31[0m] (step=0045100) Train Loss: 0.5593, Train Steps/Sec: 1.23
|
| 515 |
+
[[34m2025-10-29 23:27:49[0m] (step=0045200) Train Loss: 0.5578, Train Steps/Sec: 1.27
|
| 516 |
+
[[34m2025-10-29 23:29:08[0m] (step=0045300) Train Loss: 0.5587, Train Steps/Sec: 1.27
|
| 517 |
+
[[34m2025-10-29 23:30:27[0m] (step=0045400) Train Loss: 0.5585, Train Steps/Sec: 1.27
|
| 518 |
+
[[34m2025-10-29 23:31:45[0m] (step=0045500) Train Loss: 0.5586, Train Steps/Sec: 1.27
|
| 519 |
+
[[34m2025-10-29 23:33:04[0m] (step=0045600) Train Loss: 0.5587, Train Steps/Sec: 1.27
|
| 520 |
+
[[34m2025-10-29 23:34:22[0m] (step=0045700) Train Loss: 0.5599, Train Steps/Sec: 1.27
|
| 521 |
+
[[34m2025-10-29 23:35:41[0m] (step=0045800) Train Loss: 0.5591, Train Steps/Sec: 1.27
|
| 522 |
+
[[34m2025-10-29 23:36:59[0m] (step=0045900) Train Loss: 0.5579, Train Steps/Sec: 1.27
|
| 523 |
+
[[34m2025-10-29 23:38:18[0m] (step=0046000) Train Loss: 0.5576, Train Steps/Sec: 1.27
|
| 524 |
+
[[34m2025-10-29 23:39:37[0m] (step=0046100) Train Loss: 0.5586, Train Steps/Sec: 1.26
|
| 525 |
+
[[34m2025-10-29 23:40:56[0m] (step=0046200) Train Loss: 0.5579, Train Steps/Sec: 1.26
|
| 526 |
+
[[34m2025-10-29 23:42:05[0m] Beginning epoch 37...
|
| 527 |
+
[[34m2025-10-29 23:42:18[0m] (step=0046300) Train Loss: 0.5582, Train Steps/Sec: 1.23
|
| 528 |
+
[[34m2025-10-29 23:43:36[0m] (step=0046400) Train Loss: 0.5580, Train Steps/Sec: 1.27
|
| 529 |
+
[[34m2025-10-29 23:44:55[0m] (step=0046500) Train Loss: 0.5573, Train Steps/Sec: 1.27
|
| 530 |
+
[[34m2025-10-29 23:46:13[0m] (step=0046600) Train Loss: 0.5577, Train Steps/Sec: 1.27
|
| 531 |
+
[[34m2025-10-29 23:47:32[0m] (step=0046700) Train Loss: 0.5577, Train Steps/Sec: 1.27
|
| 532 |
+
[[34m2025-10-29 23:48:51[0m] (step=0046800) Train Loss: 0.5576, Train Steps/Sec: 1.27
|
| 533 |
+
[[34m2025-10-29 23:50:09[0m] (step=0046900) Train Loss: 0.5568, Train Steps/Sec: 1.27
|
| 534 |
+
[[34m2025-10-29 23:51:28[0m] (step=0047000) Train Loss: 0.5571, Train Steps/Sec: 1.27
|
| 535 |
+
[[34m2025-10-29 23:52:46[0m] (step=0047100) Train Loss: 0.5577, Train Steps/Sec: 1.27
|
| 536 |
+
[[34m2025-10-29 23:54:05[0m] (step=0047200) Train Loss: 0.5573, Train Steps/Sec: 1.27
|
| 537 |
+
[[34m2025-10-29 23:55:23[0m] (step=0047300) Train Loss: 0.5589, Train Steps/Sec: 1.27
|
| 538 |
+
[[34m2025-10-29 23:56:42[0m] (step=0047400) Train Loss: 0.5582, Train Steps/Sec: 1.27
|
| 539 |
+
[[34m2025-10-29 23:58:01[0m] (step=0047500) Train Loss: 0.5589, Train Steps/Sec: 1.27
|
| 540 |
+
[[34m2025-10-29 23:58:31[0m] Beginning epoch 38...
|
| 541 |
+
[[34m2025-10-29 23:59:22[0m] (step=0047600) Train Loss: 0.5562, Train Steps/Sec: 1.23
|
| 542 |
+
[[34m2025-10-30 00:00:41[0m] (step=0047700) Train Loss: 0.5575, Train Steps/Sec: 1.27
|
| 543 |
+
[[34m2025-10-30 00:02:00[0m] (step=0047800) Train Loss: 0.5583, Train Steps/Sec: 1.26
|
| 544 |
+
[[34m2025-10-30 00:03:19[0m] (step=0047900) Train Loss: 0.5586, Train Steps/Sec: 1.27
|
| 545 |
+
[[34m2025-10-30 00:04:37[0m] (step=0048000) Train Loss: 0.5560, Train Steps/Sec: 1.27
|
| 546 |
+
[[34m2025-10-30 00:05:56[0m] (step=0048100) Train Loss: 0.5583, Train Steps/Sec: 1.27
|
| 547 |
+
[[34m2025-10-30 00:07:15[0m] (step=0048200) Train Loss: 0.5574, Train Steps/Sec: 1.27
|
| 548 |
+
[[34m2025-10-30 00:08:33[0m] (step=0048300) Train Loss: 0.5575, Train Steps/Sec: 1.27
|
| 549 |
+
[[34m2025-10-30 00:09:52[0m] (step=0048400) Train Loss: 0.5569, Train Steps/Sec: 1.27
|
| 550 |
+
[[34m2025-10-30 00:11:10[0m] (step=0048500) Train Loss: 0.5589, Train Steps/Sec: 1.27
|
| 551 |
+
[[34m2025-10-30 00:12:29[0m] (step=0048600) Train Loss: 0.5562, Train Steps/Sec: 1.27
|
| 552 |
+
[[34m2025-10-30 00:13:48[0m] (step=0048700) Train Loss: 0.5557, Train Steps/Sec: 1.27
|
| 553 |
+
[[34m2025-10-30 00:14:58[0m] Beginning epoch 39...
|
| 554 |
+
[[34m2025-10-30 00:15:09[0m] (step=0048800) Train Loss: 0.5565, Train Steps/Sec: 1.23
|
| 555 |
+
[[34m2025-10-30 00:16:28[0m] (step=0048900) Train Loss: 0.5553, Train Steps/Sec: 1.27
|
| 556 |
+
[[34m2025-10-30 00:17:46[0m] (step=0049000) Train Loss: 0.5556, Train Steps/Sec: 1.27
|
| 557 |
+
[[34m2025-10-30 00:19:05[0m] (step=0049100) Train Loss: 0.5564, Train Steps/Sec: 1.27
|
| 558 |
+
[[34m2025-10-30 00:20:23[0m] (step=0049200) Train Loss: 0.5570, Train Steps/Sec: 1.27
|
| 559 |
+
[[34m2025-10-30 00:21:42[0m] (step=0049300) Train Loss: 0.5565, Train Steps/Sec: 1.27
|
| 560 |
+
[[34m2025-10-30 00:23:01[0m] (step=0049400) Train Loss: 0.5540, Train Steps/Sec: 1.27
|
| 561 |
+
[[34m2025-10-30 00:24:20[0m] (step=0049500) Train Loss: 0.5555, Train Steps/Sec: 1.26
|
| 562 |
+
[[34m2025-10-30 00:25:39[0m] (step=0049600) Train Loss: 0.5554, Train Steps/Sec: 1.27
|
| 563 |
+
[[34m2025-10-30 00:26:57[0m] (step=0049700) Train Loss: 0.5553, Train Steps/Sec: 1.27
|
| 564 |
+
[[34m2025-10-30 00:28:16[0m] (step=0049800) Train Loss: 0.5569, Train Steps/Sec: 1.27
|
| 565 |
+
[[34m2025-10-30 00:29:34[0m] (step=0049900) Train Loss: 0.5559, Train Steps/Sec: 1.27
|
| 566 |
+
[[34m2025-10-30 00:30:53[0m] (step=0050000) Train Loss: 0.5557, Train Steps/Sec: 1.27
|
| 567 |
+
[[34m2025-10-30 00:31:54[0m] Saved checkpoint to results/stage2/hfdata/lightningdit-xl-dinov2-vit-s-spnorm-bf16/checkpoints/0050000.pt
|
| 568 |
+
[[34m2025-10-30 00:31:54[0m] Generating EMA samples...
|
| 569 |
+
[[34m2025-10-30 00:32:22[0m] Generating EMA samples done.
|
| 570 |
+
[[34m2025-10-30 00:32:54[0m] Beginning epoch 40...
|
| 571 |
+
[[34m2025-10-30 00:33:43[0m] (step=0050100) Train Loss: 0.5554, Train Steps/Sec: 0.59
|
| 572 |
+
[[34m2025-10-30 00:35:02[0m] (step=0050200) Train Loss: 0.5542, Train Steps/Sec: 1.27
|
| 573 |
+
[[34m2025-10-30 00:36:20[0m] (step=0050300) Train Loss: 0.5547, Train Steps/Sec: 1.27
|
| 574 |
+
[[34m2025-10-30 00:37:39[0m] (step=0050400) Train Loss: 0.5555, Train Steps/Sec: 1.27
|
| 575 |
+
[[34m2025-10-30 00:38:58[0m] (step=0050500) Train Loss: 0.5542, Train Steps/Sec: 1.27
|
| 576 |
+
[[34m2025-10-30 00:40:16[0m] (step=0050600) Train Loss: 0.5550, Train Steps/Sec: 1.27
|
| 577 |
+
[[34m2025-10-30 00:41:35[0m] (step=0050700) Train Loss: 0.5563, Train Steps/Sec: 1.27
|
| 578 |
+
[[34m2025-10-30 00:42:53[0m] (step=0050800) Train Loss: 0.5549, Train Steps/Sec: 1.27
|
| 579 |
+
[[34m2025-10-30 00:44:12[0m] (step=0050900) Train Loss: 0.5554, Train Steps/Sec: 1.27
|
| 580 |
+
[[34m2025-10-30 00:45:30[0m] (step=0051000) Train Loss: 0.5565, Train Steps/Sec: 1.27
|
| 581 |
+
[[34m2025-10-30 00:46:49[0m] (step=0051100) Train Loss: 0.5551, Train Steps/Sec: 1.27
|
| 582 |
+
[[34m2025-10-30 00:48:08[0m] (step=0051200) Train Loss: 0.5549, Train Steps/Sec: 1.27
|
| 583 |
+
[[34m2025-10-30 00:49:20[0m] Beginning epoch 41...
|
| 584 |
+
[[34m2025-10-30 00:49:30[0m] (step=0051300) Train Loss: 0.5543, Train Steps/Sec: 1.23
|
| 585 |
+
[[34m2025-10-30 00:50:48[0m] (step=0051400) Train Loss: 0.5545, Train Steps/Sec: 1.27
|
| 586 |
+
[[34m2025-10-30 00:52:07[0m] (step=0051500) Train Loss: 0.5550, Train Steps/Sec: 1.27
|
| 587 |
+
[[34m2025-10-30 00:53:26[0m] (step=0051600) Train Loss: 0.5536, Train Steps/Sec: 1.27
|
| 588 |
+
[[34m2025-10-30 00:54:44[0m] (step=0051700) Train Loss: 0.5542, Train Steps/Sec: 1.27
|
| 589 |
+
[[34m2025-10-30 00:56:03[0m] (step=0051800) Train Loss: 0.5543, Train Steps/Sec: 1.27
|
| 590 |
+
[[34m2025-10-30 00:57:21[0m] (step=0051900) Train Loss: 0.5536, Train Steps/Sec: 1.27
|
| 591 |
+
[[34m2025-10-30 00:58:40[0m] (step=0052000) Train Loss: 0.5537, Train Steps/Sec: 1.27
|
| 592 |
+
[[34m2025-10-30 00:59:59[0m] (step=0052100) Train Loss: 0.5524, Train Steps/Sec: 1.27
|
| 593 |
+
[[34m2025-10-30 01:01:17[0m] (step=0052200) Train Loss: 0.5534, Train Steps/Sec: 1.27
|
| 594 |
+
[[34m2025-10-30 01:02:36[0m] (step=0052300) Train Loss: 0.5524, Train Steps/Sec: 1.27
|
| 595 |
+
[[34m2025-10-30 01:03:54[0m] (step=0052400) Train Loss: 0.5544, Train Steps/Sec: 1.27
|
| 596 |
+
[[34m2025-10-30 01:05:13[0m] (step=0052500) Train Loss: 0.5549, Train Steps/Sec: 1.27
|
| 597 |
+
[[34m2025-10-30 01:05:46[0m] Beginning epoch 42...
|
| 598 |
+
[[34m2025-10-30 01:06:34[0m] (step=0052600) Train Loss: 0.5545, Train Steps/Sec: 1.23
|
| 599 |
+
[[34m2025-10-30 01:07:53[0m] (step=0052700) Train Loss: 0.5526, Train Steps/Sec: 1.27
|
| 600 |
+
[[34m2025-10-30 01:09:12[0m] (step=0052800) Train Loss: 0.5530, Train Steps/Sec: 1.26
|
| 601 |
+
[[34m2025-10-30 01:10:31[0m] (step=0052900) Train Loss: 0.5539, Train Steps/Sec: 1.27
|
| 602 |
+
[[34m2025-10-30 01:11:50[0m] (step=0053000) Train Loss: 0.5540, Train Steps/Sec: 1.27
|
| 603 |
+
[[34m2025-10-30 01:13:08[0m] (step=0053100) Train Loss: 0.5537, Train Steps/Sec: 1.27
|
| 604 |
+
[[34m2025-10-30 01:14:27[0m] (step=0053200) Train Loss: 0.5532, Train Steps/Sec: 1.27
|
| 605 |
+
[[34m2025-10-30 01:15:46[0m] (step=0053300) Train Loss: 0.5539, Train Steps/Sec: 1.27
|
| 606 |
+
[[34m2025-10-30 01:17:04[0m] (step=0053400) Train Loss: 0.5526, Train Steps/Sec: 1.27
|
| 607 |
+
[[34m2025-10-30 01:18:23[0m] (step=0053500) Train Loss: 0.5530, Train Steps/Sec: 1.27
|
| 608 |
+
[[34m2025-10-30 01:19:41[0m] (step=0053600) Train Loss: 0.5539, Train Steps/Sec: 1.27
|
| 609 |
+
[[34m2025-10-30 01:21:00[0m] (step=0053700) Train Loss: 0.5523, Train Steps/Sec: 1.27
|
| 610 |
+
[[34m2025-10-30 01:22:13[0m] Beginning epoch 43...
|
| 611 |
+
[[34m2025-10-30 01:22:21[0m] (step=0053800) Train Loss: 0.5538, Train Steps/Sec: 1.23
|
| 612 |
+
[[34m2025-10-30 01:23:40[0m] (step=0053900) Train Loss: 0.5535, Train Steps/Sec: 1.27
|
| 613 |
+
[[34m2025-10-30 01:24:58[0m] (step=0054000) Train Loss: 0.5528, Train Steps/Sec: 1.27
|
| 614 |
+
[[34m2025-10-30 01:26:17[0m] (step=0054100) Train Loss: 0.5527, Train Steps/Sec: 1.27
|
| 615 |
+
[[34m2025-10-30 01:27:36[0m] (step=0054200) Train Loss: 0.5513, Train Steps/Sec: 1.27
|
| 616 |
+
[[34m2025-10-30 01:28:54[0m] (step=0054300) Train Loss: 0.5520, Train Steps/Sec: 1.27
|
| 617 |
+
[[34m2025-10-30 01:30:13[0m] (step=0054400) Train Loss: 0.5527, Train Steps/Sec: 1.27
|
| 618 |
+
[[34m2025-10-30 01:31:32[0m] (step=0054500) Train Loss: 0.5535, Train Steps/Sec: 1.26
|
| 619 |
+
[[34m2025-10-30 01:32:51[0m] (step=0054600) Train Loss: 0.5529, Train Steps/Sec: 1.27
|
| 620 |
+
[[34m2025-10-30 01:34:10[0m] (step=0054700) Train Loss: 0.5508, Train Steps/Sec: 1.27
|
| 621 |
+
[[34m2025-10-30 01:35:28[0m] (step=0054800) Train Loss: 0.5533, Train Steps/Sec: 1.27
|
| 622 |
+
[[34m2025-10-30 01:36:47[0m] (step=0054900) Train Loss: 0.5522, Train Steps/Sec: 1.27
|
| 623 |
+
[[34m2025-10-30 01:38:05[0m] (step=0055000) Train Loss: 0.5530, Train Steps/Sec: 1.27
|
| 624 |
+
[[34m2025-10-30 01:38:40[0m] Beginning epoch 44...
|
| 625 |
+
[[34m2025-10-30 01:39:27[0m] (step=0055100) Train Loss: 0.5534, Train Steps/Sec: 1.23
|
| 626 |
+
[[34m2025-10-30 01:40:45[0m] (step=0055200) Train Loss: 0.5533, Train Steps/Sec: 1.27
|
| 627 |
+
[[34m2025-10-30 01:42:04[0m] (step=0055300) Train Loss: 0.5523, Train Steps/Sec: 1.27
|
| 628 |
+
[[34m2025-10-30 01:43:22[0m] (step=0055400) Train Loss: 0.5528, Train Steps/Sec: 1.27
|
| 629 |
+
[[34m2025-10-30 01:44:41[0m] (step=0055500) Train Loss: 0.5511, Train Steps/Sec: 1.27
|
| 630 |
+
[[34m2025-10-30 01:46:00[0m] (step=0055600) Train Loss: 0.5527, Train Steps/Sec: 1.27
|
| 631 |
+
[[34m2025-10-30 01:47:18[0m] (step=0055700) Train Loss: 0.5527, Train Steps/Sec: 1.27
|
| 632 |
+
[[34m2025-10-30 01:48:37[0m] (step=0055800) Train Loss: 0.5528, Train Steps/Sec: 1.27
|
| 633 |
+
[[34m2025-10-30 01:49:55[0m] (step=0055900) Train Loss: 0.5517, Train Steps/Sec: 1.27
|
| 634 |
+
[[34m2025-10-30 01:51:14[0m] (step=0056000) Train Loss: 0.5541, Train Steps/Sec: 1.27
|
| 635 |
+
[[34m2025-10-30 01:52:33[0m] (step=0056100) Train Loss: 0.5516, Train Steps/Sec: 1.27
|
| 636 |
+
[[34m2025-10-30 01:53:52[0m] (step=0056200) Train Loss: 0.5554, Train Steps/Sec: 1.26
|
| 637 |
+
[[34m2025-10-30 01:55:07[0m] Beginning epoch 45...
|
| 638 |
+
[[34m2025-10-30 01:55:14[0m] (step=0056300) Train Loss: 0.5581, Train Steps/Sec: 1.23
|
| 639 |
+
[[34m2025-10-30 01:56:32[0m] (step=0056400) Train Loss: 0.5602, Train Steps/Sec: 1.27
|
| 640 |
+
[[34m2025-10-30 01:57:51[0m] (step=0056500) Train Loss: 0.5575, Train Steps/Sec: 1.27
|
| 641 |
+
[[34m2025-10-30 01:59:09[0m] (step=0056600) Train Loss: 0.5592, Train Steps/Sec: 1.27
|
| 642 |
+
[[34m2025-10-30 02:00:28[0m] (step=0056700) Train Loss: 0.5689, Train Steps/Sec: 1.27
|
| 643 |
+
[[34m2025-10-30 02:01:46[0m] (step=0056800) Train Loss: 1.4227, Train Steps/Sec: 1.27
|
| 644 |
+
[[34m2025-10-30 02:03:05[0m] (step=0056900) Train Loss: 2.0115, Train Steps/Sec: 1.27
|
| 645 |
+
[[34m2025-10-30 02:04:24[0m] (step=0057000) Train Loss: 1.9414, Train Steps/Sec: 1.27
|
| 646 |
+
[[34m2025-10-30 02:05:42[0m] (step=0057100) Train Loss: 1.6740, Train Steps/Sec: 1.27
|
| 647 |
+
[[34m2025-10-30 02:07:01[0m] (step=0057200) Train Loss: 1.6082, Train Steps/Sec: 1.27
|
| 648 |
+
[[34m2025-10-30 02:08:19[0m] (step=0057300) Train Loss: 1.3177, Train Steps/Sec: 1.27
|
| 649 |
+
[[34m2025-10-30 02:09:38[0m] (step=0057400) Train Loss: 0.7391, Train Steps/Sec: 1.27
|
| 650 |
+
[[34m2025-10-30 02:10:56[0m] (step=0057500) Train Loss: 0.6675, Train Steps/Sec: 1.27
|
| 651 |
+
[[34m2025-10-30 02:11:33[0m] Beginning epoch 46...
|
| 652 |
+
[[34m2025-10-30 02:12:18[0m] (step=0057600) Train Loss: 0.6626, Train Steps/Sec: 1.22
|
| 653 |
+
[[34m2025-10-30 02:13:37[0m] (step=0057700) Train Loss: 0.6753, Train Steps/Sec: 1.27
|
| 654 |
+
[[34m2025-10-30 02:14:56[0m] (step=0057800) Train Loss: 0.6694, Train Steps/Sec: 1.27
|
| 655 |
+
[[34m2025-10-30 02:16:14[0m] (step=0057900) Train Loss: 0.6624, Train Steps/Sec: 1.27
|
| 656 |
+
[[34m2025-10-30 02:17:33[0m] (step=0058000) Train Loss: 0.6798, Train Steps/Sec: 1.27
|
| 657 |
+
[[34m2025-10-30 02:18:51[0m] (step=0058100) Train Loss: 0.6752, Train Steps/Sec: 1.27
|
| 658 |
+
[[34m2025-10-30 02:20:10[0m] (step=0058200) Train Loss: 0.7096, Train Steps/Sec: 1.27
|
| 659 |
+
[[34m2025-10-30 02:21:28[0m] (step=0058300) Train Loss: 0.7234, Train Steps/Sec: 1.27
|
| 660 |
+
[[34m2025-10-30 02:22:47[0m] (step=0058400) Train Loss: 0.7318, Train Steps/Sec: 1.27
|
| 661 |
+
[[34m2025-10-30 02:24:05[0m] (step=0058500) Train Loss: 0.7374, Train Steps/Sec: 1.27
|
| 662 |
+
[[34m2025-10-30 02:25:24[0m] (step=0058600) Train Loss: 0.6907, Train Steps/Sec: 1.27
|
| 663 |
+
[[34m2025-10-30 02:26:42[0m] (step=0058700) Train Loss: 0.6444, Train Steps/Sec: 1.27
|
| 664 |
+
[[34m2025-10-30 02:27:59[0m] Beginning epoch 47...
|
| 665 |
+
[[34m2025-10-30 02:28:04[0m] (step=0058800) Train Loss: 0.6353, Train Steps/Sec: 1.23
|
| 666 |
+
[[34m2025-10-30 02:29:22[0m] (step=0058900) Train Loss: 0.6586, Train Steps/Sec: 1.28
|
| 667 |
+
[[34m2025-10-30 02:30:41[0m] (step=0059000) Train Loss: 0.6600, Train Steps/Sec: 1.27
|