2026-03-08 20:34:22,544 [Rank train_rank0] Building UniDiT model... 2026-03-08 20:39:16,640 [Rank train_rank0] Building UniDiT model... 2026-03-08 20:40:00,676 [Rank train_rank0] Building RAE (frozen)... 2026-03-08 20:40:09,663 [Rank train_rank0] Loaded RAE decoder from '/scratch/e1539128/RAE/ckpts/stage1/qwen3vlrae_gram_v1.4_448/checkpoints/ep-0000029.pt' (source='ema', decoder_keys=271, to_pixels_keys_in_ckpt=2). 2026-03-08 20:40:10,389 [Rank train_rank0] UniDiT trainable: 1455.91M / total: 5894.39M 2026-03-08 20:40:10,390 [Rank train_rank0] Noise schedule shift: enabled=True, ratio=14.000000 2026-03-08 20:40:10,391 [Rank train_rank0] Latent normalization: enabled=True, path='ckpts/latent_stats.pt', global_mean=0.287388, global_std=28.285812, token_count=40140800 2026-03-08 20:40:10,391 [Rank train_rank0] CFG context dropout: 0.0 2026-03-08 20:40:10,392 [Rank train_rank0] EMA maintenance: disabled (no EMA weights tracked during training). 2026-03-08 20:40:11,500 [Rank train_rank0] Optimizer: AdamW(lr=0.0002, betas=(0.9, 0.95), wd=0.01) 2026-03-08 20:40:11,501 [Rank train_rank0] Optimizer override: set weight_decay=0.0 for trainable query-token embeddings. 2026-03-08 20:40:11,501 [Rank train_rank0] Configuring streaming data loader... 2026-03-08 20:40:11,502 [Rank train_rank0] Streaming retry policy: max_consecutive_failures=32, rebuild_after_consecutive_failures=6, base_sleep=2.0s, max_sleep=20.0s. 2026-03-08 20:40:11,503 [Rank train_rank0] Loader config: num_workers=2, prefetch_factor=2, shuffle_buffer_size=2000, loader_timeout=600s, curl(connect_timeout=30, max_time=0, retry=15, speed_time=120, speed_limit=65536) 2026-03-08 20:40:11,503 [Rank train_rank0] Mixed dataset plan (configured weight -> normalized ratio): 2026-03-08 20:40:11,504 [Rank train_rank0] [1] fine_t2i-enhanced_prompt_square_resolution: weight=0.5, ratio=50.00%, url=https://huggingface.co/datasets/ma-xu/fine-t2i/resolve/main/synthetic_enhanced_prompt_square_resolution/train-{000000..001543}.tar 2026-03-08 20:40:11,504 [Rank train_rank0] [2] fine_t2i-original_prompt_square_resolution: weight=0.5, ratio=50.00%, url=https://huggingface.co/datasets/ma-xu/fine-t2i/resolve/main/synthetic_original_prompt_square_resolution/train-{000000..001310}.tar 2026-03-08 20:40:11,505 [Rank train_rank0] Scheduler: cosine(warmup=0.07ep/2100steps, decay_end=30.0ep/900000steps, base_lr=0.0002, final_lr=1e-06) 2026-03-08 20:40:11,507 [Rank train_rank0] Resuming from /scratch/e1539128/ckpt-8gpu-new/dit_training/checkpoints/step-0000018000.pt... 2026-03-08 20:40:29,400 [Rank train_rank0] Resume mode: loaded model/EMA only; reset epoch/step to 0 and restarted scheduler. 2026-03-08 20:40:29,401 [Rank train_rank0] Resumed at epoch=0, step=0 2026-03-08 20:40:37,235 [Rank train_rank0] Sample visualization: sample_with_ema=False, log_non_ema_samples=False, sample_num_steps=10 2026-03-08 20:40:37,236 [Rank train_rank0] Starting training: 30 epochs, micro_bs=8/gpu, accum=4, global_bs=128, image_size=448, steps_per_epoch=30000 2026-03-08 20:40:37,237 [Rank train_rank0] Checkpoint cadence: every 1000 steps. 2026-03-08 20:40:37,238 [Rank train_rank0] Building stream loader (stream_epoch=0)... 2026-03-08 20:40:37,243 [Rank train_rank0] Creating stream iterator... 2026-03-08 20:40:37,244 [Rank train_rank0] Stream iterator is ready. 2026-03-08 20:42:11,595 [Rank train_rank0] [Epoch 0 | Step 20] train/loss: 0.520191, train/mse: 0.520191, train/data_wait_sec: 0.192465, train/step_wall_sec: 4.307414, train/data_wait_ratio: 0.044682, train/lr: 0.000002, train/epoch: 0.000000, train/step: 20.000000 2026-03-08 20:43:38,513 [Rank train_rank0] [Epoch 0 | Step 40] train/loss: 0.564690, train/mse: 0.564690, train/data_wait_sec: 0.094033, train/step_wall_sec: 4.259959, train/data_wait_ratio: 0.022074, train/lr: 0.000004, train/epoch: 0.000000, train/step: 40.000000 2026-03-08 20:45:05,574 [Rank train_rank0] [Epoch 0 | Step 60] train/loss: 0.499717, train/mse: 0.499717, train/data_wait_sec: 0.125661, train/step_wall_sec: 4.285154, train/data_wait_ratio: 0.029325, train/lr: 0.000006, train/epoch: 0.000000, train/step: 60.000000 2026-03-08 20:46:32,699 [Rank train_rank0] [Epoch 0 | Step 80] train/loss: 0.478646, train/mse: 0.478646, train/data_wait_sec: 0.140119, train/step_wall_sec: 4.248167, train/data_wait_ratio: 0.032983, train/lr: 0.000008, train/epoch: 0.000000, train/step: 80.000000 2026-03-08 20:47:59,341 [Rank train_rank0] [Epoch 0 | Step 100] train/loss: 0.488427, train/mse: 0.488427, train/data_wait_sec: 0.130530, train/step_wall_sec: 4.305884, train/data_wait_ratio: 0.030314, train/lr: 0.000010, train/epoch: 0.000000, train/step: 100.000000 2026-03-08 20:50:49,754 [Rank train_rank0] Building UniDiT model... 2026-03-08 20:51:23,759 [Rank train_rank0] Building RAE (frozen)... 2026-03-08 20:51:32,310 [Rank train_rank0] Loaded RAE decoder from '/scratch/e1539128/RAE/ckpts/stage1/qwen3vlrae_gram_v1.4_448/checkpoints/ep-0000029.pt' (source='ema', decoder_keys=271, to_pixels_keys_in_ckpt=2). 2026-03-08 20:51:32,704 [Rank train_rank0] UniDiT trainable: 1455.91M / total: 5894.39M 2026-03-08 20:51:32,705 [Rank train_rank0] Noise schedule shift: enabled=True, ratio=14.000000 2026-03-08 20:51:32,705 [Rank train_rank0] Latent normalization: enabled=True, path='ckpts/latent_stats.pt', global_mean=0.287388, global_std=28.285812, token_count=40140800 2026-03-08 20:51:32,706 [Rank train_rank0] CFG context dropout: 0.0 2026-03-08 20:51:32,706 [Rank train_rank0] EMA maintenance: disabled (no EMA weights tracked during training). 2026-03-08 20:51:33,833 [Rank train_rank0] Optimizer: AdamW(lr=0.0002, betas=(0.9, 0.95), wd=0.01) 2026-03-08 20:51:33,834 [Rank train_rank0] Optimizer override: set weight_decay=0.0 for trainable query-token embeddings. 2026-03-08 20:51:33,834 [Rank train_rank0] Configuring streaming data loader... 2026-03-08 20:51:33,835 [Rank train_rank0] Streaming retry policy: max_consecutive_failures=32, rebuild_after_consecutive_failures=6, base_sleep=2.0s, max_sleep=20.0s. 2026-03-08 20:51:33,836 [Rank train_rank0] Loader config: num_workers=2, prefetch_factor=2, shuffle_buffer_size=2000, loader_timeout=600s, curl(connect_timeout=30, max_time=0, retry=15, speed_time=120, speed_limit=65536) 2026-03-08 20:51:33,837 [Rank train_rank0] Mixed dataset plan (configured weight -> normalized ratio): 2026-03-08 20:51:33,837 [Rank train_rank0] [1] fine_t2i-enhanced_prompt_square_resolution: weight=0.5, ratio=50.00%, url=https://huggingface.co/datasets/ma-xu/fine-t2i/resolve/main/synthetic_enhanced_prompt_square_resolution/train-{000000..001543}.tar 2026-03-08 20:51:33,838 [Rank train_rank0] [2] fine_t2i-original_prompt_square_resolution: weight=0.5, ratio=50.00%, url=https://huggingface.co/datasets/ma-xu/fine-t2i/resolve/main/synthetic_original_prompt_square_resolution/train-{000000..001310}.tar 2026-03-08 20:51:33,839 [Rank train_rank0] Scheduler: cosine(warmup=0.07ep/2100steps, decay_end=30.0ep/900000steps, base_lr=0.0002, final_lr=1e-06) 2026-03-08 20:51:33,841 [Rank train_rank0] Resuming from /scratch/e1539128/ckpt-8gpu-new/dit_training/checkpoints/step-0000018000.pt... 2026-03-08 20:51:51,096 [Rank train_rank0] Resume mode: loaded model/EMA only; reset epoch/step to 0 and restarted scheduler. 2026-03-08 20:51:51,097 [Rank train_rank0] Resumed at epoch=0, step=0 2026-03-08 20:51:58,179 [Rank train_rank0] Sample visualization: sample_with_ema=False, log_non_ema_samples=False, sample_num_steps=10 2026-03-08 20:51:58,180 [Rank train_rank0] Starting training: 30 epochs, micro_bs=10/gpu, accum=4, global_bs=160, image_size=448, steps_per_epoch=30000 2026-03-08 20:51:58,181 [Rank train_rank0] Checkpoint cadence: every 1000 steps. 2026-03-08 20:51:58,182 [Rank train_rank0] Building stream loader (stream_epoch=0)... 2026-03-08 20:51:58,188 [Rank train_rank0] Creating stream iterator... 2026-03-08 20:51:58,188 [Rank train_rank0] Stream iterator is ready. 2026-03-08 20:53:52,288 [Rank train_rank0] [Epoch 0 | Step 20] train/loss: 0.547013, train/mse: 0.547013, train/data_wait_sec: 0.223749, train/step_wall_sec: 5.481787, train/data_wait_ratio: 0.040817, train/lr: 0.000002, train/epoch: 0.000000, train/step: 20.000000 2026-03-08 20:55:41,329 [Rank train_rank0] [Epoch 0 | Step 40] train/loss: 0.514379, train/mse: 0.514379, train/data_wait_sec: 0.150323, train/step_wall_sec: 5.460392, train/data_wait_ratio: 0.027530, train/lr: 0.000004, train/epoch: 0.000000, train/step: 40.000000 2026-03-08 20:57:30,113 [Rank train_rank0] [Epoch 0 | Step 60] train/loss: 0.477623, train/mse: 0.477623, train/data_wait_sec: 0.093980, train/step_wall_sec: 5.320937, train/data_wait_ratio: 0.017662, train/lr: 0.000006, train/epoch: 0.000000, train/step: 60.000000 2026-03-08 20:59:19,441 [Rank train_rank0] [Epoch 0 | Step 80] train/loss: 0.470775, train/mse: 0.470775, train/data_wait_sec: 0.163414, train/step_wall_sec: 5.421515, train/data_wait_ratio: 0.030142, train/lr: 0.000008, train/epoch: 0.000000, train/step: 80.000000 2026-03-08 21:01:07,836 [Rank train_rank0] [Epoch 0 | Step 100] train/loss: 0.476011, train/mse: 0.476011, train/data_wait_sec: 0.117472, train/step_wall_sec: 5.413506, train/data_wait_ratio: 0.021700, train/lr: 0.000010, train/epoch: 0.000000, train/step: 100.000000 2026-03-08 21:02:56,832 [Rank train_rank0] [Epoch 0 | Step 120] train/loss: 0.487018, train/mse: 0.487018, train/data_wait_sec: 0.148194, train/step_wall_sec: 5.408652, train/data_wait_ratio: 0.027399, train/lr: 0.000011, train/epoch: 0.000000, train/step: 120.000000 2026-03-08 21:04:45,524 [Rank train_rank0] [Epoch 0 | Step 140] train/loss: 0.460542, train/mse: 0.460542, train/data_wait_sec: 0.174132, train/step_wall_sec: 5.513119, train/data_wait_ratio: 0.031585, train/lr: 0.000013, train/epoch: 0.000000, train/step: 140.000000 2026-03-08 21:06:34,918 [Rank train_rank0] [Epoch 0 | Step 160] train/loss: 0.459310, train/mse: 0.459310, train/data_wait_sec: 0.077221, train/step_wall_sec: 5.346796, train/data_wait_ratio: 0.014443, train/lr: 0.000015, train/epoch: 0.000000, train/step: 160.000000 2026-03-08 21:08:23,781 [Rank train_rank0] [Epoch 0 | Step 180] train/loss: 0.489676, train/mse: 0.489676, train/data_wait_sec: 0.121184, train/step_wall_sec: 5.432119, train/data_wait_ratio: 0.022309, train/lr: 0.000017, train/epoch: 0.000000, train/step: 180.000000 2026-03-08 21:10:12,664 [Rank train_rank0] [Epoch 0 | Step 200] train/loss: 0.455036, train/mse: 0.455036, train/data_wait_sec: 0.106095, train/step_wall_sec: 5.365625, train/data_wait_ratio: 0.019773, train/lr: 0.000019, train/epoch: 0.000000, train/step: 200.000000 2026-03-08 21:12:01,736 [Rank train_rank0] [Epoch 0 | Step 220] train/loss: 0.487731, train/mse: 0.487731, train/data_wait_sec: 0.147974, train/step_wall_sec: 5.385463, train/data_wait_ratio: 0.027476, train/lr: 0.000021, train/epoch: 0.000000, train/step: 220.000000 2026-03-08 21:13:50,400 [Rank train_rank0] [Epoch 0 | Step 240] train/loss: 0.437401, train/mse: 0.437401, train/data_wait_sec: 0.119918, train/step_wall_sec: 5.360691, train/data_wait_ratio: 0.022370, train/lr: 0.000023, train/epoch: 0.000000, train/step: 240.000000 2026-03-08 21:15:39,258 [Rank train_rank0] [Epoch 0 | Step 260] train/loss: 0.461089, train/mse: 0.461089, train/data_wait_sec: 0.160618, train/step_wall_sec: 5.405825, train/data_wait_ratio: 0.029712, train/lr: 0.000025, train/epoch: 0.000000, train/step: 260.000000 2026-03-08 21:17:28,476 [Rank train_rank0] [Epoch 0 | Step 280] train/loss: 0.456767, train/mse: 0.456767, train/data_wait_sec: 0.095814, train/step_wall_sec: 5.336097, train/data_wait_ratio: 0.017956, train/lr: 0.000027, train/epoch: 0.000000, train/step: 280.000000 2026-03-08 21:19:16,927 [Rank train_rank0] [Epoch 0 | Step 300] train/loss: 0.475074, train/mse: 0.475074, train/data_wait_sec: 0.157410, train/step_wall_sec: 5.334435, train/data_wait_ratio: 0.029508, train/lr: 0.000029, train/epoch: 0.000000, train/step: 300.000000 2026-03-08 21:21:05,762 [Rank train_rank0] [Epoch 0 | Step 320] train/loss: 0.463954, train/mse: 0.463954, train/data_wait_sec: 0.125450, train/step_wall_sec: 5.391949, train/data_wait_ratio: 0.023266, train/lr: 0.000030, train/epoch: 0.000000, train/step: 320.000000 2026-03-08 21:22:54,541 [Rank train_rank0] [Epoch 0 | Step 340] train/loss: 0.456039, train/mse: 0.456039, train/data_wait_sec: 0.124405, train/step_wall_sec: 5.430477, train/data_wait_ratio: 0.022909, train/lr: 0.000032, train/epoch: 0.000000, train/step: 340.000000 2026-03-08 21:24:43,040 [Rank train_rank0] [Epoch 0 | Step 360] train/loss: 0.446811, train/mse: 0.446811, train/data_wait_sec: 0.126252, train/step_wall_sec: 5.295104, train/data_wait_ratio: 0.023843, train/lr: 0.000034, train/epoch: 0.000000, train/step: 360.000000 2026-03-08 21:26:31,912 [Rank train_rank0] [Epoch 0 | Step 380] train/loss: 0.486607, train/mse: 0.486607, train/data_wait_sec: 0.172775, train/step_wall_sec: 5.405606, train/data_wait_ratio: 0.031962, train/lr: 0.000036, train/epoch: 0.000000, train/step: 380.000000 2026-03-08 21:28:20,728 [Rank train_rank0] [Epoch 0 | Step 400] train/loss: 0.454976, train/mse: 0.454976, train/data_wait_sec: 0.103295, train/step_wall_sec: 5.421176, train/data_wait_ratio: 0.019054, train/lr: 0.000038, train/epoch: 0.000000, train/step: 400.000000 2026-03-08 21:30:09,449 [Rank train_rank0] [Epoch 0 | Step 420] train/loss: 0.465427, train/mse: 0.465427, train/data_wait_sec: 0.151678, train/step_wall_sec: 5.390993, train/data_wait_ratio: 0.028135, train/lr: 0.000040, train/epoch: 0.000000, train/step: 420.000000 2026-03-08 21:31:58,372 [Rank train_rank0] [Epoch 0 | Step 440] train/loss: 0.438997, train/mse: 0.438997, train/data_wait_sec: 0.120272, train/step_wall_sec: 5.332174, train/data_wait_ratio: 0.022556, train/lr: 0.000042, train/epoch: 0.000000, train/step: 440.000000 2026-03-08 21:33:47,111 [Rank train_rank0] [Epoch 0 | Step 460] train/loss: 0.475074, train/mse: 0.475074, train/data_wait_sec: 0.195787, train/step_wall_sec: 5.462268, train/data_wait_ratio: 0.035843, train/lr: 0.000044, train/epoch: 0.000000, train/step: 460.000000 2026-03-08 21:35:35,575 [Rank train_rank0] [Epoch 0 | Step 480] train/loss: 0.471543, train/mse: 0.471543, train/data_wait_sec: 0.075564, train/step_wall_sec: 5.298366, train/data_wait_ratio: 0.014262, train/lr: 0.000046, train/epoch: 0.000000, train/step: 480.000000 2026-03-08 21:37:24,138 [Rank train_rank0] [Epoch 0 | Step 500] train/loss: 0.464237, train/mse: 0.464237, train/data_wait_sec: 0.128918, train/step_wall_sec: 5.378256, train/data_wait_ratio: 0.023970, train/lr: 0.000048, train/epoch: 0.000000, train/step: 500.000000 2026-03-08 21:39:13,532 [Rank train_rank0] [Epoch 0 | Step 520] train/loss: 0.448553, train/mse: 0.448553, train/data_wait_sec: 0.168594, train/step_wall_sec: 5.404853, train/data_wait_ratio: 0.031193, train/lr: 0.000050, train/epoch: 0.000000, train/step: 520.000000 2026-03-08 21:41:02,428 [Rank train_rank0] [Epoch 0 | Step 540] train/loss: 0.449332, train/mse: 0.449332, train/data_wait_sec: 0.175267, train/step_wall_sec: 5.427399, train/data_wait_ratio: 0.032293, train/lr: 0.000051, train/epoch: 0.000000, train/step: 540.000000 2026-03-08 21:42:51,132 [Rank train_rank0] [Epoch 0 | Step 560] train/loss: 0.458501, train/mse: 0.458501, train/data_wait_sec: 0.233797, train/step_wall_sec: 5.431725, train/data_wait_ratio: 0.043043, train/lr: 0.000053, train/epoch: 0.000000, train/step: 560.000000 2026-03-08 21:44:39,991 [Rank train_rank0] [Epoch 0 | Step 580] train/loss: 0.469743, train/mse: 0.469743, train/data_wait_sec: 0.142472, train/step_wall_sec: 5.326207, train/data_wait_ratio: 0.026749, train/lr: 0.000055, train/epoch: 0.000000, train/step: 580.000000 2026-03-08 21:46:29,063 [Rank train_rank0] [Epoch 0 | Step 600] train/loss: 0.458141, train/mse: 0.458141, train/data_wait_sec: 0.097585, train/step_wall_sec: 5.412449, train/data_wait_ratio: 0.018030, train/lr: 0.000057, train/epoch: 0.000000, train/step: 600.000000 2026-03-08 21:48:17,652 [Rank train_rank0] [Epoch 0 | Step 620] train/loss: 0.470807, train/mse: 0.470807, train/data_wait_sec: 0.204230, train/step_wall_sec: 5.435657, train/data_wait_ratio: 0.037572, train/lr: 0.000059, train/epoch: 0.000000, train/step: 620.000000 2026-03-08 21:50:06,626 [Rank train_rank0] [Epoch 0 | Step 640] train/loss: 0.468638, train/mse: 0.468638, train/data_wait_sec: 0.137086, train/step_wall_sec: 5.363351, train/data_wait_ratio: 0.025560, train/lr: 0.000061, train/epoch: 0.000000, train/step: 640.000000 2026-03-08 21:51:55,660 [Rank train_rank0] [Epoch 0 | Step 660] train/loss: 0.452559, train/mse: 0.452559, train/data_wait_sec: 0.101116, train/step_wall_sec: 5.371146, train/data_wait_ratio: 0.018826, train/lr: 0.000063, train/epoch: 0.000000, train/step: 660.000000 2026-03-08 21:53:44,227 [Rank train_rank0] [Epoch 0 | Step 680] train/loss: 0.454775, train/mse: 0.454775, train/data_wait_sec: 0.196892, train/step_wall_sec: 5.451818, train/data_wait_ratio: 0.036115, train/lr: 0.000065, train/epoch: 0.000000, train/step: 680.000000 2026-03-08 21:55:33,097 [Rank train_rank0] [Epoch 0 | Step 700] train/loss: 0.492381, train/mse: 0.492381, train/data_wait_sec: 0.091893, train/step_wall_sec: 5.357524, train/data_wait_ratio: 0.017152, train/lr: 0.000067, train/epoch: 0.000000, train/step: 700.000000 2026-03-08 21:57:21,794 [Rank train_rank0] [Epoch 0 | Step 720] train/loss: 0.466496, train/mse: 0.466496, train/data_wait_sec: 0.137865, train/step_wall_sec: 5.377726, train/data_wait_ratio: 0.025636, train/lr: 0.000069, train/epoch: 0.000000, train/step: 720.000000 2026-03-08 21:59:10,990 [Rank train_rank0] [Epoch 0 | Step 740] train/loss: 0.492215, train/mse: 0.492215, train/data_wait_sec: 0.153661, train/step_wall_sec: 5.452626, train/data_wait_ratio: 0.028181, train/lr: 0.000070, train/epoch: 0.000000, train/step: 740.000000 2026-03-08 22:00:59,949 [Rank train_rank0] [Epoch 0 | Step 760] train/loss: 0.453624, train/mse: 0.453624, train/data_wait_sec: 0.123247, train/step_wall_sec: 5.351594, train/data_wait_ratio: 0.023030, train/lr: 0.000072, train/epoch: 0.000000, train/step: 760.000000 2026-03-08 22:02:48,963 [Rank train_rank0] [Epoch 0 | Step 780] train/loss: 0.465036, train/mse: 0.465036, train/data_wait_sec: 0.130796, train/step_wall_sec: 5.433473, train/data_wait_ratio: 0.024072, train/lr: 0.000074, train/epoch: 0.000000, train/step: 780.000000 2026-03-08 22:04:37,588 [Rank train_rank0] [Epoch 0 | Step 800] train/loss: 0.473741, train/mse: 0.473741, train/data_wait_sec: 0.172938, train/step_wall_sec: 5.507158, train/data_wait_ratio: 0.031402, train/lr: 0.000076, train/epoch: 0.000000, train/step: 800.000000 2026-03-08 22:06:26,593 [Rank train_rank0] [Epoch 0 | Step 820] train/loss: 0.454996, train/mse: 0.454996, train/data_wait_sec: 0.176798, train/step_wall_sec: 5.428974, train/data_wait_ratio: 0.032566, train/lr: 0.000078, train/epoch: 0.000000, train/step: 820.000000 2026-03-08 22:08:15,093 [Rank train_rank0] [Epoch 0 | Step 840] train/loss: 0.481142, train/mse: 0.481142, train/data_wait_sec: 0.194577, train/step_wall_sec: 5.394446, train/data_wait_ratio: 0.036070, train/lr: 0.000080, train/epoch: 0.000000, train/step: 840.000000 2026-03-08 22:10:03,934 [Rank train_rank0] [Epoch 0 | Step 860] train/loss: 0.466172, train/mse: 0.466172, train/data_wait_sec: 0.102682, train/step_wall_sec: 5.331469, train/data_wait_ratio: 0.019260, train/lr: 0.000082, train/epoch: 0.000000, train/step: 860.000000 2026-03-08 22:11:52,947 [Rank train_rank0] [Epoch 0 | Step 880] train/loss: 0.457527, train/mse: 0.457527, train/data_wait_sec: 0.094723, train/step_wall_sec: 5.332883, train/data_wait_ratio: 0.017762, train/lr: 0.000084, train/epoch: 0.000000, train/step: 880.000000 2026-03-08 22:13:41,872 [Rank train_rank0] [Epoch 0 | Step 900] train/loss: 0.472166, train/mse: 0.472166, train/data_wait_sec: 0.172783, train/step_wall_sec: 5.482841, train/data_wait_ratio: 0.031513, train/lr: 0.000086, train/epoch: 0.000000, train/step: 900.000000 2026-03-08 22:15:30,754 [Rank train_rank0] [Epoch 0 | Step 920] train/loss: 0.475747, train/mse: 0.475747, train/data_wait_sec: 0.191762, train/step_wall_sec: 5.441937, train/data_wait_ratio: 0.035238, train/lr: 0.000088, train/epoch: 0.000000, train/step: 920.000000 2026-03-08 22:17:19,708 [Rank train_rank0] [Epoch 0 | Step 940] train/loss: 0.477062, train/mse: 0.477062, train/data_wait_sec: 0.124684, train/step_wall_sec: 5.341643, train/data_wait_ratio: 0.023342, train/lr: 0.000090, train/epoch: 0.000000, train/step: 940.000000 2026-03-08 22:19:09,064 [Rank train_rank0] [Epoch 0 | Step 960] train/loss: 0.457286, train/mse: 0.457286, train/data_wait_sec: 0.257220, train/step_wall_sec: 5.465852, train/data_wait_ratio: 0.047059, train/lr: 0.000091, train/epoch: 0.000000, train/step: 960.000000 2026-03-08 22:20:57,936 [Rank train_rank0] [Epoch 0 | Step 980] train/loss: 0.459431, train/mse: 0.459431, train/data_wait_sec: 0.185744, train/step_wall_sec: 5.512186, train/data_wait_ratio: 0.033697, train/lr: 0.000093, train/epoch: 0.000000, train/step: 980.000000 2026-03-08 22:23:12,712 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000001000.pt 2026-03-08 22:23:12,714 [Rank train_rank0] [Epoch 0 | Step 1000] train/loss: 0.450517, train/mse: 0.450517, train/data_wait_sec: 0.111019, train/step_wall_sec: 5.459261, train/data_wait_ratio: 0.020336, train/lr: 0.000095, train/epoch: 0.000000, train/step: 1000.000000 2026-03-08 22:23:12,716 [Rank train_rank0] Generating samples for visualization... 2026-03-08 22:23:17,933 [Rank train_rank0] Sample generation done. 2026-03-08 22:25:07,119 [Rank train_rank0] [Epoch 0 | Step 1020] train/loss: 0.451829, train/mse: 0.451829, train/data_wait_sec: 0.134668, train/step_wall_sec: 5.324781, train/data_wait_ratio: 0.025291, train/lr: 0.000097, train/epoch: 0.000000, train/step: 1020.000000 2026-03-08 22:26:56,369 [Rank train_rank0] [Epoch 0 | Step 1040] train/loss: 0.473831, train/mse: 0.473831, train/data_wait_sec: 0.141548, train/step_wall_sec: 5.374008, train/data_wait_ratio: 0.026339, train/lr: 0.000099, train/epoch: 0.000000, train/step: 1040.000000 2026-03-08 22:28:44,895 [Rank train_rank0] [Epoch 0 | Step 1060] train/loss: 0.468956, train/mse: 0.468956, train/data_wait_sec: 0.187890, train/step_wall_sec: 5.496477, train/data_wait_ratio: 0.034184, train/lr: 0.000101, train/epoch: 0.000000, train/step: 1060.000000 2026-03-08 22:30:33,445 [Rank train_rank0] [Epoch 0 | Step 1080] train/loss: 0.417721, train/mse: 0.417721, train/data_wait_sec: 0.149168, train/step_wall_sec: 5.437478, train/data_wait_ratio: 0.027433, train/lr: 0.000103, train/epoch: 0.000000, train/step: 1080.000000 2026-03-08 22:32:22,622 [Rank train_rank0] [Epoch 0 | Step 1100] train/loss: 0.494283, train/mse: 0.494283, train/data_wait_sec: 0.177796, train/step_wall_sec: 5.454914, train/data_wait_ratio: 0.032594, train/lr: 0.000105, train/epoch: 0.000000, train/step: 1100.000000 2026-03-08 22:33:49,129 [Rank train_rank0] [Epoch 0 | Step 1120] train/loss: 0.452837, train/mse: 0.452837, train/data_wait_sec: 0.074247, train/step_wall_sec: 3.668529, train/data_wait_ratio: 0.020239, train/lr: 0.000107, train/epoch: 0.000000, train/step: 1120.000000 2026-03-08 22:35:03,818 [Rank train_rank0] [Epoch 0 | Step 1140] train/loss: 0.432794, train/mse: 0.432794, train/data_wait_sec: 0.111889, train/step_wall_sec: 3.834358, train/data_wait_ratio: 0.029181, train/lr: 0.000109, train/epoch: 0.000000, train/step: 1140.000000 2026-03-08 22:36:18,107 [Rank train_rank0] [Epoch 0 | Step 1160] train/loss: 0.490389, train/mse: 0.490389, train/data_wait_sec: 0.058816, train/step_wall_sec: 3.669933, train/data_wait_ratio: 0.016027, train/lr: 0.000110, train/epoch: 0.000000, train/step: 1160.000000 2026-03-08 22:37:32,732 [Rank train_rank0] [Epoch 0 | Step 1180] train/loss: 0.476351, train/mse: 0.476351, train/data_wait_sec: 0.060024, train/step_wall_sec: 3.705589, train/data_wait_ratio: 0.016198, train/lr: 0.000112, train/epoch: 0.000000, train/step: 1180.000000 2026-03-08 22:38:47,253 [Rank train_rank0] [Epoch 0 | Step 1200] train/loss: 0.462635, train/mse: 0.462635, train/data_wait_sec: 0.048454, train/step_wall_sec: 3.624363, train/data_wait_ratio: 0.013369, train/lr: 0.000114, train/epoch: 0.000000, train/step: 1200.000000 2026-03-08 22:40:01,756 [Rank train_rank0] [Epoch 0 | Step 1220] train/loss: 0.470201, train/mse: 0.470201, train/data_wait_sec: 0.054696, train/step_wall_sec: 3.666815, train/data_wait_ratio: 0.014916, train/lr: 0.000116, train/epoch: 0.000000, train/step: 1220.000000 2026-03-08 22:41:16,178 [Rank train_rank0] [Epoch 0 | Step 1240] train/loss: 0.443304, train/mse: 0.443304, train/data_wait_sec: 0.068692, train/step_wall_sec: 3.696455, train/data_wait_ratio: 0.018583, train/lr: 0.000118, train/epoch: 0.000000, train/step: 1240.000000 2026-03-08 22:42:30,854 [Rank train_rank0] [Epoch 0 | Step 1260] train/loss: 0.448767, train/mse: 0.448767, train/data_wait_sec: 0.096709, train/step_wall_sec: 3.700676, train/data_wait_ratio: 0.026133, train/lr: 0.000120, train/epoch: 0.000000, train/step: 1260.000000 2026-03-08 22:43:45,661 [Rank train_rank0] [Epoch 0 | Step 1280] train/loss: 0.474153, train/mse: 0.474153, train/data_wait_sec: 0.063423, train/step_wall_sec: 3.693656, train/data_wait_ratio: 0.017171, train/lr: 0.000122, train/epoch: 0.000000, train/step: 1280.000000 2026-03-08 22:45:00,076 [Rank train_rank0] [Epoch 0 | Step 1300] train/loss: 0.434658, train/mse: 0.434658, train/data_wait_sec: 0.064119, train/step_wall_sec: 3.652915, train/data_wait_ratio: 0.017553, train/lr: 0.000124, train/epoch: 0.000000, train/step: 1300.000000 2026-03-08 22:46:14,506 [Rank train_rank0] [Epoch 0 | Step 1320] train/loss: 0.441684, train/mse: 0.441684, train/data_wait_sec: 0.055859, train/step_wall_sec: 3.654280, train/data_wait_ratio: 0.015286, train/lr: 0.000126, train/epoch: 0.000000, train/step: 1320.000000 2026-03-08 22:47:29,126 [Rank train_rank0] [Epoch 0 | Step 1340] train/loss: 0.473924, train/mse: 0.473924, train/data_wait_sec: 0.077330, train/step_wall_sec: 3.688441, train/data_wait_ratio: 0.020965, train/lr: 0.000128, train/epoch: 0.000000, train/step: 1340.000000 2026-03-08 22:48:43,502 [Rank train_rank0] [Epoch 0 | Step 1360] train/loss: 0.440400, train/mse: 0.440400, train/data_wait_sec: 0.049228, train/step_wall_sec: 3.636356, train/data_wait_ratio: 0.013538, train/lr: 0.000130, train/epoch: 0.000000, train/step: 1360.000000 2026-03-08 22:49:58,061 [Rank train_rank0] [Epoch 0 | Step 1380] train/loss: 0.459153, train/mse: 0.459153, train/data_wait_sec: 0.154274, train/step_wall_sec: 3.809374, train/data_wait_ratio: 0.040499, train/lr: 0.000131, train/epoch: 0.000000, train/step: 1380.000000 2026-03-08 22:51:12,142 [Rank train_rank0] [Epoch 0 | Step 1400] train/loss: 0.454499, train/mse: 0.454499, train/data_wait_sec: 0.066761, train/step_wall_sec: 3.674802, train/data_wait_ratio: 0.018167, train/lr: 0.000133, train/epoch: 0.000000, train/step: 1400.000000 2026-03-08 22:52:26,653 [Rank train_rank0] [Epoch 0 | Step 1420] train/loss: 0.461901, train/mse: 0.461901, train/data_wait_sec: 0.066595, train/step_wall_sec: 3.673735, train/data_wait_ratio: 0.018127, train/lr: 0.000135, train/epoch: 0.000000, train/step: 1420.000000 2026-03-08 22:53:41,357 [Rank train_rank0] [Epoch 0 | Step 1440] train/loss: 0.470190, train/mse: 0.470190, train/data_wait_sec: 0.051723, train/step_wall_sec: 3.657007, train/data_wait_ratio: 0.014143, train/lr: 0.000137, train/epoch: 0.000000, train/step: 1440.000000 2026-03-08 22:54:55,921 [Rank train_rank0] [Epoch 0 | Step 1460] train/loss: 0.437505, train/mse: 0.437505, train/data_wait_sec: 0.076347, train/step_wall_sec: 3.683641, train/data_wait_ratio: 0.020726, train/lr: 0.000139, train/epoch: 0.000000, train/step: 1460.000000 2026-03-08 22:56:10,366 [Rank train_rank0] [Epoch 0 | Step 1480] train/loss: 0.466654, train/mse: 0.466654, train/data_wait_sec: 0.154961, train/step_wall_sec: 3.757269, train/data_wait_ratio: 0.041243, train/lr: 0.000141, train/epoch: 0.000000, train/step: 1480.000000 2026-03-08 22:57:24,880 [Rank train_rank0] [Epoch 0 | Step 1500] train/loss: 0.477297, train/mse: 0.477297, train/data_wait_sec: 0.050130, train/step_wall_sec: 3.650776, train/data_wait_ratio: 0.013731, train/lr: 0.000143, train/epoch: 0.000000, train/step: 1500.000000 2026-03-08 22:58:39,308 [Rank train_rank0] [Epoch 0 | Step 1520] train/loss: 0.440295, train/mse: 0.440295, train/data_wait_sec: 0.054404, train/step_wall_sec: 3.672404, train/data_wait_ratio: 0.014814, train/lr: 0.000145, train/epoch: 0.000000, train/step: 1520.000000 2026-03-08 22:59:53,866 [Rank train_rank0] [Epoch 0 | Step 1540] train/loss: 0.453597, train/mse: 0.453597, train/data_wait_sec: 0.098909, train/step_wall_sec: 3.701006, train/data_wait_ratio: 0.026725, train/lr: 0.000147, train/epoch: 0.000000, train/step: 1540.000000 2026-03-08 23:01:08,067 [Rank train_rank0] [Epoch 0 | Step 1560] train/loss: 0.483381, train/mse: 0.483381, train/data_wait_sec: 0.142168, train/step_wall_sec: 3.787310, train/data_wait_ratio: 0.037538, train/lr: 0.000149, train/epoch: 0.000000, train/step: 1560.000000 2026-03-08 23:02:22,176 [Rank train_rank0] [Epoch 0 | Step 1580] train/loss: 0.434037, train/mse: 0.434037, train/data_wait_sec: 0.047675, train/step_wall_sec: 3.670329, train/data_wait_ratio: 0.012989, train/lr: 0.000150, train/epoch: 0.000000, train/step: 1580.000000 2026-03-08 23:03:36,870 [Rank train_rank0] [Epoch 0 | Step 1600] train/loss: 0.445722, train/mse: 0.445722, train/data_wait_sec: 0.043952, train/step_wall_sec: 3.647890, train/data_wait_ratio: 0.012049, train/lr: 0.000152, train/epoch: 0.000000, train/step: 1600.000000 2026-03-08 23:04:51,197 [Rank train_rank0] [Epoch 0 | Step 1620] train/loss: 0.471794, train/mse: 0.471794, train/data_wait_sec: 0.066371, train/step_wall_sec: 3.688092, train/data_wait_ratio: 0.017996, train/lr: 0.000154, train/epoch: 0.000000, train/step: 1620.000000 2026-03-08 23:06:05,459 [Rank train_rank0] [Epoch 0 | Step 1640] train/loss: 0.451515, train/mse: 0.451515, train/data_wait_sec: 0.069441, train/step_wall_sec: 3.676994, train/data_wait_ratio: 0.018885, train/lr: 0.000156, train/epoch: 0.000000, train/step: 1640.000000 2026-03-08 23:07:19,921 [Rank train_rank0] [Epoch 0 | Step 1660] train/loss: 0.462220, train/mse: 0.462220, train/data_wait_sec: 0.053560, train/step_wall_sec: 3.678356, train/data_wait_ratio: 0.014561, train/lr: 0.000158, train/epoch: 0.000000, train/step: 1660.000000 2026-03-08 23:08:34,508 [Rank train_rank0] [Epoch 0 | Step 1680] train/loss: 0.469290, train/mse: 0.469290, train/data_wait_sec: 0.052694, train/step_wall_sec: 3.670894, train/data_wait_ratio: 0.014354, train/lr: 0.000160, train/epoch: 0.000000, train/step: 1680.000000 2026-03-08 23:09:49,145 [Rank train_rank0] [Epoch 0 | Step 1700] train/loss: 0.452756, train/mse: 0.452756, train/data_wait_sec: 0.063223, train/step_wall_sec: 3.691274, train/data_wait_ratio: 0.017128, train/lr: 0.000162, train/epoch: 0.000000, train/step: 1700.000000 2026-03-08 23:11:03,493 [Rank train_rank0] [Epoch 0 | Step 1720] train/loss: 0.453702, train/mse: 0.453702, train/data_wait_sec: 0.060119, train/step_wall_sec: 3.641442, train/data_wait_ratio: 0.016510, train/lr: 0.000164, train/epoch: 0.000000, train/step: 1720.000000 2026-03-08 23:12:17,825 [Rank train_rank0] [Epoch 0 | Step 1740] train/loss: 0.435368, train/mse: 0.435368, train/data_wait_sec: 0.061582, train/step_wall_sec: 3.679026, train/data_wait_ratio: 0.016739, train/lr: 0.000166, train/epoch: 0.000000, train/step: 1740.000000 2026-03-08 23:13:32,427 [Rank train_rank0] [Epoch 0 | Step 1760] train/loss: 0.439066, train/mse: 0.439066, train/data_wait_sec: 0.063537, train/step_wall_sec: 3.702068, train/data_wait_ratio: 0.017163, train/lr: 0.000168, train/epoch: 0.000000, train/step: 1760.000000 2026-03-08 23:14:46,915 [Rank train_rank0] [Epoch 0 | Step 1780] train/loss: 0.445977, train/mse: 0.445977, train/data_wait_sec: 0.046805, train/step_wall_sec: 3.650255, train/data_wait_ratio: 0.012822, train/lr: 0.000170, train/epoch: 0.000000, train/step: 1780.000000 2026-03-08 23:16:01,339 [Rank train_rank0] [Epoch 0 | Step 1800] train/loss: 0.471398, train/mse: 0.471398, train/data_wait_sec: 0.160600, train/step_wall_sec: 3.810021, train/data_wait_ratio: 0.042152, train/lr: 0.000171, train/epoch: 0.000000, train/step: 1800.000000 2026-03-08 23:17:15,553 [Rank train_rank0] [Epoch 0 | Step 1820] train/loss: 0.435804, train/mse: 0.435804, train/data_wait_sec: 0.090581, train/step_wall_sec: 3.680735, train/data_wait_ratio: 0.024609, train/lr: 0.000173, train/epoch: 0.000000, train/step: 1820.000000 2026-03-08 23:18:30,074 [Rank train_rank0] [Epoch 0 | Step 1840] train/loss: 0.444199, train/mse: 0.444199, train/data_wait_sec: 0.098581, train/step_wall_sec: 3.729451, train/data_wait_ratio: 0.026433, train/lr: 0.000175, train/epoch: 0.000000, train/step: 1840.000000 2026-03-08 23:19:44,576 [Rank train_rank0] [Epoch 0 | Step 1860] train/loss: 0.483314, train/mse: 0.483314, train/data_wait_sec: 0.073055, train/step_wall_sec: 3.685300, train/data_wait_ratio: 0.019823, train/lr: 0.000177, train/epoch: 0.000000, train/step: 1860.000000 2026-03-08 23:20:59,050 [Rank train_rank0] [Epoch 0 | Step 1880] train/loss: 0.438133, train/mse: 0.438133, train/data_wait_sec: 0.126394, train/step_wall_sec: 3.859834, train/data_wait_ratio: 0.032746, train/lr: 0.000179, train/epoch: 0.000000, train/step: 1880.000000 2026-03-08 23:22:13,026 [Rank train_rank0] [Epoch 0 | Step 1900] train/loss: 0.467699, train/mse: 0.467699, train/data_wait_sec: 0.055455, train/step_wall_sec: 3.658494, train/data_wait_ratio: 0.015158, train/lr: 0.000181, train/epoch: 0.000000, train/step: 1900.000000 2026-03-08 23:23:27,565 [Rank train_rank0] [Epoch 0 | Step 1920] train/loss: 0.463658, train/mse: 0.463658, train/data_wait_sec: 0.073399, train/step_wall_sec: 3.648836, train/data_wait_ratio: 0.020116, train/lr: 0.000183, train/epoch: 0.000000, train/step: 1920.000000 2026-03-08 23:24:42,031 [Rank train_rank0] [Epoch 0 | Step 1940] train/loss: 0.447923, train/mse: 0.447923, train/data_wait_sec: 0.042841, train/step_wall_sec: 3.648725, train/data_wait_ratio: 0.011741, train/lr: 0.000185, train/epoch: 0.000000, train/step: 1940.000000 2026-03-08 23:25:56,719 [Rank train_rank0] [Epoch 0 | Step 1960] train/loss: 0.456921, train/mse: 0.456921, train/data_wait_sec: 0.092033, train/step_wall_sec: 3.696839, train/data_wait_ratio: 0.024895, train/lr: 0.000187, train/epoch: 0.000000, train/step: 1960.000000 2026-03-08 23:27:11,286 [Rank train_rank0] [Epoch 0 | Step 1980] train/loss: 0.459269, train/mse: 0.459269, train/data_wait_sec: 0.040479, train/step_wall_sec: 3.737359, train/data_wait_ratio: 0.010831, train/lr: 0.000189, train/epoch: 0.000000, train/step: 1980.000000 2026-03-08 23:28:51,820 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000002000.pt 2026-03-08 23:28:51,823 [Rank train_rank0] [Epoch 0 | Step 2000] train/loss: 0.461657, train/mse: 0.461657, train/data_wait_sec: 0.089515, train/step_wall_sec: 3.711029, train/data_wait_ratio: 0.024121, train/lr: 0.000190, train/epoch: 0.000000, train/step: 2000.000000 2026-03-08 23:28:51,824 [Rank train_rank0] Generating samples for visualization... 2026-03-08 23:28:54,881 [Rank train_rank0] Sample generation done. 2026-03-08 23:30:09,548 [Rank train_rank0] [Epoch 0 | Step 2020] train/loss: 0.480879, train/mse: 0.480879, train/data_wait_sec: 0.051624, train/step_wall_sec: 3.665566, train/data_wait_ratio: 0.014083, train/lr: 0.000192, train/epoch: 0.000000, train/step: 2020.000000 2026-03-08 23:31:24,182 [Rank train_rank0] [Epoch 0 | Step 2040] train/loss: 0.447941, train/mse: 0.447941, train/data_wait_sec: 0.052802, train/step_wall_sec: 3.806385, train/data_wait_ratio: 0.013872, train/lr: 0.000194, train/epoch: 0.000000, train/step: 2040.000000 2026-03-08 23:32:38,402 [Rank train_rank0] [Epoch 0 | Step 2060] train/loss: 0.489779, train/mse: 0.489779, train/data_wait_sec: 0.075168, train/step_wall_sec: 3.686834, train/data_wait_ratio: 0.020388, train/lr: 0.000196, train/epoch: 0.000000, train/step: 2060.000000 2026-03-08 23:33:52,835 [Rank train_rank0] [Epoch 0 | Step 2080] train/loss: 0.471626, train/mse: 0.471626, train/data_wait_sec: 0.058889, train/step_wall_sec: 3.660151, train/data_wait_ratio: 0.016089, train/lr: 0.000198, train/epoch: 0.000000, train/step: 2080.000000 2026-03-08 23:35:07,410 [Rank train_rank0] [Epoch 0 | Step 2100] train/loss: 0.468390, train/mse: 0.468390, train/data_wait_sec: 0.083959, train/step_wall_sec: 3.705974, train/data_wait_ratio: 0.022655, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2100.000000 2026-03-08 23:36:22,067 [Rank train_rank0] [Epoch 0 | Step 2120] train/loss: 0.433043, train/mse: 0.433043, train/data_wait_sec: 0.136121, train/step_wall_sec: 3.779782, train/data_wait_ratio: 0.036013, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2120.000000 2026-03-08 23:37:36,323 [Rank train_rank0] [Epoch 0 | Step 2140] train/loss: 0.460251, train/mse: 0.460251, train/data_wait_sec: 0.067922, train/step_wall_sec: 3.681179, train/data_wait_ratio: 0.018451, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2140.000000 2026-03-08 23:38:50,625 [Rank train_rank0] [Epoch 0 | Step 2160] train/loss: 0.444735, train/mse: 0.444735, train/data_wait_sec: 0.055300, train/step_wall_sec: 3.637437, train/data_wait_ratio: 0.015203, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2160.000000 2026-03-08 23:40:05,085 [Rank train_rank0] [Epoch 0 | Step 2180] train/loss: 0.470364, train/mse: 0.470364, train/data_wait_sec: 0.077839, train/step_wall_sec: 3.653068, train/data_wait_ratio: 0.021308, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2180.000000 2026-03-08 23:41:19,743 [Rank train_rank0] [Epoch 0 | Step 2200] train/loss: 0.468185, train/mse: 0.468185, train/data_wait_sec: 0.072916, train/step_wall_sec: 3.710588, train/data_wait_ratio: 0.019651, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2200.000000 2026-03-08 23:42:33,948 [Rank train_rank0] [Epoch 0 | Step 2220] train/loss: 0.460639, train/mse: 0.460639, train/data_wait_sec: 0.066027, train/step_wall_sec: 3.664422, train/data_wait_ratio: 0.018019, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2220.000000 2026-03-08 23:43:48,406 [Rank train_rank0] [Epoch 0 | Step 2240] train/loss: 0.466393, train/mse: 0.466393, train/data_wait_sec: 0.061669, train/step_wall_sec: 3.674311, train/data_wait_ratio: 0.016784, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2240.000000 2026-03-08 23:45:03,159 [Rank train_rank0] [Epoch 0 | Step 2260] train/loss: 0.471053, train/mse: 0.471053, train/data_wait_sec: 0.086401, train/step_wall_sec: 3.711807, train/data_wait_ratio: 0.023277, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2260.000000 2026-03-08 23:46:17,386 [Rank train_rank0] [Epoch 0 | Step 2280] train/loss: 0.450216, train/mse: 0.450216, train/data_wait_sec: 0.078294, train/step_wall_sec: 3.652052, train/data_wait_ratio: 0.021438, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2280.000000 2026-03-08 23:47:31,744 [Rank train_rank0] [Epoch 0 | Step 2300] train/loss: 0.468115, train/mse: 0.468115, train/data_wait_sec: 0.064477, train/step_wall_sec: 3.783301, train/data_wait_ratio: 0.017042, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2300.000000 2026-03-08 23:48:45,968 [Rank train_rank0] [Epoch 0 | Step 2320] train/loss: 0.440459, train/mse: 0.440459, train/data_wait_sec: 0.040248, train/step_wall_sec: 3.655690, train/data_wait_ratio: 0.011010, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2320.000000 2026-03-08 23:50:00,426 [Rank train_rank0] [Epoch 0 | Step 2340] train/loss: 0.458722, train/mse: 0.458722, train/data_wait_sec: 0.101585, train/step_wall_sec: 3.723291, train/data_wait_ratio: 0.027284, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2340.000000 2026-03-08 23:51:15,197 [Rank train_rank0] [Epoch 0 | Step 2360] train/loss: 0.478104, train/mse: 0.478104, train/data_wait_sec: 0.052172, train/step_wall_sec: 3.633536, train/data_wait_ratio: 0.014358, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2360.000000 2026-03-08 23:52:29,637 [Rank train_rank0] [Epoch 0 | Step 2380] train/loss: 0.439645, train/mse: 0.439645, train/data_wait_sec: 0.063499, train/step_wall_sec: 3.650028, train/data_wait_ratio: 0.017397, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2380.000000 2026-03-08 23:53:44,095 [Rank train_rank0] [Epoch 0 | Step 2400] train/loss: 0.451696, train/mse: 0.451696, train/data_wait_sec: 0.088226, train/step_wall_sec: 3.821056, train/data_wait_ratio: 0.023090, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2400.000000 2026-03-08 23:54:58,587 [Rank train_rank0] [Epoch 0 | Step 2420] train/loss: 0.439001, train/mse: 0.439001, train/data_wait_sec: 0.068819, train/step_wall_sec: 3.694210, train/data_wait_ratio: 0.018629, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2420.000000 2026-03-08 23:56:13,084 [Rank train_rank0] [Epoch 0 | Step 2440] train/loss: 0.459189, train/mse: 0.459189, train/data_wait_sec: 0.085676, train/step_wall_sec: 3.703457, train/data_wait_ratio: 0.023134, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2440.000000 2026-03-08 23:57:27,384 [Rank train_rank0] [Epoch 0 | Step 2460] train/loss: 0.442242, train/mse: 0.442242, train/data_wait_sec: 0.081742, train/step_wall_sec: 3.651259, train/data_wait_ratio: 0.022387, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2460.000000 2026-03-08 23:58:41,761 [Rank train_rank0] [Epoch 0 | Step 2480] train/loss: 0.454469, train/mse: 0.454469, train/data_wait_sec: 0.034464, train/step_wall_sec: 3.641364, train/data_wait_ratio: 0.009465, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2480.000000 2026-03-08 23:59:56,054 [Rank train_rank0] [Epoch 0 | Step 2500] train/loss: 0.447736, train/mse: 0.447736, train/data_wait_sec: 0.077003, train/step_wall_sec: 3.701072, train/data_wait_ratio: 0.020805, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2500.000000 2026-03-09 00:01:10,554 [Rank train_rank0] [Epoch 0 | Step 2520] train/loss: 0.482683, train/mse: 0.482683, train/data_wait_sec: 0.045406, train/step_wall_sec: 3.676451, train/data_wait_ratio: 0.012351, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2520.000000 2026-03-09 00:02:24,933 [Rank train_rank0] [Epoch 0 | Step 2540] train/loss: 0.450033, train/mse: 0.450033, train/data_wait_sec: 0.042987, train/step_wall_sec: 3.780562, train/data_wait_ratio: 0.011371, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2540.000000 2026-03-09 00:03:39,108 [Rank train_rank0] [Epoch 0 | Step 2560] train/loss: 0.453551, train/mse: 0.453551, train/data_wait_sec: 0.082730, train/step_wall_sec: 3.703778, train/data_wait_ratio: 0.022337, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2560.000000 2026-03-09 00:04:53,602 [Rank train_rank0] [Epoch 0 | Step 2580] train/loss: 0.465378, train/mse: 0.465378, train/data_wait_sec: 0.058900, train/step_wall_sec: 3.654700, train/data_wait_ratio: 0.016116, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2580.000000 2026-03-09 00:06:08,183 [Rank train_rank0] [Epoch 0 | Step 2600] train/loss: 0.472092, train/mse: 0.472092, train/data_wait_sec: 0.042558, train/step_wall_sec: 3.640999, train/data_wait_ratio: 0.011688, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2600.000000 2026-03-09 00:07:22,704 [Rank train_rank0] [Epoch 0 | Step 2620] train/loss: 0.434022, train/mse: 0.434022, train/data_wait_sec: 0.059216, train/step_wall_sec: 3.658364, train/data_wait_ratio: 0.016186, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2620.000000 2026-03-09 00:08:37,229 [Rank train_rank0] [Epoch 0 | Step 2640] train/loss: 0.446153, train/mse: 0.446153, train/data_wait_sec: 0.082652, train/step_wall_sec: 3.701486, train/data_wait_ratio: 0.022329, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2640.000000 2026-03-09 00:09:51,658 [Rank train_rank0] [Epoch 0 | Step 2660] train/loss: 0.490647, train/mse: 0.490647, train/data_wait_sec: 0.046889, train/step_wall_sec: 3.683606, train/data_wait_ratio: 0.012729, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2660.000000 2026-03-09 00:11:06,141 [Rank train_rank0] [Epoch 0 | Step 2680] train/loss: 0.459313, train/mse: 0.459313, train/data_wait_sec: 0.058740, train/step_wall_sec: 3.673109, train/data_wait_ratio: 0.015992, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2680.000000 2026-03-09 00:12:20,724 [Rank train_rank0] [Epoch 0 | Step 2700] train/loss: 0.450299, train/mse: 0.450299, train/data_wait_sec: 0.156002, train/step_wall_sec: 3.756032, train/data_wait_ratio: 0.041534, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2700.000000 2026-03-09 00:13:34,804 [Rank train_rank0] [Epoch 0 | Step 2720] train/loss: 0.440675, train/mse: 0.440675, train/data_wait_sec: 0.124426, train/step_wall_sec: 3.691930, train/data_wait_ratio: 0.033702, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2720.000000 2026-03-09 00:14:49,366 [Rank train_rank0] [Epoch 0 | Step 2740] train/loss: 0.460762, train/mse: 0.460762, train/data_wait_sec: 0.032147, train/step_wall_sec: 3.659054, train/data_wait_ratio: 0.008786, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2740.000000 2026-03-09 00:16:03,698 [Rank train_rank0] [Epoch 0 | Step 2760] train/loss: 0.444483, train/mse: 0.444483, train/data_wait_sec: 0.043352, train/step_wall_sec: 3.629818, train/data_wait_ratio: 0.011943, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2760.000000 2026-03-09 00:17:18,364 [Rank train_rank0] [Epoch 0 | Step 2780] train/loss: 0.471132, train/mse: 0.471132, train/data_wait_sec: 0.099369, train/step_wall_sec: 3.681741, train/data_wait_ratio: 0.026990, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2780.000000 2026-03-09 00:18:32,882 [Rank train_rank0] [Epoch 0 | Step 2800] train/loss: 0.454776, train/mse: 0.454776, train/data_wait_sec: 0.171587, train/step_wall_sec: 3.831554, train/data_wait_ratio: 0.044783, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2800.000000 2026-03-09 00:19:47,266 [Rank train_rank0] [Epoch 0 | Step 2820] train/loss: 0.470300, train/mse: 0.470300, train/data_wait_sec: 0.050994, train/step_wall_sec: 3.671662, train/data_wait_ratio: 0.013889, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2820.000000 2026-03-09 00:21:01,730 [Rank train_rank0] [Epoch 0 | Step 2840] train/loss: 0.441890, train/mse: 0.441890, train/data_wait_sec: 0.040943, train/step_wall_sec: 3.634641, train/data_wait_ratio: 0.011265, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2840.000000 2026-03-09 00:22:16,279 [Rank train_rank0] [Epoch 0 | Step 2860] train/loss: 0.504582, train/mse: 0.504582, train/data_wait_sec: 0.193158, train/step_wall_sec: 3.827738, train/data_wait_ratio: 0.050463, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2860.000000 2026-03-09 00:23:30,614 [Rank train_rank0] [Epoch 0 | Step 2880] train/loss: 0.451246, train/mse: 0.451246, train/data_wait_sec: 0.081561, train/step_wall_sec: 3.695865, train/data_wait_ratio: 0.022068, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2880.000000 2026-03-09 00:24:44,959 [Rank train_rank0] [Epoch 0 | Step 2900] train/loss: 0.491186, train/mse: 0.491186, train/data_wait_sec: 0.037113, train/step_wall_sec: 3.666784, train/data_wait_ratio: 0.010121, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2900.000000 2026-03-09 00:25:59,165 [Rank train_rank0] [Epoch 0 | Step 2920] train/loss: 0.450930, train/mse: 0.450930, train/data_wait_sec: 0.054131, train/step_wall_sec: 3.683348, train/data_wait_ratio: 0.014696, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2920.000000 2026-03-09 00:27:13,855 [Rank train_rank0] [Epoch 0 | Step 2940] train/loss: 0.445548, train/mse: 0.445548, train/data_wait_sec: 0.056962, train/step_wall_sec: 3.670458, train/data_wait_ratio: 0.015519, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2940.000000 2026-03-09 00:28:28,181 [Rank train_rank0] [Epoch 0 | Step 2960] train/loss: 0.469422, train/mse: 0.469422, train/data_wait_sec: 0.060674, train/step_wall_sec: 3.675544, train/data_wait_ratio: 0.016508, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2960.000000 2026-03-09 00:29:42,346 [Rank train_rank0] [Epoch 0 | Step 2980] train/loss: 0.501032, train/mse: 0.501032, train/data_wait_sec: 0.057774, train/step_wall_sec: 3.752107, train/data_wait_ratio: 0.015398, train/lr: 0.000200, train/epoch: 0.000000, train/step: 2980.000000 2026-03-09 00:31:22,625 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000003000.pt 2026-03-09 00:31:22,628 [Rank train_rank0] [Epoch 0 | Step 3000] train/loss: 0.473010, train/mse: 0.473010, train/data_wait_sec: 0.058620, train/step_wall_sec: 3.675627, train/data_wait_ratio: 0.015948, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3000.000000 2026-03-09 00:31:22,629 [Rank train_rank0] Generating samples for visualization... 2026-03-09 00:31:25,530 [Rank train_rank0] Sample generation done. 2026-03-09 00:32:39,884 [Rank train_rank0] [Epoch 0 | Step 3020] train/loss: 0.466013, train/mse: 0.466013, train/data_wait_sec: 0.056552, train/step_wall_sec: 3.651168, train/data_wait_ratio: 0.015489, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3020.000000 2026-03-09 00:33:54,128 [Rank train_rank0] [Epoch 0 | Step 3040] train/loss: 0.461154, train/mse: 0.461154, train/data_wait_sec: 0.064485, train/step_wall_sec: 3.696316, train/data_wait_ratio: 0.017446, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3040.000000 2026-03-09 00:35:08,460 [Rank train_rank0] [Epoch 0 | Step 3060] train/loss: 0.453744, train/mse: 0.453744, train/data_wait_sec: 0.065101, train/step_wall_sec: 3.681254, train/data_wait_ratio: 0.017684, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3060.000000 2026-03-09 00:36:23,040 [Rank train_rank0] [Epoch 0 | Step 3080] train/loss: 0.460426, train/mse: 0.460426, train/data_wait_sec: 0.070439, train/step_wall_sec: 3.637117, train/data_wait_ratio: 0.019367, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3080.000000 2026-03-09 00:37:37,295 [Rank train_rank0] [Epoch 0 | Step 3100] train/loss: 0.480264, train/mse: 0.480264, train/data_wait_sec: 0.099261, train/step_wall_sec: 3.697901, train/data_wait_ratio: 0.026843, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3100.000000 2026-03-09 00:38:51,639 [Rank train_rank0] [Epoch 0 | Step 3120] train/loss: 0.439837, train/mse: 0.439837, train/data_wait_sec: 0.040808, train/step_wall_sec: 3.653340, train/data_wait_ratio: 0.011170, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3120.000000 2026-03-09 00:40:05,836 [Rank train_rank0] [Epoch 0 | Step 3140] train/loss: 0.436848, train/mse: 0.436848, train/data_wait_sec: 0.045582, train/step_wall_sec: 3.637045, train/data_wait_ratio: 0.012533, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3140.000000 2026-03-09 00:41:20,150 [Rank train_rank0] [Epoch 0 | Step 3160] train/loss: 0.436702, train/mse: 0.436702, train/data_wait_sec: 0.076863, train/step_wall_sec: 3.679273, train/data_wait_ratio: 0.020891, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3160.000000 2026-03-09 00:42:34,505 [Rank train_rank0] [Epoch 0 | Step 3180] train/loss: 0.454147, train/mse: 0.454147, train/data_wait_sec: 0.050723, train/step_wall_sec: 3.653464, train/data_wait_ratio: 0.013883, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3180.000000 2026-03-09 00:43:48,668 [Rank train_rank0] [Epoch 0 | Step 3200] train/loss: 0.467045, train/mse: 0.467045, train/data_wait_sec: 0.057852, train/step_wall_sec: 3.654034, train/data_wait_ratio: 0.015832, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3200.000000 2026-03-09 00:45:02,945 [Rank train_rank0] [Epoch 0 | Step 3220] train/loss: 0.448636, train/mse: 0.448636, train/data_wait_sec: 0.123493, train/step_wall_sec: 3.730386, train/data_wait_ratio: 0.033105, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3220.000000 2026-03-09 00:46:16,938 [Rank train_rank0] [Epoch 0 | Step 3240] train/loss: 0.451651, train/mse: 0.451651, train/data_wait_sec: 0.089154, train/step_wall_sec: 3.652489, train/data_wait_ratio: 0.024409, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3240.000000 2026-03-09 00:47:31,279 [Rank train_rank0] [Epoch 0 | Step 3260] train/loss: 0.455864, train/mse: 0.455864, train/data_wait_sec: 0.068811, train/step_wall_sec: 3.694395, train/data_wait_ratio: 0.018626, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3260.000000 2026-03-09 00:48:45,476 [Rank train_rank0] [Epoch 0 | Step 3280] train/loss: 0.474445, train/mse: 0.474445, train/data_wait_sec: 0.089249, train/step_wall_sec: 3.690633, train/data_wait_ratio: 0.024183, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3280.000000 2026-03-09 00:49:59,654 [Rank train_rank0] [Epoch 0 | Step 3300] train/loss: 0.463518, train/mse: 0.463518, train/data_wait_sec: 0.136774, train/step_wall_sec: 3.789411, train/data_wait_ratio: 0.036094, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3300.000000 2026-03-09 00:51:13,686 [Rank train_rank0] [Epoch 0 | Step 3320] train/loss: 0.445590, train/mse: 0.445590, train/data_wait_sec: 0.058306, train/step_wall_sec: 3.652190, train/data_wait_ratio: 0.015965, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3320.000000 2026-03-09 00:52:28,213 [Rank train_rank0] [Epoch 0 | Step 3340] train/loss: 0.476638, train/mse: 0.476638, train/data_wait_sec: 0.064124, train/step_wall_sec: 3.674215, train/data_wait_ratio: 0.017452, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3340.000000 2026-03-09 00:53:42,420 [Rank train_rank0] [Epoch 0 | Step 3360] train/loss: 0.476440, train/mse: 0.476440, train/data_wait_sec: 0.081278, train/step_wall_sec: 3.672888, train/data_wait_ratio: 0.022129, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3360.000000 2026-03-09 00:54:56,512 [Rank train_rank0] [Epoch 0 | Step 3380] train/loss: 0.470038, train/mse: 0.470038, train/data_wait_sec: 0.131792, train/step_wall_sec: 3.721638, train/data_wait_ratio: 0.035412, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3380.000000 2026-03-09 00:56:10,819 [Rank train_rank0] [Epoch 0 | Step 3400] train/loss: 0.487910, train/mse: 0.487910, train/data_wait_sec: 0.078724, train/step_wall_sec: 3.666789, train/data_wait_ratio: 0.021469, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3400.000000 2026-03-09 00:57:25,056 [Rank train_rank0] [Epoch 0 | Step 3420] train/loss: 0.464457, train/mse: 0.464457, train/data_wait_sec: 0.074789, train/step_wall_sec: 3.685139, train/data_wait_ratio: 0.020295, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3420.000000 2026-03-09 00:58:39,351 [Rank train_rank0] [Epoch 0 | Step 3440] train/loss: 0.501801, train/mse: 0.501801, train/data_wait_sec: 0.084731, train/step_wall_sec: 3.653305, train/data_wait_ratio: 0.023193, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3440.000000 2026-03-09 00:59:53,636 [Rank train_rank0] [Epoch 0 | Step 3460] train/loss: 0.445272, train/mse: 0.445272, train/data_wait_sec: 0.095487, train/step_wall_sec: 3.740894, train/data_wait_ratio: 0.025525, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3460.000000 2026-03-09 01:01:08,023 [Rank train_rank0] [Epoch 0 | Step 3480] train/loss: 0.466280, train/mse: 0.466280, train/data_wait_sec: 0.076433, train/step_wall_sec: 3.685902, train/data_wait_ratio: 0.020737, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3480.000000 2026-03-09 01:02:22,240 [Rank train_rank0] [Epoch 0 | Step 3500] train/loss: 0.460149, train/mse: 0.460149, train/data_wait_sec: 0.078771, train/step_wall_sec: 3.685498, train/data_wait_ratio: 0.021373, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3500.000000 2026-03-09 01:03:36,530 [Rank train_rank0] [Epoch 0 | Step 3520] train/loss: 0.439890, train/mse: 0.439890, train/data_wait_sec: 0.061952, train/step_wall_sec: 3.664577, train/data_wait_ratio: 0.016906, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3520.000000 2026-03-09 01:04:51,103 [Rank train_rank0] [Epoch 0 | Step 3540] train/loss: 0.495790, train/mse: 0.495790, train/data_wait_sec: 0.136082, train/step_wall_sec: 3.772935, train/data_wait_ratio: 0.036068, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3540.000000 2026-03-09 01:06:05,454 [Rank train_rank0] [Epoch 0 | Step 3560] train/loss: 0.447128, train/mse: 0.447128, train/data_wait_sec: 0.095409, train/step_wall_sec: 3.696427, train/data_wait_ratio: 0.025811, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3560.000000 2026-03-09 01:07:19,549 [Rank train_rank0] [Epoch 0 | Step 3580] train/loss: 0.465572, train/mse: 0.465572, train/data_wait_sec: 0.087927, train/step_wall_sec: 3.672133, train/data_wait_ratio: 0.023944, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3580.000000 2026-03-09 01:08:33,982 [Rank train_rank0] [Epoch 0 | Step 3600] train/loss: 0.465993, train/mse: 0.465993, train/data_wait_sec: 0.073085, train/step_wall_sec: 3.661759, train/data_wait_ratio: 0.019959, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3600.000000 2026-03-09 01:09:48,092 [Rank train_rank0] [Epoch 0 | Step 3620] train/loss: 0.467248, train/mse: 0.467248, train/data_wait_sec: 0.182419, train/step_wall_sec: 3.839263, train/data_wait_ratio: 0.047514, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3620.000000 2026-03-09 01:11:02,459 [Rank train_rank0] [Epoch 0 | Step 3640] train/loss: 0.471051, train/mse: 0.471051, train/data_wait_sec: 0.047074, train/step_wall_sec: 3.653364, train/data_wait_ratio: 0.012885, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3640.000000 2026-03-09 01:12:16,844 [Rank train_rank0] [Epoch 0 | Step 3660] train/loss: 0.472276, train/mse: 0.472276, train/data_wait_sec: 0.055128, train/step_wall_sec: 3.643292, train/data_wait_ratio: 0.015131, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3660.000000 2026-03-09 01:13:30,812 [Rank train_rank0] [Epoch 0 | Step 3680] train/loss: 0.464599, train/mse: 0.464599, train/data_wait_sec: 0.085622, train/step_wall_sec: 3.676289, train/data_wait_ratio: 0.023290, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3680.000000 2026-03-09 01:14:44,940 [Rank train_rank0] [Epoch 0 | Step 3700] train/loss: 0.447999, train/mse: 0.447999, train/data_wait_sec: 0.071488, train/step_wall_sec: 3.628716, train/data_wait_ratio: 0.019701, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3700.000000 2026-03-09 01:15:58,726 [Rank train_rank0] [Epoch 0 | Step 3720] train/loss: 0.444961, train/mse: 0.444961, train/data_wait_sec: 0.059144, train/step_wall_sec: 3.655025, train/data_wait_ratio: 0.016181, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3720.000000 2026-03-09 01:17:12,503 [Rank train_rank0] [Epoch 0 | Step 3740] train/loss: 0.461321, train/mse: 0.461321, train/data_wait_sec: 0.075117, train/step_wall_sec: 3.649221, train/data_wait_ratio: 0.020584, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3740.000000 2026-03-09 01:18:26,784 [Rank train_rank0] [Epoch 0 | Step 3760] train/loss: 0.423173, train/mse: 0.423173, train/data_wait_sec: 0.079102, train/step_wall_sec: 3.666929, train/data_wait_ratio: 0.021572, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3760.000000 2026-03-09 01:19:40,620 [Rank train_rank0] [Epoch 0 | Step 3780] train/loss: 0.445588, train/mse: 0.445588, train/data_wait_sec: 0.079760, train/step_wall_sec: 3.649735, train/data_wait_ratio: 0.021854, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3780.000000 2026-03-09 01:20:54,635 [Rank train_rank0] [Epoch 0 | Step 3800] train/loss: 0.453255, train/mse: 0.453255, train/data_wait_sec: 0.054099, train/step_wall_sec: 3.639463, train/data_wait_ratio: 0.014865, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3800.000000 2026-03-09 01:22:08,615 [Rank train_rank0] [Epoch 0 | Step 3820] train/loss: 0.437340, train/mse: 0.437340, train/data_wait_sec: 0.089139, train/step_wall_sec: 3.667736, train/data_wait_ratio: 0.024304, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3820.000000 2026-03-09 01:23:22,746 [Rank train_rank0] [Epoch 0 | Step 3840] train/loss: 0.466348, train/mse: 0.466348, train/data_wait_sec: 0.073462, train/step_wall_sec: 3.652014, train/data_wait_ratio: 0.020116, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3840.000000 2026-03-09 01:24:37,101 [Rank train_rank0] [Epoch 0 | Step 3860] train/loss: 0.436073, train/mse: 0.436073, train/data_wait_sec: 0.057941, train/step_wall_sec: 3.685720, train/data_wait_ratio: 0.015720, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3860.000000 2026-03-09 01:25:51,087 [Rank train_rank0] [Epoch 0 | Step 3880] train/loss: 0.455780, train/mse: 0.455780, train/data_wait_sec: 0.068140, train/step_wall_sec: 3.673101, train/data_wait_ratio: 0.018551, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3880.000000 2026-03-09 01:27:05,065 [Rank train_rank0] [Epoch 0 | Step 3900] train/loss: 0.430256, train/mse: 0.430256, train/data_wait_sec: 0.092525, train/step_wall_sec: 3.653731, train/data_wait_ratio: 0.025324, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3900.000000 2026-03-09 01:28:19,515 [Rank train_rank0] [Epoch 0 | Step 3920] train/loss: 0.448931, train/mse: 0.448931, train/data_wait_sec: 0.059353, train/step_wall_sec: 3.638957, train/data_wait_ratio: 0.016311, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3920.000000 2026-03-09 01:29:33,709 [Rank train_rank0] [Epoch 0 | Step 3940] train/loss: 0.476853, train/mse: 0.476853, train/data_wait_sec: 0.154022, train/step_wall_sec: 3.796142, train/data_wait_ratio: 0.040573, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3940.000000 2026-03-09 01:30:47,781 [Rank train_rank0] [Epoch 0 | Step 3960] train/loss: 0.465485, train/mse: 0.465485, train/data_wait_sec: 0.105929, train/step_wall_sec: 3.707402, train/data_wait_ratio: 0.028572, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3960.000000 2026-03-09 01:32:02,039 [Rank train_rank0] [Epoch 0 | Step 3980] train/loss: 0.438778, train/mse: 0.438778, train/data_wait_sec: 0.063565, train/step_wall_sec: 3.795678, train/data_wait_ratio: 0.016747, train/lr: 0.000200, train/epoch: 0.000000, train/step: 3980.000000 2026-03-09 01:33:42,084 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000004000.pt 2026-03-09 01:33:42,087 [Rank train_rank0] [Epoch 0 | Step 4000] train/loss: 0.463266, train/mse: 0.463266, train/data_wait_sec: 0.050810, train/step_wall_sec: 3.614110, train/data_wait_ratio: 0.014059, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4000.000000 2026-03-09 01:33:42,088 [Rank train_rank0] Generating samples for visualization... 2026-03-09 01:33:44,954 [Rank train_rank0] Sample generation done. 2026-03-09 01:34:59,046 [Rank train_rank0] [Epoch 0 | Step 4020] train/loss: 0.452225, train/mse: 0.452225, train/data_wait_sec: 0.069166, train/step_wall_sec: 3.649707, train/data_wait_ratio: 0.018951, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4020.000000 2026-03-09 01:36:12,965 [Rank train_rank0] [Epoch 0 | Step 4040] train/loss: 0.454748, train/mse: 0.454748, train/data_wait_sec: 0.043470, train/step_wall_sec: 3.657713, train/data_wait_ratio: 0.011884, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4040.000000 2026-03-09 01:37:26,891 [Rank train_rank0] [Epoch 0 | Step 4060] train/loss: 0.485034, train/mse: 0.485034, train/data_wait_sec: 0.075394, train/step_wall_sec: 3.667166, train/data_wait_ratio: 0.020559, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4060.000000 2026-03-09 01:38:41,256 [Rank train_rank0] [Epoch 0 | Step 4080] train/loss: 0.468838, train/mse: 0.468838, train/data_wait_sec: 0.063282, train/step_wall_sec: 3.670197, train/data_wait_ratio: 0.017242, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4080.000000 2026-03-09 01:39:55,287 [Rank train_rank0] [Epoch 0 | Step 4100] train/loss: 0.474075, train/mse: 0.474075, train/data_wait_sec: 0.065421, train/step_wall_sec: 3.652972, train/data_wait_ratio: 0.017909, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4100.000000 2026-03-09 01:41:09,216 [Rank train_rank0] [Epoch 0 | Step 4120] train/loss: 0.453097, train/mse: 0.453097, train/data_wait_sec: 0.075290, train/step_wall_sec: 3.648048, train/data_wait_ratio: 0.020638, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4120.000000 2026-03-09 01:42:23,274 [Rank train_rank0] [Epoch 0 | Step 4140] train/loss: 0.446865, train/mse: 0.446865, train/data_wait_sec: 0.050813, train/step_wall_sec: 3.640493, train/data_wait_ratio: 0.013958, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4140.000000 2026-03-09 01:43:37,317 [Rank train_rank0] [Epoch 0 | Step 4160] train/loss: 0.445562, train/mse: 0.445562, train/data_wait_sec: 0.056582, train/step_wall_sec: 3.643280, train/data_wait_ratio: 0.015531, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4160.000000 2026-03-09 01:44:51,269 [Rank train_rank0] [Epoch 0 | Step 4180] train/loss: 0.487422, train/mse: 0.487422, train/data_wait_sec: 0.028345, train/step_wall_sec: 3.607617, train/data_wait_ratio: 0.007857, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4180.000000 2026-03-09 01:46:05,678 [Rank train_rank0] [Epoch 0 | Step 4200] train/loss: 0.470520, train/mse: 0.470520, train/data_wait_sec: 0.061367, train/step_wall_sec: 3.700449, train/data_wait_ratio: 0.016584, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4200.000000 2026-03-09 01:47:19,806 [Rank train_rank0] [Epoch 0 | Step 4220] train/loss: 0.451037, train/mse: 0.451037, train/data_wait_sec: 0.066296, train/step_wall_sec: 3.680995, train/data_wait_ratio: 0.018010, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4220.000000 2026-03-09 01:48:34,205 [Rank train_rank0] [Epoch 0 | Step 4240] train/loss: 0.454752, train/mse: 0.454752, train/data_wait_sec: 0.127616, train/step_wall_sec: 3.727076, train/data_wait_ratio: 0.034240, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4240.000000 2026-03-09 01:49:48,447 [Rank train_rank0] [Epoch 0 | Step 4260] train/loss: 0.446966, train/mse: 0.446966, train/data_wait_sec: 0.080162, train/step_wall_sec: 3.687741, train/data_wait_ratio: 0.021737, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4260.000000 2026-03-09 01:51:02,485 [Rank train_rank0] [Epoch 0 | Step 4280] train/loss: 0.474578, train/mse: 0.474578, train/data_wait_sec: 0.074945, train/step_wall_sec: 3.676082, train/data_wait_ratio: 0.020387, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4280.000000 2026-03-09 01:52:16,709 [Rank train_rank0] [Epoch 0 | Step 4300] train/loss: 0.440602, train/mse: 0.440602, train/data_wait_sec: 0.180142, train/step_wall_sec: 3.835004, train/data_wait_ratio: 0.046973, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4300.000000 2026-03-09 01:53:30,555 [Rank train_rank0] [Epoch 0 | Step 4320] train/loss: 0.473609, train/mse: 0.473609, train/data_wait_sec: 0.050887, train/step_wall_sec: 3.646781, train/data_wait_ratio: 0.013954, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4320.000000 2026-03-09 01:54:44,550 [Rank train_rank0] [Epoch 0 | Step 4340] train/loss: 0.451507, train/mse: 0.451507, train/data_wait_sec: 0.056140, train/step_wall_sec: 3.682192, train/data_wait_ratio: 0.015246, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4340.000000 2026-03-09 01:55:58,369 [Rank train_rank0] [Epoch 0 | Step 4360] train/loss: 0.475656, train/mse: 0.475656, train/data_wait_sec: 0.052280, train/step_wall_sec: 3.642728, train/data_wait_ratio: 0.014352, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4360.000000 2026-03-09 01:57:12,334 [Rank train_rank0] [Epoch 0 | Step 4380] train/loss: 0.449106, train/mse: 0.449106, train/data_wait_sec: 0.056360, train/step_wall_sec: 3.679775, train/data_wait_ratio: 0.015316, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4380.000000 2026-03-09 01:58:26,342 [Rank train_rank0] [Epoch 0 | Step 4400] train/loss: 0.452063, train/mse: 0.452063, train/data_wait_sec: 0.070761, train/step_wall_sec: 3.662558, train/data_wait_ratio: 0.019320, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4400.000000 2026-03-09 01:59:40,275 [Rank train_rank0] [Epoch 0 | Step 4420] train/loss: 0.441637, train/mse: 0.441637, train/data_wait_sec: 0.066369, train/step_wall_sec: 3.652434, train/data_wait_ratio: 0.018171, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4420.000000 2026-03-09 02:00:53,997 [Rank train_rank0] [Epoch 0 | Step 4440] train/loss: 0.466373, train/mse: 0.466373, train/data_wait_sec: 0.088704, train/step_wall_sec: 3.686027, train/data_wait_ratio: 0.024065, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4440.000000 2026-03-09 02:02:07,885 [Rank train_rank0] [Epoch 0 | Step 4460] train/loss: 0.451789, train/mse: 0.451789, train/data_wait_sec: 0.050518, train/step_wall_sec: 3.629831, train/data_wait_ratio: 0.013918, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4460.000000 2026-03-09 02:03:21,835 [Rank train_rank0] [Epoch 0 | Step 4480] train/loss: 0.469362, train/mse: 0.469362, train/data_wait_sec: 0.051567, train/step_wall_sec: 3.641771, train/data_wait_ratio: 0.014160, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4480.000000 2026-03-09 02:04:35,775 [Rank train_rank0] [Epoch 0 | Step 4500] train/loss: 0.463758, train/mse: 0.463758, train/data_wait_sec: 0.065971, train/step_wall_sec: 3.652992, train/data_wait_ratio: 0.018059, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4500.000000 2026-03-09 02:05:49,690 [Rank train_rank0] [Epoch 0 | Step 4520] train/loss: 0.437401, train/mse: 0.437401, train/data_wait_sec: 0.070986, train/step_wall_sec: 3.651492, train/data_wait_ratio: 0.019440, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4520.000000 2026-03-09 02:07:03,728 [Rank train_rank0] [Epoch 0 | Step 4540] train/loss: 0.446881, train/mse: 0.446881, train/data_wait_sec: 0.067398, train/step_wall_sec: 3.652556, train/data_wait_ratio: 0.018452, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4540.000000 2026-03-09 02:08:17,541 [Rank train_rank0] [Epoch 0 | Step 4560] train/loss: 0.478122, train/mse: 0.478122, train/data_wait_sec: 0.056958, train/step_wall_sec: 3.665804, train/data_wait_ratio: 0.015538, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4560.000000 2026-03-09 02:09:31,470 [Rank train_rank0] [Epoch 0 | Step 4580] train/loss: 0.466837, train/mse: 0.466837, train/data_wait_sec: 0.064370, train/step_wall_sec: 3.650091, train/data_wait_ratio: 0.017635, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4580.000000 2026-03-09 02:10:45,566 [Rank train_rank0] [Epoch 0 | Step 4600] train/loss: 0.458886, train/mse: 0.458886, train/data_wait_sec: 0.063410, train/step_wall_sec: 3.667730, train/data_wait_ratio: 0.017289, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4600.000000 2026-03-09 02:11:59,318 [Rank train_rank0] [Epoch 0 | Step 4620] train/loss: 0.430383, train/mse: 0.430383, train/data_wait_sec: 0.064956, train/step_wall_sec: 3.677531, train/data_wait_ratio: 0.017663, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4620.000000 2026-03-09 02:13:13,473 [Rank train_rank0] [Epoch 0 | Step 4640] train/loss: 0.432504, train/mse: 0.432504, train/data_wait_sec: 0.055442, train/step_wall_sec: 3.638842, train/data_wait_ratio: 0.015236, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4640.000000 2026-03-09 02:14:27,044 [Rank train_rank0] [Epoch 0 | Step 4660] train/loss: 0.445864, train/mse: 0.445864, train/data_wait_sec: 0.063221, train/step_wall_sec: 3.611700, train/data_wait_ratio: 0.017505, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4660.000000 2026-03-09 02:15:41,160 [Rank train_rank0] [Epoch 0 | Step 4680] train/loss: 0.484764, train/mse: 0.484764, train/data_wait_sec: 0.063487, train/step_wall_sec: 3.656910, train/data_wait_ratio: 0.017361, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4680.000000 2026-03-09 02:16:55,200 [Rank train_rank0] [Epoch 0 | Step 4700] train/loss: 0.448463, train/mse: 0.448463, train/data_wait_sec: 0.063406, train/step_wall_sec: 3.664465, train/data_wait_ratio: 0.017303, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4700.000000 2026-03-09 02:18:08,981 [Rank train_rank0] [Epoch 0 | Step 4720] train/loss: 0.452545, train/mse: 0.452545, train/data_wait_sec: 0.042796, train/step_wall_sec: 3.649122, train/data_wait_ratio: 0.011728, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4720.000000 2026-03-09 02:19:23,069 [Rank train_rank0] [Epoch 0 | Step 4740] train/loss: 0.463730, train/mse: 0.463730, train/data_wait_sec: 0.067144, train/step_wall_sec: 3.655833, train/data_wait_ratio: 0.018366, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4740.000000 2026-03-09 02:20:36,831 [Rank train_rank0] [Epoch 0 | Step 4760] train/loss: 0.488804, train/mse: 0.488804, train/data_wait_sec: 0.054186, train/step_wall_sec: 3.641561, train/data_wait_ratio: 0.014880, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4760.000000 2026-03-09 02:21:50,687 [Rank train_rank0] [Epoch 0 | Step 4780] train/loss: 0.455002, train/mse: 0.455002, train/data_wait_sec: 0.035572, train/step_wall_sec: 3.632524, train/data_wait_ratio: 0.009793, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4780.000000 2026-03-09 02:23:04,488 [Rank train_rank0] [Epoch 0 | Step 4800] train/loss: 0.463465, train/mse: 0.463465, train/data_wait_sec: 0.072904, train/step_wall_sec: 3.622144, train/data_wait_ratio: 0.020127, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4800.000000 2026-03-09 02:24:18,691 [Rank train_rank0] [Epoch 0 | Step 4820] train/loss: 0.467501, train/mse: 0.467501, train/data_wait_sec: 0.058372, train/step_wall_sec: 3.684758, train/data_wait_ratio: 0.015842, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4820.000000 2026-03-09 02:25:32,435 [Rank train_rank0] [Epoch 0 | Step 4840] train/loss: 0.463471, train/mse: 0.463471, train/data_wait_sec: 0.101415, train/step_wall_sec: 3.716439, train/data_wait_ratio: 0.027288, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4840.000000 2026-03-09 02:26:46,656 [Rank train_rank0] [Epoch 0 | Step 4860] train/loss: 0.466810, train/mse: 0.466810, train/data_wait_sec: 0.133061, train/step_wall_sec: 3.738872, train/data_wait_ratio: 0.035589, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4860.000000 2026-03-09 02:28:00,456 [Rank train_rank0] [Epoch 0 | Step 4880] train/loss: 0.475532, train/mse: 0.475532, train/data_wait_sec: 0.091720, train/step_wall_sec: 3.691323, train/data_wait_ratio: 0.024848, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4880.000000 2026-03-09 02:29:14,439 [Rank train_rank0] [Epoch 0 | Step 4900] train/loss: 0.471230, train/mse: 0.471230, train/data_wait_sec: 0.115375, train/step_wall_sec: 3.675112, train/data_wait_ratio: 0.031394, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4900.000000 2026-03-09 02:30:28,431 [Rank train_rank0] [Epoch 0 | Step 4920] train/loss: 0.457615, train/mse: 0.457615, train/data_wait_sec: 0.086689, train/step_wall_sec: 3.657248, train/data_wait_ratio: 0.023703, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4920.000000 2026-03-09 02:31:42,505 [Rank train_rank0] [Epoch 0 | Step 4940] train/loss: 0.474696, train/mse: 0.474696, train/data_wait_sec: 0.050412, train/step_wall_sec: 3.699496, train/data_wait_ratio: 0.013627, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4940.000000 2026-03-09 02:32:56,276 [Rank train_rank0] [Epoch 0 | Step 4960] train/loss: 0.463401, train/mse: 0.463401, train/data_wait_sec: 0.033181, train/step_wall_sec: 3.636757, train/data_wait_ratio: 0.009124, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4960.000000 2026-03-09 02:34:10,050 [Rank train_rank0] [Epoch 0 | Step 4980] train/loss: 0.476255, train/mse: 0.476255, train/data_wait_sec: 0.087760, train/step_wall_sec: 3.698397, train/data_wait_ratio: 0.023729, train/lr: 0.000200, train/epoch: 0.000000, train/step: 4980.000000 2026-03-09 02:35:49,803 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000005000.pt 2026-03-09 02:35:49,806 [Rank train_rank0] [Epoch 0 | Step 5000] train/loss: 0.483437, train/mse: 0.483437, train/data_wait_sec: 0.046721, train/step_wall_sec: 3.772771, train/data_wait_ratio: 0.012384, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5000.000000 2026-03-09 02:35:49,807 [Rank train_rank0] Generating samples for visualization... 2026-03-09 02:35:52,536 [Rank train_rank0] Sample generation done. 2026-03-09 02:37:06,280 [Rank train_rank0] [Epoch 0 | Step 5020] train/loss: 0.466274, train/mse: 0.466274, train/data_wait_sec: 0.040987, train/step_wall_sec: 3.621999, train/data_wait_ratio: 0.011316, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5020.000000 2026-03-09 02:38:20,202 [Rank train_rank0] [Epoch 0 | Step 5040] train/loss: 0.462976, train/mse: 0.462976, train/data_wait_sec: 0.050539, train/step_wall_sec: 3.612560, train/data_wait_ratio: 0.013990, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5040.000000 2026-03-09 02:39:34,375 [Rank train_rank0] [Epoch 0 | Step 5060] train/loss: 0.461560, train/mse: 0.461560, train/data_wait_sec: 0.063005, train/step_wall_sec: 3.638148, train/data_wait_ratio: 0.017318, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5060.000000 2026-03-09 02:40:48,224 [Rank train_rank0] [Epoch 0 | Step 5080] train/loss: 0.454463, train/mse: 0.454463, train/data_wait_sec: 0.059693, train/step_wall_sec: 3.635715, train/data_wait_ratio: 0.016418, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5080.000000 2026-03-09 02:42:02,232 [Rank train_rank0] [Epoch 0 | Step 5100] train/loss: 0.453752, train/mse: 0.453752, train/data_wait_sec: 0.050378, train/step_wall_sec: 3.675493, train/data_wait_ratio: 0.013706, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5100.000000 2026-03-09 02:43:16,355 [Rank train_rank0] [Epoch 0 | Step 5120] train/loss: 0.487896, train/mse: 0.487896, train/data_wait_sec: 0.079793, train/step_wall_sec: 3.733939, train/data_wait_ratio: 0.021370, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5120.000000 2026-03-09 02:44:30,427 [Rank train_rank0] [Epoch 0 | Step 5140] train/loss: 0.465228, train/mse: 0.465228, train/data_wait_sec: 0.040598, train/step_wall_sec: 3.628419, train/data_wait_ratio: 0.011189, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5140.000000 2026-03-09 02:45:44,359 [Rank train_rank0] [Epoch 0 | Step 5160] train/loss: 0.473353, train/mse: 0.473353, train/data_wait_sec: 0.062844, train/step_wall_sec: 3.664629, train/data_wait_ratio: 0.017149, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5160.000000 2026-03-09 02:46:58,131 [Rank train_rank0] [Epoch 0 | Step 5180] train/loss: 0.446129, train/mse: 0.446129, train/data_wait_sec: 0.074968, train/step_wall_sec: 3.660283, train/data_wait_ratio: 0.020481, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5180.000000 2026-03-09 02:48:12,090 [Rank train_rank0] [Epoch 0 | Step 5200] train/loss: 0.438120, train/mse: 0.438120, train/data_wait_sec: 0.065278, train/step_wall_sec: 3.671113, train/data_wait_ratio: 0.017781, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5200.000000 2026-03-09 02:49:26,105 [Rank train_rank0] [Epoch 0 | Step 5220] train/loss: 0.443395, train/mse: 0.443395, train/data_wait_sec: 0.089379, train/step_wall_sec: 3.674943, train/data_wait_ratio: 0.024321, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5220.000000 2026-03-09 02:50:40,130 [Rank train_rank0] [Epoch 0 | Step 5240] train/loss: 0.473951, train/mse: 0.473951, train/data_wait_sec: 0.055171, train/step_wall_sec: 3.664961, train/data_wait_ratio: 0.015054, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5240.000000 2026-03-09 02:51:53,839 [Rank train_rank0] [Epoch 0 | Step 5260] train/loss: 0.456098, train/mse: 0.456098, train/data_wait_sec: 0.081233, train/step_wall_sec: 3.650713, train/data_wait_ratio: 0.022251, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5260.000000 2026-03-09 02:53:07,749 [Rank train_rank0] [Epoch 0 | Step 5280] train/loss: 0.440015, train/mse: 0.440015, train/data_wait_sec: 0.104415, train/step_wall_sec: 3.706837, train/data_wait_ratio: 0.028168, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5280.000000 2026-03-09 02:54:21,638 [Rank train_rank0] [Epoch 0 | Step 5300] train/loss: 0.463738, train/mse: 0.463738, train/data_wait_sec: 0.070687, train/step_wall_sec: 3.624321, train/data_wait_ratio: 0.019504, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5300.000000 2026-03-09 02:55:35,383 [Rank train_rank0] [Epoch 0 | Step 5320] train/loss: 0.432485, train/mse: 0.432485, train/data_wait_sec: 0.056879, train/step_wall_sec: 3.654213, train/data_wait_ratio: 0.015565, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5320.000000 2026-03-09 02:56:49,605 [Rank train_rank0] [Epoch 0 | Step 5340] train/loss: 0.417496, train/mse: 0.417496, train/data_wait_sec: 0.109737, train/step_wall_sec: 3.689686, train/data_wait_ratio: 0.029741, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5340.000000 2026-03-09 02:58:03,652 [Rank train_rank0] [Epoch 0 | Step 5360] train/loss: 0.443100, train/mse: 0.443100, train/data_wait_sec: 0.113635, train/step_wall_sec: 3.701353, train/data_wait_ratio: 0.030701, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5360.000000 2026-03-09 02:59:17,420 [Rank train_rank0] [Epoch 0 | Step 5380] train/loss: 0.448293, train/mse: 0.448293, train/data_wait_sec: 0.056470, train/step_wall_sec: 3.631221, train/data_wait_ratio: 0.015551, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5380.000000 2026-03-09 03:00:31,483 [Rank train_rank0] [Epoch 0 | Step 5400] train/loss: 0.490297, train/mse: 0.490297, train/data_wait_sec: 0.084012, train/step_wall_sec: 3.739473, train/data_wait_ratio: 0.022466, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5400.000000 2026-03-09 03:01:45,235 [Rank train_rank0] [Epoch 0 | Step 5420] train/loss: 0.471983, train/mse: 0.471983, train/data_wait_sec: 0.116681, train/step_wall_sec: 3.740127, train/data_wait_ratio: 0.031197, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5420.000000 2026-03-09 03:02:59,104 [Rank train_rank0] [Epoch 0 | Step 5440] train/loss: 0.449235, train/mse: 0.449235, train/data_wait_sec: 0.040988, train/step_wall_sec: 3.628997, train/data_wait_ratio: 0.011295, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5440.000000 2026-03-09 03:04:13,178 [Rank train_rank0] [Epoch 0 | Step 5460] train/loss: 0.466065, train/mse: 0.466065, train/data_wait_sec: 0.095405, train/step_wall_sec: 3.742468, train/data_wait_ratio: 0.025493, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5460.000000 2026-03-09 03:05:27,132 [Rank train_rank0] [Epoch 0 | Step 5480] train/loss: 0.458111, train/mse: 0.458111, train/data_wait_sec: 0.106042, train/step_wall_sec: 3.615729, train/data_wait_ratio: 0.029328, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5480.000000 2026-03-09 03:06:41,084 [Rank train_rank0] [Epoch 0 | Step 5500] train/loss: 0.466866, train/mse: 0.466866, train/data_wait_sec: 0.068811, train/step_wall_sec: 3.636386, train/data_wait_ratio: 0.018923, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5500.000000 2026-03-09 03:07:55,122 [Rank train_rank0] [Epoch 0 | Step 5520] train/loss: 0.459880, train/mse: 0.459880, train/data_wait_sec: 0.119831, train/step_wall_sec: 3.737083, train/data_wait_ratio: 0.032066, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5520.000000 2026-03-09 03:09:08,851 [Rank train_rank0] [Epoch 0 | Step 5540] train/loss: 0.432374, train/mse: 0.432374, train/data_wait_sec: 0.063250, train/step_wall_sec: 3.631484, train/data_wait_ratio: 0.017417, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5540.000000 2026-03-09 03:10:22,988 [Rank train_rank0] [Epoch 0 | Step 5560] train/loss: 0.462860, train/mse: 0.462860, train/data_wait_sec: 0.038444, train/step_wall_sec: 3.635219, train/data_wait_ratio: 0.010575, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5560.000000 2026-03-09 03:11:36,909 [Rank train_rank0] [Epoch 0 | Step 5580] train/loss: 0.453575, train/mse: 0.453575, train/data_wait_sec: 0.143737, train/step_wall_sec: 3.793274, train/data_wait_ratio: 0.037892, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5580.000000 2026-03-09 03:12:50,752 [Rank train_rank0] [Epoch 0 | Step 5600] train/loss: 0.474835, train/mse: 0.474835, train/data_wait_sec: 0.039282, train/step_wall_sec: 3.629997, train/data_wait_ratio: 0.010821, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5600.000000 2026-03-09 03:14:04,767 [Rank train_rank0] [Epoch 0 | Step 5620] train/loss: 0.465394, train/mse: 0.465394, train/data_wait_sec: 0.077057, train/step_wall_sec: 3.661391, train/data_wait_ratio: 0.021046, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5620.000000 2026-03-09 03:15:18,757 [Rank train_rank0] [Epoch 0 | Step 5640] train/loss: 0.470227, train/mse: 0.470227, train/data_wait_sec: 0.065827, train/step_wall_sec: 3.641621, train/data_wait_ratio: 0.018076, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5640.000000 2026-03-09 03:16:32,767 [Rank train_rank0] [Epoch 0 | Step 5660] train/loss: 0.458028, train/mse: 0.458028, train/data_wait_sec: 0.068929, train/step_wall_sec: 3.795062, train/data_wait_ratio: 0.018163, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5660.000000 2026-03-09 03:17:46,949 [Rank train_rank0] [Epoch 0 | Step 5680] train/loss: 0.462374, train/mse: 0.462374, train/data_wait_sec: 0.108731, train/step_wall_sec: 3.708831, train/data_wait_ratio: 0.029317, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5680.000000 2026-03-09 03:19:00,883 [Rank train_rank0] [Epoch 0 | Step 5700] train/loss: 0.447075, train/mse: 0.447075, train/data_wait_sec: 0.073375, train/step_wall_sec: 3.665977, train/data_wait_ratio: 0.020015, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5700.000000 2026-03-09 03:20:14,818 [Rank train_rank0] [Epoch 0 | Step 5720] train/loss: 0.441228, train/mse: 0.441228, train/data_wait_sec: 0.033064, train/step_wall_sec: 3.665744, train/data_wait_ratio: 0.009020, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5720.000000 2026-03-09 03:21:28,642 [Rank train_rank0] [Epoch 0 | Step 5740] train/loss: 0.453674, train/mse: 0.453674, train/data_wait_sec: 0.058602, train/step_wall_sec: 3.636558, train/data_wait_ratio: 0.016115, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5740.000000 2026-03-09 03:22:42,492 [Rank train_rank0] [Epoch 0 | Step 5760] train/loss: 0.495379, train/mse: 0.495379, train/data_wait_sec: 0.075600, train/step_wall_sec: 3.648713, train/data_wait_ratio: 0.020720, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5760.000000 2026-03-09 03:23:56,399 [Rank train_rank0] [Epoch 0 | Step 5780] train/loss: 0.460274, train/mse: 0.460274, train/data_wait_sec: 0.066255, train/step_wall_sec: 3.614779, train/data_wait_ratio: 0.018329, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5780.000000 2026-03-09 03:25:10,393 [Rank train_rank0] [Epoch 0 | Step 5800] train/loss: 0.475686, train/mse: 0.475686, train/data_wait_sec: 0.141858, train/step_wall_sec: 3.766006, train/data_wait_ratio: 0.037668, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5800.000000 2026-03-09 03:26:24,309 [Rank train_rank0] [Epoch 0 | Step 5820] train/loss: 0.444598, train/mse: 0.444598, train/data_wait_sec: 0.078322, train/step_wall_sec: 3.611926, train/data_wait_ratio: 0.021684, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5820.000000 2026-03-09 03:27:38,384 [Rank train_rank0] [Epoch 0 | Step 5840] train/loss: 0.472109, train/mse: 0.472109, train/data_wait_sec: 0.057720, train/step_wall_sec: 3.656385, train/data_wait_ratio: 0.015786, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5840.000000 2026-03-09 03:28:52,154 [Rank train_rank0] [Epoch 0 | Step 5860] train/loss: 0.435890, train/mse: 0.435890, train/data_wait_sec: 0.061067, train/step_wall_sec: 3.649979, train/data_wait_ratio: 0.016731, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5860.000000 2026-03-09 03:30:06,025 [Rank train_rank0] [Epoch 0 | Step 5880] train/loss: 0.465335, train/mse: 0.465335, train/data_wait_sec: 0.068051, train/step_wall_sec: 3.653155, train/data_wait_ratio: 0.018628, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5880.000000 2026-03-09 03:31:19,950 [Rank train_rank0] [Epoch 0 | Step 5900] train/loss: 0.466834, train/mse: 0.466834, train/data_wait_sec: 0.084704, train/step_wall_sec: 3.674872, train/data_wait_ratio: 0.023049, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5900.000000 2026-03-09 03:32:33,771 [Rank train_rank0] [Epoch 0 | Step 5920] train/loss: 0.472529, train/mse: 0.472529, train/data_wait_sec: 0.064014, train/step_wall_sec: 3.664029, train/data_wait_ratio: 0.017471, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5920.000000 2026-03-09 03:33:47,884 [Rank train_rank0] [Epoch 0 | Step 5940] train/loss: 0.436659, train/mse: 0.436659, train/data_wait_sec: 0.091354, train/step_wall_sec: 3.776544, train/data_wait_ratio: 0.024190, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5940.000000 2026-03-09 03:35:01,673 [Rank train_rank0] [Epoch 0 | Step 5960] train/loss: 0.463063, train/mse: 0.463063, train/data_wait_sec: 0.094742, train/step_wall_sec: 3.737336, train/data_wait_ratio: 0.025350, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5960.000000 2026-03-09 03:36:15,558 [Rank train_rank0] [Epoch 0 | Step 5980] train/loss: 0.449897, train/mse: 0.449897, train/data_wait_sec: 0.110206, train/step_wall_sec: 3.725947, train/data_wait_ratio: 0.029578, train/lr: 0.000200, train/epoch: 0.000000, train/step: 5980.000000 2026-03-09 03:37:54,712 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000006000.pt 2026-03-09 03:37:54,714 [Rank train_rank0] [Epoch 0 | Step 6000] train/loss: 0.444615, train/mse: 0.444615, train/data_wait_sec: 0.077114, train/step_wall_sec: 3.636777, train/data_wait_ratio: 0.021204, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6000.000000 2026-03-09 03:37:54,716 [Rank train_rank0] Generating samples for visualization... 2026-03-09 03:37:57,686 [Rank train_rank0] Sample generation done. 2026-03-09 03:39:11,518 [Rank train_rank0] [Epoch 0 | Step 6020] train/loss: 0.435371, train/mse: 0.435371, train/data_wait_sec: 0.163138, train/step_wall_sec: 3.790405, train/data_wait_ratio: 0.043040, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6020.000000 2026-03-09 03:40:25,378 [Rank train_rank0] [Epoch 0 | Step 6040] train/loss: 0.467690, train/mse: 0.467690, train/data_wait_sec: 0.085929, train/step_wall_sec: 3.657648, train/data_wait_ratio: 0.023493, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6040.000000 2026-03-09 03:41:39,262 [Rank train_rank0] [Epoch 0 | Step 6060] train/loss: 0.455085, train/mse: 0.455085, train/data_wait_sec: 0.051185, train/step_wall_sec: 3.630032, train/data_wait_ratio: 0.014100, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6060.000000 2026-03-09 03:42:52,943 [Rank train_rank0] [Epoch 0 | Step 6080] train/loss: 0.447690, train/mse: 0.447690, train/data_wait_sec: 0.060326, train/step_wall_sec: 3.646096, train/data_wait_ratio: 0.016545, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6080.000000 2026-03-09 03:44:06,695 [Rank train_rank0] [Epoch 0 | Step 6100] train/loss: 0.478508, train/mse: 0.478508, train/data_wait_sec: 0.042825, train/step_wall_sec: 3.640647, train/data_wait_ratio: 0.011763, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6100.000000 2026-03-09 03:45:20,540 [Rank train_rank0] [Epoch 0 | Step 6120] train/loss: 0.452640, train/mse: 0.452640, train/data_wait_sec: 0.030075, train/step_wall_sec: 3.629407, train/data_wait_ratio: 0.008287, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6120.000000 2026-03-09 03:46:34,453 [Rank train_rank0] [Epoch 0 | Step 6140] train/loss: 0.459874, train/mse: 0.459874, train/data_wait_sec: 0.069290, train/step_wall_sec: 3.637665, train/data_wait_ratio: 0.019048, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6140.000000 2026-03-09 03:47:48,189 [Rank train_rank0] [Epoch 0 | Step 6160] train/loss: 0.467930, train/mse: 0.467930, train/data_wait_sec: 0.048347, train/step_wall_sec: 3.651039, train/data_wait_ratio: 0.013242, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6160.000000 2026-03-09 03:49:02,219 [Rank train_rank0] [Epoch 0 | Step 6180] train/loss: 0.472822, train/mse: 0.472822, train/data_wait_sec: 0.177804, train/step_wall_sec: 3.764376, train/data_wait_ratio: 0.047233, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6180.000000 2026-03-09 03:50:15,973 [Rank train_rank0] [Epoch 0 | Step 6200] train/loss: 0.462943, train/mse: 0.462943, train/data_wait_sec: 0.084793, train/step_wall_sec: 3.703911, train/data_wait_ratio: 0.022893, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6200.000000 2026-03-09 03:51:30,025 [Rank train_rank0] [Epoch 0 | Step 6220] train/loss: 0.466734, train/mse: 0.466734, train/data_wait_sec: 0.113466, train/step_wall_sec: 3.737837, train/data_wait_ratio: 0.030356, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6220.000000 2026-03-09 03:52:43,802 [Rank train_rank0] [Epoch 0 | Step 6240] train/loss: 0.466471, train/mse: 0.466471, train/data_wait_sec: 0.076236, train/step_wall_sec: 3.635871, train/data_wait_ratio: 0.020968, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6240.000000 2026-03-09 03:53:58,122 [Rank train_rank0] [Epoch 0 | Step 6260] train/loss: 0.474943, train/mse: 0.474943, train/data_wait_sec: 0.048813, train/step_wall_sec: 3.617546, train/data_wait_ratio: 0.013493, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6260.000000 2026-03-09 03:55:12,069 [Rank train_rank0] [Epoch 0 | Step 6280] train/loss: 0.474347, train/mse: 0.474347, train/data_wait_sec: 0.064453, train/step_wall_sec: 3.644236, train/data_wait_ratio: 0.017686, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6280.000000 2026-03-09 03:56:26,038 [Rank train_rank0] [Epoch 0 | Step 6300] train/loss: 0.465469, train/mse: 0.465469, train/data_wait_sec: 0.033092, train/step_wall_sec: 3.617174, train/data_wait_ratio: 0.009148, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6300.000000 2026-03-09 03:57:39,874 [Rank train_rank0] [Epoch 0 | Step 6320] train/loss: 0.428342, train/mse: 0.428342, train/data_wait_sec: 0.087911, train/step_wall_sec: 3.689712, train/data_wait_ratio: 0.023826, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6320.000000 2026-03-09 03:58:54,098 [Rank train_rank0] [Epoch 0 | Step 6340] train/loss: 0.447186, train/mse: 0.447186, train/data_wait_sec: 0.184068, train/step_wall_sec: 3.831513, train/data_wait_ratio: 0.048041, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6340.000000 2026-03-09 04:00:07,930 [Rank train_rank0] [Epoch 0 | Step 6360] train/loss: 0.463422, train/mse: 0.463422, train/data_wait_sec: 0.066910, train/step_wall_sec: 3.655845, train/data_wait_ratio: 0.018302, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6360.000000 2026-03-09 04:01:22,020 [Rank train_rank0] [Epoch 0 | Step 6380] train/loss: 0.445156, train/mse: 0.445156, train/data_wait_sec: 0.061956, train/step_wall_sec: 3.646100, train/data_wait_ratio: 0.016992, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6380.000000 2026-03-09 04:02:35,605 [Rank train_rank0] [Epoch 0 | Step 6400] train/loss: 0.482764, train/mse: 0.482764, train/data_wait_sec: 0.030739, train/step_wall_sec: 3.639436, train/data_wait_ratio: 0.008446, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6400.000000 2026-03-09 04:03:49,928 [Rank train_rank0] [Epoch 0 | Step 6420] train/loss: 0.473249, train/mse: 0.473249, train/data_wait_sec: 0.057952, train/step_wall_sec: 3.632254, train/data_wait_ratio: 0.015955, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6420.000000 2026-03-09 04:05:04,044 [Rank train_rank0] [Epoch 0 | Step 6440] train/loss: 0.480237, train/mse: 0.480237, train/data_wait_sec: 0.121531, train/step_wall_sec: 3.735092, train/data_wait_ratio: 0.032538, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6440.000000 2026-03-09 04:06:17,805 [Rank train_rank0] [Epoch 0 | Step 6460] train/loss: 0.432243, train/mse: 0.432243, train/data_wait_sec: 0.060638, train/step_wall_sec: 3.665534, train/data_wait_ratio: 0.016543, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6460.000000 2026-03-09 04:07:31,746 [Rank train_rank0] [Epoch 0 | Step 6480] train/loss: 0.486565, train/mse: 0.486565, train/data_wait_sec: 0.044344, train/step_wall_sec: 3.615776, train/data_wait_ratio: 0.012264, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6480.000000 2026-03-09 04:08:45,815 [Rank train_rank0] [Epoch 0 | Step 6500] train/loss: 0.444464, train/mse: 0.444464, train/data_wait_sec: 0.064061, train/step_wall_sec: 3.623037, train/data_wait_ratio: 0.017682, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6500.000000 2026-03-09 04:09:59,769 [Rank train_rank0] [Epoch 0 | Step 6520] train/loss: 0.458254, train/mse: 0.458254, train/data_wait_sec: 0.046498, train/step_wall_sec: 3.652405, train/data_wait_ratio: 0.012731, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6520.000000 2026-03-09 04:11:13,617 [Rank train_rank0] [Epoch 0 | Step 6540] train/loss: 0.468701, train/mse: 0.468701, train/data_wait_sec: 0.029735, train/step_wall_sec: 3.647634, train/data_wait_ratio: 0.008152, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6540.000000 2026-03-09 04:12:27,008 [Rank train_rank0] [Epoch 0 | Step 6560] train/loss: 0.473226, train/mse: 0.473226, train/data_wait_sec: 0.066933, train/step_wall_sec: 3.645727, train/data_wait_ratio: 0.018359, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6560.000000 2026-03-09 04:13:40,942 [Rank train_rank0] [Epoch 0 | Step 6580] train/loss: 0.477176, train/mse: 0.477176, train/data_wait_sec: 0.079780, train/step_wall_sec: 3.632337, train/data_wait_ratio: 0.021964, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6580.000000 2026-03-09 04:14:55,145 [Rank train_rank0] [Epoch 0 | Step 6600] train/loss: 0.463612, train/mse: 0.463612, train/data_wait_sec: 0.062103, train/step_wall_sec: 3.635307, train/data_wait_ratio: 0.017083, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6600.000000 2026-03-09 04:16:09,233 [Rank train_rank0] [Epoch 0 | Step 6620] train/loss: 0.462423, train/mse: 0.462423, train/data_wait_sec: 0.054295, train/step_wall_sec: 3.657879, train/data_wait_ratio: 0.014843, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6620.000000 2026-03-09 04:17:23,090 [Rank train_rank0] [Epoch 0 | Step 6640] train/loss: 0.476840, train/mse: 0.476840, train/data_wait_sec: 0.053056, train/step_wall_sec: 3.693637, train/data_wait_ratio: 0.014364, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6640.000000 2026-03-09 04:18:37,012 [Rank train_rank0] [Epoch 0 | Step 6660] train/loss: 0.440844, train/mse: 0.440844, train/data_wait_sec: 0.085503, train/step_wall_sec: 3.669594, train/data_wait_ratio: 0.023300, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6660.000000 2026-03-09 04:19:50,950 [Rank train_rank0] [Epoch 0 | Step 6680] train/loss: 0.474149, train/mse: 0.474149, train/data_wait_sec: 0.060096, train/step_wall_sec: 3.637655, train/data_wait_ratio: 0.016521, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6680.000000 2026-03-09 04:21:04,806 [Rank train_rank0] [Epoch 0 | Step 6700] train/loss: 0.444625, train/mse: 0.444625, train/data_wait_sec: 0.076246, train/step_wall_sec: 3.650290, train/data_wait_ratio: 0.020888, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6700.000000 2026-03-09 04:22:18,987 [Rank train_rank0] [Epoch 0 | Step 6720] train/loss: 0.471362, train/mse: 0.471362, train/data_wait_sec: 0.287421, train/step_wall_sec: 3.953530, train/data_wait_ratio: 0.072700, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6720.000000 2026-03-09 04:23:32,944 [Rank train_rank0] [Epoch 0 | Step 6740] train/loss: 0.437408, train/mse: 0.437408, train/data_wait_sec: 0.042732, train/step_wall_sec: 3.616205, train/data_wait_ratio: 0.011817, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6740.000000 2026-03-09 04:24:47,170 [Rank train_rank0] [Epoch 0 | Step 6760] train/loss: 0.490771, train/mse: 0.490771, train/data_wait_sec: 0.057458, train/step_wall_sec: 3.615643, train/data_wait_ratio: 0.015892, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6760.000000 2026-03-09 04:26:01,225 [Rank train_rank0] [Epoch 0 | Step 6780] train/loss: 0.483820, train/mse: 0.483820, train/data_wait_sec: 0.082631, train/step_wall_sec: 3.636057, train/data_wait_ratio: 0.022725, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6780.000000 2026-03-09 04:27:14,965 [Rank train_rank0] [Epoch 0 | Step 6800] train/loss: 0.485761, train/mse: 0.485761, train/data_wait_sec: 0.064550, train/step_wall_sec: 3.649085, train/data_wait_ratio: 0.017689, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6800.000000 2026-03-09 04:28:28,643 [Rank train_rank0] [Epoch 0 | Step 6820] train/loss: 0.469139, train/mse: 0.469139, train/data_wait_sec: 0.041948, train/step_wall_sec: 3.644916, train/data_wait_ratio: 0.011509, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6820.000000 2026-03-09 04:29:42,610 [Rank train_rank0] [Epoch 0 | Step 6840] train/loss: 0.445342, train/mse: 0.445342, train/data_wait_sec: 0.078879, train/step_wall_sec: 3.656801, train/data_wait_ratio: 0.021570, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6840.000000 2026-03-09 04:30:56,476 [Rank train_rank0] [Epoch 0 | Step 6860] train/loss: 0.452377, train/mse: 0.452377, train/data_wait_sec: 0.088821, train/step_wall_sec: 3.682054, train/data_wait_ratio: 0.024123, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6860.000000 2026-03-09 04:32:10,383 [Rank train_rank0] [Epoch 0 | Step 6880] train/loss: 0.458713, train/mse: 0.458713, train/data_wait_sec: 0.082051, train/step_wall_sec: 3.653496, train/data_wait_ratio: 0.022458, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6880.000000 2026-03-09 04:33:24,081 [Rank train_rank0] [Epoch 0 | Step 6900] train/loss: 0.455871, train/mse: 0.455871, train/data_wait_sec: 0.054013, train/step_wall_sec: 3.643684, train/data_wait_ratio: 0.014824, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6900.000000 2026-03-09 04:34:38,243 [Rank train_rank0] [Epoch 0 | Step 6920] train/loss: 0.450253, train/mse: 0.450253, train/data_wait_sec: 0.091143, train/step_wall_sec: 3.675529, train/data_wait_ratio: 0.024797, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6920.000000 2026-03-09 04:35:51,921 [Rank train_rank0] [Epoch 0 | Step 6940] train/loss: 0.447910, train/mse: 0.447910, train/data_wait_sec: 0.065121, train/step_wall_sec: 3.645981, train/data_wait_ratio: 0.017861, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6940.000000 2026-03-09 04:37:05,920 [Rank train_rank0] [Epoch 0 | Step 6960] train/loss: 0.488852, train/mse: 0.488852, train/data_wait_sec: 0.082329, train/step_wall_sec: 3.673909, train/data_wait_ratio: 0.022409, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6960.000000 2026-03-09 04:38:19,743 [Rank train_rank0] [Epoch 0 | Step 6980] train/loss: 0.456782, train/mse: 0.456782, train/data_wait_sec: 0.068910, train/step_wall_sec: 3.630508, train/data_wait_ratio: 0.018981, train/lr: 0.000200, train/epoch: 0.000000, train/step: 6980.000000 2026-03-09 04:40:00,163 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000007000.pt 2026-03-09 04:40:00,165 [Rank train_rank0] [Epoch 0 | Step 7000] train/loss: 0.480102, train/mse: 0.480102, train/data_wait_sec: 0.054286, train/step_wall_sec: 3.642349, train/data_wait_ratio: 0.014904, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7000.000000 2026-03-09 04:40:00,167 [Rank train_rank0] Generating samples for visualization... 2026-03-09 04:40:03,012 [Rank train_rank0] Sample generation done. 2026-03-09 04:41:16,752 [Rank train_rank0] [Epoch 0 | Step 7020] train/loss: 0.456758, train/mse: 0.456758, train/data_wait_sec: 0.086425, train/step_wall_sec: 3.651111, train/data_wait_ratio: 0.023671, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7020.000000 2026-03-09 04:42:30,917 [Rank train_rank0] [Epoch 0 | Step 7040] train/loss: 0.449241, train/mse: 0.449241, train/data_wait_sec: 0.081699, train/step_wall_sec: 3.667637, train/data_wait_ratio: 0.022276, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7040.000000 2026-03-09 04:43:44,779 [Rank train_rank0] [Epoch 0 | Step 7060] train/loss: 0.464657, train/mse: 0.464657, train/data_wait_sec: 0.155949, train/step_wall_sec: 3.794408, train/data_wait_ratio: 0.041100, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7060.000000 2026-03-09 04:44:58,847 [Rank train_rank0] [Epoch 0 | Step 7080] train/loss: 0.451806, train/mse: 0.451806, train/data_wait_sec: 0.049882, train/step_wall_sec: 3.660898, train/data_wait_ratio: 0.013626, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7080.000000 2026-03-09 04:46:12,674 [Rank train_rank0] [Epoch 0 | Step 7100] train/loss: 0.480468, train/mse: 0.480468, train/data_wait_sec: 0.075573, train/step_wall_sec: 3.675067, train/data_wait_ratio: 0.020564, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7100.000000 2026-03-09 04:47:26,380 [Rank train_rank0] [Epoch 0 | Step 7120] train/loss: 0.454198, train/mse: 0.454198, train/data_wait_sec: 0.071526, train/step_wall_sec: 3.643368, train/data_wait_ratio: 0.019632, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7120.000000 2026-03-09 04:48:40,457 [Rank train_rank0] [Epoch 0 | Step 7140] train/loss: 0.453230, train/mse: 0.453230, train/data_wait_sec: 0.083786, train/step_wall_sec: 3.673809, train/data_wait_ratio: 0.022806, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7140.000000 2026-03-09 04:49:54,418 [Rank train_rank0] [Epoch 0 | Step 7160] train/loss: 0.459042, train/mse: 0.459042, train/data_wait_sec: 0.075691, train/step_wall_sec: 3.655242, train/data_wait_ratio: 0.020707, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7160.000000 2026-03-09 04:51:08,000 [Rank train_rank0] [Epoch 0 | Step 7180] train/loss: 0.472450, train/mse: 0.472450, train/data_wait_sec: 0.073783, train/step_wall_sec: 3.624114, train/data_wait_ratio: 0.020359, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7180.000000 2026-03-09 04:52:22,134 [Rank train_rank0] [Epoch 0 | Step 7200] train/loss: 0.454348, train/mse: 0.454348, train/data_wait_sec: 0.109633, train/step_wall_sec: 3.709569, train/data_wait_ratio: 0.029554, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7200.000000 2026-03-09 04:53:36,024 [Rank train_rank0] [Epoch 0 | Step 7220] train/loss: 0.447680, train/mse: 0.447680, train/data_wait_sec: 0.055895, train/step_wall_sec: 3.657587, train/data_wait_ratio: 0.015282, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7220.000000 2026-03-09 04:54:49,767 [Rank train_rank0] [Epoch 0 | Step 7240] train/loss: 0.458301, train/mse: 0.458301, train/data_wait_sec: 0.067622, train/step_wall_sec: 3.625781, train/data_wait_ratio: 0.018650, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7240.000000 2026-03-09 04:56:03,738 [Rank train_rank0] [Epoch 0 | Step 7260] train/loss: 0.485541, train/mse: 0.485541, train/data_wait_sec: 0.172617, train/step_wall_sec: 3.763278, train/data_wait_ratio: 0.045869, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7260.000000 2026-03-09 04:57:17,434 [Rank train_rank0] [Epoch 0 | Step 7280] train/loss: 0.454104, train/mse: 0.454104, train/data_wait_sec: 0.070311, train/step_wall_sec: 3.643943, train/data_wait_ratio: 0.019295, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7280.000000 2026-03-09 04:58:31,406 [Rank train_rank0] [Epoch 0 | Step 7300] train/loss: 0.475264, train/mse: 0.475264, train/data_wait_sec: 0.059825, train/step_wall_sec: 3.634987, train/data_wait_ratio: 0.016458, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7300.000000 2026-03-09 04:59:45,360 [Rank train_rank0] [Epoch 0 | Step 7320] train/loss: 0.478548, train/mse: 0.478548, train/data_wait_sec: 0.079056, train/step_wall_sec: 3.705571, train/data_wait_ratio: 0.021334, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7320.000000 2026-03-09 05:00:59,086 [Rank train_rank0] [Epoch 0 | Step 7340] train/loss: 0.456902, train/mse: 0.456902, train/data_wait_sec: 0.073288, train/step_wall_sec: 3.641657, train/data_wait_ratio: 0.020125, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7340.000000 2026-03-09 05:02:13,393 [Rank train_rank0] [Epoch 0 | Step 7360] train/loss: 0.452058, train/mse: 0.452058, train/data_wait_sec: 0.048784, train/step_wall_sec: 3.657964, train/data_wait_ratio: 0.013336, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7360.000000 2026-03-09 05:03:27,458 [Rank train_rank0] [Epoch 0 | Step 7380] train/loss: 0.464978, train/mse: 0.464978, train/data_wait_sec: 0.063454, train/step_wall_sec: 3.638743, train/data_wait_ratio: 0.017438, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7380.000000 2026-03-09 05:04:41,219 [Rank train_rank0] [Epoch 0 | Step 7400] train/loss: 0.462049, train/mse: 0.462049, train/data_wait_sec: 0.061087, train/step_wall_sec: 3.640008, train/data_wait_ratio: 0.016782, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7400.000000 2026-03-09 05:05:55,359 [Rank train_rank0] [Epoch 0 | Step 7420] train/loss: 0.474427, train/mse: 0.474427, train/data_wait_sec: 0.057452, train/step_wall_sec: 3.633104, train/data_wait_ratio: 0.015813, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7420.000000 2026-03-09 05:07:09,117 [Rank train_rank0] [Epoch 0 | Step 7440] train/loss: 0.485083, train/mse: 0.485083, train/data_wait_sec: 0.078978, train/step_wall_sec: 3.663536, train/data_wait_ratio: 0.021558, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7440.000000 2026-03-09 05:08:22,934 [Rank train_rank0] [Epoch 0 | Step 7460] train/loss: 0.455576, train/mse: 0.455576, train/data_wait_sec: 0.176779, train/step_wall_sec: 3.837091, train/data_wait_ratio: 0.046071, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7460.000000 2026-03-09 05:09:36,826 [Rank train_rank0] [Epoch 0 | Step 7480] train/loss: 0.450395, train/mse: 0.450395, train/data_wait_sec: 0.075631, train/step_wall_sec: 3.665980, train/data_wait_ratio: 0.020631, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7480.000000 2026-03-09 05:10:50,650 [Rank train_rank0] [Epoch 0 | Step 7500] train/loss: 0.463712, train/mse: 0.463712, train/data_wait_sec: 0.092184, train/step_wall_sec: 3.648360, train/data_wait_ratio: 0.025267, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7500.000000 2026-03-09 05:12:04,562 [Rank train_rank0] [Epoch 0 | Step 7520] train/loss: 0.454927, train/mse: 0.454927, train/data_wait_sec: 0.086162, train/step_wall_sec: 3.698672, train/data_wait_ratio: 0.023295, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7520.000000 2026-03-09 05:13:18,623 [Rank train_rank0] [Epoch 0 | Step 7540] train/loss: 0.470491, train/mse: 0.470491, train/data_wait_sec: 0.163835, train/step_wall_sec: 3.735733, train/data_wait_ratio: 0.043856, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7540.000000 2026-03-09 05:14:32,241 [Rank train_rank0] [Epoch 0 | Step 7560] train/loss: 0.502955, train/mse: 0.502955, train/data_wait_sec: 0.056153, train/step_wall_sec: 3.590210, train/data_wait_ratio: 0.015640, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7560.000000 2026-03-09 05:15:46,529 [Rank train_rank0] [Epoch 0 | Step 7580] train/loss: 0.445317, train/mse: 0.445317, train/data_wait_sec: 0.037000, train/step_wall_sec: 3.655067, train/data_wait_ratio: 0.010123, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7580.000000 2026-03-09 05:17:00,398 [Rank train_rank0] [Epoch 0 | Step 7600] train/loss: 0.459152, train/mse: 0.459152, train/data_wait_sec: 0.073375, train/step_wall_sec: 3.664792, train/data_wait_ratio: 0.020022, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7600.000000 2026-03-09 05:18:14,514 [Rank train_rank0] [Epoch 0 | Step 7620] train/loss: 0.477375, train/mse: 0.477375, train/data_wait_sec: 0.061109, train/step_wall_sec: 3.613518, train/data_wait_ratio: 0.016911, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7620.000000 2026-03-09 05:19:28,335 [Rank train_rank0] [Epoch 0 | Step 7640] train/loss: 0.439686, train/mse: 0.439686, train/data_wait_sec: 0.033312, train/step_wall_sec: 3.620003, train/data_wait_ratio: 0.009202, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7640.000000 2026-03-09 05:20:42,237 [Rank train_rank0] [Epoch 0 | Step 7660] train/loss: 0.438170, train/mse: 0.438170, train/data_wait_sec: 0.085957, train/step_wall_sec: 3.676143, train/data_wait_ratio: 0.023382, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7660.000000 2026-03-09 05:21:56,339 [Rank train_rank0] [Epoch 0 | Step 7680] train/loss: 0.463031, train/mse: 0.463031, train/data_wait_sec: 0.061472, train/step_wall_sec: 3.662467, train/data_wait_ratio: 0.016784, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7680.000000 2026-03-09 05:23:10,201 [Rank train_rank0] [Epoch 0 | Step 7700] train/loss: 0.482855, train/mse: 0.482855, train/data_wait_sec: 0.098593, train/step_wall_sec: 3.675897, train/data_wait_ratio: 0.026821, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7700.000000 2026-03-09 05:24:23,883 [Rank train_rank0] [Epoch 0 | Step 7720] train/loss: 0.452744, train/mse: 0.452744, train/data_wait_sec: 0.073270, train/step_wall_sec: 3.673705, train/data_wait_ratio: 0.019945, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7720.000000 2026-03-09 05:25:38,029 [Rank train_rank0] [Epoch 0 | Step 7740] train/loss: 0.456546, train/mse: 0.456546, train/data_wait_sec: 0.085730, train/step_wall_sec: 3.682627, train/data_wait_ratio: 0.023280, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7740.000000 2026-03-09 05:26:51,878 [Rank train_rank0] [Epoch 0 | Step 7760] train/loss: 0.458085, train/mse: 0.458085, train/data_wait_sec: 0.073268, train/step_wall_sec: 3.691160, train/data_wait_ratio: 0.019850, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7760.000000 2026-03-09 05:28:05,994 [Rank train_rank0] [Epoch 0 | Step 7780] train/loss: 0.462700, train/mse: 0.462700, train/data_wait_sec: 0.060839, train/step_wall_sec: 3.677383, train/data_wait_ratio: 0.016544, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7780.000000 2026-03-09 05:29:19,926 [Rank train_rank0] [Epoch 0 | Step 7800] train/loss: 0.479598, train/mse: 0.479598, train/data_wait_sec: 0.064358, train/step_wall_sec: 3.635616, train/data_wait_ratio: 0.017702, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7800.000000 2026-03-09 05:30:34,136 [Rank train_rank0] [Epoch 0 | Step 7820] train/loss: 0.463043, train/mse: 0.463043, train/data_wait_sec: 0.098721, train/step_wall_sec: 3.668198, train/data_wait_ratio: 0.026913, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7820.000000 2026-03-09 05:31:48,028 [Rank train_rank0] [Epoch 0 | Step 7840] train/loss: 0.444822, train/mse: 0.444822, train/data_wait_sec: 0.049520, train/step_wall_sec: 3.658905, train/data_wait_ratio: 0.013534, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7840.000000 2026-03-09 05:33:01,736 [Rank train_rank0] [Epoch 0 | Step 7860] train/loss: 0.480696, train/mse: 0.480696, train/data_wait_sec: 0.090921, train/step_wall_sec: 3.686778, train/data_wait_ratio: 0.024661, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7860.000000 2026-03-09 05:34:15,380 [Rank train_rank0] [Epoch 0 | Step 7880] train/loss: 0.469530, train/mse: 0.469530, train/data_wait_sec: 0.057559, train/step_wall_sec: 3.634012, train/data_wait_ratio: 0.015839, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7880.000000 2026-03-09 05:35:29,413 [Rank train_rank0] [Epoch 0 | Step 7900] train/loss: 0.433880, train/mse: 0.433880, train/data_wait_sec: 0.059904, train/step_wall_sec: 3.622757, train/data_wait_ratio: 0.016535, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7900.000000 2026-03-09 05:36:43,646 [Rank train_rank0] [Epoch 0 | Step 7920] train/loss: 0.444797, train/mse: 0.444797, train/data_wait_sec: 0.065594, train/step_wall_sec: 3.619267, train/data_wait_ratio: 0.018124, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7920.000000 2026-03-09 05:37:57,585 [Rank train_rank0] [Epoch 0 | Step 7940] train/loss: 0.447257, train/mse: 0.447257, train/data_wait_sec: 0.075525, train/step_wall_sec: 3.667333, train/data_wait_ratio: 0.020594, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7940.000000 2026-03-09 05:39:11,444 [Rank train_rank0] [Epoch 0 | Step 7960] train/loss: 0.465497, train/mse: 0.465497, train/data_wait_sec: 0.059847, train/step_wall_sec: 3.641191, train/data_wait_ratio: 0.016436, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7960.000000 2026-03-09 05:40:25,311 [Rank train_rank0] [Epoch 0 | Step 7980] train/loss: 0.444236, train/mse: 0.444236, train/data_wait_sec: 0.077360, train/step_wall_sec: 3.658327, train/data_wait_ratio: 0.021146, train/lr: 0.000200, train/epoch: 0.000000, train/step: 7980.000000 2026-03-09 05:42:04,842 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000008000.pt 2026-03-09 05:42:04,845 [Rank train_rank0] [Epoch 0 | Step 8000] train/loss: 0.435769, train/mse: 0.435769, train/data_wait_sec: 0.082243, train/step_wall_sec: 3.660086, train/data_wait_ratio: 0.022470, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8000.000000 2026-03-09 05:42:04,846 [Rank train_rank0] Generating samples for visualization... 2026-03-09 05:42:07,692 [Rank train_rank0] Sample generation done. 2026-03-09 05:43:21,516 [Rank train_rank0] [Epoch 0 | Step 8020] train/loss: 0.475433, train/mse: 0.475433, train/data_wait_sec: 0.078131, train/step_wall_sec: 3.626228, train/data_wait_ratio: 0.021546, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8020.000000 2026-03-09 05:44:35,300 [Rank train_rank0] [Epoch 0 | Step 8040] train/loss: 0.474970, train/mse: 0.474970, train/data_wait_sec: 0.057716, train/step_wall_sec: 3.797049, train/data_wait_ratio: 0.015200, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8040.000000 2026-03-09 05:45:49,228 [Rank train_rank0] [Epoch 0 | Step 8060] train/loss: 0.466201, train/mse: 0.466201, train/data_wait_sec: 0.048970, train/step_wall_sec: 3.658416, train/data_wait_ratio: 0.013386, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8060.000000 2026-03-09 05:47:03,198 [Rank train_rank0] [Epoch 0 | Step 8080] train/loss: 0.446099, train/mse: 0.446099, train/data_wait_sec: 0.065233, train/step_wall_sec: 3.660330, train/data_wait_ratio: 0.017822, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8080.000000 2026-03-09 05:48:17,232 [Rank train_rank0] [Epoch 0 | Step 8100] train/loss: 0.433448, train/mse: 0.433448, train/data_wait_sec: 0.060443, train/step_wall_sec: 3.651635, train/data_wait_ratio: 0.016552, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8100.000000 2026-03-09 05:49:31,185 [Rank train_rank0] [Epoch 0 | Step 8120] train/loss: 0.467169, train/mse: 0.467169, train/data_wait_sec: 0.053136, train/step_wall_sec: 3.658903, train/data_wait_ratio: 0.014522, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8120.000000 2026-03-09 05:50:45,034 [Rank train_rank0] [Epoch 0 | Step 8140] train/loss: 0.475650, train/mse: 0.475650, train/data_wait_sec: 0.051240, train/step_wall_sec: 3.621715, train/data_wait_ratio: 0.014148, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8140.000000 2026-03-09 05:51:59,012 [Rank train_rank0] [Epoch 0 | Step 8160] train/loss: 0.462841, train/mse: 0.462841, train/data_wait_sec: 0.057467, train/step_wall_sec: 3.635989, train/data_wait_ratio: 0.015805, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8160.000000 2026-03-09 05:53:13,169 [Rank train_rank0] [Epoch 0 | Step 8180] train/loss: 0.465001, train/mse: 0.465001, train/data_wait_sec: 0.072986, train/step_wall_sec: 3.668193, train/data_wait_ratio: 0.019897, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8180.000000 2026-03-09 05:54:27,292 [Rank train_rank0] [Epoch 0 | Step 8200] train/loss: 0.460801, train/mse: 0.460801, train/data_wait_sec: 0.181062, train/step_wall_sec: 3.835405, train/data_wait_ratio: 0.047208, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8200.000000 2026-03-09 05:55:41,022 [Rank train_rank0] [Epoch 0 | Step 8220] train/loss: 0.447726, train/mse: 0.447726, train/data_wait_sec: 0.064262, train/step_wall_sec: 3.648977, train/data_wait_ratio: 0.017611, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8220.000000 2026-03-09 05:56:54,775 [Rank train_rank0] [Epoch 0 | Step 8240] train/loss: 0.442626, train/mse: 0.442626, train/data_wait_sec: 0.053429, train/step_wall_sec: 3.661682, train/data_wait_ratio: 0.014591, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8240.000000 2026-03-09 05:58:08,496 [Rank train_rank0] [Epoch 0 | Step 8260] train/loss: 0.446193, train/mse: 0.446193, train/data_wait_sec: 0.055336, train/step_wall_sec: 3.618569, train/data_wait_ratio: 0.015292, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8260.000000 2026-03-09 05:59:22,442 [Rank train_rank0] [Epoch 0 | Step 8280] train/loss: 0.442677, train/mse: 0.442677, train/data_wait_sec: 0.039667, train/step_wall_sec: 3.637895, train/data_wait_ratio: 0.010904, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8280.000000 2026-03-09 06:00:36,314 [Rank train_rank0] [Epoch 0 | Step 8300] train/loss: 0.470587, train/mse: 0.470587, train/data_wait_sec: 0.047070, train/step_wall_sec: 3.659817, train/data_wait_ratio: 0.012861, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8300.000000 2026-03-09 06:01:50,209 [Rank train_rank0] [Epoch 0 | Step 8320] train/loss: 0.467160, train/mse: 0.467160, train/data_wait_sec: 0.152226, train/step_wall_sec: 3.783094, train/data_wait_ratio: 0.040239, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8320.000000 2026-03-09 06:03:04,049 [Rank train_rank0] [Epoch 0 | Step 8340] train/loss: 0.490010, train/mse: 0.490010, train/data_wait_sec: 0.080807, train/step_wall_sec: 3.665725, train/data_wait_ratio: 0.022044, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8340.000000 2026-03-09 06:04:17,939 [Rank train_rank0] [Epoch 0 | Step 8360] train/loss: 0.446921, train/mse: 0.446921, train/data_wait_sec: 0.079363, train/step_wall_sec: 3.669938, train/data_wait_ratio: 0.021625, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8360.000000 2026-03-09 06:05:31,579 [Rank train_rank0] [Epoch 0 | Step 8380] train/loss: 0.452243, train/mse: 0.452243, train/data_wait_sec: 0.073048, train/step_wall_sec: 3.630342, train/data_wait_ratio: 0.020122, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8380.000000 2026-03-09 06:06:45,394 [Rank train_rank0] [Epoch 0 | Step 8400] train/loss: 0.459303, train/mse: 0.459303, train/data_wait_sec: 0.069342, train/step_wall_sec: 3.656197, train/data_wait_ratio: 0.018966, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8400.000000 2026-03-09 06:07:59,254 [Rank train_rank0] [Epoch 0 | Step 8420] train/loss: 0.454760, train/mse: 0.454760, train/data_wait_sec: 0.057298, train/step_wall_sec: 3.630196, train/data_wait_ratio: 0.015784, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8420.000000 2026-03-09 06:09:13,167 [Rank train_rank0] [Epoch 0 | Step 8440] train/loss: 0.470088, train/mse: 0.470088, train/data_wait_sec: 0.049049, train/step_wall_sec: 3.645425, train/data_wait_ratio: 0.013455, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8440.000000 2026-03-09 06:10:26,895 [Rank train_rank0] [Epoch 0 | Step 8460] train/loss: 0.459870, train/mse: 0.459870, train/data_wait_sec: 0.059894, train/step_wall_sec: 3.650435, train/data_wait_ratio: 0.016407, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8460.000000 2026-03-09 06:11:40,720 [Rank train_rank0] [Epoch 0 | Step 8480] train/loss: 0.446712, train/mse: 0.446712, train/data_wait_sec: 0.071669, train/step_wall_sec: 3.609589, train/data_wait_ratio: 0.019855, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8480.000000 2026-03-09 06:12:54,544 [Rank train_rank0] [Epoch 0 | Step 8500] train/loss: 0.465227, train/mse: 0.465227, train/data_wait_sec: 0.041188, train/step_wall_sec: 3.604088, train/data_wait_ratio: 0.011428, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8500.000000 2026-03-09 06:14:08,393 [Rank train_rank0] [Epoch 0 | Step 8520] train/loss: 0.456051, train/mse: 0.456051, train/data_wait_sec: 0.054441, train/step_wall_sec: 3.652024, train/data_wait_ratio: 0.014907, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8520.000000 2026-03-09 06:15:22,191 [Rank train_rank0] [Epoch 0 | Step 8540] train/loss: 0.453467, train/mse: 0.453467, train/data_wait_sec: 0.182891, train/step_wall_sec: 3.790090, train/data_wait_ratio: 0.048255, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8540.000000 2026-03-09 06:16:36,080 [Rank train_rank0] [Epoch 0 | Step 8560] train/loss: 0.447060, train/mse: 0.447060, train/data_wait_sec: 0.047956, train/step_wall_sec: 3.629172, train/data_wait_ratio: 0.013214, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8560.000000 2026-03-09 06:17:49,839 [Rank train_rank0] [Epoch 0 | Step 8580] train/loss: 0.478841, train/mse: 0.478841, train/data_wait_sec: 0.057203, train/step_wall_sec: 3.626186, train/data_wait_ratio: 0.015775, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8580.000000 2026-03-09 06:19:03,962 [Rank train_rank0] [Epoch 0 | Step 8600] train/loss: 0.469029, train/mse: 0.469029, train/data_wait_sec: 0.054249, train/step_wall_sec: 3.650862, train/data_wait_ratio: 0.014859, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8600.000000 2026-03-09 06:20:17,840 [Rank train_rank0] [Epoch 0 | Step 8620] train/loss: 0.457790, train/mse: 0.457790, train/data_wait_sec: 0.134849, train/step_wall_sec: 3.771400, train/data_wait_ratio: 0.035756, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8620.000000 2026-03-09 06:21:31,583 [Rank train_rank0] [Epoch 0 | Step 8640] train/loss: 0.440546, train/mse: 0.440546, train/data_wait_sec: 0.089724, train/step_wall_sec: 3.681627, train/data_wait_ratio: 0.024371, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8640.000000 2026-03-09 06:22:45,618 [Rank train_rank0] [Epoch 0 | Step 8660] train/loss: 0.438316, train/mse: 0.438316, train/data_wait_sec: 0.054358, train/step_wall_sec: 3.648269, train/data_wait_ratio: 0.014900, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8660.000000 2026-03-09 06:23:59,671 [Rank train_rank0] [Epoch 0 | Step 8680] train/loss: 0.430611, train/mse: 0.430611, train/data_wait_sec: 0.046791, train/step_wall_sec: 3.650674, train/data_wait_ratio: 0.012817, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8680.000000 2026-03-09 06:25:13,550 [Rank train_rank0] [Epoch 0 | Step 8700] train/loss: 0.444669, train/mse: 0.444669, train/data_wait_sec: 0.055131, train/step_wall_sec: 3.622109, train/data_wait_ratio: 0.015221, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8700.000000 2026-03-09 06:26:27,461 [Rank train_rank0] [Epoch 0 | Step 8720] train/loss: 0.458017, train/mse: 0.458017, train/data_wait_sec: 0.106355, train/step_wall_sec: 3.708119, train/data_wait_ratio: 0.028682, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8720.000000 2026-03-09 06:27:41,217 [Rank train_rank0] [Epoch 0 | Step 8740] train/loss: 0.486551, train/mse: 0.486551, train/data_wait_sec: 0.079714, train/step_wall_sec: 3.693981, train/data_wait_ratio: 0.021580, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8740.000000 2026-03-09 06:28:55,232 [Rank train_rank0] [Epoch 0 | Step 8760] train/loss: 0.468511, train/mse: 0.468511, train/data_wait_sec: 0.072044, train/step_wall_sec: 3.672133, train/data_wait_ratio: 0.019619, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8760.000000 2026-03-09 06:30:09,248 [Rank train_rank0] [Epoch 0 | Step 8780] train/loss: 0.468654, train/mse: 0.468654, train/data_wait_sec: 0.126726, train/step_wall_sec: 3.727377, train/data_wait_ratio: 0.033999, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8780.000000 2026-03-09 06:31:22,991 [Rank train_rank0] [Epoch 0 | Step 8800] train/loss: 0.463378, train/mse: 0.463378, train/data_wait_sec: 0.060995, train/step_wall_sec: 3.657053, train/data_wait_ratio: 0.016679, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8800.000000 2026-03-09 06:32:36,732 [Rank train_rank0] [Epoch 0 | Step 8820] train/loss: 0.467005, train/mse: 0.467005, train/data_wait_sec: 0.081177, train/step_wall_sec: 3.670300, train/data_wait_ratio: 0.022117, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8820.000000 2026-03-09 06:33:50,706 [Rank train_rank0] [Epoch 0 | Step 8840] train/loss: 0.446022, train/mse: 0.446022, train/data_wait_sec: 0.052724, train/step_wall_sec: 3.615129, train/data_wait_ratio: 0.014584, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8840.000000 2026-03-09 06:35:04,576 [Rank train_rank0] [Epoch 0 | Step 8860] train/loss: 0.468607, train/mse: 0.468607, train/data_wait_sec: 0.054381, train/step_wall_sec: 3.651637, train/data_wait_ratio: 0.014892, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8860.000000 2026-03-09 06:36:18,571 [Rank train_rank0] [Epoch 0 | Step 8880] train/loss: 0.438647, train/mse: 0.438647, train/data_wait_sec: 0.075681, train/step_wall_sec: 3.671259, train/data_wait_ratio: 0.020614, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8880.000000 2026-03-09 06:37:32,480 [Rank train_rank0] [Epoch 0 | Step 8900] train/loss: 0.436947, train/mse: 0.436947, train/data_wait_sec: 0.164345, train/step_wall_sec: 3.753249, train/data_wait_ratio: 0.043787, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8900.000000 2026-03-09 06:38:46,400 [Rank train_rank0] [Epoch 0 | Step 8920] train/loss: 0.494890, train/mse: 0.494890, train/data_wait_sec: 0.043467, train/step_wall_sec: 3.615128, train/data_wait_ratio: 0.012024, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8920.000000 2026-03-09 06:40:00,397 [Rank train_rank0] [Epoch 0 | Step 8940] train/loss: 0.461754, train/mse: 0.461754, train/data_wait_sec: 0.061573, train/step_wall_sec: 3.659443, train/data_wait_ratio: 0.016826, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8940.000000 2026-03-09 06:41:14,238 [Rank train_rank0] [Epoch 0 | Step 8960] train/loss: 0.451327, train/mse: 0.451327, train/data_wait_sec: 0.068005, train/step_wall_sec: 3.630312, train/data_wait_ratio: 0.018733, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8960.000000 2026-03-09 06:42:28,380 [Rank train_rank0] [Epoch 0 | Step 8980] train/loss: 0.438443, train/mse: 0.438443, train/data_wait_sec: 0.060881, train/step_wall_sec: 3.681199, train/data_wait_ratio: 0.016538, train/lr: 0.000200, train/epoch: 0.000000, train/step: 8980.000000 2026-03-09 06:44:07,823 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000009000.pt 2026-03-09 06:44:07,826 [Rank train_rank0] [Epoch 0 | Step 9000] train/loss: 0.454961, train/mse: 0.454961, train/data_wait_sec: 0.075225, train/step_wall_sec: 3.628419, train/data_wait_ratio: 0.020732, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9000.000000 2026-03-09 06:44:07,827 [Rank train_rank0] Generating samples for visualization... 2026-03-09 06:44:10,759 [Rank train_rank0] Sample generation done. 2026-03-09 06:45:24,733 [Rank train_rank0] [Epoch 0 | Step 9020] train/loss: 0.469845, train/mse: 0.469845, train/data_wait_sec: 0.046354, train/step_wall_sec: 3.667966, train/data_wait_ratio: 0.012638, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9020.000000 2026-03-09 06:46:38,587 [Rank train_rank0] [Epoch 0 | Step 9040] train/loss: 0.495624, train/mse: 0.495624, train/data_wait_sec: 0.083150, train/step_wall_sec: 3.796262, train/data_wait_ratio: 0.021903, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9040.000000 2026-03-09 06:47:52,417 [Rank train_rank0] [Epoch 0 | Step 9060] train/loss: 0.464442, train/mse: 0.464442, train/data_wait_sec: 0.073352, train/step_wall_sec: 3.746777, train/data_wait_ratio: 0.019577, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9060.000000 2026-03-09 06:49:06,192 [Rank train_rank0] [Epoch 0 | Step 9080] train/loss: 0.453844, train/mse: 0.453844, train/data_wait_sec: 0.032703, train/step_wall_sec: 3.604174, train/data_wait_ratio: 0.009074, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9080.000000 2026-03-09 06:50:20,341 [Rank train_rank0] [Epoch 0 | Step 9100] train/loss: 0.453850, train/mse: 0.453850, train/data_wait_sec: 0.083687, train/step_wall_sec: 3.668592, train/data_wait_ratio: 0.022812, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9100.000000 2026-03-09 06:51:34,607 [Rank train_rank0] [Epoch 0 | Step 9120] train/loss: 0.437493, train/mse: 0.437493, train/data_wait_sec: 0.167868, train/step_wall_sec: 3.792797, train/data_wait_ratio: 0.044260, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9120.000000 2026-03-09 06:52:48,465 [Rank train_rank0] [Epoch 0 | Step 9140] train/loss: 0.441124, train/mse: 0.441124, train/data_wait_sec: 0.055238, train/step_wall_sec: 3.656538, train/data_wait_ratio: 0.015107, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9140.000000 2026-03-09 06:54:02,430 [Rank train_rank0] [Epoch 0 | Step 9160] train/loss: 0.456676, train/mse: 0.456676, train/data_wait_sec: 0.111870, train/step_wall_sec: 3.689254, train/data_wait_ratio: 0.030323, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9160.000000 2026-03-09 06:55:16,437 [Rank train_rank0] [Epoch 0 | Step 9180] train/loss: 0.452802, train/mse: 0.452802, train/data_wait_sec: 0.067407, train/step_wall_sec: 3.664438, train/data_wait_ratio: 0.018395, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9180.000000 2026-03-09 06:56:30,615 [Rank train_rank0] [Epoch 0 | Step 9200] train/loss: 0.473781, train/mse: 0.473781, train/data_wait_sec: 0.081408, train/step_wall_sec: 3.654543, train/data_wait_ratio: 0.022276, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9200.000000 2026-03-09 06:57:44,396 [Rank train_rank0] [Epoch 0 | Step 9220] train/loss: 0.453514, train/mse: 0.453514, train/data_wait_sec: 0.064108, train/step_wall_sec: 3.641234, train/data_wait_ratio: 0.017606, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9220.000000 2026-03-09 06:58:58,274 [Rank train_rank0] [Epoch 0 | Step 9240] train/loss: 0.474019, train/mse: 0.474019, train/data_wait_sec: 0.077958, train/step_wall_sec: 3.636999, train/data_wait_ratio: 0.021435, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9240.000000 2026-03-09 07:00:12,388 [Rank train_rank0] [Epoch 0 | Step 9260] train/loss: 0.426224, train/mse: 0.426224, train/data_wait_sec: 0.108967, train/step_wall_sec: 3.693763, train/data_wait_ratio: 0.029500, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9260.000000 2026-03-09 07:01:26,057 [Rank train_rank0] [Epoch 0 | Step 9280] train/loss: 0.470715, train/mse: 0.470715, train/data_wait_sec: 0.037032, train/step_wall_sec: 3.606337, train/data_wait_ratio: 0.010269, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9280.000000 2026-03-09 07:02:40,022 [Rank train_rank0] [Epoch 0 | Step 9300] train/loss: 0.436034, train/mse: 0.436034, train/data_wait_sec: 0.069294, train/step_wall_sec: 3.779630, train/data_wait_ratio: 0.018333, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9300.000000 2026-03-09 07:13:38,241 [Rank train_rank0] Discarding a locally fetched batch to keep DDP ranks synchronized after a peer streaming failure. 2026-03-09 07:13:38,244 [Rank train_rank0] Recoverable streaming error (1/32): RuntimeError: peer rank reported a recoverable streaming failure. Skipping this batch without rebuilding (threshold=6). 2026-03-09 07:14:38,341 [Rank train_rank0] Streaming dataloader recovered after 1 retry attempt(s). 2026-03-14 03:50:03,297 [Rank train_rank0] Building UniDiT model... 2026-03-14 03:50:51,102 [Rank train_rank0] Building RAE (frozen)... 2026-03-14 03:51:03,075 [Rank train_rank0] Loaded RAE decoder from '/scratch/e1539128/RAE/ckpts/stage1/qwen3vlrae_gram_v1.4_448/checkpoints/ep-0000029.pt' (source='ema', decoder_keys=271, to_pixels_keys_in_ckpt=2). 2026-03-14 03:51:03,755 [Rank train_rank0] UniDiT trainable: 1455.91M / total: 5894.39M 2026-03-14 03:51:03,756 [Rank train_rank0] Noise schedule shift: enabled=True, ratio=14.000000 2026-03-14 03:51:03,757 [Rank train_rank0] Latent normalization: enabled=True, path='ckpts/latent_stats.pt', global_mean=0.287388, global_std=28.285812, token_count=40140800 2026-03-14 03:51:03,757 [Rank train_rank0] CFG uncondition dropout (null embedding): 0.1 2026-03-14 03:51:03,758 [Rank train_rank0] EMA maintenance: disabled (no EMA weights tracked during training). 2026-03-14 03:51:06,929 [Rank train_rank0] Optimizer: AdamW(lr=0.0002, betas=(0.9, 0.95), wd=0.01) 2026-03-14 03:51:06,930 [Rank train_rank0] Optimizer override: set weight_decay=0.0 for trainable query-token embeddings. 2026-03-14 03:51:06,930 [Rank train_rank0] Configuring streaming data loader... 2026-03-14 03:51:06,931 [Rank train_rank0] Streaming retry policy: max_consecutive_failures=32, rebuild_after_consecutive_failures=6, base_sleep=2.0s, max_sleep=20.0s. 2026-03-14 03:51:06,932 [Rank train_rank0] Loader config: num_workers=2, prefetch_factor=2, shuffle_buffer_size=2000, loader_timeout=600s, curl(connect_timeout=30, max_time=0, retry=15, speed_time=120, speed_limit=65536) 2026-03-14 03:51:06,932 [Rank train_rank0] Mixed dataset plan (configured weight -> normalized ratio): 2026-03-14 03:51:06,933 [Rank train_rank0] [1] fine_t2i-enhanced_prompt_square_resolution: weight=0.5, ratio=50.00%, url=https://huggingface.co/datasets/ma-xu/fine-t2i/resolve/main/synthetic_enhanced_prompt_square_resolution/train-{000000..001543}.tar 2026-03-14 03:51:06,934 [Rank train_rank0] [2] fine_t2i-original_prompt_square_resolution: weight=0.5, ratio=50.00%, url=https://huggingface.co/datasets/ma-xu/fine-t2i/resolve/main/synthetic_original_prompt_square_resolution/train-{000000..001310}.tar 2026-03-14 03:51:06,934 [Rank train_rank0] Scheduler: cosine(warmup=0.07ep/2100steps, decay_end=30.0ep/900000steps, base_lr=0.0002, final_lr=1e-06) 2026-03-14 03:51:06,936 [Rank train_rank0] Resuming from /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000009000.pt... 2026-03-14 03:51:39,659 [Rank train_rank0] Resumed at epoch=0, step=9000 2026-03-14 03:51:47,945 [Rank train_rank0] Sample visualization: sample_with_ema=False, log_non_ema_samples=False, sample_num_steps=10, sample_guidance_scale=7.0 2026-03-14 03:51:47,945 [Rank train_rank0] Starting training: 30 epochs, micro_bs=10/gpu, accum=4, global_bs=160, image_size=448, steps_per_epoch=30000 2026-03-14 03:51:47,946 [Rank train_rank0] Checkpoint cadence: every 1000 steps. 2026-03-14 03:51:47,948 [Rank train_rank0] Building stream loader (stream_epoch=0)... 2026-03-14 03:51:47,954 [Rank train_rank0] Creating stream iterator... 2026-03-14 03:51:47,954 [Rank train_rank0] Stream iterator is ready. 2026-03-14 03:53:14,749 [Rank train_rank0] [Epoch 0 | Step 9020] train/loss: 0.462013, train/mse: 0.462013, train/data_wait_sec: 0.101021, train/step_wall_sec: 3.947968, train/data_wait_ratio: 0.025588, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9020.000000 2026-03-14 03:54:34,727 [Rank train_rank0] [Epoch 0 | Step 9040] train/loss: 0.471560, train/mse: 0.471560, train/data_wait_sec: 0.145325, train/step_wall_sec: 3.929235, train/data_wait_ratio: 0.036985, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9040.000000 2026-03-14 03:55:54,054 [Rank train_rank0] [Epoch 0 | Step 9060] train/loss: 0.429303, train/mse: 0.429303, train/data_wait_sec: 0.107424, train/step_wall_sec: 3.887763, train/data_wait_ratio: 0.027631, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9060.000000 2026-03-14 03:57:13,180 [Rank train_rank0] [Epoch 0 | Step 9080] train/loss: 0.426072, train/mse: 0.426072, train/data_wait_sec: 0.241046, train/step_wall_sec: 4.091932, train/data_wait_ratio: 0.058908, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9080.000000 2026-03-14 03:58:32,067 [Rank train_rank0] [Epoch 0 | Step 9100] train/loss: 0.446846, train/mse: 0.446846, train/data_wait_sec: 0.122452, train/step_wall_sec: 3.919906, train/data_wait_ratio: 0.031239, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9100.000000 2026-03-14 03:59:51,245 [Rank train_rank0] [Epoch 0 | Step 9120] train/loss: 0.473104, train/mse: 0.473104, train/data_wait_sec: 0.122314, train/step_wall_sec: 3.953568, train/data_wait_ratio: 0.030938, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9120.000000 2026-03-14 04:01:11,123 [Rank train_rank0] [Epoch 0 | Step 9140] train/loss: 0.479210, train/mse: 0.479210, train/data_wait_sec: 0.136176, train/step_wall_sec: 3.941847, train/data_wait_ratio: 0.034546, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9140.000000 2026-03-14 04:02:30,665 [Rank train_rank0] [Epoch 0 | Step 9160] train/loss: 0.459560, train/mse: 0.459560, train/data_wait_sec: 0.083224, train/step_wall_sec: 3.872502, train/data_wait_ratio: 0.021491, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9160.000000 2026-03-14 04:03:51,008 [Rank train_rank0] [Epoch 0 | Step 9180] train/loss: 0.444284, train/mse: 0.444284, train/data_wait_sec: 0.170932, train/step_wall_sec: 3.953272, train/data_wait_ratio: 0.043238, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9180.000000 2026-03-14 04:05:10,478 [Rank train_rank0] [Epoch 0 | Step 9200] train/loss: 0.464124, train/mse: 0.464124, train/data_wait_sec: 0.125028, train/step_wall_sec: 3.936847, train/data_wait_ratio: 0.031758, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9200.000000 2026-03-14 04:06:29,701 [Rank train_rank0] [Epoch 0 | Step 9220] train/loss: 0.461431, train/mse: 0.461431, train/data_wait_sec: 0.105104, train/step_wall_sec: 3.823445, train/data_wait_ratio: 0.027489, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9220.000000 2026-03-14 04:07:49,139 [Rank train_rank0] [Epoch 0 | Step 9240] train/loss: 0.449722, train/mse: 0.449722, train/data_wait_sec: 0.167896, train/step_wall_sec: 3.957032, train/data_wait_ratio: 0.042430, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9240.000000 2026-03-14 04:09:08,448 [Rank train_rank0] [Epoch 0 | Step 9260] train/loss: 0.453457, train/mse: 0.453457, train/data_wait_sec: 0.106025, train/step_wall_sec: 3.932014, train/data_wait_ratio: 0.026964, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9260.000000 2026-03-14 04:10:28,100 [Rank train_rank0] [Epoch 0 | Step 9280] train/loss: 0.486384, train/mse: 0.486384, train/data_wait_sec: 0.171633, train/step_wall_sec: 4.022045, train/data_wait_ratio: 0.042673, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9280.000000 2026-03-14 04:11:47,862 [Rank train_rank0] [Epoch 0 | Step 9300] train/loss: 0.450794, train/mse: 0.450794, train/data_wait_sec: 0.147167, train/step_wall_sec: 4.020406, train/data_wait_ratio: 0.036605, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9300.000000 2026-03-14 04:13:07,303 [Rank train_rank0] [Epoch 0 | Step 9320] train/loss: 0.447545, train/mse: 0.447545, train/data_wait_sec: 0.108481, train/step_wall_sec: 3.924654, train/data_wait_ratio: 0.027641, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9320.000000 2026-03-14 04:14:26,902 [Rank train_rank0] [Epoch 0 | Step 9340] train/loss: 0.461355, train/mse: 0.461355, train/data_wait_sec: 0.114544, train/step_wall_sec: 3.921386, train/data_wait_ratio: 0.029210, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9340.000000 2026-03-14 04:15:46,367 [Rank train_rank0] [Epoch 0 | Step 9360] train/loss: 0.468861, train/mse: 0.468861, train/data_wait_sec: 0.062388, train/step_wall_sec: 4.025376, train/data_wait_ratio: 0.015499, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9360.000000 2026-03-14 04:17:06,013 [Rank train_rank0] [Epoch 0 | Step 9380] train/loss: 0.464122, train/mse: 0.464122, train/data_wait_sec: 0.206045, train/step_wall_sec: 4.000749, train/data_wait_ratio: 0.051502, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9380.000000 2026-03-14 04:18:25,645 [Rank train_rank0] [Epoch 0 | Step 9400] train/loss: 0.452589, train/mse: 0.452589, train/data_wait_sec: 0.149760, train/step_wall_sec: 3.957500, train/data_wait_ratio: 0.037842, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9400.000000 2026-03-14 04:19:45,365 [Rank train_rank0] [Epoch 0 | Step 9420] train/loss: 0.431113, train/mse: 0.431113, train/data_wait_sec: 0.142065, train/step_wall_sec: 3.931233, train/data_wait_ratio: 0.036137, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9420.000000 2026-03-14 04:21:04,853 [Rank train_rank0] [Epoch 0 | Step 9440] train/loss: 0.469482, train/mse: 0.469482, train/data_wait_sec: 0.129580, train/step_wall_sec: 3.976603, train/data_wait_ratio: 0.032586, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9440.000000 2026-03-14 04:22:24,619 [Rank train_rank0] [Epoch 0 | Step 9460] train/loss: 0.461776, train/mse: 0.461776, train/data_wait_sec: 0.106873, train/step_wall_sec: 3.966699, train/data_wait_ratio: 0.026943, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9460.000000 2026-03-14 04:23:44,358 [Rank train_rank0] [Epoch 0 | Step 9480] train/loss: 0.483091, train/mse: 0.483091, train/data_wait_sec: 0.097333, train/step_wall_sec: 3.912681, train/data_wait_ratio: 0.024876, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9480.000000 2026-03-14 04:25:03,426 [Rank train_rank0] [Epoch 0 | Step 9500] train/loss: 0.494796, train/mse: 0.494796, train/data_wait_sec: 0.139493, train/step_wall_sec: 3.934444, train/data_wait_ratio: 0.035454, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9500.000000 2026-03-14 04:26:22,979 [Rank train_rank0] [Epoch 0 | Step 9520] train/loss: 0.476565, train/mse: 0.476565, train/data_wait_sec: 0.088456, train/step_wall_sec: 3.926197, train/data_wait_ratio: 0.022530, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9520.000000 2026-03-14 04:27:42,852 [Rank train_rank0] [Epoch 0 | Step 9540] train/loss: 0.453595, train/mse: 0.453595, train/data_wait_sec: 0.125437, train/step_wall_sec: 3.891507, train/data_wait_ratio: 0.032233, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9540.000000 2026-03-14 04:29:02,384 [Rank train_rank0] [Epoch 0 | Step 9560] train/loss: 0.429066, train/mse: 0.429066, train/data_wait_sec: 0.112283, train/step_wall_sec: 3.932357, train/data_wait_ratio: 0.028554, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9560.000000 2026-03-14 04:30:22,075 [Rank train_rank0] [Epoch 0 | Step 9580] train/loss: 0.470725, train/mse: 0.470725, train/data_wait_sec: 0.144068, train/step_wall_sec: 3.901093, train/data_wait_ratio: 0.036930, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9580.000000 2026-03-14 04:31:41,748 [Rank train_rank0] [Epoch 0 | Step 9600] train/loss: 0.468201, train/mse: 0.468201, train/data_wait_sec: 0.146296, train/step_wall_sec: 3.954827, train/data_wait_ratio: 0.036992, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9600.000000 2026-03-14 04:33:01,448 [Rank train_rank0] [Epoch 0 | Step 9620] train/loss: 0.458444, train/mse: 0.458444, train/data_wait_sec: 0.168923, train/step_wall_sec: 3.957053, train/data_wait_ratio: 0.042689, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9620.000000 2026-03-14 04:34:21,306 [Rank train_rank0] [Epoch 0 | Step 9640] train/loss: 0.454515, train/mse: 0.454515, train/data_wait_sec: 0.130422, train/step_wall_sec: 3.996822, train/data_wait_ratio: 0.032631, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9640.000000 2026-03-14 04:35:40,853 [Rank train_rank0] [Epoch 0 | Step 9660] train/loss: 0.461293, train/mse: 0.461293, train/data_wait_sec: 0.143407, train/step_wall_sec: 4.015293, train/data_wait_ratio: 0.035715, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9660.000000 2026-03-14 04:37:01,190 [Rank train_rank0] [Epoch 0 | Step 9680] train/loss: 0.475454, train/mse: 0.475454, train/data_wait_sec: 0.101081, train/step_wall_sec: 3.954585, train/data_wait_ratio: 0.025560, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9680.000000 2026-03-14 04:38:21,072 [Rank train_rank0] [Epoch 0 | Step 9700] train/loss: 0.440963, train/mse: 0.440963, train/data_wait_sec: 0.150450, train/step_wall_sec: 3.986635, train/data_wait_ratio: 0.037739, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9700.000000 2026-03-14 04:39:40,548 [Rank train_rank0] [Epoch 0 | Step 9720] train/loss: 0.443793, train/mse: 0.443793, train/data_wait_sec: 0.107731, train/step_wall_sec: 3.895943, train/data_wait_ratio: 0.027652, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9720.000000 2026-03-14 04:41:00,411 [Rank train_rank0] [Epoch 0 | Step 9740] train/loss: 0.464772, train/mse: 0.464772, train/data_wait_sec: 0.097114, train/step_wall_sec: 4.060736, train/data_wait_ratio: 0.023915, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9740.000000 2026-03-14 04:42:20,725 [Rank train_rank0] [Epoch 0 | Step 9760] train/loss: 0.448324, train/mse: 0.448324, train/data_wait_sec: 0.144817, train/step_wall_sec: 3.955472, train/data_wait_ratio: 0.036612, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9760.000000 2026-03-14 04:43:40,369 [Rank train_rank0] [Epoch 0 | Step 9780] train/loss: 0.461724, train/mse: 0.461724, train/data_wait_sec: 0.145337, train/step_wall_sec: 3.938260, train/data_wait_ratio: 0.036904, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9780.000000 2026-03-14 04:44:59,675 [Rank train_rank0] [Epoch 0 | Step 9800] train/loss: 0.490446, train/mse: 0.490446, train/data_wait_sec: 0.232159, train/step_wall_sec: 4.068358, train/data_wait_ratio: 0.057064, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9800.000000 2026-03-14 04:46:19,170 [Rank train_rank0] [Epoch 0 | Step 9820] train/loss: 0.455038, train/mse: 0.455038, train/data_wait_sec: 0.143080, train/step_wall_sec: 3.974727, train/data_wait_ratio: 0.035997, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9820.000000 2026-03-14 04:47:38,828 [Rank train_rank0] [Epoch 0 | Step 9840] train/loss: 0.483610, train/mse: 0.483610, train/data_wait_sec: 0.192050, train/step_wall_sec: 3.929452, train/data_wait_ratio: 0.048874, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9840.000000 2026-03-14 04:48:58,609 [Rank train_rank0] [Epoch 0 | Step 9860] train/loss: 0.430267, train/mse: 0.430267, train/data_wait_sec: 0.109301, train/step_wall_sec: 3.932129, train/data_wait_ratio: 0.027797, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9860.000000 2026-03-14 04:50:18,335 [Rank train_rank0] [Epoch 0 | Step 9880] train/loss: 0.459069, train/mse: 0.459069, train/data_wait_sec: 0.128071, train/step_wall_sec: 3.891285, train/data_wait_ratio: 0.032912, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9880.000000 2026-03-14 04:51:37,475 [Rank train_rank0] [Epoch 0 | Step 9900] train/loss: 0.441744, train/mse: 0.441744, train/data_wait_sec: 0.119599, train/step_wall_sec: 3.916325, train/data_wait_ratio: 0.030539, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9900.000000 2026-03-14 04:52:56,977 [Rank train_rank0] [Epoch 0 | Step 9920] train/loss: 0.437470, train/mse: 0.437470, train/data_wait_sec: 0.143326, train/step_wall_sec: 3.906829, train/data_wait_ratio: 0.036686, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9920.000000 2026-03-14 04:54:16,446 [Rank train_rank0] [Epoch 0 | Step 9940] train/loss: 0.471442, train/mse: 0.471442, train/data_wait_sec: 0.150888, train/step_wall_sec: 3.928538, train/data_wait_ratio: 0.038408, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9940.000000 2026-03-14 04:55:35,919 [Rank train_rank0] [Epoch 0 | Step 9960] train/loss: 0.464004, train/mse: 0.464004, train/data_wait_sec: 0.075000, train/step_wall_sec: 3.889163, train/data_wait_ratio: 0.019284, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9960.000000 2026-03-14 04:56:55,366 [Rank train_rank0] [Epoch 0 | Step 9980] train/loss: 0.442414, train/mse: 0.442414, train/data_wait_sec: 0.258928, train/step_wall_sec: 4.072652, train/data_wait_ratio: 0.063577, train/lr: 0.000200, train/epoch: 0.000000, train/step: 9980.000000 2026-03-14 04:58:39,527 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000010000.pt 2026-03-14 04:58:39,532 [Rank train_rank0] [Epoch 0 | Step 10000] train/loss: 0.452164, train/mse: 0.452164, train/data_wait_sec: 0.054944, train/step_wall_sec: 3.857065, train/data_wait_ratio: 0.014245, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10000.000000 2026-03-14 04:58:39,534 [Rank train_rank0] Generating samples for visualization... 2026-03-14 04:58:45,432 [Rank train_rank0] Sample generation done. 2026-03-14 05:00:04,677 [Rank train_rank0] [Epoch 0 | Step 10020] train/loss: 0.454966, train/mse: 0.454966, train/data_wait_sec: 0.096731, train/step_wall_sec: 4.068557, train/data_wait_ratio: 0.023775, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10020.000000 2026-03-14 05:01:23,846 [Rank train_rank0] [Epoch 0 | Step 10040] train/loss: 0.449355, train/mse: 0.449355, train/data_wait_sec: 0.144255, train/step_wall_sec: 3.886384, train/data_wait_ratio: 0.037118, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10040.000000 2026-03-14 05:02:43,187 [Rank train_rank0] [Epoch 0 | Step 10060] train/loss: 0.450796, train/mse: 0.450796, train/data_wait_sec: 0.118679, train/step_wall_sec: 3.918657, train/data_wait_ratio: 0.030286, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10060.000000 2026-03-14 05:04:02,638 [Rank train_rank0] [Epoch 0 | Step 10080] train/loss: 0.467331, train/mse: 0.467331, train/data_wait_sec: 0.242090, train/step_wall_sec: 4.029713, train/data_wait_ratio: 0.060076, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10080.000000 2026-03-14 05:05:22,257 [Rank train_rank0] [Epoch 0 | Step 10100] train/loss: 0.454785, train/mse: 0.454785, train/data_wait_sec: 0.124072, train/step_wall_sec: 3.943261, train/data_wait_ratio: 0.031464, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10100.000000 2026-03-14 05:06:41,793 [Rank train_rank0] [Epoch 0 | Step 10120] train/loss: 0.456677, train/mse: 0.456677, train/data_wait_sec: 0.164190, train/step_wall_sec: 3.959938, train/data_wait_ratio: 0.041463, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10120.000000 2026-03-14 05:08:01,229 [Rank train_rank0] [Epoch 0 | Step 10140] train/loss: 0.445452, train/mse: 0.445452, train/data_wait_sec: 0.132849, train/step_wall_sec: 3.891703, train/data_wait_ratio: 0.034136, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10140.000000 2026-03-14 05:09:20,506 [Rank train_rank0] [Epoch 0 | Step 10160] train/loss: 0.453858, train/mse: 0.453858, train/data_wait_sec: 0.106164, train/step_wall_sec: 3.956059, train/data_wait_ratio: 0.026836, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10160.000000 2026-03-14 05:10:39,753 [Rank train_rank0] [Epoch 0 | Step 10180] train/loss: 0.450083, train/mse: 0.450083, train/data_wait_sec: 0.153750, train/step_wall_sec: 3.956974, train/data_wait_ratio: 0.038855, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10180.000000 2026-03-14 05:11:58,743 [Rank train_rank0] [Epoch 0 | Step 10200] train/loss: 0.451366, train/mse: 0.451366, train/data_wait_sec: 0.131101, train/step_wall_sec: 3.899242, train/data_wait_ratio: 0.033622, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10200.000000 2026-03-14 05:13:17,826 [Rank train_rank0] [Epoch 0 | Step 10220] train/loss: 0.433826, train/mse: 0.433826, train/data_wait_sec: 0.099826, train/step_wall_sec: 3.877400, train/data_wait_ratio: 0.025746, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10220.000000 2026-03-14 05:14:37,073 [Rank train_rank0] [Epoch 0 | Step 10240] train/loss: 0.440249, train/mse: 0.440249, train/data_wait_sec: 0.082406, train/step_wall_sec: 3.969191, train/data_wait_ratio: 0.020761, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10240.000000 2026-03-14 05:15:56,476 [Rank train_rank0] [Epoch 0 | Step 10260] train/loss: 0.482996, train/mse: 0.482996, train/data_wait_sec: 0.186271, train/step_wall_sec: 4.000764, train/data_wait_ratio: 0.046559, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10260.000000 2026-03-14 05:17:15,928 [Rank train_rank0] [Epoch 0 | Step 10280] train/loss: 0.487177, train/mse: 0.487177, train/data_wait_sec: 0.085817, train/step_wall_sec: 3.873371, train/data_wait_ratio: 0.022156, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10280.000000 2026-03-14 05:18:35,058 [Rank train_rank0] [Epoch 0 | Step 10300] train/loss: 0.470677, train/mse: 0.470677, train/data_wait_sec: 0.087094, train/step_wall_sec: 3.981887, train/data_wait_ratio: 0.021873, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10300.000000 2026-03-14 05:19:54,101 [Rank train_rank0] [Epoch 0 | Step 10320] train/loss: 0.475506, train/mse: 0.475506, train/data_wait_sec: 0.136820, train/step_wall_sec: 4.094862, train/data_wait_ratio: 0.033413, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10320.000000 2026-03-14 05:21:13,162 [Rank train_rank0] [Epoch 0 | Step 10340] train/loss: 0.446808, train/mse: 0.446808, train/data_wait_sec: 0.170282, train/step_wall_sec: 3.943179, train/data_wait_ratio: 0.043184, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10340.000000 2026-03-14 05:22:32,276 [Rank train_rank0] [Epoch 0 | Step 10360] train/loss: 0.455018, train/mse: 0.455018, train/data_wait_sec: 0.129218, train/step_wall_sec: 3.965192, train/data_wait_ratio: 0.032588, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10360.000000 2026-03-14 05:23:51,001 [Rank train_rank0] [Epoch 0 | Step 10380] train/loss: 0.471218, train/mse: 0.471218, train/data_wait_sec: 0.241397, train/step_wall_sec: 4.022597, train/data_wait_ratio: 0.060010, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10380.000000 2026-03-14 05:25:10,466 [Rank train_rank0] [Epoch 0 | Step 10400] train/loss: 0.468062, train/mse: 0.468062, train/data_wait_sec: 0.189821, train/step_wall_sec: 4.069446, train/data_wait_ratio: 0.046645, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10400.000000 2026-03-14 05:26:30,023 [Rank train_rank0] [Epoch 0 | Step 10420] train/loss: 0.469596, train/mse: 0.469596, train/data_wait_sec: 0.047068, train/step_wall_sec: 3.947759, train/data_wait_ratio: 0.011923, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10420.000000 2026-03-14 05:27:49,811 [Rank train_rank0] [Epoch 0 | Step 10440] train/loss: 0.469594, train/mse: 0.469594, train/data_wait_sec: 0.084893, train/step_wall_sec: 3.931591, train/data_wait_ratio: 0.021592, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10440.000000 2026-03-14 05:29:09,113 [Rank train_rank0] [Epoch 0 | Step 10460] train/loss: 0.471866, train/mse: 0.471866, train/data_wait_sec: 0.121222, train/step_wall_sec: 3.940188, train/data_wait_ratio: 0.030765, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10460.000000 2026-03-14 05:30:28,281 [Rank train_rank0] [Epoch 0 | Step 10480] train/loss: 0.460853, train/mse: 0.460853, train/data_wait_sec: 0.120115, train/step_wall_sec: 3.894411, train/data_wait_ratio: 0.030843, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10480.000000 2026-03-14 05:31:47,398 [Rank train_rank0] [Epoch 0 | Step 10500] train/loss: 0.439641, train/mse: 0.439641, train/data_wait_sec: 0.127909, train/step_wall_sec: 3.921242, train/data_wait_ratio: 0.032619, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10500.000000 2026-03-14 05:33:06,580 [Rank train_rank0] [Epoch 0 | Step 10520] train/loss: 0.484949, train/mse: 0.484949, train/data_wait_sec: 0.146304, train/step_wall_sec: 3.951842, train/data_wait_ratio: 0.037022, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10520.000000 2026-03-14 05:34:26,198 [Rank train_rank0] [Epoch 0 | Step 10540] train/loss: 0.507784, train/mse: 0.507784, train/data_wait_sec: 0.137967, train/step_wall_sec: 3.975290, train/data_wait_ratio: 0.034706, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10540.000000 2026-03-14 05:35:45,043 [Rank train_rank0] [Epoch 0 | Step 10560] train/loss: 0.446296, train/mse: 0.446296, train/data_wait_sec: 0.156254, train/step_wall_sec: 3.888661, train/data_wait_ratio: 0.040182, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10560.000000 2026-03-14 05:37:04,436 [Rank train_rank0] [Epoch 0 | Step 10580] train/loss: 0.448134, train/mse: 0.448134, train/data_wait_sec: 0.103797, train/step_wall_sec: 3.908556, train/data_wait_ratio: 0.026556, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10580.000000 2026-03-14 05:38:23,903 [Rank train_rank0] [Epoch 0 | Step 10600] train/loss: 0.454295, train/mse: 0.454295, train/data_wait_sec: 0.124774, train/step_wall_sec: 3.926400, train/data_wait_ratio: 0.031778, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10600.000000 2026-03-14 05:39:43,224 [Rank train_rank0] [Epoch 0 | Step 10620] train/loss: 0.443168, train/mse: 0.443168, train/data_wait_sec: 0.167356, train/step_wall_sec: 3.949094, train/data_wait_ratio: 0.042378, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10620.000000 2026-03-14 05:41:01,967 [Rank train_rank0] [Epoch 0 | Step 10640] train/loss: 0.484796, train/mse: 0.484796, train/data_wait_sec: 0.104918, train/step_wall_sec: 3.916830, train/data_wait_ratio: 0.026786, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10640.000000 2026-03-14 05:42:21,122 [Rank train_rank0] [Epoch 0 | Step 10660] train/loss: 0.466446, train/mse: 0.466446, train/data_wait_sec: 0.121004, train/step_wall_sec: 3.952992, train/data_wait_ratio: 0.030611, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10660.000000 2026-03-14 05:43:40,189 [Rank train_rank0] [Epoch 0 | Step 10680] train/loss: 0.450190, train/mse: 0.450190, train/data_wait_sec: 0.089212, train/step_wall_sec: 3.879843, train/data_wait_ratio: 0.022994, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10680.000000 2026-03-14 05:44:59,140 [Rank train_rank0] [Epoch 0 | Step 10700] train/loss: 0.465371, train/mse: 0.465371, train/data_wait_sec: 0.105501, train/step_wall_sec: 3.869923, train/data_wait_ratio: 0.027262, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10700.000000 2026-03-14 05:46:18,299 [Rank train_rank0] [Epoch 0 | Step 10720] train/loss: 0.486125, train/mse: 0.486125, train/data_wait_sec: 0.124959, train/step_wall_sec: 3.935575, train/data_wait_ratio: 0.031751, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10720.000000 2026-03-14 05:47:37,145 [Rank train_rank0] [Epoch 0 | Step 10740] train/loss: 0.476525, train/mse: 0.476525, train/data_wait_sec: 0.128405, train/step_wall_sec: 3.914922, train/data_wait_ratio: 0.032799, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10740.000000 2026-03-14 05:48:56,201 [Rank train_rank0] [Epoch 0 | Step 10760] train/loss: 0.438493, train/mse: 0.438493, train/data_wait_sec: 0.119315, train/step_wall_sec: 3.896154, train/data_wait_ratio: 0.030624, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10760.000000 2026-03-14 05:50:14,878 [Rank train_rank0] [Epoch 0 | Step 10780] train/loss: 0.468143, train/mse: 0.468143, train/data_wait_sec: 0.102581, train/step_wall_sec: 3.968326, train/data_wait_ratio: 0.025850, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10780.000000 2026-03-14 05:51:33,920 [Rank train_rank0] [Epoch 0 | Step 10800] train/loss: 0.462135, train/mse: 0.462135, train/data_wait_sec: 0.129299, train/step_wall_sec: 3.931800, train/data_wait_ratio: 0.032885, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10800.000000 2026-03-14 05:52:52,717 [Rank train_rank0] [Epoch 0 | Step 10820] train/loss: 0.433616, train/mse: 0.433616, train/data_wait_sec: 0.100408, train/step_wall_sec: 3.892536, train/data_wait_ratio: 0.025795, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10820.000000 2026-03-14 05:54:11,752 [Rank train_rank0] [Epoch 0 | Step 10840] train/loss: 0.451882, train/mse: 0.451882, train/data_wait_sec: 0.118435, train/step_wall_sec: 3.865906, train/data_wait_ratio: 0.030636, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10840.000000 2026-03-14 05:55:30,609 [Rank train_rank0] [Epoch 0 | Step 10860] train/loss: 0.456374, train/mse: 0.456374, train/data_wait_sec: 0.189296, train/step_wall_sec: 3.960694, train/data_wait_ratio: 0.047794, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10860.000000 2026-03-14 05:56:49,782 [Rank train_rank0] [Epoch 0 | Step 10880] train/loss: 0.456486, train/mse: 0.456486, train/data_wait_sec: 0.118367, train/step_wall_sec: 3.902002, train/data_wait_ratio: 0.030335, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10880.000000 2026-03-14 05:58:08,499 [Rank train_rank0] [Epoch 0 | Step 10900] train/loss: 0.501967, train/mse: 0.501967, train/data_wait_sec: 0.095690, train/step_wall_sec: 3.902155, train/data_wait_ratio: 0.024522, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10900.000000 2026-03-14 05:59:27,376 [Rank train_rank0] [Epoch 0 | Step 10920] train/loss: 0.474798, train/mse: 0.474798, train/data_wait_sec: 0.121301, train/step_wall_sec: 3.884841, train/data_wait_ratio: 0.031224, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10920.000000 2026-03-14 06:00:46,229 [Rank train_rank0] [Epoch 0 | Step 10940] train/loss: 0.480880, train/mse: 0.480880, train/data_wait_sec: 0.103285, train/step_wall_sec: 3.903776, train/data_wait_ratio: 0.026458, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10940.000000 2026-03-14 06:02:05,143 [Rank train_rank0] [Epoch 0 | Step 10960] train/loss: 0.440428, train/mse: 0.440428, train/data_wait_sec: 0.128258, train/step_wall_sec: 3.909555, train/data_wait_ratio: 0.032806, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10960.000000 2026-03-14 06:03:24,310 [Rank train_rank0] [Epoch 0 | Step 10980] train/loss: 0.498826, train/mse: 0.498826, train/data_wait_sec: 0.122714, train/step_wall_sec: 4.076237, train/data_wait_ratio: 0.030105, train/lr: 0.000200, train/epoch: 0.000000, train/step: 10980.000000 2026-03-14 06:05:08,537 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000011000.pt 2026-03-14 06:05:08,540 [Rank train_rank0] [Epoch 0 | Step 11000] train/loss: 0.477892, train/mse: 0.477892, train/data_wait_sec: 0.105508, train/step_wall_sec: 3.888462, train/data_wait_ratio: 0.027134, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11000.000000 2026-03-14 06:05:08,541 [Rank train_rank0] Generating samples for visualization... 2026-03-14 06:05:11,719 [Rank train_rank0] Sample generation done. 2026-03-14 06:06:30,748 [Rank train_rank0] [Epoch 0 | Step 11020] train/loss: 0.460690, train/mse: 0.460690, train/data_wait_sec: 0.139364, train/step_wall_sec: 3.911823, train/data_wait_ratio: 0.035626, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11020.000000 2026-03-14 06:07:49,618 [Rank train_rank0] [Epoch 0 | Step 11040] train/loss: 0.439447, train/mse: 0.439447, train/data_wait_sec: 0.155028, train/step_wall_sec: 3.917434, train/data_wait_ratio: 0.039574, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11040.000000 2026-03-14 06:09:08,792 [Rank train_rank0] [Epoch 0 | Step 11060] train/loss: 0.443503, train/mse: 0.443503, train/data_wait_sec: 0.129572, train/step_wall_sec: 3.902391, train/data_wait_ratio: 0.033203, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11060.000000 2026-03-14 06:10:28,026 [Rank train_rank0] [Epoch 0 | Step 11080] train/loss: 0.477028, train/mse: 0.477028, train/data_wait_sec: 0.133862, train/step_wall_sec: 3.930445, train/data_wait_ratio: 0.034058, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11080.000000 2026-03-14 06:11:47,403 [Rank train_rank0] [Epoch 0 | Step 11100] train/loss: 0.463597, train/mse: 0.463597, train/data_wait_sec: 0.081247, train/step_wall_sec: 3.847201, train/data_wait_ratio: 0.021119, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11100.000000 2026-03-14 06:13:06,583 [Rank train_rank0] [Epoch 0 | Step 11120] train/loss: 0.460555, train/mse: 0.460555, train/data_wait_sec: 0.099901, train/step_wall_sec: 3.893852, train/data_wait_ratio: 0.025656, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11120.000000 2026-03-14 06:14:25,520 [Rank train_rank0] [Epoch 0 | Step 11140] train/loss: 0.476440, train/mse: 0.476440, train/data_wait_sec: 0.144297, train/step_wall_sec: 3.899819, train/data_wait_ratio: 0.037001, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11140.000000 2026-03-14 06:15:44,482 [Rank train_rank0] [Epoch 0 | Step 11160] train/loss: 0.466943, train/mse: 0.466943, train/data_wait_sec: 0.120852, train/step_wall_sec: 3.906067, train/data_wait_ratio: 0.030940, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11160.000000 2026-03-14 06:17:03,905 [Rank train_rank0] [Epoch 0 | Step 11180] train/loss: 0.443721, train/mse: 0.443721, train/data_wait_sec: 0.096420, train/step_wall_sec: 3.909478, train/data_wait_ratio: 0.024663, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11180.000000 2026-03-14 06:18:22,860 [Rank train_rank0] [Epoch 0 | Step 11200] train/loss: 0.443690, train/mse: 0.443690, train/data_wait_sec: 0.145886, train/step_wall_sec: 3.967049, train/data_wait_ratio: 0.036774, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11200.000000 2026-03-14 06:19:42,021 [Rank train_rank0] [Epoch 0 | Step 11220] train/loss: 0.473506, train/mse: 0.473506, train/data_wait_sec: 0.135487, train/step_wall_sec: 3.890675, train/data_wait_ratio: 0.034823, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11220.000000 2026-03-14 06:21:00,988 [Rank train_rank0] [Epoch 0 | Step 11240] train/loss: 0.457013, train/mse: 0.457013, train/data_wait_sec: 0.118887, train/step_wall_sec: 3.925640, train/data_wait_ratio: 0.030285, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11240.000000 2026-03-14 06:22:20,126 [Rank train_rank0] [Epoch 0 | Step 11260] train/loss: 0.456357, train/mse: 0.456357, train/data_wait_sec: 0.099298, train/step_wall_sec: 3.877613, train/data_wait_ratio: 0.025608, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11260.000000 2026-03-14 06:23:39,016 [Rank train_rank0] [Epoch 0 | Step 11280] train/loss: 0.471688, train/mse: 0.471688, train/data_wait_sec: 0.112236, train/step_wall_sec: 3.919989, train/data_wait_ratio: 0.028632, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11280.000000 2026-03-14 06:24:57,905 [Rank train_rank0] [Epoch 0 | Step 11300] train/loss: 0.439340, train/mse: 0.439340, train/data_wait_sec: 0.108746, train/step_wall_sec: 3.912509, train/data_wait_ratio: 0.027795, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11300.000000 2026-03-14 06:26:16,855 [Rank train_rank0] [Epoch 0 | Step 11320] train/loss: 0.467617, train/mse: 0.467617, train/data_wait_sec: 0.066281, train/step_wall_sec: 3.927342, train/data_wait_ratio: 0.016877, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11320.000000 2026-03-14 06:27:35,957 [Rank train_rank0] [Epoch 0 | Step 11340] train/loss: 0.439640, train/mse: 0.439640, train/data_wait_sec: 0.086902, train/step_wall_sec: 3.882962, train/data_wait_ratio: 0.022380, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11340.000000 2026-03-14 06:28:55,141 [Rank train_rank0] [Epoch 0 | Step 11360] train/loss: 0.490242, train/mse: 0.490242, train/data_wait_sec: 0.094839, train/step_wall_sec: 3.886571, train/data_wait_ratio: 0.024402, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11360.000000 2026-03-14 06:30:14,538 [Rank train_rank0] [Epoch 0 | Step 11380] train/loss: 0.478105, train/mse: 0.478105, train/data_wait_sec: 0.126285, train/step_wall_sec: 3.978597, train/data_wait_ratio: 0.031741, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11380.000000 2026-03-14 06:31:33,513 [Rank train_rank0] [Epoch 0 | Step 11400] train/loss: 0.463964, train/mse: 0.463964, train/data_wait_sec: 0.134496, train/step_wall_sec: 3.930527, train/data_wait_ratio: 0.034218, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11400.000000 2026-03-14 06:32:52,729 [Rank train_rank0] [Epoch 0 | Step 11420] train/loss: 0.457510, train/mse: 0.457510, train/data_wait_sec: 0.183361, train/step_wall_sec: 4.008471, train/data_wait_ratio: 0.045743, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11420.000000 2026-03-14 06:34:11,930 [Rank train_rank0] [Epoch 0 | Step 11440] train/loss: 0.440851, train/mse: 0.440851, train/data_wait_sec: 0.132013, train/step_wall_sec: 3.913730, train/data_wait_ratio: 0.033731, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11440.000000 2026-03-14 06:35:31,239 [Rank train_rank0] [Epoch 0 | Step 11460] train/loss: 0.453246, train/mse: 0.453246, train/data_wait_sec: 0.135613, train/step_wall_sec: 3.893897, train/data_wait_ratio: 0.034827, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11460.000000 2026-03-14 06:36:50,172 [Rank train_rank0] [Epoch 0 | Step 11480] train/loss: 0.474720, train/mse: 0.474720, train/data_wait_sec: 0.104332, train/step_wall_sec: 4.073345, train/data_wait_ratio: 0.025613, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11480.000000 2026-03-14 06:38:09,905 [Rank train_rank0] [Epoch 0 | Step 11500] train/loss: 0.484936, train/mse: 0.484936, train/data_wait_sec: 0.091960, train/step_wall_sec: 3.902614, train/data_wait_ratio: 0.023564, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11500.000000 2026-03-14 06:39:29,074 [Rank train_rank0] [Epoch 0 | Step 11520] train/loss: 0.469182, train/mse: 0.469182, train/data_wait_sec: 0.114179, train/step_wall_sec: 3.894580, train/data_wait_ratio: 0.029317, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11520.000000 2026-03-14 06:40:47,901 [Rank train_rank0] [Epoch 0 | Step 11540] train/loss: 0.454599, train/mse: 0.454599, train/data_wait_sec: 0.125843, train/step_wall_sec: 3.932360, train/data_wait_ratio: 0.032002, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11540.000000 2026-03-14 06:42:07,148 [Rank train_rank0] [Epoch 0 | Step 11560] train/loss: 0.468676, train/mse: 0.468676, train/data_wait_sec: 0.194042, train/step_wall_sec: 3.913621, train/data_wait_ratio: 0.049581, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11560.000000 2026-03-14 06:43:26,076 [Rank train_rank0] [Epoch 0 | Step 11580] train/loss: 0.497133, train/mse: 0.497133, train/data_wait_sec: 0.108620, train/step_wall_sec: 3.895488, train/data_wait_ratio: 0.027883, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11580.000000 2026-03-14 06:44:45,361 [Rank train_rank0] [Epoch 0 | Step 11600] train/loss: 0.462127, train/mse: 0.462127, train/data_wait_sec: 0.134656, train/step_wall_sec: 3.909987, train/data_wait_ratio: 0.034439, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11600.000000 2026-03-14 06:46:04,339 [Rank train_rank0] [Epoch 0 | Step 11620] train/loss: 0.442816, train/mse: 0.442816, train/data_wait_sec: 0.203461, train/step_wall_sec: 3.974611, train/data_wait_ratio: 0.051190, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11620.000000 2026-03-14 06:47:23,499 [Rank train_rank0] [Epoch 0 | Step 11640] train/loss: 0.446102, train/mse: 0.446102, train/data_wait_sec: 0.231986, train/step_wall_sec: 4.085231, train/data_wait_ratio: 0.056786, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11640.000000 2026-03-14 06:48:42,628 [Rank train_rank0] [Epoch 0 | Step 11660] train/loss: 0.479280, train/mse: 0.479280, train/data_wait_sec: 0.126003, train/step_wall_sec: 3.886377, train/data_wait_ratio: 0.032422, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11660.000000 2026-03-14 06:50:01,819 [Rank train_rank0] [Epoch 0 | Step 11680] train/loss: 0.477360, train/mse: 0.477360, train/data_wait_sec: 0.166511, train/step_wall_sec: 3.945430, train/data_wait_ratio: 0.042203, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11680.000000 2026-03-14 06:51:20,993 [Rank train_rank0] [Epoch 0 | Step 11700] train/loss: 0.462522, train/mse: 0.462522, train/data_wait_sec: 0.107587, train/step_wall_sec: 3.850645, train/data_wait_ratio: 0.027940, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11700.000000 2026-03-14 06:52:40,083 [Rank train_rank0] [Epoch 0 | Step 11720] train/loss: 0.468568, train/mse: 0.468568, train/data_wait_sec: 0.156131, train/step_wall_sec: 3.909105, train/data_wait_ratio: 0.039940, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11720.000000 2026-03-14 06:53:59,372 [Rank train_rank0] [Epoch 0 | Step 11740] train/loss: 0.459150, train/mse: 0.459150, train/data_wait_sec: 0.179306, train/step_wall_sec: 3.954933, train/data_wait_ratio: 0.045337, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11740.000000 2026-03-14 06:55:18,690 [Rank train_rank0] [Epoch 0 | Step 11760] train/loss: 0.446969, train/mse: 0.446969, train/data_wait_sec: 0.114334, train/step_wall_sec: 3.903960, train/data_wait_ratio: 0.029287, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11760.000000 2026-03-14 06:56:37,729 [Rank train_rank0] [Epoch 0 | Step 11780] train/loss: 0.474600, train/mse: 0.474600, train/data_wait_sec: 0.108141, train/step_wall_sec: 3.928369, train/data_wait_ratio: 0.027528, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11780.000000 2026-03-14 06:57:56,686 [Rank train_rank0] [Epoch 0 | Step 11800] train/loss: 0.457292, train/mse: 0.457292, train/data_wait_sec: 0.126747, train/step_wall_sec: 3.869054, train/data_wait_ratio: 0.032759, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11800.000000 2026-03-14 06:59:15,888 [Rank train_rank0] [Epoch 0 | Step 11820] train/loss: 0.484138, train/mse: 0.484138, train/data_wait_sec: 0.117938, train/step_wall_sec: 3.919240, train/data_wait_ratio: 0.030092, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11820.000000 2026-03-14 07:00:34,724 [Rank train_rank0] [Epoch 0 | Step 11840] train/loss: 0.455185, train/mse: 0.455185, train/data_wait_sec: 0.095400, train/step_wall_sec: 3.843770, train/data_wait_ratio: 0.024819, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11840.000000 2026-03-14 07:01:53,783 [Rank train_rank0] [Epoch 0 | Step 11860] train/loss: 0.476535, train/mse: 0.476535, train/data_wait_sec: 0.070815, train/step_wall_sec: 3.891947, train/data_wait_ratio: 0.018195, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11860.000000 2026-03-14 07:03:12,851 [Rank train_rank0] [Epoch 0 | Step 11880] train/loss: 0.454352, train/mse: 0.454352, train/data_wait_sec: 0.090485, train/step_wall_sec: 3.921623, train/data_wait_ratio: 0.023073, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11880.000000 2026-03-14 07:04:32,003 [Rank train_rank0] [Epoch 0 | Step 11900] train/loss: 0.462867, train/mse: 0.462867, train/data_wait_sec: 0.115092, train/step_wall_sec: 3.900100, train/data_wait_ratio: 0.029510, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11900.000000 2026-03-14 07:05:51,645 [Rank train_rank0] [Epoch 0 | Step 11920] train/loss: 0.448613, train/mse: 0.448613, train/data_wait_sec: 0.103769, train/step_wall_sec: 3.903175, train/data_wait_ratio: 0.026586, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11920.000000 2026-03-14 07:07:10,575 [Rank train_rank0] [Epoch 0 | Step 11940] train/loss: 0.453448, train/mse: 0.453448, train/data_wait_sec: 0.104715, train/step_wall_sec: 3.894255, train/data_wait_ratio: 0.026890, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11940.000000 2026-03-14 07:08:29,897 [Rank train_rank0] [Epoch 0 | Step 11960] train/loss: 0.475938, train/mse: 0.475938, train/data_wait_sec: 0.149355, train/step_wall_sec: 3.995448, train/data_wait_ratio: 0.037381, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11960.000000 2026-03-14 07:09:49,012 [Rank train_rank0] [Epoch 0 | Step 11980] train/loss: 0.462888, train/mse: 0.462888, train/data_wait_sec: 0.112683, train/step_wall_sec: 3.905819, train/data_wait_ratio: 0.028850, train/lr: 0.000200, train/epoch: 0.000000, train/step: 11980.000000 2026-03-14 07:11:33,419 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000012000.pt 2026-03-14 07:11:33,422 [Rank train_rank0] [Epoch 0 | Step 12000] train/loss: 0.483200, train/mse: 0.483200, train/data_wait_sec: 0.092394, train/step_wall_sec: 3.928225, train/data_wait_ratio: 0.023521, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12000.000000 2026-03-14 07:11:33,423 [Rank train_rank0] Generating samples for visualization... 2026-03-14 07:11:36,629 [Rank train_rank0] Sample generation done. 2026-03-14 07:12:56,054 [Rank train_rank0] [Epoch 0 | Step 12020] train/loss: 0.498823, train/mse: 0.498823, train/data_wait_sec: 0.164136, train/step_wall_sec: 3.944954, train/data_wait_ratio: 0.041606, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12020.000000 2026-03-14 07:14:15,131 [Rank train_rank0] [Epoch 0 | Step 12040] train/loss: 0.465317, train/mse: 0.465317, train/data_wait_sec: 0.095867, train/step_wall_sec: 3.971391, train/data_wait_ratio: 0.024140, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12040.000000 2026-03-14 07:15:34,438 [Rank train_rank0] [Epoch 0 | Step 12060] train/loss: 0.460599, train/mse: 0.460599, train/data_wait_sec: 0.128849, train/step_wall_sec: 3.911167, train/data_wait_ratio: 0.032944, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12060.000000 2026-03-14 07:16:53,554 [Rank train_rank0] [Epoch 0 | Step 12080] train/loss: 0.472502, train/mse: 0.472502, train/data_wait_sec: 0.140199, train/step_wall_sec: 3.927883, train/data_wait_ratio: 0.035693, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12080.000000 2026-03-14 07:18:12,602 [Rank train_rank0] [Epoch 0 | Step 12100] train/loss: 0.477065, train/mse: 0.477065, train/data_wait_sec: 0.109200, train/step_wall_sec: 3.876956, train/data_wait_ratio: 0.028167, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12100.000000 2026-03-14 07:19:31,850 [Rank train_rank0] [Epoch 0 | Step 12120] train/loss: 0.451212, train/mse: 0.451212, train/data_wait_sec: 0.200010, train/step_wall_sec: 3.974355, train/data_wait_ratio: 0.050325, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12120.000000 2026-03-14 07:20:51,089 [Rank train_rank0] [Epoch 0 | Step 12140] train/loss: 0.444522, train/mse: 0.444522, train/data_wait_sec: 0.056783, train/step_wall_sec: 3.971489, train/data_wait_ratio: 0.014298, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12140.000000 2026-03-14 07:22:10,331 [Rank train_rank0] [Epoch 0 | Step 12160] train/loss: 0.455014, train/mse: 0.455014, train/data_wait_sec: 0.091720, train/step_wall_sec: 3.906621, train/data_wait_ratio: 0.023478, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12160.000000 2026-03-14 07:23:29,723 [Rank train_rank0] [Epoch 0 | Step 12180] train/loss: 0.472382, train/mse: 0.472382, train/data_wait_sec: 0.143624, train/step_wall_sec: 3.905358, train/data_wait_ratio: 0.036776, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12180.000000 2026-03-14 07:24:48,767 [Rank train_rank0] [Epoch 0 | Step 12200] train/loss: 0.484599, train/mse: 0.484599, train/data_wait_sec: 0.125345, train/step_wall_sec: 3.883091, train/data_wait_ratio: 0.032280, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12200.000000 2026-03-14 07:26:07,643 [Rank train_rank0] [Epoch 0 | Step 12220] train/loss: 0.464965, train/mse: 0.464965, train/data_wait_sec: 0.110532, train/step_wall_sec: 3.875233, train/data_wait_ratio: 0.028523, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12220.000000 2026-03-14 07:27:26,694 [Rank train_rank0] [Epoch 0 | Step 12240] train/loss: 0.485913, train/mse: 0.485913, train/data_wait_sec: 0.114923, train/step_wall_sec: 3.817146, train/data_wait_ratio: 0.030107, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12240.000000 2026-03-14 07:28:45,892 [Rank train_rank0] [Epoch 0 | Step 12260] train/loss: 0.453714, train/mse: 0.453714, train/data_wait_sec: 0.095056, train/step_wall_sec: 3.888077, train/data_wait_ratio: 0.024448, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12260.000000 2026-03-14 07:30:04,823 [Rank train_rank0] [Epoch 0 | Step 12280] train/loss: 0.491826, train/mse: 0.491826, train/data_wait_sec: 0.106376, train/step_wall_sec: 3.877181, train/data_wait_ratio: 0.027436, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12280.000000 2026-03-14 07:31:24,264 [Rank train_rank0] [Epoch 0 | Step 12300] train/loss: 0.462271, train/mse: 0.462271, train/data_wait_sec: 0.092808, train/step_wall_sec: 3.884027, train/data_wait_ratio: 0.023895, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12300.000000 2026-03-14 07:32:43,239 [Rank train_rank0] [Epoch 0 | Step 12320] train/loss: 0.459133, train/mse: 0.459133, train/data_wait_sec: 0.121739, train/step_wall_sec: 3.907537, train/data_wait_ratio: 0.031155, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12320.000000 2026-03-14 07:34:03,188 [Rank train_rank0] [Epoch 0 | Step 12340] train/loss: 0.449332, train/mse: 0.449332, train/data_wait_sec: 0.092663, train/step_wall_sec: 3.900973, train/data_wait_ratio: 0.023754, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12340.000000 2026-03-14 07:35:22,479 [Rank train_rank0] [Epoch 0 | Step 12360] train/loss: 0.452989, train/mse: 0.452989, train/data_wait_sec: 0.125210, train/step_wall_sec: 3.993589, train/data_wait_ratio: 0.031353, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12360.000000 2026-03-14 07:36:41,564 [Rank train_rank0] [Epoch 0 | Step 12380] train/loss: 0.444590, train/mse: 0.444590, train/data_wait_sec: 0.074463, train/step_wall_sec: 3.903840, train/data_wait_ratio: 0.019074, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12380.000000 2026-03-14 07:38:00,592 [Rank train_rank0] [Epoch 0 | Step 12400] train/loss: 0.461146, train/mse: 0.461146, train/data_wait_sec: 0.102005, train/step_wall_sec: 3.912724, train/data_wait_ratio: 0.026070, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12400.000000 2026-03-14 07:39:19,713 [Rank train_rank0] [Epoch 0 | Step 12420] train/loss: 0.456338, train/mse: 0.456338, train/data_wait_sec: 0.175767, train/step_wall_sec: 4.022934, train/data_wait_ratio: 0.043691, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12420.000000 2026-03-14 07:40:39,071 [Rank train_rank0] [Epoch 0 | Step 12440] train/loss: 0.477617, train/mse: 0.477617, train/data_wait_sec: 0.088755, train/step_wall_sec: 3.958093, train/data_wait_ratio: 0.022424, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12440.000000 2026-03-14 07:41:58,637 [Rank train_rank0] [Epoch 0 | Step 12460] train/loss: 0.465031, train/mse: 0.465031, train/data_wait_sec: 0.096750, train/step_wall_sec: 4.121744, train/data_wait_ratio: 0.023473, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12460.000000 2026-03-14 07:43:17,362 [Rank train_rank0] [Epoch 0 | Step 12480] train/loss: 0.463309, train/mse: 0.463309, train/data_wait_sec: 0.137194, train/step_wall_sec: 3.911419, train/data_wait_ratio: 0.035075, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12480.000000 2026-03-14 07:44:36,904 [Rank train_rank0] [Epoch 0 | Step 12500] train/loss: 0.435453, train/mse: 0.435453, train/data_wait_sec: 0.105013, train/step_wall_sec: 3.882329, train/data_wait_ratio: 0.027049, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12500.000000 2026-03-14 07:45:56,349 [Rank train_rank0] [Epoch 0 | Step 12520] train/loss: 0.450044, train/mse: 0.450044, train/data_wait_sec: 0.124794, train/step_wall_sec: 3.935571, train/data_wait_ratio: 0.031709, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12520.000000 2026-03-14 07:47:15,546 [Rank train_rank0] [Epoch 0 | Step 12540] train/loss: 0.460729, train/mse: 0.460729, train/data_wait_sec: 0.193725, train/step_wall_sec: 3.997026, train/data_wait_ratio: 0.048467, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12540.000000 2026-03-14 07:48:34,667 [Rank train_rank0] [Epoch 0 | Step 12560] train/loss: 0.442586, train/mse: 0.442586, train/data_wait_sec: 0.067120, train/step_wall_sec: 3.886568, train/data_wait_ratio: 0.017270, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12560.000000 2026-03-14 07:49:53,985 [Rank train_rank0] [Epoch 0 | Step 12580] train/loss: 0.484658, train/mse: 0.484658, train/data_wait_sec: 0.105879, train/step_wall_sec: 3.901053, train/data_wait_ratio: 0.027141, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12580.000000 2026-03-14 07:51:12,655 [Rank train_rank0] [Epoch 0 | Step 12600] train/loss: 0.484092, train/mse: 0.484092, train/data_wait_sec: 0.058693, train/step_wall_sec: 3.927941, train/data_wait_ratio: 0.014942, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12600.000000 2026-03-14 07:52:31,793 [Rank train_rank0] [Epoch 0 | Step 12620] train/loss: 0.454254, train/mse: 0.454254, train/data_wait_sec: 0.100556, train/step_wall_sec: 3.925756, train/data_wait_ratio: 0.025614, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12620.000000 2026-03-14 07:53:50,855 [Rank train_rank0] [Epoch 0 | Step 12640] train/loss: 0.438569, train/mse: 0.438569, train/data_wait_sec: 0.185577, train/step_wall_sec: 3.987450, train/data_wait_ratio: 0.046540, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12640.000000 2026-03-14 07:55:09,552 [Rank train_rank0] [Epoch 0 | Step 12660] train/loss: 0.454535, train/mse: 0.454535, train/data_wait_sec: 0.096116, train/step_wall_sec: 3.896283, train/data_wait_ratio: 0.024669, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12660.000000 2026-03-14 07:56:28,784 [Rank train_rank0] [Epoch 0 | Step 12680] train/loss: 0.442591, train/mse: 0.442591, train/data_wait_sec: 0.157186, train/step_wall_sec: 3.930153, train/data_wait_ratio: 0.039995, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12680.000000 2026-03-14 07:57:48,001 [Rank train_rank0] [Epoch 0 | Step 12700] train/loss: 0.464590, train/mse: 0.464590, train/data_wait_sec: 0.097688, train/step_wall_sec: 3.857095, train/data_wait_ratio: 0.025327, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12700.000000 2026-03-14 07:59:07,457 [Rank train_rank0] [Epoch 0 | Step 12720] train/loss: 0.499574, train/mse: 0.499574, train/data_wait_sec: 0.093487, train/step_wall_sec: 3.919208, train/data_wait_ratio: 0.023854, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12720.000000 2026-03-14 08:00:26,680 [Rank train_rank0] [Epoch 0 | Step 12740] train/loss: 0.448581, train/mse: 0.448581, train/data_wait_sec: 0.105585, train/step_wall_sec: 3.912135, train/data_wait_ratio: 0.026989, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12740.000000 2026-03-14 08:01:45,855 [Rank train_rank0] [Epoch 0 | Step 12760] train/loss: 0.432956, train/mse: 0.432956, train/data_wait_sec: 0.139023, train/step_wall_sec: 3.922843, train/data_wait_ratio: 0.035439, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12760.000000 2026-03-14 08:03:04,757 [Rank train_rank0] [Epoch 0 | Step 12780] train/loss: 0.478683, train/mse: 0.478683, train/data_wait_sec: 0.119561, train/step_wall_sec: 3.901025, train/data_wait_ratio: 0.030649, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12780.000000 2026-03-14 08:04:23,970 [Rank train_rank0] [Epoch 0 | Step 12800] train/loss: 0.440873, train/mse: 0.440873, train/data_wait_sec: 0.085962, train/step_wall_sec: 3.859740, train/data_wait_ratio: 0.022271, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12800.000000 2026-03-14 08:05:42,955 [Rank train_rank0] [Epoch 0 | Step 12820] train/loss: 0.465819, train/mse: 0.465819, train/data_wait_sec: 0.075187, train/step_wall_sec: 3.932852, train/data_wait_ratio: 0.019118, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12820.000000 2026-03-14 08:07:02,596 [Rank train_rank0] [Epoch 0 | Step 12840] train/loss: 0.464306, train/mse: 0.464306, train/data_wait_sec: 0.077391, train/step_wall_sec: 3.931172, train/data_wait_ratio: 0.019686, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12840.000000 2026-03-14 08:08:21,587 [Rank train_rank0] [Epoch 0 | Step 12860] train/loss: 0.455381, train/mse: 0.455381, train/data_wait_sec: 0.114032, train/step_wall_sec: 3.893921, train/data_wait_ratio: 0.029285, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12860.000000 2026-03-14 08:09:40,708 [Rank train_rank0] [Epoch 0 | Step 12880] train/loss: 0.465759, train/mse: 0.465759, train/data_wait_sec: 0.131199, train/step_wall_sec: 3.960450, train/data_wait_ratio: 0.033127, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12880.000000 2026-03-14 08:11:00,026 [Rank train_rank0] [Epoch 0 | Step 12900] train/loss: 0.469938, train/mse: 0.469938, train/data_wait_sec: 0.115913, train/step_wall_sec: 3.940739, train/data_wait_ratio: 0.029414, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12900.000000 2026-03-14 08:12:19,113 [Rank train_rank0] [Epoch 0 | Step 12920] train/loss: 0.458758, train/mse: 0.458758, train/data_wait_sec: 0.159450, train/step_wall_sec: 3.944073, train/data_wait_ratio: 0.040428, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12920.000000 2026-03-14 08:13:38,638 [Rank train_rank0] [Epoch 0 | Step 12940] train/loss: 0.454762, train/mse: 0.454762, train/data_wait_sec: 0.088538, train/step_wall_sec: 3.909786, train/data_wait_ratio: 0.022645, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12940.000000 2026-03-14 08:14:58,119 [Rank train_rank0] [Epoch 0 | Step 12960] train/loss: 0.481272, train/mse: 0.481272, train/data_wait_sec: 0.106478, train/step_wall_sec: 3.895041, train/data_wait_ratio: 0.027337, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12960.000000 2026-03-14 08:16:17,638 [Rank train_rank0] [Epoch 0 | Step 12980] train/loss: 0.449831, train/mse: 0.449831, train/data_wait_sec: 0.055020, train/step_wall_sec: 4.111414, train/data_wait_ratio: 0.013382, train/lr: 0.000200, train/epoch: 0.000000, train/step: 12980.000000 2026-03-14 08:18:01,595 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000013000.pt 2026-03-14 08:18:01,599 [Rank train_rank0] [Epoch 0 | Step 13000] train/loss: 0.472707, train/mse: 0.472707, train/data_wait_sec: 0.103139, train/step_wall_sec: 3.954133, train/data_wait_ratio: 0.026084, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13000.000000 2026-03-14 08:18:01,600 [Rank train_rank0] Generating samples for visualization... 2026-03-14 08:18:04,761 [Rank train_rank0] Sample generation done. 2026-03-14 08:19:24,358 [Rank train_rank0] [Epoch 0 | Step 13020] train/loss: 0.459856, train/mse: 0.459856, train/data_wait_sec: 0.090334, train/step_wall_sec: 3.934767, train/data_wait_ratio: 0.022958, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13020.000000 2026-03-14 08:20:43,940 [Rank train_rank0] [Epoch 0 | Step 13040] train/loss: 0.478788, train/mse: 0.478788, train/data_wait_sec: 0.110896, train/step_wall_sec: 3.953181, train/data_wait_ratio: 0.028052, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13040.000000 2026-03-14 08:22:03,426 [Rank train_rank0] [Epoch 0 | Step 13060] train/loss: 0.484403, train/mse: 0.484403, train/data_wait_sec: 0.126672, train/step_wall_sec: 3.938777, train/data_wait_ratio: 0.032160, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13060.000000 2026-03-14 08:23:22,816 [Rank train_rank0] [Epoch 0 | Step 13080] train/loss: 0.464816, train/mse: 0.464816, train/data_wait_sec: 0.105716, train/step_wall_sec: 3.894753, train/data_wait_ratio: 0.027143, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13080.000000 2026-03-14 08:24:42,172 [Rank train_rank0] [Epoch 0 | Step 13100] train/loss: 0.466278, train/mse: 0.466278, train/data_wait_sec: 0.116904, train/step_wall_sec: 3.860849, train/data_wait_ratio: 0.030279, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13100.000000 2026-03-14 08:26:01,132 [Rank train_rank0] [Epoch 0 | Step 13120] train/loss: 0.451844, train/mse: 0.451844, train/data_wait_sec: 0.151996, train/step_wall_sec: 3.990906, train/data_wait_ratio: 0.038086, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13120.000000 2026-03-14 08:27:20,812 [Rank train_rank0] [Epoch 0 | Step 13140] train/loss: 0.428900, train/mse: 0.428900, train/data_wait_sec: 0.131129, train/step_wall_sec: 3.895385, train/data_wait_ratio: 0.033663, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13140.000000 2026-03-14 08:28:40,318 [Rank train_rank0] [Epoch 0 | Step 13160] train/loss: 0.452504, train/mse: 0.452504, train/data_wait_sec: 0.095919, train/step_wall_sec: 3.940385, train/data_wait_ratio: 0.024343, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13160.000000 2026-03-14 08:29:59,712 [Rank train_rank0] [Epoch 0 | Step 13180] train/loss: 0.455793, train/mse: 0.455793, train/data_wait_sec: 0.196945, train/step_wall_sec: 3.977409, train/data_wait_ratio: 0.049516, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13180.000000 2026-03-14 08:31:19,464 [Rank train_rank0] [Epoch 0 | Step 13200] train/loss: 0.487251, train/mse: 0.487251, train/data_wait_sec: 0.095134, train/step_wall_sec: 3.897706, train/data_wait_ratio: 0.024408, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13200.000000 2026-03-14 08:32:38,927 [Rank train_rank0] [Epoch 0 | Step 13220] train/loss: 0.440166, train/mse: 0.440166, train/data_wait_sec: 0.099388, train/step_wall_sec: 3.918256, train/data_wait_ratio: 0.025365, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13220.000000 2026-03-14 08:33:57,960 [Rank train_rank0] [Epoch 0 | Step 13240] train/loss: 0.438689, train/mse: 0.438689, train/data_wait_sec: 0.109732, train/step_wall_sec: 3.913899, train/data_wait_ratio: 0.028036, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13240.000000 2026-03-14 08:35:17,324 [Rank train_rank0] [Epoch 0 | Step 13260] train/loss: 0.466377, train/mse: 0.466377, train/data_wait_sec: 0.061708, train/step_wall_sec: 3.882455, train/data_wait_ratio: 0.015894, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13260.000000 2026-03-14 08:36:36,859 [Rank train_rank0] [Epoch 0 | Step 13280] train/loss: 0.473908, train/mse: 0.473908, train/data_wait_sec: 0.121191, train/step_wall_sec: 3.958353, train/data_wait_ratio: 0.030617, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13280.000000 2026-03-14 08:37:56,072 [Rank train_rank0] [Epoch 0 | Step 13300] train/loss: 0.474701, train/mse: 0.474701, train/data_wait_sec: 0.118894, train/step_wall_sec: 3.927315, train/data_wait_ratio: 0.030274, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13300.000000 2026-03-14 08:39:15,234 [Rank train_rank0] [Epoch 0 | Step 13320] train/loss: 0.450768, train/mse: 0.450768, train/data_wait_sec: 0.119887, train/step_wall_sec: 3.915915, train/data_wait_ratio: 0.030615, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13320.000000 2026-03-14 08:40:34,701 [Rank train_rank0] [Epoch 0 | Step 13340] train/loss: 0.449894, train/mse: 0.449894, train/data_wait_sec: 0.058647, train/step_wall_sec: 3.913201, train/data_wait_ratio: 0.014987, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13340.000000 2026-03-14 08:41:53,711 [Rank train_rank0] [Epoch 0 | Step 13360] train/loss: 0.448953, train/mse: 0.448953, train/data_wait_sec: 0.112503, train/step_wall_sec: 3.924355, train/data_wait_ratio: 0.028668, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13360.000000 2026-03-14 08:43:12,840 [Rank train_rank0] [Epoch 0 | Step 13380] train/loss: 0.462417, train/mse: 0.462417, train/data_wait_sec: 0.106587, train/step_wall_sec: 3.918678, train/data_wait_ratio: 0.027200, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13380.000000 2026-03-14 08:44:32,449 [Rank train_rank0] [Epoch 0 | Step 13400] train/loss: 0.439937, train/mse: 0.439937, train/data_wait_sec: 0.149856, train/step_wall_sec: 3.934077, train/data_wait_ratio: 0.038092, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13400.000000 2026-03-14 08:45:51,906 [Rank train_rank0] [Epoch 0 | Step 13420] train/loss: 0.460267, train/mse: 0.460267, train/data_wait_sec: 0.152059, train/step_wall_sec: 3.972354, train/data_wait_ratio: 0.038279, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13420.000000 2026-03-14 08:47:11,273 [Rank train_rank0] [Epoch 0 | Step 13440] train/loss: 0.438936, train/mse: 0.438936, train/data_wait_sec: 0.134056, train/step_wall_sec: 3.975079, train/data_wait_ratio: 0.033724, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13440.000000 2026-03-14 08:48:31,089 [Rank train_rank0] [Epoch 0 | Step 13460] train/loss: 0.446343, train/mse: 0.446343, train/data_wait_sec: 0.124342, train/step_wall_sec: 3.939528, train/data_wait_ratio: 0.031563, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13460.000000 2026-03-14 08:49:50,670 [Rank train_rank0] [Epoch 0 | Step 13480] train/loss: 0.434605, train/mse: 0.434605, train/data_wait_sec: 0.098370, train/step_wall_sec: 4.014806, train/data_wait_ratio: 0.024502, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13480.000000 2026-03-14 08:51:09,758 [Rank train_rank0] [Epoch 0 | Step 13500] train/loss: 0.478954, train/mse: 0.478954, train/data_wait_sec: 0.102628, train/step_wall_sec: 3.911704, train/data_wait_ratio: 0.026236, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13500.000000 2026-03-14 08:52:28,902 [Rank train_rank0] [Epoch 0 | Step 13520] train/loss: 0.451920, train/mse: 0.451920, train/data_wait_sec: 0.103401, train/step_wall_sec: 3.924112, train/data_wait_ratio: 0.026350, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13520.000000 2026-03-14 08:53:48,298 [Rank train_rank0] [Epoch 0 | Step 13540] train/loss: 0.488556, train/mse: 0.488556, train/data_wait_sec: 0.154395, train/step_wall_sec: 3.898663, train/data_wait_ratio: 0.039602, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13540.000000 2026-03-14 08:55:07,641 [Rank train_rank0] [Epoch 0 | Step 13560] train/loss: 0.466127, train/mse: 0.466127, train/data_wait_sec: 0.097097, train/step_wall_sec: 3.933705, train/data_wait_ratio: 0.024683, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13560.000000 2026-03-14 08:56:26,827 [Rank train_rank0] [Epoch 0 | Step 13580] train/loss: 0.477995, train/mse: 0.477995, train/data_wait_sec: 0.082295, train/step_wall_sec: 3.918808, train/data_wait_ratio: 0.021000, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13580.000000 2026-03-14 08:57:46,342 [Rank train_rank0] [Epoch 0 | Step 13600] train/loss: 0.502618, train/mse: 0.502618, train/data_wait_sec: 0.135120, train/step_wall_sec: 3.965754, train/data_wait_ratio: 0.034072, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13600.000000 2026-03-14 08:59:05,370 [Rank train_rank0] [Epoch 0 | Step 13620] train/loss: 0.467065, train/mse: 0.467065, train/data_wait_sec: 0.122389, train/step_wall_sec: 3.960885, train/data_wait_ratio: 0.030899, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13620.000000 2026-03-14 09:00:24,328 [Rank train_rank0] [Epoch 0 | Step 13640] train/loss: 0.464094, train/mse: 0.464094, train/data_wait_sec: 0.137044, train/step_wall_sec: 3.921302, train/data_wait_ratio: 0.034949, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13640.000000 2026-03-14 09:01:43,296 [Rank train_rank0] [Epoch 0 | Step 13660] train/loss: 0.467260, train/mse: 0.467260, train/data_wait_sec: 0.156536, train/step_wall_sec: 3.907544, train/data_wait_ratio: 0.040060, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13660.000000 2026-03-14 09:03:02,458 [Rank train_rank0] [Epoch 0 | Step 13680] train/loss: 0.462533, train/mse: 0.462533, train/data_wait_sec: 0.124489, train/step_wall_sec: 3.859276, train/data_wait_ratio: 0.032257, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13680.000000 2026-03-14 09:04:21,633 [Rank train_rank0] [Epoch 0 | Step 13700] train/loss: 0.466858, train/mse: 0.466858, train/data_wait_sec: 0.089930, train/step_wall_sec: 3.855298, train/data_wait_ratio: 0.023326, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13700.000000 2026-03-14 09:05:41,024 [Rank train_rank0] [Epoch 0 | Step 13720] train/loss: 0.445949, train/mse: 0.445949, train/data_wait_sec: 0.240782, train/step_wall_sec: 4.181036, train/data_wait_ratio: 0.057589, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13720.000000 2026-03-14 09:07:00,270 [Rank train_rank0] [Epoch 0 | Step 13740] train/loss: 0.506284, train/mse: 0.506284, train/data_wait_sec: 0.119681, train/step_wall_sec: 3.914402, train/data_wait_ratio: 0.030575, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13740.000000 2026-03-14 09:08:19,863 [Rank train_rank0] [Epoch 0 | Step 13760] train/loss: 0.461945, train/mse: 0.461945, train/data_wait_sec: 0.122601, train/step_wall_sec: 3.925014, train/data_wait_ratio: 0.031236, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13760.000000 2026-03-14 09:09:39,344 [Rank train_rank0] [Epoch 0 | Step 13780] train/loss: 0.484862, train/mse: 0.484862, train/data_wait_sec: 0.065930, train/step_wall_sec: 3.907345, train/data_wait_ratio: 0.016873, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13780.000000 2026-03-14 09:10:59,106 [Rank train_rank0] [Epoch 0 | Step 13800] train/loss: 0.436879, train/mse: 0.436879, train/data_wait_sec: 0.134719, train/step_wall_sec: 3.920690, train/data_wait_ratio: 0.034361, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13800.000000 2026-03-14 09:12:18,180 [Rank train_rank0] [Epoch 0 | Step 13820] train/loss: 0.477928, train/mse: 0.477928, train/data_wait_sec: 0.117768, train/step_wall_sec: 3.929170, train/data_wait_ratio: 0.029973, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13820.000000 2026-03-14 09:13:37,789 [Rank train_rank0] [Epoch 0 | Step 13840] train/loss: 0.463721, train/mse: 0.463721, train/data_wait_sec: 0.253288, train/step_wall_sec: 4.104505, train/data_wait_ratio: 0.061710, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13840.000000 2026-03-14 09:14:56,862 [Rank train_rank0] [Epoch 0 | Step 13860] train/loss: 0.474486, train/mse: 0.474486, train/data_wait_sec: 0.084535, train/step_wall_sec: 3.911908, train/data_wait_ratio: 0.021610, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13860.000000 2026-03-14 09:16:16,504 [Rank train_rank0] [Epoch 0 | Step 13880] train/loss: 0.476327, train/mse: 0.476327, train/data_wait_sec: 0.068963, train/step_wall_sec: 3.954125, train/data_wait_ratio: 0.017441, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13880.000000 2026-03-14 09:17:35,916 [Rank train_rank0] [Epoch 0 | Step 13900] train/loss: 0.453375, train/mse: 0.453375, train/data_wait_sec: 0.121266, train/step_wall_sec: 3.960525, train/data_wait_ratio: 0.030619, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13900.000000 2026-03-14 09:18:54,809 [Rank train_rank0] [Epoch 0 | Step 13920] train/loss: 0.451698, train/mse: 0.451698, train/data_wait_sec: 0.147944, train/step_wall_sec: 3.916750, train/data_wait_ratio: 0.037772, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13920.000000 2026-03-14 09:20:14,303 [Rank train_rank0] [Epoch 0 | Step 13940] train/loss: 0.494760, train/mse: 0.494760, train/data_wait_sec: 0.089197, train/step_wall_sec: 3.897092, train/data_wait_ratio: 0.022888, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13940.000000 2026-03-14 09:21:33,701 [Rank train_rank0] [Epoch 0 | Step 13960] train/loss: 0.461288, train/mse: 0.461288, train/data_wait_sec: 0.211687, train/step_wall_sec: 4.088127, train/data_wait_ratio: 0.051781, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13960.000000 2026-03-14 09:22:52,844 [Rank train_rank0] [Epoch 0 | Step 13980] train/loss: 0.483510, train/mse: 0.483510, train/data_wait_sec: 0.127429, train/step_wall_sec: 3.942061, train/data_wait_ratio: 0.032325, train/lr: 0.000200, train/epoch: 0.000000, train/step: 13980.000000 2026-03-14 09:24:37,693 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000014000.pt 2026-03-14 09:24:37,696 [Rank train_rank0] [Epoch 0 | Step 14000] train/loss: 0.468404, train/mse: 0.468404, train/data_wait_sec: 0.119080, train/step_wall_sec: 3.936564, train/data_wait_ratio: 0.030250, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14000.000000 2026-03-14 09:24:37,697 [Rank train_rank0] Generating samples for visualization... 2026-03-14 09:24:40,870 [Rank train_rank0] Sample generation done. 2026-03-14 09:26:00,013 [Rank train_rank0] [Epoch 0 | Step 14020] train/loss: 0.439449, train/mse: 0.439449, train/data_wait_sec: 0.116649, train/step_wall_sec: 4.039528, train/data_wait_ratio: 0.028877, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14020.000000 2026-03-14 09:27:19,408 [Rank train_rank0] [Epoch 0 | Step 14040] train/loss: 0.454071, train/mse: 0.454071, train/data_wait_sec: 0.102376, train/step_wall_sec: 3.882016, train/data_wait_ratio: 0.026372, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14040.000000 2026-03-14 09:28:38,321 [Rank train_rank0] [Epoch 0 | Step 14060] train/loss: 0.446189, train/mse: 0.446189, train/data_wait_sec: 0.124859, train/step_wall_sec: 3.893144, train/data_wait_ratio: 0.032071, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14060.000000 2026-03-14 09:29:57,422 [Rank train_rank0] [Epoch 0 | Step 14080] train/loss: 0.466557, train/mse: 0.466557, train/data_wait_sec: 0.115260, train/step_wall_sec: 3.965893, train/data_wait_ratio: 0.029063, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14080.000000 2026-03-14 09:31:16,566 [Rank train_rank0] [Epoch 0 | Step 14100] train/loss: 0.446763, train/mse: 0.446763, train/data_wait_sec: 0.051730, train/step_wall_sec: 3.953766, train/data_wait_ratio: 0.013084, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14100.000000 2026-03-14 09:32:35,463 [Rank train_rank0] [Epoch 0 | Step 14120] train/loss: 0.450352, train/mse: 0.450352, train/data_wait_sec: 0.085416, train/step_wall_sec: 3.903938, train/data_wait_ratio: 0.021880, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14120.000000 2026-03-14 09:33:55,008 [Rank train_rank0] [Epoch 0 | Step 14140] train/loss: 0.455810, train/mse: 0.455810, train/data_wait_sec: 0.097228, train/step_wall_sec: 3.939720, train/data_wait_ratio: 0.024679, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14140.000000 2026-03-14 09:35:13,844 [Rank train_rank0] [Epoch 0 | Step 14160] train/loss: 0.450427, train/mse: 0.450427, train/data_wait_sec: 0.095021, train/step_wall_sec: 3.859308, train/data_wait_ratio: 0.024621, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14160.000000 2026-03-14 09:36:32,877 [Rank train_rank0] [Epoch 0 | Step 14180] train/loss: 0.465306, train/mse: 0.465306, train/data_wait_sec: 0.143545, train/step_wall_sec: 4.026129, train/data_wait_ratio: 0.035653, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14180.000000 2026-03-14 09:37:51,977 [Rank train_rank0] [Epoch 0 | Step 14200] train/loss: 0.466956, train/mse: 0.466956, train/data_wait_sec: 0.092437, train/step_wall_sec: 3.942739, train/data_wait_ratio: 0.023445, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14200.000000 2026-03-14 09:39:11,146 [Rank train_rank0] [Epoch 0 | Step 14220] train/loss: 0.512970, train/mse: 0.512970, train/data_wait_sec: 0.092071, train/step_wall_sec: 3.928395, train/data_wait_ratio: 0.023437, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14220.000000 2026-03-14 09:40:30,432 [Rank train_rank0] [Epoch 0 | Step 14240] train/loss: 0.443482, train/mse: 0.443482, train/data_wait_sec: 0.106976, train/step_wall_sec: 3.923074, train/data_wait_ratio: 0.027268, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14240.000000 2026-03-14 09:41:50,172 [Rank train_rank0] [Epoch 0 | Step 14260] train/loss: 0.473027, train/mse: 0.473027, train/data_wait_sec: 0.205311, train/step_wall_sec: 4.000486, train/data_wait_ratio: 0.051322, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14260.000000 2026-03-14 09:43:09,754 [Rank train_rank0] [Epoch 0 | Step 14280] train/loss: 0.450321, train/mse: 0.450321, train/data_wait_sec: 0.115845, train/step_wall_sec: 3.911593, train/data_wait_ratio: 0.029616, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14280.000000 2026-03-14 09:44:29,425 [Rank train_rank0] [Epoch 0 | Step 14300] train/loss: 0.444118, train/mse: 0.444118, train/data_wait_sec: 0.110958, train/step_wall_sec: 4.019353, train/data_wait_ratio: 0.027606, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14300.000000 2026-03-14 09:45:48,624 [Rank train_rank0] [Epoch 0 | Step 14320] train/loss: 0.433332, train/mse: 0.433332, train/data_wait_sec: 0.146114, train/step_wall_sec: 3.863461, train/data_wait_ratio: 0.037820, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14320.000000 2026-03-14 09:47:07,990 [Rank train_rank0] [Epoch 0 | Step 14340] train/loss: 0.467960, train/mse: 0.467960, train/data_wait_sec: 0.120747, train/step_wall_sec: 3.992029, train/data_wait_ratio: 0.030247, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14340.000000 2026-03-14 09:48:27,050 [Rank train_rank0] [Epoch 0 | Step 14360] train/loss: 0.470614, train/mse: 0.470614, train/data_wait_sec: 0.121451, train/step_wall_sec: 3.941983, train/data_wait_ratio: 0.030810, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14360.000000 2026-03-14 09:49:45,962 [Rank train_rank0] [Epoch 0 | Step 14380] train/loss: 0.442492, train/mse: 0.442492, train/data_wait_sec: 0.095189, train/step_wall_sec: 3.807296, train/data_wait_ratio: 0.025002, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14380.000000 2026-03-14 09:51:05,079 [Rank train_rank0] [Epoch 0 | Step 14400] train/loss: 0.459971, train/mse: 0.459971, train/data_wait_sec: 0.228909, train/step_wall_sec: 4.067444, train/data_wait_ratio: 0.056278, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14400.000000 2026-03-14 09:52:23,935 [Rank train_rank0] [Epoch 0 | Step 14420] train/loss: 0.467652, train/mse: 0.467652, train/data_wait_sec: 0.136391, train/step_wall_sec: 3.954259, train/data_wait_ratio: 0.034492, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14420.000000 2026-03-14 09:53:43,070 [Rank train_rank0] [Epoch 0 | Step 14440] train/loss: 0.450804, train/mse: 0.450804, train/data_wait_sec: 0.086116, train/step_wall_sec: 3.891912, train/data_wait_ratio: 0.022127, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14440.000000 2026-03-14 09:55:02,424 [Rank train_rank0] [Epoch 0 | Step 14460] train/loss: 0.499606, train/mse: 0.499606, train/data_wait_sec: 0.092929, train/step_wall_sec: 3.829068, train/data_wait_ratio: 0.024269, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14460.000000 2026-03-14 09:56:21,618 [Rank train_rank0] [Epoch 0 | Step 14480] train/loss: 0.457820, train/mse: 0.457820, train/data_wait_sec: 0.269995, train/step_wall_sec: 4.026577, train/data_wait_ratio: 0.067053, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14480.000000 2026-03-14 09:57:40,479 [Rank train_rank0] [Epoch 0 | Step 14500] train/loss: 0.460358, train/mse: 0.460358, train/data_wait_sec: 0.128128, train/step_wall_sec: 3.840247, train/data_wait_ratio: 0.033365, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14500.000000 2026-03-14 09:58:59,829 [Rank train_rank0] [Epoch 0 | Step 14520] train/loss: 0.445845, train/mse: 0.445845, train/data_wait_sec: 0.135931, train/step_wall_sec: 3.953941, train/data_wait_ratio: 0.034379, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14520.000000 2026-03-14 10:00:18,942 [Rank train_rank0] [Epoch 0 | Step 14540] train/loss: 0.488755, train/mse: 0.488755, train/data_wait_sec: 0.095236, train/step_wall_sec: 3.919406, train/data_wait_ratio: 0.024298, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14540.000000 2026-03-14 10:01:38,695 [Rank train_rank0] [Epoch 0 | Step 14560] train/loss: 0.446984, train/mse: 0.446984, train/data_wait_sec: 0.097446, train/step_wall_sec: 3.929315, train/data_wait_ratio: 0.024800, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14560.000000 2026-03-14 10:02:57,963 [Rank train_rank0] [Epoch 0 | Step 14580] train/loss: 0.468558, train/mse: 0.468558, train/data_wait_sec: 0.094228, train/step_wall_sec: 3.883792, train/data_wait_ratio: 0.024262, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14580.000000 2026-03-14 10:04:16,782 [Rank train_rank0] [Epoch 0 | Step 14600] train/loss: 0.488895, train/mse: 0.488895, train/data_wait_sec: 0.093410, train/step_wall_sec: 3.921967, train/data_wait_ratio: 0.023817, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14600.000000 2026-03-14 10:05:36,472 [Rank train_rank0] [Epoch 0 | Step 14620] train/loss: 0.460150, train/mse: 0.460150, train/data_wait_sec: 0.104775, train/step_wall_sec: 3.884867, train/data_wait_ratio: 0.026970, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14620.000000 2026-03-14 10:06:56,225 [Rank train_rank0] [Epoch 0 | Step 14640] train/loss: 0.455638, train/mse: 0.455638, train/data_wait_sec: 0.210755, train/step_wall_sec: 4.052693, train/data_wait_ratio: 0.052004, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14640.000000 2026-03-14 10:08:15,640 [Rank train_rank0] [Epoch 0 | Step 14660] train/loss: 0.466394, train/mse: 0.466394, train/data_wait_sec: 0.143370, train/step_wall_sec: 3.976321, train/data_wait_ratio: 0.036056, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14660.000000 2026-03-14 10:09:34,878 [Rank train_rank0] [Epoch 0 | Step 14680] train/loss: 0.442847, train/mse: 0.442847, train/data_wait_sec: 0.123468, train/step_wall_sec: 3.957700, train/data_wait_ratio: 0.031197, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14680.000000 2026-03-14 10:10:53,931 [Rank train_rank0] [Epoch 0 | Step 14700] train/loss: 0.451375, train/mse: 0.451375, train/data_wait_sec: 0.129547, train/step_wall_sec: 3.905305, train/data_wait_ratio: 0.033172, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14700.000000 2026-03-14 10:12:12,901 [Rank train_rank0] [Epoch 0 | Step 14720] train/loss: 0.468584, train/mse: 0.468584, train/data_wait_sec: 0.098943, train/step_wall_sec: 3.942151, train/data_wait_ratio: 0.025099, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14720.000000 2026-03-14 10:13:32,213 [Rank train_rank0] [Epoch 0 | Step 14740] train/loss: 0.458140, train/mse: 0.458140, train/data_wait_sec: 0.107095, train/step_wall_sec: 3.892929, train/data_wait_ratio: 0.027510, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14740.000000 2026-03-14 10:14:51,345 [Rank train_rank0] [Epoch 0 | Step 14760] train/loss: 0.459171, train/mse: 0.459171, train/data_wait_sec: 0.148332, train/step_wall_sec: 3.961565, train/data_wait_ratio: 0.037443, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14760.000000 2026-03-14 10:16:11,073 [Rank train_rank0] [Epoch 0 | Step 14780] train/loss: 0.464065, train/mse: 0.464065, train/data_wait_sec: 0.141055, train/step_wall_sec: 3.865957, train/data_wait_ratio: 0.036486, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14780.000000 2026-03-14 10:17:30,454 [Rank train_rank0] [Epoch 0 | Step 14800] train/loss: 0.435986, train/mse: 0.435986, train/data_wait_sec: 0.089913, train/step_wall_sec: 3.854671, train/data_wait_ratio: 0.023326, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14800.000000 2026-03-14 10:18:49,929 [Rank train_rank0] [Epoch 0 | Step 14820] train/loss: 0.450433, train/mse: 0.450433, train/data_wait_sec: 0.123795, train/step_wall_sec: 3.891171, train/data_wait_ratio: 0.031814, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14820.000000 2026-03-14 10:20:09,087 [Rank train_rank0] [Epoch 0 | Step 14840] train/loss: 0.449088, train/mse: 0.449088, train/data_wait_sec: 0.106566, train/step_wall_sec: 3.890386, train/data_wait_ratio: 0.027392, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14840.000000 2026-03-14 10:21:28,639 [Rank train_rank0] [Epoch 0 | Step 14860] train/loss: 0.485499, train/mse: 0.485499, train/data_wait_sec: 0.143042, train/step_wall_sec: 3.957530, train/data_wait_ratio: 0.036144, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14860.000000 2026-03-14 10:22:48,296 [Rank train_rank0] [Epoch 0 | Step 14880] train/loss: 0.451925, train/mse: 0.451925, train/data_wait_sec: 0.135225, train/step_wall_sec: 3.911475, train/data_wait_ratio: 0.034571, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14880.000000 2026-03-14 10:24:07,448 [Rank train_rank0] [Epoch 0 | Step 14900] train/loss: 0.462003, train/mse: 0.462003, train/data_wait_sec: 0.111848, train/step_wall_sec: 3.888785, train/data_wait_ratio: 0.028762, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14900.000000 2026-03-14 10:25:26,979 [Rank train_rank0] [Epoch 0 | Step 14920] train/loss: 0.446273, train/mse: 0.446273, train/data_wait_sec: 0.144615, train/step_wall_sec: 3.956645, train/data_wait_ratio: 0.036550, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14920.000000 2026-03-14 10:26:46,631 [Rank train_rank0] [Epoch 0 | Step 14940] train/loss: 0.434408, train/mse: 0.434408, train/data_wait_sec: 0.060848, train/step_wall_sec: 3.935634, train/data_wait_ratio: 0.015461, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14940.000000 2026-03-14 10:28:06,308 [Rank train_rank0] [Epoch 0 | Step 14960] train/loss: 0.449885, train/mse: 0.449885, train/data_wait_sec: 0.126887, train/step_wall_sec: 3.981616, train/data_wait_ratio: 0.031868, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14960.000000 2026-03-14 10:29:26,062 [Rank train_rank0] [Epoch 0 | Step 14980] train/loss: 0.461690, train/mse: 0.461690, train/data_wait_sec: 0.099611, train/step_wall_sec: 3.965684, train/data_wait_ratio: 0.025118, train/lr: 0.000200, train/epoch: 0.000000, train/step: 14980.000000 2026-03-14 10:31:09,550 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000015000.pt 2026-03-14 10:31:09,553 [Rank train_rank0] [Epoch 0 | Step 15000] train/loss: 0.492070, train/mse: 0.492070, train/data_wait_sec: 0.248777, train/step_wall_sec: 4.035429, train/data_wait_ratio: 0.061648, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15000.000000 2026-03-14 10:31:09,555 [Rank train_rank0] Generating samples for visualization... 2026-03-14 10:31:12,632 [Rank train_rank0] Sample generation done. 2026-03-14 10:32:31,891 [Rank train_rank0] [Epoch 0 | Step 15020] train/loss: 0.452331, train/mse: 0.452331, train/data_wait_sec: 0.124584, train/step_wall_sec: 3.934764, train/data_wait_ratio: 0.031662, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15020.000000 2026-03-14 10:33:51,050 [Rank train_rank0] [Epoch 0 | Step 15040] train/loss: 0.453052, train/mse: 0.453052, train/data_wait_sec: 0.119535, train/step_wall_sec: 3.916461, train/data_wait_ratio: 0.030521, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15040.000000 2026-03-14 10:35:11,000 [Rank train_rank0] [Epoch 0 | Step 15060] train/loss: 0.428922, train/mse: 0.428922, train/data_wait_sec: 0.112613, train/step_wall_sec: 3.947679, train/data_wait_ratio: 0.028526, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15060.000000 2026-03-14 10:36:30,817 [Rank train_rank0] [Epoch 0 | Step 15080] train/loss: 0.469093, train/mse: 0.469093, train/data_wait_sec: 0.097828, train/step_wall_sec: 3.945900, train/data_wait_ratio: 0.024792, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15080.000000 2026-03-14 10:37:50,196 [Rank train_rank0] [Epoch 0 | Step 15100] train/loss: 0.453489, train/mse: 0.453489, train/data_wait_sec: 0.090056, train/step_wall_sec: 3.959355, train/data_wait_ratio: 0.022745, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15100.000000 2026-03-14 10:39:09,665 [Rank train_rank0] [Epoch 0 | Step 15120] train/loss: 0.466911, train/mse: 0.466911, train/data_wait_sec: 0.122967, train/step_wall_sec: 4.000634, train/data_wait_ratio: 0.030737, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15120.000000 2026-03-14 10:40:28,664 [Rank train_rank0] [Epoch 0 | Step 15140] train/loss: 0.458265, train/mse: 0.458265, train/data_wait_sec: 0.069060, train/step_wall_sec: 3.925729, train/data_wait_ratio: 0.017592, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15140.000000 2026-03-14 10:41:48,362 [Rank train_rank0] [Epoch 0 | Step 15160] train/loss: 0.472380, train/mse: 0.472380, train/data_wait_sec: 0.091374, train/step_wall_sec: 3.972299, train/data_wait_ratio: 0.023003, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15160.000000 2026-03-14 10:43:07,861 [Rank train_rank0] [Epoch 0 | Step 15180] train/loss: 0.419382, train/mse: 0.419382, train/data_wait_sec: 0.286764, train/step_wall_sec: 4.125806, train/data_wait_ratio: 0.069505, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15180.000000 2026-03-14 10:44:27,117 [Rank train_rank0] [Epoch 0 | Step 15200] train/loss: 0.447884, train/mse: 0.447884, train/data_wait_sec: 0.131633, train/step_wall_sec: 3.959828, train/data_wait_ratio: 0.033242, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15200.000000 2026-03-14 10:45:46,477 [Rank train_rank0] [Epoch 0 | Step 15220] train/loss: 0.476553, train/mse: 0.476553, train/data_wait_sec: 0.117445, train/step_wall_sec: 4.060181, train/data_wait_ratio: 0.028926, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15220.000000 2026-03-14 10:47:05,691 [Rank train_rank0] [Epoch 0 | Step 15240] train/loss: 0.439176, train/mse: 0.439176, train/data_wait_sec: 0.090233, train/step_wall_sec: 3.938585, train/data_wait_ratio: 0.022910, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15240.000000 2026-03-14 10:48:24,630 [Rank train_rank0] [Epoch 0 | Step 15260] train/loss: 0.461326, train/mse: 0.461326, train/data_wait_sec: 0.081964, train/step_wall_sec: 3.872099, train/data_wait_ratio: 0.021168, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15260.000000 2026-03-14 10:49:44,190 [Rank train_rank0] [Epoch 0 | Step 15280] train/loss: 0.460523, train/mse: 0.460523, train/data_wait_sec: 0.105437, train/step_wall_sec: 3.899859, train/data_wait_ratio: 0.027036, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15280.000000 2026-03-14 10:51:03,957 [Rank train_rank0] [Epoch 0 | Step 15300] train/loss: 0.470535, train/mse: 0.470535, train/data_wait_sec: 0.173056, train/step_wall_sec: 3.941849, train/data_wait_ratio: 0.043902, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15300.000000 2026-03-14 10:52:23,759 [Rank train_rank0] [Epoch 0 | Step 15320] train/loss: 0.445890, train/mse: 0.445890, train/data_wait_sec: 0.143207, train/step_wall_sec: 3.952289, train/data_wait_ratio: 0.036234, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15320.000000 2026-03-14 10:53:43,374 [Rank train_rank0] [Epoch 0 | Step 15340] train/loss: 0.470357, train/mse: 0.470357, train/data_wait_sec: 0.057227, train/step_wall_sec: 3.911808, train/data_wait_ratio: 0.014629, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15340.000000 2026-03-14 10:55:02,722 [Rank train_rank0] [Epoch 0 | Step 15360] train/loss: 0.457704, train/mse: 0.457704, train/data_wait_sec: 0.121665, train/step_wall_sec: 3.946159, train/data_wait_ratio: 0.030831, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15360.000000 2026-03-14 10:56:22,412 [Rank train_rank0] [Epoch 0 | Step 15380] train/loss: 0.438564, train/mse: 0.438564, train/data_wait_sec: 0.212398, train/step_wall_sec: 4.047797, train/data_wait_ratio: 0.052473, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15380.000000 2026-03-14 10:57:42,148 [Rank train_rank0] [Epoch 0 | Step 15400] train/loss: 0.423142, train/mse: 0.423142, train/data_wait_sec: 0.076431, train/step_wall_sec: 3.899809, train/data_wait_ratio: 0.019599, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15400.000000 2026-03-14 10:59:01,354 [Rank train_rank0] [Epoch 0 | Step 15420] train/loss: 0.462044, train/mse: 0.462044, train/data_wait_sec: 0.117353, train/step_wall_sec: 3.928159, train/data_wait_ratio: 0.029875, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15420.000000 2026-03-14 11:00:21,115 [Rank train_rank0] [Epoch 0 | Step 15440] train/loss: 0.452124, train/mse: 0.452124, train/data_wait_sec: 0.140354, train/step_wall_sec: 3.957608, train/data_wait_ratio: 0.035464, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15440.000000 2026-03-14 11:01:41,088 [Rank train_rank0] [Epoch 0 | Step 15460] train/loss: 0.471836, train/mse: 0.471836, train/data_wait_sec: 0.269739, train/step_wall_sec: 4.213837, train/data_wait_ratio: 0.064013, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15460.000000 2026-03-14 11:03:00,716 [Rank train_rank0] [Epoch 0 | Step 15480] train/loss: 0.452170, train/mse: 0.452170, train/data_wait_sec: 0.155424, train/step_wall_sec: 3.930182, train/data_wait_ratio: 0.039546, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15480.000000 2026-03-14 11:04:20,424 [Rank train_rank0] [Epoch 0 | Step 15500] train/loss: 0.446283, train/mse: 0.446283, train/data_wait_sec: 0.071362, train/step_wall_sec: 3.907807, train/data_wait_ratio: 0.018261, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15500.000000 2026-03-14 11:05:40,127 [Rank train_rank0] [Epoch 0 | Step 15520] train/loss: 0.435885, train/mse: 0.435885, train/data_wait_sec: 0.122339, train/step_wall_sec: 3.959078, train/data_wait_ratio: 0.030901, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15520.000000 2026-03-14 11:06:59,412 [Rank train_rank0] [Epoch 0 | Step 15540] train/loss: 0.460531, train/mse: 0.460531, train/data_wait_sec: 0.145857, train/step_wall_sec: 3.906534, train/data_wait_ratio: 0.037337, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15540.000000 2026-03-14 11:08:18,925 [Rank train_rank0] [Epoch 0 | Step 15560] train/loss: 0.468933, train/mse: 0.468933, train/data_wait_sec: 0.103159, train/step_wall_sec: 3.933249, train/data_wait_ratio: 0.026227, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15560.000000 2026-03-14 11:09:38,389 [Rank train_rank0] [Epoch 0 | Step 15580] train/loss: 0.445571, train/mse: 0.445571, train/data_wait_sec: 0.106571, train/step_wall_sec: 3.995156, train/data_wait_ratio: 0.026675, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15580.000000 2026-03-14 11:10:57,839 [Rank train_rank0] [Epoch 0 | Step 15600] train/loss: 0.481555, train/mse: 0.481555, train/data_wait_sec: 0.066555, train/step_wall_sec: 3.841163, train/data_wait_ratio: 0.017327, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15600.000000 2026-03-14 11:12:17,195 [Rank train_rank0] [Epoch 0 | Step 15620] train/loss: 0.477289, train/mse: 0.477289, train/data_wait_sec: 0.149987, train/step_wall_sec: 3.967986, train/data_wait_ratio: 0.037799, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15620.000000 2026-03-14 11:13:36,898 [Rank train_rank0] [Epoch 0 | Step 15640] train/loss: 0.467236, train/mse: 0.467236, train/data_wait_sec: 0.145779, train/step_wall_sec: 3.949434, train/data_wait_ratio: 0.036911, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15640.000000 2026-03-14 11:14:56,698 [Rank train_rank0] [Epoch 0 | Step 15660] train/loss: 0.467090, train/mse: 0.467090, train/data_wait_sec: 0.126634, train/step_wall_sec: 3.941086, train/data_wait_ratio: 0.032132, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15660.000000 2026-03-14 11:16:16,206 [Rank train_rank0] [Epoch 0 | Step 15680] train/loss: 0.448459, train/mse: 0.448459, train/data_wait_sec: 0.110438, train/step_wall_sec: 3.920330, train/data_wait_ratio: 0.028170, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15680.000000 2026-03-14 11:17:35,588 [Rank train_rank0] [Epoch 0 | Step 15700] train/loss: 0.436911, train/mse: 0.436911, train/data_wait_sec: 0.121206, train/step_wall_sec: 3.935354, train/data_wait_ratio: 0.030799, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15700.000000 2026-03-14 11:18:55,415 [Rank train_rank0] [Epoch 0 | Step 15720] train/loss: 0.454527, train/mse: 0.454527, train/data_wait_sec: 0.147449, train/step_wall_sec: 3.999497, train/data_wait_ratio: 0.036867, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15720.000000 2026-03-14 11:20:15,190 [Rank train_rank0] [Epoch 0 | Step 15740] train/loss: 0.464691, train/mse: 0.464691, train/data_wait_sec: 0.124706, train/step_wall_sec: 3.937693, train/data_wait_ratio: 0.031670, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15740.000000 2026-03-14 11:21:34,641 [Rank train_rank0] [Epoch 0 | Step 15760] train/loss: 0.474657, train/mse: 0.474657, train/data_wait_sec: 0.175011, train/step_wall_sec: 3.955072, train/data_wait_ratio: 0.044250, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15760.000000 2026-03-14 11:22:54,716 [Rank train_rank0] [Epoch 0 | Step 15780] train/loss: 0.452466, train/mse: 0.452466, train/data_wait_sec: 0.101292, train/step_wall_sec: 3.946717, train/data_wait_ratio: 0.025665, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15780.000000 2026-03-14 11:24:13,963 [Rank train_rank0] [Epoch 0 | Step 15800] train/loss: 0.459958, train/mse: 0.459958, train/data_wait_sec: 0.090918, train/step_wall_sec: 3.905650, train/data_wait_ratio: 0.023279, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15800.000000 2026-03-14 11:25:33,364 [Rank train_rank0] [Epoch 0 | Step 15820] train/loss: 0.468350, train/mse: 0.468350, train/data_wait_sec: 0.102592, train/step_wall_sec: 3.923156, train/data_wait_ratio: 0.026150, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15820.000000 2026-03-14 11:26:53,049 [Rank train_rank0] [Epoch 0 | Step 15840] train/loss: 0.457973, train/mse: 0.457973, train/data_wait_sec: 0.099919, train/step_wall_sec: 3.907039, train/data_wait_ratio: 0.025574, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15840.000000 2026-03-14 11:28:12,523 [Rank train_rank0] [Epoch 0 | Step 15860] train/loss: 0.452545, train/mse: 0.452545, train/data_wait_sec: 0.100704, train/step_wall_sec: 3.956767, train/data_wait_ratio: 0.025451, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15860.000000 2026-03-14 11:29:32,195 [Rank train_rank0] [Epoch 0 | Step 15880] train/loss: 0.452641, train/mse: 0.452641, train/data_wait_sec: 0.127726, train/step_wall_sec: 3.920370, train/data_wait_ratio: 0.032580, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15880.000000 2026-03-14 11:30:52,086 [Rank train_rank0] [Epoch 0 | Step 15900] train/loss: 0.457254, train/mse: 0.457254, train/data_wait_sec: 0.083228, train/step_wall_sec: 3.960884, train/data_wait_ratio: 0.021013, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15900.000000 2026-03-14 11:32:11,823 [Rank train_rank0] [Epoch 0 | Step 15920] train/loss: 0.435043, train/mse: 0.435043, train/data_wait_sec: 0.153444, train/step_wall_sec: 3.956559, train/data_wait_ratio: 0.038782, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15920.000000 2026-03-14 11:33:31,371 [Rank train_rank0] [Epoch 0 | Step 15940] train/loss: 0.423124, train/mse: 0.423124, train/data_wait_sec: 0.116260, train/step_wall_sec: 3.938264, train/data_wait_ratio: 0.029521, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15940.000000 2026-03-14 11:34:50,938 [Rank train_rank0] [Epoch 0 | Step 15960] train/loss: 0.413024, train/mse: 0.413024, train/data_wait_sec: 0.090792, train/step_wall_sec: 3.878624, train/data_wait_ratio: 0.023408, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15960.000000 2026-03-14 11:36:10,666 [Rank train_rank0] [Epoch 0 | Step 15980] train/loss: 0.444695, train/mse: 0.444695, train/data_wait_sec: 0.104238, train/step_wall_sec: 3.948472, train/data_wait_ratio: 0.026400, train/lr: 0.000200, train/epoch: 0.000000, train/step: 15980.000000 2026-03-14 11:37:54,872 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000016000.pt 2026-03-14 11:37:54,875 [Rank train_rank0] [Epoch 0 | Step 16000] train/loss: 0.441806, train/mse: 0.441806, train/data_wait_sec: 0.096043, train/step_wall_sec: 4.106114, train/data_wait_ratio: 0.023390, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16000.000000 2026-03-14 11:37:54,877 [Rank train_rank0] Generating samples for visualization... 2026-03-14 11:37:57,999 [Rank train_rank0] Sample generation done. 2026-03-14 11:39:17,339 [Rank train_rank0] [Epoch 0 | Step 16020] train/loss: 0.459696, train/mse: 0.459696, train/data_wait_sec: 0.076640, train/step_wall_sec: 3.898401, train/data_wait_ratio: 0.019659, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16020.000000 2026-03-14 11:40:36,736 [Rank train_rank0] [Epoch 0 | Step 16040] train/loss: 0.446632, train/mse: 0.446632, train/data_wait_sec: 0.142464, train/step_wall_sec: 3.907427, train/data_wait_ratio: 0.036460, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16040.000000 2026-03-14 11:41:55,704 [Rank train_rank0] [Epoch 0 | Step 16060] train/loss: 0.447666, train/mse: 0.447666, train/data_wait_sec: 0.083031, train/step_wall_sec: 3.902731, train/data_wait_ratio: 0.021275, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16060.000000 2026-03-14 11:43:14,943 [Rank train_rank0] [Epoch 0 | Step 16080] train/loss: 0.454645, train/mse: 0.454645, train/data_wait_sec: 0.118051, train/step_wall_sec: 3.906867, train/data_wait_ratio: 0.030216, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16080.000000 2026-03-14 11:44:34,164 [Rank train_rank0] [Epoch 0 | Step 16100] train/loss: 0.453952, train/mse: 0.453952, train/data_wait_sec: 0.117222, train/step_wall_sec: 3.948743, train/data_wait_ratio: 0.029686, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16100.000000 2026-03-14 11:45:53,937 [Rank train_rank0] [Epoch 0 | Step 16120] train/loss: 0.456231, train/mse: 0.456231, train/data_wait_sec: 0.123303, train/step_wall_sec: 3.942848, train/data_wait_ratio: 0.031273, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16120.000000 2026-03-14 11:47:13,351 [Rank train_rank0] [Epoch 0 | Step 16140] train/loss: 0.440516, train/mse: 0.440516, train/data_wait_sec: 0.111533, train/step_wall_sec: 3.920064, train/data_wait_ratio: 0.028452, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16140.000000 2026-03-14 11:48:32,571 [Rank train_rank0] [Epoch 0 | Step 16160] train/loss: 0.445697, train/mse: 0.445697, train/data_wait_sec: 0.083603, train/step_wall_sec: 3.972500, train/data_wait_ratio: 0.021046, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16160.000000 2026-03-14 11:49:51,999 [Rank train_rank0] [Epoch 0 | Step 16180] train/loss: 0.467775, train/mse: 0.467775, train/data_wait_sec: 0.127592, train/step_wall_sec: 3.978135, train/data_wait_ratio: 0.032073, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16180.000000 2026-03-14 11:51:11,579 [Rank train_rank0] [Epoch 0 | Step 16200] train/loss: 0.426703, train/mse: 0.426703, train/data_wait_sec: 0.181224, train/step_wall_sec: 3.984017, train/data_wait_ratio: 0.045488, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16200.000000 2026-03-14 11:52:31,056 [Rank train_rank0] [Epoch 0 | Step 16220] train/loss: 0.450107, train/mse: 0.450107, train/data_wait_sec: 0.101129, train/step_wall_sec: 3.905099, train/data_wait_ratio: 0.025897, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16220.000000 2026-03-14 11:53:50,271 [Rank train_rank0] [Epoch 0 | Step 16240] train/loss: 0.425899, train/mse: 0.425899, train/data_wait_sec: 0.148759, train/step_wall_sec: 3.944941, train/data_wait_ratio: 0.037709, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16240.000000 2026-03-14 11:55:09,897 [Rank train_rank0] [Epoch 0 | Step 16260] train/loss: 0.473428, train/mse: 0.473428, train/data_wait_sec: 0.116681, train/step_wall_sec: 3.937113, train/data_wait_ratio: 0.029636, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16260.000000 2026-03-14 11:56:29,250 [Rank train_rank0] [Epoch 0 | Step 16280] train/loss: 0.468503, train/mse: 0.468503, train/data_wait_sec: 0.078088, train/step_wall_sec: 3.863149, train/data_wait_ratio: 0.020214, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16280.000000 2026-03-14 11:57:48,143 [Rank train_rank0] [Epoch 0 | Step 16300] train/loss: 0.441971, train/mse: 0.441971, train/data_wait_sec: 0.117493, train/step_wall_sec: 3.889184, train/data_wait_ratio: 0.030210, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16300.000000 2026-03-14 11:59:07,438 [Rank train_rank0] [Epoch 0 | Step 16320] train/loss: 0.454330, train/mse: 0.454330, train/data_wait_sec: 0.129589, train/step_wall_sec: 3.892506, train/data_wait_ratio: 0.033292, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16320.000000 2026-03-14 12:00:26,850 [Rank train_rank0] [Epoch 0 | Step 16340] train/loss: 0.484569, train/mse: 0.484569, train/data_wait_sec: 0.101770, train/step_wall_sec: 3.917431, train/data_wait_ratio: 0.025979, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16340.000000 2026-03-14 12:01:46,569 [Rank train_rank0] [Epoch 0 | Step 16360] train/loss: 0.449388, train/mse: 0.449388, train/data_wait_sec: 0.091219, train/step_wall_sec: 3.963792, train/data_wait_ratio: 0.023013, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16360.000000 2026-03-14 12:03:05,684 [Rank train_rank0] [Epoch 0 | Step 16380] train/loss: 0.451269, train/mse: 0.451269, train/data_wait_sec: 0.102557, train/step_wall_sec: 3.920467, train/data_wait_ratio: 0.026159, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16380.000000 2026-03-14 12:04:24,916 [Rank train_rank0] [Epoch 0 | Step 16400] train/loss: 0.445846, train/mse: 0.445846, train/data_wait_sec: 0.119629, train/step_wall_sec: 3.904663, train/data_wait_ratio: 0.030637, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16400.000000 2026-03-14 12:05:43,754 [Rank train_rank0] [Epoch 0 | Step 16420] train/loss: 0.442566, train/mse: 0.442566, train/data_wait_sec: 0.106299, train/step_wall_sec: 3.960414, train/data_wait_ratio: 0.026840, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16420.000000 2026-03-14 12:07:02,667 [Rank train_rank0] [Epoch 0 | Step 16440] train/loss: 0.482459, train/mse: 0.482459, train/data_wait_sec: 0.122903, train/step_wall_sec: 3.919475, train/data_wait_ratio: 0.031357, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16440.000000 2026-03-14 12:08:22,196 [Rank train_rank0] [Epoch 0 | Step 16460] train/loss: 0.464854, train/mse: 0.464854, train/data_wait_sec: 0.235495, train/step_wall_sec: 4.038209, train/data_wait_ratio: 0.058317, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16460.000000 2026-03-14 12:09:41,548 [Rank train_rank0] [Epoch 0 | Step 16480] train/loss: 0.459606, train/mse: 0.459606, train/data_wait_sec: 0.089991, train/step_wall_sec: 3.920207, train/data_wait_ratio: 0.022956, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16480.000000 2026-03-14 12:11:01,234 [Rank train_rank0] [Epoch 0 | Step 16500] train/loss: 0.445286, train/mse: 0.445286, train/data_wait_sec: 0.120034, train/step_wall_sec: 4.066840, train/data_wait_ratio: 0.029515, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16500.000000 2026-03-14 12:12:20,210 [Rank train_rank0] [Epoch 0 | Step 16520] train/loss: 0.469643, train/mse: 0.469643, train/data_wait_sec: 0.113836, train/step_wall_sec: 3.921279, train/data_wait_ratio: 0.029030, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16520.000000 2026-03-14 12:13:39,418 [Rank train_rank0] [Epoch 0 | Step 16540] train/loss: 0.461556, train/mse: 0.461556, train/data_wait_sec: 0.106352, train/step_wall_sec: 3.874965, train/data_wait_ratio: 0.027446, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16540.000000 2026-03-14 12:14:58,958 [Rank train_rank0] [Epoch 0 | Step 16560] train/loss: 0.477658, train/mse: 0.477658, train/data_wait_sec: 0.099477, train/step_wall_sec: 3.853769, train/data_wait_ratio: 0.025813, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16560.000000 2026-03-14 12:16:17,731 [Rank train_rank0] [Epoch 0 | Step 16580] train/loss: 0.468160, train/mse: 0.468160, train/data_wait_sec: 0.100139, train/step_wall_sec: 3.915671, train/data_wait_ratio: 0.025574, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16580.000000 2026-03-14 12:17:36,745 [Rank train_rank0] [Epoch 0 | Step 16600] train/loss: 0.480055, train/mse: 0.480055, train/data_wait_sec: 0.111988, train/step_wall_sec: 3.923920, train/data_wait_ratio: 0.028540, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16600.000000 2026-03-14 12:18:56,339 [Rank train_rank0] [Epoch 0 | Step 16620] train/loss: 0.440687, train/mse: 0.440687, train/data_wait_sec: 0.119178, train/step_wall_sec: 3.912627, train/data_wait_ratio: 0.030460, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16620.000000 2026-03-14 12:20:15,692 [Rank train_rank0] [Epoch 0 | Step 16640] train/loss: 0.468705, train/mse: 0.468705, train/data_wait_sec: 0.105852, train/step_wall_sec: 3.966878, train/data_wait_ratio: 0.026684, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16640.000000 2026-03-14 12:21:35,507 [Rank train_rank0] [Epoch 0 | Step 16660] train/loss: 0.458118, train/mse: 0.458118, train/data_wait_sec: 0.112362, train/step_wall_sec: 3.903550, train/data_wait_ratio: 0.028785, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16660.000000 2026-03-14 12:22:55,187 [Rank train_rank0] [Epoch 0 | Step 16680] train/loss: 0.465822, train/mse: 0.465822, train/data_wait_sec: 0.123459, train/step_wall_sec: 3.940768, train/data_wait_ratio: 0.031329, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16680.000000 2026-03-14 12:24:14,614 [Rank train_rank0] [Epoch 0 | Step 16700] train/loss: 0.467821, train/mse: 0.467821, train/data_wait_sec: 0.090889, train/step_wall_sec: 3.868088, train/data_wait_ratio: 0.023497, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16700.000000 2026-03-14 12:25:34,385 [Rank train_rank0] [Epoch 0 | Step 16720] train/loss: 0.438199, train/mse: 0.438199, train/data_wait_sec: 0.093938, train/step_wall_sec: 3.880765, train/data_wait_ratio: 0.024206, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16720.000000 2026-03-14 12:26:53,839 [Rank train_rank0] [Epoch 0 | Step 16740] train/loss: 0.443139, train/mse: 0.443139, train/data_wait_sec: 0.088010, train/step_wall_sec: 3.983000, train/data_wait_ratio: 0.022096, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16740.000000 2026-03-14 12:28:13,446 [Rank train_rank0] [Epoch 0 | Step 16760] train/loss: 0.446533, train/mse: 0.446533, train/data_wait_sec: 0.094043, train/step_wall_sec: 3.913044, train/data_wait_ratio: 0.024033, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16760.000000 2026-03-14 12:29:33,114 [Rank train_rank0] [Epoch 0 | Step 16780] train/loss: 0.485041, train/mse: 0.485041, train/data_wait_sec: 0.135819, train/step_wall_sec: 3.911453, train/data_wait_ratio: 0.034724, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16780.000000 2026-03-14 12:30:52,417 [Rank train_rank0] [Epoch 0 | Step 16800] train/loss: 0.432202, train/mse: 0.432202, train/data_wait_sec: 0.120113, train/step_wall_sec: 3.913600, train/data_wait_ratio: 0.030691, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16800.000000 2026-03-14 12:32:11,847 [Rank train_rank0] [Epoch 0 | Step 16820] train/loss: 0.471202, train/mse: 0.471202, train/data_wait_sec: 0.095797, train/step_wall_sec: 3.917779, train/data_wait_ratio: 0.024452, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16820.000000 2026-03-14 12:33:30,944 [Rank train_rank0] [Epoch 0 | Step 16840] train/loss: 0.468565, train/mse: 0.468565, train/data_wait_sec: 0.101683, train/step_wall_sec: 3.923662, train/data_wait_ratio: 0.025915, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16840.000000 2026-03-14 12:34:50,666 [Rank train_rank0] [Epoch 0 | Step 16860] train/loss: 0.466790, train/mse: 0.466790, train/data_wait_sec: 0.137237, train/step_wall_sec: 3.964646, train/data_wait_ratio: 0.034615, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16860.000000 2026-03-14 12:36:09,744 [Rank train_rank0] [Epoch 0 | Step 16880] train/loss: 0.462860, train/mse: 0.462860, train/data_wait_sec: 0.103009, train/step_wall_sec: 3.886658, train/data_wait_ratio: 0.026503, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16880.000000 2026-03-14 12:37:29,329 [Rank train_rank0] [Epoch 0 | Step 16900] train/loss: 0.462083, train/mse: 0.462083, train/data_wait_sec: 0.112509, train/step_wall_sec: 3.932922, train/data_wait_ratio: 0.028607, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16900.000000 2026-03-14 12:38:49,104 [Rank train_rank0] [Epoch 0 | Step 16920] train/loss: 0.476727, train/mse: 0.476727, train/data_wait_sec: 0.116861, train/step_wall_sec: 3.914750, train/data_wait_ratio: 0.029851, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16920.000000 2026-03-14 12:40:08,157 [Rank train_rank0] [Epoch 0 | Step 16940] train/loss: 0.445568, train/mse: 0.445568, train/data_wait_sec: 0.086101, train/step_wall_sec: 3.964729, train/data_wait_ratio: 0.021717, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16940.000000 2026-03-14 12:41:27,704 [Rank train_rank0] [Epoch 0 | Step 16960] train/loss: 0.449036, train/mse: 0.449036, train/data_wait_sec: 0.070972, train/step_wall_sec: 3.942281, train/data_wait_ratio: 0.018003, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16960.000000 2026-03-14 12:42:47,011 [Rank train_rank0] [Epoch 0 | Step 16980] train/loss: 0.465117, train/mse: 0.465117, train/data_wait_sec: 0.081083, train/step_wall_sec: 3.898007, train/data_wait_ratio: 0.020801, train/lr: 0.000200, train/epoch: 0.000000, train/step: 16980.000000 2026-03-14 12:44:31,011 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000017000.pt 2026-03-14 12:44:31,016 [Rank train_rank0] [Epoch 0 | Step 17000] train/loss: 0.422229, train/mse: 0.422229, train/data_wait_sec: 0.148288, train/step_wall_sec: 3.961275, train/data_wait_ratio: 0.037434, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17000.000000 2026-03-14 12:44:31,018 [Rank train_rank0] Generating samples for visualization... 2026-03-14 12:44:34,137 [Rank train_rank0] Sample generation done. 2026-03-14 12:45:53,717 [Rank train_rank0] [Epoch 0 | Step 17020] train/loss: 0.478784, train/mse: 0.478784, train/data_wait_sec: 0.108091, train/step_wall_sec: 3.926080, train/data_wait_ratio: 0.027532, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17020.000000 2026-03-14 12:47:13,116 [Rank train_rank0] [Epoch 0 | Step 17040] train/loss: 0.421475, train/mse: 0.421475, train/data_wait_sec: 0.159453, train/step_wall_sec: 3.993468, train/data_wait_ratio: 0.039929, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17040.000000 2026-03-14 12:48:32,794 [Rank train_rank0] [Epoch 0 | Step 17060] train/loss: 0.462569, train/mse: 0.462569, train/data_wait_sec: 0.129643, train/step_wall_sec: 3.932513, train/data_wait_ratio: 0.032967, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17060.000000 2026-03-14 12:49:52,138 [Rank train_rank0] [Epoch 0 | Step 17080] train/loss: 0.452376, train/mse: 0.452376, train/data_wait_sec: 0.160094, train/step_wall_sec: 3.947680, train/data_wait_ratio: 0.040554, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17080.000000 2026-03-14 12:51:11,334 [Rank train_rank0] [Epoch 0 | Step 17100] train/loss: 0.474331, train/mse: 0.474331, train/data_wait_sec: 0.109948, train/step_wall_sec: 3.902382, train/data_wait_ratio: 0.028175, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17100.000000 2026-03-14 12:52:30,807 [Rank train_rank0] [Epoch 0 | Step 17120] train/loss: 0.456829, train/mse: 0.456829, train/data_wait_sec: 0.142043, train/step_wall_sec: 3.985214, train/data_wait_ratio: 0.035643, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17120.000000 2026-03-14 12:53:51,000 [Rank train_rank0] [Epoch 0 | Step 17140] train/loss: 0.470788, train/mse: 0.470788, train/data_wait_sec: 0.171726, train/step_wall_sec: 4.065769, train/data_wait_ratio: 0.042237, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17140.000000 2026-03-14 12:55:10,370 [Rank train_rank0] [Epoch 0 | Step 17160] train/loss: 0.441425, train/mse: 0.441425, train/data_wait_sec: 0.081657, train/step_wall_sec: 3.952775, train/data_wait_ratio: 0.020658, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17160.000000 2026-03-14 12:56:29,759 [Rank train_rank0] [Epoch 0 | Step 17180] train/loss: 0.472313, train/mse: 0.472313, train/data_wait_sec: 0.114385, train/step_wall_sec: 3.903490, train/data_wait_ratio: 0.029303, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17180.000000 2026-03-14 12:57:49,446 [Rank train_rank0] [Epoch 0 | Step 17200] train/loss: 0.475116, train/mse: 0.475116, train/data_wait_sec: 0.130592, train/step_wall_sec: 3.926527, train/data_wait_ratio: 0.033259, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17200.000000 2026-03-14 12:59:08,757 [Rank train_rank0] [Epoch 0 | Step 17220] train/loss: 0.443860, train/mse: 0.443860, train/data_wait_sec: 0.093544, train/step_wall_sec: 3.847247, train/data_wait_ratio: 0.024315, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17220.000000 2026-03-14 13:00:28,293 [Rank train_rank0] [Epoch 0 | Step 17240] train/loss: 0.489191, train/mse: 0.489191, train/data_wait_sec: 0.133772, train/step_wall_sec: 3.993295, train/data_wait_ratio: 0.033499, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17240.000000 2026-03-14 13:01:47,788 [Rank train_rank0] [Epoch 0 | Step 17260] train/loss: 0.429291, train/mse: 0.429291, train/data_wait_sec: 0.144691, train/step_wall_sec: 3.904876, train/data_wait_ratio: 0.037054, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17260.000000 2026-03-14 13:03:07,060 [Rank train_rank0] [Epoch 0 | Step 17280] train/loss: 0.465245, train/mse: 0.465245, train/data_wait_sec: 0.086080, train/step_wall_sec: 3.896061, train/data_wait_ratio: 0.022094, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17280.000000 2026-03-14 13:04:26,459 [Rank train_rank0] [Epoch 0 | Step 17300] train/loss: 0.444404, train/mse: 0.444404, train/data_wait_sec: 0.184472, train/step_wall_sec: 4.001102, train/data_wait_ratio: 0.046105, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17300.000000 2026-03-14 13:05:45,889 [Rank train_rank0] [Epoch 0 | Step 17320] train/loss: 0.448128, train/mse: 0.448128, train/data_wait_sec: 0.108514, train/step_wall_sec: 3.896618, train/data_wait_ratio: 0.027848, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17320.000000 2026-03-14 13:07:04,722 [Rank train_rank0] [Epoch 0 | Step 17340] train/loss: 0.459749, train/mse: 0.459749, train/data_wait_sec: 0.186530, train/step_wall_sec: 4.103908, train/data_wait_ratio: 0.045452, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17340.000000 2026-03-14 13:08:24,186 [Rank train_rank0] [Epoch 0 | Step 17360] train/loss: 0.450024, train/mse: 0.450024, train/data_wait_sec: 0.170318, train/step_wall_sec: 3.935768, train/data_wait_ratio: 0.043274, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17360.000000 2026-03-14 13:09:43,295 [Rank train_rank0] [Epoch 0 | Step 17380] train/loss: 0.431690, train/mse: 0.431690, train/data_wait_sec: 0.106430, train/step_wall_sec: 3.902917, train/data_wait_ratio: 0.027269, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17380.000000 2026-03-14 13:11:02,580 [Rank train_rank0] [Epoch 0 | Step 17400] train/loss: 0.447643, train/mse: 0.447643, train/data_wait_sec: 0.137542, train/step_wall_sec: 3.966968, train/data_wait_ratio: 0.034672, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17400.000000 2026-03-14 13:12:22,645 [Rank train_rank0] [Epoch 0 | Step 17420] train/loss: 0.446611, train/mse: 0.446611, train/data_wait_sec: 0.122152, train/step_wall_sec: 3.908003, train/data_wait_ratio: 0.031257, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17420.000000 2026-03-14 13:13:41,592 [Rank train_rank0] [Epoch 0 | Step 17440] train/loss: 0.476473, train/mse: 0.476473, train/data_wait_sec: 0.113040, train/step_wall_sec: 3.942535, train/data_wait_ratio: 0.028672, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17440.000000 2026-03-14 13:15:00,931 [Rank train_rank0] [Epoch 0 | Step 17460] train/loss: 0.457809, train/mse: 0.457809, train/data_wait_sec: 0.109703, train/step_wall_sec: 3.890456, train/data_wait_ratio: 0.028198, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17460.000000 2026-03-14 13:16:20,567 [Rank train_rank0] [Epoch 0 | Step 17480] train/loss: 0.443087, train/mse: 0.443087, train/data_wait_sec: 0.151274, train/step_wall_sec: 3.932014, train/data_wait_ratio: 0.038472, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17480.000000 2026-03-14 13:17:39,943 [Rank train_rank0] [Epoch 0 | Step 17500] train/loss: 0.471170, train/mse: 0.471170, train/data_wait_sec: 0.077057, train/step_wall_sec: 3.945168, train/data_wait_ratio: 0.019532, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17500.000000 2026-03-14 13:18:59,192 [Rank train_rank0] [Epoch 0 | Step 17520] train/loss: 0.485883, train/mse: 0.485883, train/data_wait_sec: 0.132486, train/step_wall_sec: 3.976318, train/data_wait_ratio: 0.033319, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17520.000000 2026-03-14 13:20:18,623 [Rank train_rank0] [Epoch 0 | Step 17540] train/loss: 0.461021, train/mse: 0.461021, train/data_wait_sec: 0.111711, train/step_wall_sec: 3.967303, train/data_wait_ratio: 0.028158, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17540.000000 2026-03-14 13:21:37,821 [Rank train_rank0] [Epoch 0 | Step 17560] train/loss: 0.483622, train/mse: 0.483622, train/data_wait_sec: 0.098265, train/step_wall_sec: 3.945147, train/data_wait_ratio: 0.024908, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17560.000000 2026-03-14 13:22:57,467 [Rank train_rank0] [Epoch 0 | Step 17580] train/loss: 0.459462, train/mse: 0.459462, train/data_wait_sec: 0.097023, train/step_wall_sec: 3.952565, train/data_wait_ratio: 0.024547, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17580.000000 2026-03-14 13:24:16,847 [Rank train_rank0] [Epoch 0 | Step 17600] train/loss: 0.490355, train/mse: 0.490355, train/data_wait_sec: 0.118536, train/step_wall_sec: 3.949851, train/data_wait_ratio: 0.030010, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17600.000000 2026-03-14 13:25:36,203 [Rank train_rank0] [Epoch 0 | Step 17620] train/loss: 0.449131, train/mse: 0.449131, train/data_wait_sec: 0.180953, train/step_wall_sec: 3.954515, train/data_wait_ratio: 0.045758, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17620.000000 2026-03-14 13:26:55,469 [Rank train_rank0] [Epoch 0 | Step 17640] train/loss: 0.446819, train/mse: 0.446819, train/data_wait_sec: 0.120144, train/step_wall_sec: 3.913316, train/data_wait_ratio: 0.030701, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17640.000000 2026-03-14 13:28:15,022 [Rank train_rank0] [Epoch 0 | Step 17660] train/loss: 0.463221, train/mse: 0.463221, train/data_wait_sec: 0.119734, train/step_wall_sec: 3.911509, train/data_wait_ratio: 0.030611, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17660.000000 2026-03-14 13:29:34,272 [Rank train_rank0] [Epoch 0 | Step 17680] train/loss: 0.444758, train/mse: 0.444758, train/data_wait_sec: 0.127365, train/step_wall_sec: 3.991025, train/data_wait_ratio: 0.031913, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17680.000000 2026-03-14 13:30:53,699 [Rank train_rank0] [Epoch 0 | Step 17700] train/loss: 0.475494, train/mse: 0.475494, train/data_wait_sec: 0.103613, train/step_wall_sec: 3.874484, train/data_wait_ratio: 0.026743, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17700.000000 2026-03-14 13:32:13,169 [Rank train_rank0] [Epoch 0 | Step 17720] train/loss: 0.470371, train/mse: 0.470371, train/data_wait_sec: 0.101523, train/step_wall_sec: 3.914012, train/data_wait_ratio: 0.025938, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17720.000000 2026-03-14 13:33:32,505 [Rank train_rank0] [Epoch 0 | Step 17740] train/loss: 0.462809, train/mse: 0.462809, train/data_wait_sec: 0.118046, train/step_wall_sec: 3.940580, train/data_wait_ratio: 0.029956, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17740.000000 2026-03-14 13:34:51,655 [Rank train_rank0] [Epoch 0 | Step 17760] train/loss: 0.485261, train/mse: 0.485261, train/data_wait_sec: 0.114432, train/step_wall_sec: 3.927951, train/data_wait_ratio: 0.029133, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17760.000000 2026-03-14 13:36:11,310 [Rank train_rank0] [Epoch 0 | Step 17780] train/loss: 0.484215, train/mse: 0.484215, train/data_wait_sec: 0.112775, train/step_wall_sec: 3.906857, train/data_wait_ratio: 0.028866, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17780.000000 2026-03-14 13:37:30,430 [Rank train_rank0] [Epoch 0 | Step 17800] train/loss: 0.465933, train/mse: 0.465933, train/data_wait_sec: 0.120271, train/step_wall_sec: 3.924759, train/data_wait_ratio: 0.030644, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17800.000000 2026-03-14 13:38:50,159 [Rank train_rank0] [Epoch 0 | Step 17820] train/loss: 0.471316, train/mse: 0.471316, train/data_wait_sec: 0.158295, train/step_wall_sec: 3.930814, train/data_wait_ratio: 0.040270, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17820.000000 2026-03-14 13:40:09,690 [Rank train_rank0] [Epoch 0 | Step 17840] train/loss: 0.437409, train/mse: 0.437409, train/data_wait_sec: 0.136187, train/step_wall_sec: 4.009332, train/data_wait_ratio: 0.033968, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17840.000000 2026-03-14 13:41:28,776 [Rank train_rank0] [Epoch 0 | Step 17860] train/loss: 0.442419, train/mse: 0.442419, train/data_wait_sec: 0.140683, train/step_wall_sec: 3.960568, train/data_wait_ratio: 0.035521, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17860.000000 2026-03-14 13:42:47,916 [Rank train_rank0] [Epoch 0 | Step 17880] train/loss: 0.456848, train/mse: 0.456848, train/data_wait_sec: 0.087787, train/step_wall_sec: 3.922202, train/data_wait_ratio: 0.022382, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17880.000000 2026-03-14 13:44:06,981 [Rank train_rank0] [Epoch 0 | Step 17900] train/loss: 0.453438, train/mse: 0.453438, train/data_wait_sec: 0.090999, train/step_wall_sec: 3.915697, train/data_wait_ratio: 0.023240, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17900.000000 2026-03-14 13:45:26,121 [Rank train_rank0] [Epoch 0 | Step 17920] train/loss: 0.472404, train/mse: 0.472404, train/data_wait_sec: 0.108145, train/step_wall_sec: 3.906704, train/data_wait_ratio: 0.027682, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17920.000000 2026-03-14 13:46:45,219 [Rank train_rank0] [Epoch 0 | Step 17940] train/loss: 0.467173, train/mse: 0.467173, train/data_wait_sec: 0.113162, train/step_wall_sec: 3.887413, train/data_wait_ratio: 0.029110, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17940.000000 2026-03-14 13:48:04,838 [Rank train_rank0] [Epoch 0 | Step 17960] train/loss: 0.439955, train/mse: 0.439955, train/data_wait_sec: 0.094449, train/step_wall_sec: 4.165392, train/data_wait_ratio: 0.022675, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17960.000000 2026-03-14 13:49:58,065 [Rank train_rank0] [Epoch 0 | Step 17980] train/loss: 0.463636, train/mse: 0.463636, train/data_wait_sec: 0.215517, train/step_wall_sec: 5.735522, train/data_wait_ratio: 0.037576, train/lr: 0.000200, train/epoch: 0.000000, train/step: 17980.000000 2026-03-14 13:52:19,404 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000018000.pt 2026-03-14 13:52:19,407 [Rank train_rank0] [Epoch 0 | Step 18000] train/loss: 0.407720, train/mse: 0.407720, train/data_wait_sec: 0.398847, train/step_wall_sec: 5.879138, train/data_wait_ratio: 0.067841, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18000.000000 2026-03-14 13:52:19,409 [Rank train_rank0] Generating samples for visualization... 2026-03-14 13:52:22,414 [Rank train_rank0] Sample generation done. 2026-03-14 13:54:20,559 [Rank train_rank0] [Epoch 0 | Step 18020] train/loss: 0.474552, train/mse: 0.474552, train/data_wait_sec: 0.325626, train/step_wall_sec: 5.819891, train/data_wait_ratio: 0.055950, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18020.000000 2026-03-14 13:56:18,635 [Rank train_rank0] [Epoch 0 | Step 18040] train/loss: 0.465106, train/mse: 0.465106, train/data_wait_sec: 0.391275, train/step_wall_sec: 5.820119, train/data_wait_ratio: 0.067228, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18040.000000 2026-03-14 14:02:52,887 [Rank train_rank0] [Epoch 0 | Step 18060] train/loss: 0.458162, train/mse: 0.458162, train/data_wait_sec: 1.068865, train/step_wall_sec: 24.993305, train/data_wait_ratio: 0.042766, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18060.000000 2026-03-14 14:15:37,156 [Rank train_rank0] Discarding a locally fetched batch to keep DDP ranks synchronized after a peer streaming failure. 2026-03-14 14:18:38,968 [Rank train_rank0] Recoverable streaming error (1/32): RuntimeError: peer rank reported a recoverable streaming failure. Skipping this batch without rebuilding (threshold=6). 2026-03-14 14:18:39,012 [Rank train_rank0] Building stream loader (stream_epoch=1)... 2026-03-14 14:18:39,095 [Rank train_rank0] Creating stream iterator... 2026-03-14 14:18:41,631 [Rank train_rank0] Stream iterator is ready. 2026-03-14 14:18:56,262 [Rank train_rank0] Streaming dataloader recovered after 1 retry attempt(s). 2026-03-14 14:25:37,327 [Rank train_rank0] [Epoch 0 | Step 18080] train/loss: 0.429478, train/mse: 0.429478, train/data_wait_sec: 1.067594, train/step_wall_sec: 25.953121, train/data_wait_ratio: 0.041135, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18080.000000 2026-03-14 14:34:22,562 [Rank train_rank0] [Epoch 0 | Step 18100] train/loss: 0.458051, train/mse: 0.458051, train/data_wait_sec: 1.531388, train/step_wall_sec: 25.820583, train/data_wait_ratio: 0.059309, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18100.000000 2026-03-14 14:43:10,071 [Rank train_rank0] [Epoch 0 | Step 18120] train/loss: 0.443574, train/mse: 0.443574, train/data_wait_sec: 1.306713, train/step_wall_sec: 26.567509, train/data_wait_ratio: 0.049185, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18120.000000 2026-03-14 14:51:53,320 [Rank train_rank0] [Epoch 0 | Step 18140] train/loss: 0.475263, train/mse: 0.475263, train/data_wait_sec: 1.860236, train/step_wall_sec: 26.526707, train/data_wait_ratio: 0.070127, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18140.000000 2026-03-14 15:00:36,836 [Rank train_rank0] [Epoch 0 | Step 18160] train/loss: 0.449087, train/mse: 0.449087, train/data_wait_sec: 1.811251, train/step_wall_sec: 26.383099, train/data_wait_ratio: 0.068652, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18160.000000 2026-03-14 15:09:21,677 [Rank train_rank0] [Epoch 0 | Step 18180] train/loss: 0.478961, train/mse: 0.478961, train/data_wait_sec: 1.681186, train/step_wall_sec: 26.969368, train/data_wait_ratio: 0.062337, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18180.000000 2026-03-14 15:18:07,681 [Rank train_rank0] [Epoch 0 | Step 18200] train/loss: 0.456762, train/mse: 0.456762, train/data_wait_sec: 2.170518, train/step_wall_sec: 27.362070, train/data_wait_ratio: 0.079326, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18200.000000 2026-03-14 15:26:53,414 [Rank train_rank0] [Epoch 0 | Step 18220] train/loss: 0.476594, train/mse: 0.476594, train/data_wait_sec: 1.629286, train/step_wall_sec: 26.244101, train/data_wait_ratio: 0.062082, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18220.000000 2026-03-14 15:35:37,948 [Rank train_rank0] [Epoch 0 | Step 18240] train/loss: 0.473955, train/mse: 0.473955, train/data_wait_sec: 2.330287, train/step_wall_sec: 27.205921, train/data_wait_ratio: 0.085654, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18240.000000 2026-03-14 15:44:23,947 [Rank train_rank0] [Epoch 0 | Step 18260] train/loss: 0.446622, train/mse: 0.446622, train/data_wait_sec: 0.903538, train/step_wall_sec: 25.651709, train/data_wait_ratio: 0.035223, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18260.000000 2026-03-14 15:53:10,139 [Rank train_rank0] [Epoch 0 | Step 18280] train/loss: 0.492960, train/mse: 0.492960, train/data_wait_sec: 1.682460, train/step_wall_sec: 26.045952, train/data_wait_ratio: 0.064596, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18280.000000 2026-03-14 16:01:53,828 [Rank train_rank0] [Epoch 0 | Step 18300] train/loss: 0.459203, train/mse: 0.459203, train/data_wait_sec: 1.139274, train/step_wall_sec: 26.223068, train/data_wait_ratio: 0.043445, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18300.000000 2026-03-14 16:10:36,152 [Rank train_rank0] [Epoch 0 | Step 18320] train/loss: 0.463344, train/mse: 0.463344, train/data_wait_sec: 1.691808, train/step_wall_sec: 25.991482, train/data_wait_ratio: 0.065091, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18320.000000 2026-03-14 16:20:31,432 [Rank train_rank0] [Epoch 0 | Step 18340] train/loss: 0.430154, train/mse: 0.430154, train/data_wait_sec: 1.174521, train/step_wall_sec: 25.366745, train/data_wait_ratio: 0.046302, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18340.000000 2026-03-14 16:29:14,127 [Rank train_rank0] [Epoch 0 | Step 18360] train/loss: 0.456125, train/mse: 0.456125, train/data_wait_sec: 1.210848, train/step_wall_sec: 26.067175, train/data_wait_ratio: 0.046451, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18360.000000 2026-03-14 16:37:53,898 [Rank train_rank0] [Epoch 0 | Step 18380] train/loss: 0.403590, train/mse: 0.403590, train/data_wait_sec: 1.750041, train/step_wall_sec: 26.445768, train/data_wait_ratio: 0.066175, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18380.000000 2026-03-14 16:46:36,460 [Rank train_rank0] [Epoch 0 | Step 18400] train/loss: 0.453101, train/mse: 0.453101, train/data_wait_sec: 1.491199, train/step_wall_sec: 26.262142, train/data_wait_ratio: 0.056781, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18400.000000 2026-03-14 16:55:20,460 [Rank train_rank0] [Epoch 0 | Step 18420] train/loss: 0.485599, train/mse: 0.485599, train/data_wait_sec: 1.250109, train/step_wall_sec: 26.305305, train/data_wait_ratio: 0.047523, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18420.000000 2026-03-14 17:04:02,859 [Rank train_rank0] [Epoch 0 | Step 18440] train/loss: 0.464017, train/mse: 0.464017, train/data_wait_sec: 1.367677, train/step_wall_sec: 26.183472, train/data_wait_ratio: 0.052234, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18440.000000 2026-03-14 17:12:45,181 [Rank train_rank0] [Epoch 0 | Step 18460] train/loss: 0.451065, train/mse: 0.451065, train/data_wait_sec: 1.591587, train/step_wall_sec: 26.285816, train/data_wait_ratio: 0.060549, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18460.000000 2026-03-14 17:21:30,205 [Rank train_rank0] [Epoch 0 | Step 18480] train/loss: 0.461322, train/mse: 0.461322, train/data_wait_sec: 1.277678, train/step_wall_sec: 25.964539, train/data_wait_ratio: 0.049209, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18480.000000 2026-03-14 17:30:14,096 [Rank train_rank0] [Epoch 0 | Step 18500] train/loss: 0.465117, train/mse: 0.465117, train/data_wait_sec: 1.625514, train/step_wall_sec: 26.169613, train/data_wait_ratio: 0.062115, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18500.000000 2026-03-14 17:38:56,773 [Rank train_rank0] [Epoch 0 | Step 18520] train/loss: 0.476010, train/mse: 0.476010, train/data_wait_sec: 1.942698, train/step_wall_sec: 26.404938, train/data_wait_ratio: 0.073573, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18520.000000 2026-03-14 17:47:41,525 [Rank train_rank0] [Epoch 0 | Step 18540] train/loss: 0.462258, train/mse: 0.462258, train/data_wait_sec: 1.777219, train/step_wall_sec: 26.281481, train/data_wait_ratio: 0.067622, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18540.000000 2026-03-14 17:56:22,255 [Rank train_rank0] [Epoch 0 | Step 18560] train/loss: 0.414627, train/mse: 0.414627, train/data_wait_sec: 1.709926, train/step_wall_sec: 26.466829, train/data_wait_ratio: 0.064606, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18560.000000 2026-03-14 18:05:05,477 [Rank train_rank0] [Epoch 0 | Step 18580] train/loss: 0.454938, train/mse: 0.454938, train/data_wait_sec: 1.527164, train/step_wall_sec: 25.524326, train/data_wait_ratio: 0.059832, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18580.000000 2026-03-14 18:13:53,005 [Rank train_rank0] [Epoch 0 | Step 18600] train/loss: 0.465032, train/mse: 0.465032, train/data_wait_sec: 1.483110, train/step_wall_sec: 26.144157, train/data_wait_ratio: 0.056728, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18600.000000 2026-03-14 18:22:36,008 [Rank train_rank0] [Epoch 0 | Step 18620] train/loss: 0.463473, train/mse: 0.463473, train/data_wait_sec: 1.563941, train/step_wall_sec: 26.001184, train/data_wait_ratio: 0.060149, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18620.000000 2026-03-14 18:31:20,317 [Rank train_rank0] [Epoch 0 | Step 18640] train/loss: 0.447125, train/mse: 0.447125, train/data_wait_sec: 1.314948, train/step_wall_sec: 26.205795, train/data_wait_ratio: 0.050178, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18640.000000 2026-03-14 18:40:00,862 [Rank train_rank0] [Epoch 0 | Step 18660] train/loss: 0.474077, train/mse: 0.474077, train/data_wait_sec: 1.253180, train/step_wall_sec: 25.921370, train/data_wait_ratio: 0.048345, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18660.000000 2026-03-14 18:48:40,757 [Rank train_rank0] [Epoch 0 | Step 18680] train/loss: 0.458395, train/mse: 0.458395, train/data_wait_sec: 0.980620, train/step_wall_sec: 25.930405, train/data_wait_ratio: 0.037817, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18680.000000 2026-03-14 18:57:25,030 [Rank train_rank0] [Epoch 0 | Step 18700] train/loss: 0.475781, train/mse: 0.475781, train/data_wait_sec: 1.736866, train/step_wall_sec: 26.921371, train/data_wait_ratio: 0.064516, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18700.000000 2026-03-14 19:06:04,536 [Rank train_rank0] [Epoch 0 | Step 18720] train/loss: 0.446959, train/mse: 0.446959, train/data_wait_sec: 1.860829, train/step_wall_sec: 26.139393, train/data_wait_ratio: 0.071189, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18720.000000 2026-03-14 19:14:45,406 [Rank train_rank0] [Epoch 0 | Step 18740] train/loss: 0.491294, train/mse: 0.491294, train/data_wait_sec: 1.316447, train/step_wall_sec: 25.351166, train/data_wait_ratio: 0.051928, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18740.000000 2026-03-14 19:23:28,495 [Rank train_rank0] [Epoch 0 | Step 18760] train/loss: 0.492271, train/mse: 0.492271, train/data_wait_sec: 1.835199, train/step_wall_sec: 26.575499, train/data_wait_ratio: 0.069056, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18760.000000 2026-03-14 19:32:11,411 [Rank train_rank0] [Epoch 0 | Step 18780] train/loss: 0.451284, train/mse: 0.451284, train/data_wait_sec: 1.636948, train/step_wall_sec: 25.911909, train/data_wait_ratio: 0.063174, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18780.000000 2026-03-14 19:40:52,210 [Rank train_rank0] [Epoch 0 | Step 18800] train/loss: 0.479609, train/mse: 0.479609, train/data_wait_sec: 2.330031, train/step_wall_sec: 27.262995, train/data_wait_ratio: 0.085465, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18800.000000 2026-03-14 19:49:30,550 [Rank train_rank0] [Epoch 0 | Step 18820] train/loss: 0.460591, train/mse: 0.460591, train/data_wait_sec: 1.111250, train/step_wall_sec: 25.518967, train/data_wait_ratio: 0.043546, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18820.000000 2026-03-14 19:58:08,522 [Rank train_rank0] [Epoch 0 | Step 18840] train/loss: 0.452264, train/mse: 0.452264, train/data_wait_sec: 0.857105, train/step_wall_sec: 25.400555, train/data_wait_ratio: 0.033744, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18840.000000 2026-03-14 20:06:54,722 [Rank train_rank0] [Epoch 0 | Step 18860] train/loss: 0.465318, train/mse: 0.465318, train/data_wait_sec: 1.599133, train/step_wall_sec: 25.816494, train/data_wait_ratio: 0.061942, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18860.000000 2026-03-14 20:15:31,546 [Rank train_rank0] [Epoch 0 | Step 18880] train/loss: 0.462953, train/mse: 0.462953, train/data_wait_sec: 1.197191, train/step_wall_sec: 25.443018, train/data_wait_ratio: 0.047054, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18880.000000 2026-03-14 20:24:11,852 [Rank train_rank0] [Epoch 0 | Step 18900] train/loss: 0.461769, train/mse: 0.461769, train/data_wait_sec: 1.395581, train/step_wall_sec: 26.066769, train/data_wait_ratio: 0.053539, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18900.000000 2026-03-14 20:32:52,378 [Rank train_rank0] [Epoch 0 | Step 18920] train/loss: 0.453228, train/mse: 0.453228, train/data_wait_sec: 1.506794, train/step_wall_sec: 26.000183, train/data_wait_ratio: 0.057953, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18920.000000 2026-03-14 20:41:31,478 [Rank train_rank0] [Epoch 0 | Step 18940] train/loss: 0.433566, train/mse: 0.433566, train/data_wait_sec: 1.026688, train/step_wall_sec: 25.190018, train/data_wait_ratio: 0.040758, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18940.000000 2026-03-14 20:50:11,051 [Rank train_rank0] [Epoch 0 | Step 18960] train/loss: 0.439609, train/mse: 0.439609, train/data_wait_sec: 1.907814, train/step_wall_sec: 26.317463, train/data_wait_ratio: 0.072492, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18960.000000 2026-03-14 20:58:53,166 [Rank train_rank0] [Epoch 0 | Step 18980] train/loss: 0.468367, train/mse: 0.468367, train/data_wait_sec: 1.444628, train/step_wall_sec: 26.626003, train/data_wait_ratio: 0.054256, train/lr: 0.000200, train/epoch: 0.000000, train/step: 18980.000000 2026-03-14 21:10:36,814 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000019000.pt 2026-03-14 21:10:36,838 [Rank train_rank0] [Epoch 0 | Step 19000] train/loss: 0.467450, train/mse: 0.467450, train/data_wait_sec: 1.144922, train/step_wall_sec: 26.038431, train/data_wait_ratio: 0.043970, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19000.000000 2026-03-14 21:10:36,849 [Rank train_rank0] Generating samples for visualization... 2026-03-14 21:11:10,419 [Rank train_rank0] Sample generation done. 2026-03-14 21:19:53,493 [Rank train_rank0] [Epoch 0 | Step 19020] train/loss: 0.451368, train/mse: 0.451368, train/data_wait_sec: 1.812415, train/step_wall_sec: 25.917583, train/data_wait_ratio: 0.069930, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19020.000000 2026-03-14 21:28:36,686 [Rank train_rank0] [Epoch 0 | Step 19040] train/loss: 0.459959, train/mse: 0.459959, train/data_wait_sec: 1.695511, train/step_wall_sec: 25.839037, train/data_wait_ratio: 0.065618, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19040.000000 2026-03-14 21:37:20,449 [Rank train_rank0] [Epoch 0 | Step 19060] train/loss: 0.478534, train/mse: 0.478534, train/data_wait_sec: 1.403609, train/step_wall_sec: 25.861423, train/data_wait_ratio: 0.054274, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19060.000000 2026-03-14 21:45:59,426 [Rank train_rank0] [Epoch 0 | Step 19080] train/loss: 0.475282, train/mse: 0.475282, train/data_wait_sec: 1.192158, train/step_wall_sec: 26.530439, train/data_wait_ratio: 0.044935, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19080.000000 2026-03-14 21:54:39,891 [Rank train_rank0] [Epoch 0 | Step 19100] train/loss: 0.479449, train/mse: 0.479449, train/data_wait_sec: 1.552511, train/step_wall_sec: 26.396122, train/data_wait_ratio: 0.058816, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19100.000000 2026-03-14 22:03:21,168 [Rank train_rank0] [Epoch 0 | Step 19120] train/loss: 0.445783, train/mse: 0.445783, train/data_wait_sec: 1.073063, train/step_wall_sec: 26.001093, train/data_wait_ratio: 0.041270, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19120.000000 2026-03-14 22:12:04,368 [Rank train_rank0] [Epoch 0 | Step 19140] train/loss: 0.465353, train/mse: 0.465353, train/data_wait_sec: 1.886448, train/step_wall_sec: 26.614368, train/data_wait_ratio: 0.070881, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19140.000000 2026-03-14 22:20:49,989 [Rank train_rank0] [Epoch 0 | Step 19160] train/loss: 0.496591, train/mse: 0.496591, train/data_wait_sec: 1.399116, train/step_wall_sec: 27.246605, train/data_wait_ratio: 0.051350, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19160.000000 2026-03-14 22:29:29,130 [Rank train_rank0] [Epoch 0 | Step 19180] train/loss: 0.435516, train/mse: 0.435516, train/data_wait_sec: 1.193408, train/step_wall_sec: 25.944429, train/data_wait_ratio: 0.045999, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19180.000000 2026-03-14 22:38:07,237 [Rank train_rank0] [Epoch 0 | Step 19200] train/loss: 0.465175, train/mse: 0.465175, train/data_wait_sec: 1.553661, train/step_wall_sec: 26.140804, train/data_wait_ratio: 0.059434, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19200.000000 2026-03-14 22:46:47,890 [Rank train_rank0] [Epoch 0 | Step 19220] train/loss: 0.451130, train/mse: 0.451130, train/data_wait_sec: 1.778793, train/step_wall_sec: 27.041092, train/data_wait_ratio: 0.065781, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19220.000000 2026-03-14 22:55:24,863 [Rank train_rank0] [Epoch 0 | Step 19240] train/loss: 0.436274, train/mse: 0.436274, train/data_wait_sec: 1.032099, train/step_wall_sec: 25.924925, train/data_wait_ratio: 0.039811, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19240.000000 2026-03-14 23:04:03,660 [Rank train_rank0] [Epoch 0 | Step 19260] train/loss: 0.465532, train/mse: 0.465532, train/data_wait_sec: 1.686114, train/step_wall_sec: 26.631130, train/data_wait_ratio: 0.063314, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19260.000000 2026-03-14 23:12:47,705 [Rank train_rank0] [Epoch 0 | Step 19280] train/loss: 0.463575, train/mse: 0.463575, train/data_wait_sec: 1.518730, train/step_wall_sec: 25.286610, train/data_wait_ratio: 0.060061, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19280.000000 2026-03-14 23:21:29,571 [Rank train_rank0] [Epoch 0 | Step 19300] train/loss: 0.448052, train/mse: 0.448052, train/data_wait_sec: 1.263183, train/step_wall_sec: 25.953274, train/data_wait_ratio: 0.048671, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19300.000000 2026-03-14 23:30:10,646 [Rank train_rank0] [Epoch 0 | Step 19320] train/loss: 0.446693, train/mse: 0.446693, train/data_wait_sec: 1.148720, train/step_wall_sec: 25.873104, train/data_wait_ratio: 0.044398, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19320.000000 2026-03-14 23:38:54,001 [Rank train_rank0] [Epoch 0 | Step 19340] train/loss: 0.456874, train/mse: 0.456874, train/data_wait_sec: 1.642748, train/step_wall_sec: 25.534689, train/data_wait_ratio: 0.064334, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19340.000000 2026-03-14 23:47:37,728 [Rank train_rank0] [Epoch 0 | Step 19360] train/loss: 0.472376, train/mse: 0.472376, train/data_wait_sec: 1.458843, train/step_wall_sec: 26.345306, train/data_wait_ratio: 0.055374, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19360.000000 2026-03-14 23:56:16,814 [Rank train_rank0] [Epoch 0 | Step 19380] train/loss: 0.459817, train/mse: 0.459817, train/data_wait_sec: 1.063411, train/step_wall_sec: 25.559866, train/data_wait_ratio: 0.041605, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19380.000000 2026-03-15 00:04:59,672 [Rank train_rank0] [Epoch 0 | Step 19400] train/loss: 0.465447, train/mse: 0.465447, train/data_wait_sec: 1.125988, train/step_wall_sec: 26.331009, train/data_wait_ratio: 0.042763, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19400.000000 2026-03-15 00:13:48,397 [Rank train_rank0] [Epoch 0 | Step 19420] train/loss: 0.462109, train/mse: 0.462109, train/data_wait_sec: 1.416269, train/step_wall_sec: 27.449961, train/data_wait_ratio: 0.051595, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19420.000000 2026-03-15 00:22:30,579 [Rank train_rank0] [Epoch 0 | Step 19440] train/loss: 0.471002, train/mse: 0.471002, train/data_wait_sec: 1.568928, train/step_wall_sec: 25.882355, train/data_wait_ratio: 0.060618, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19440.000000 2026-03-15 00:31:14,799 [Rank train_rank0] [Epoch 0 | Step 19460] train/loss: 0.472202, train/mse: 0.472202, train/data_wait_sec: 1.271247, train/step_wall_sec: 25.909904, train/data_wait_ratio: 0.049064, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19460.000000 2026-03-15 00:39:56,761 [Rank train_rank0] [Epoch 0 | Step 19480] train/loss: 0.466960, train/mse: 0.466960, train/data_wait_sec: 2.318933, train/step_wall_sec: 27.114401, train/data_wait_ratio: 0.085524, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19480.000000 2026-03-15 00:48:40,690 [Rank train_rank0] [Epoch 0 | Step 19500] train/loss: 0.457030, train/mse: 0.457030, train/data_wait_sec: 1.186599, train/step_wall_sec: 26.196259, train/data_wait_ratio: 0.045296, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19500.000000 2026-03-15 00:57:19,528 [Rank train_rank0] [Epoch 0 | Step 19520] train/loss: 0.464185, train/mse: 0.464185, train/data_wait_sec: 1.117402, train/step_wall_sec: 25.807901, train/data_wait_ratio: 0.043297, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19520.000000 2026-03-15 01:06:01,299 [Rank train_rank0] [Epoch 0 | Step 19540] train/loss: 0.439938, train/mse: 0.439938, train/data_wait_sec: 1.398155, train/step_wall_sec: 25.851009, train/data_wait_ratio: 0.054085, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19540.000000 2026-03-15 01:14:44,935 [Rank train_rank0] [Epoch 0 | Step 19560] train/loss: 0.452535, train/mse: 0.452535, train/data_wait_sec: 1.176310, train/step_wall_sec: 26.155787, train/data_wait_ratio: 0.044973, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19560.000000 2026-03-15 01:23:28,529 [Rank train_rank0] [Epoch 0 | Step 19580] train/loss: 0.444531, train/mse: 0.444531, train/data_wait_sec: 1.555821, train/step_wall_sec: 26.201069, train/data_wait_ratio: 0.059380, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19580.000000 2026-03-15 01:32:11,060 [Rank train_rank0] [Epoch 0 | Step 19600] train/loss: 0.450804, train/mse: 0.450804, train/data_wait_sec: 1.066376, train/step_wall_sec: 25.605053, train/data_wait_ratio: 0.041647, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19600.000000 2026-03-15 01:40:51,461 [Rank train_rank0] [Epoch 0 | Step 19620] train/loss: 0.439751, train/mse: 0.439751, train/data_wait_sec: 1.234804, train/step_wall_sec: 25.614990, train/data_wait_ratio: 0.048206, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19620.000000 2026-03-15 01:49:32,994 [Rank train_rank0] [Epoch 0 | Step 19640] train/loss: 0.467645, train/mse: 0.467645, train/data_wait_sec: 1.438854, train/step_wall_sec: 26.327671, train/data_wait_ratio: 0.054652, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19640.000000 2026-03-15 01:58:17,549 [Rank train_rank0] [Epoch 0 | Step 19660] train/loss: 0.451899, train/mse: 0.451899, train/data_wait_sec: 1.185904, train/step_wall_sec: 26.029612, train/data_wait_ratio: 0.045560, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19660.000000 2026-03-15 02:07:00,754 [Rank train_rank0] [Epoch 0 | Step 19680] train/loss: 0.454875, train/mse: 0.454875, train/data_wait_sec: 1.752254, train/step_wall_sec: 26.190941, train/data_wait_ratio: 0.066903, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19680.000000 2026-03-15 02:15:42,817 [Rank train_rank0] [Epoch 0 | Step 19700] train/loss: 0.463533, train/mse: 0.463533, train/data_wait_sec: 0.965038, train/step_wall_sec: 26.315708, train/data_wait_ratio: 0.036672, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19700.000000 2026-03-15 02:24:27,389 [Rank train_rank0] [Epoch 0 | Step 19720] train/loss: 0.450412, train/mse: 0.450412, train/data_wait_sec: 1.671290, train/step_wall_sec: 26.360682, train/data_wait_ratio: 0.063401, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19720.000000 2026-03-15 02:33:08,655 [Rank train_rank0] [Epoch 0 | Step 19740] train/loss: 0.465350, train/mse: 0.465350, train/data_wait_sec: 1.385359, train/step_wall_sec: 26.666809, train/data_wait_ratio: 0.051951, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19740.000000 2026-03-15 02:41:48,439 [Rank train_rank0] [Epoch 0 | Step 19760] train/loss: 0.446956, train/mse: 0.446956, train/data_wait_sec: 0.922462, train/step_wall_sec: 25.651852, train/data_wait_ratio: 0.035961, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19760.000000 2026-03-15 02:50:26,181 [Rank train_rank0] [Epoch 0 | Step 19780] train/loss: 0.471463, train/mse: 0.471463, train/data_wait_sec: 1.268906, train/step_wall_sec: 26.461311, train/data_wait_ratio: 0.047953, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19780.000000 2026-03-15 02:59:07,679 [Rank train_rank0] [Epoch 0 | Step 19800] train/loss: 0.450882, train/mse: 0.450882, train/data_wait_sec: 1.271987, train/step_wall_sec: 25.392014, train/data_wait_ratio: 0.050094, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19800.000000 2026-03-15 03:07:51,943 [Rank train_rank0] [Epoch 0 | Step 19820] train/loss: 0.475958, train/mse: 0.475958, train/data_wait_sec: 1.863113, train/step_wall_sec: 25.932825, train/data_wait_ratio: 0.071844, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19820.000000 2026-03-15 03:16:37,157 [Rank train_rank0] [Epoch 0 | Step 19840] train/loss: 0.440997, train/mse: 0.440997, train/data_wait_sec: 0.926814, train/step_wall_sec: 25.536945, train/data_wait_ratio: 0.036293, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19840.000000 2026-03-15 03:25:17,094 [Rank train_rank0] [Epoch 0 | Step 19860] train/loss: 0.456655, train/mse: 0.456655, train/data_wait_sec: 1.421679, train/step_wall_sec: 24.947889, train/data_wait_ratio: 0.056986, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19860.000000 2026-03-15 03:34:01,039 [Rank train_rank0] [Epoch 0 | Step 19880] train/loss: 0.434780, train/mse: 0.434780, train/data_wait_sec: 1.527480, train/step_wall_sec: 26.227741, train/data_wait_ratio: 0.058239, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19880.000000 2026-03-15 03:42:44,294 [Rank train_rank0] [Epoch 0 | Step 19900] train/loss: 0.467562, train/mse: 0.467562, train/data_wait_sec: 1.068149, train/step_wall_sec: 25.614147, train/data_wait_ratio: 0.041702, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19900.000000 2026-03-15 03:51:23,729 [Rank train_rank0] [Epoch 0 | Step 19920] train/loss: 0.466410, train/mse: 0.466410, train/data_wait_sec: 1.753970, train/step_wall_sec: 26.039036, train/data_wait_ratio: 0.067359, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19920.000000 2026-03-15 04:00:07,432 [Rank train_rank0] [Epoch 0 | Step 19940] train/loss: 0.468611, train/mse: 0.468611, train/data_wait_sec: 1.513757, train/step_wall_sec: 25.847778, train/data_wait_ratio: 0.058564, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19940.000000 2026-03-15 04:08:48,693 [Rank train_rank0] [Epoch 0 | Step 19960] train/loss: 0.454133, train/mse: 0.454133, train/data_wait_sec: 1.787841, train/step_wall_sec: 25.544857, train/data_wait_ratio: 0.069988, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19960.000000 2026-03-15 04:17:28,229 [Rank train_rank0] [Epoch 0 | Step 19980] train/loss: 0.430847, train/mse: 0.430847, train/data_wait_sec: 1.445336, train/step_wall_sec: 25.806408, train/data_wait_ratio: 0.056007, train/lr: 0.000200, train/epoch: 0.000000, train/step: 19980.000000 2026-03-15 04:29:07,161 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000020000.pt 2026-03-15 04:29:07,182 [Rank train_rank0] [Epoch 0 | Step 20000] train/loss: 0.443640, train/mse: 0.443640, train/data_wait_sec: 1.198303, train/step_wall_sec: 25.862625, train/data_wait_ratio: 0.046333, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20000.000000 2026-03-15 04:29:07,190 [Rank train_rank0] Generating samples for visualization... 2026-03-15 04:29:41,314 [Rank train_rank0] Sample generation done. 2026-03-15 04:39:01,589 [Rank train_rank0] [Epoch 0 | Step 20020] train/loss: 0.474450, train/mse: 0.474450, train/data_wait_sec: 1.016222, train/step_wall_sec: 25.991814, train/data_wait_ratio: 0.039098, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20020.000000 2026-03-15 04:47:45,716 [Rank train_rank0] [Epoch 0 | Step 20040] train/loss: 0.463295, train/mse: 0.463295, train/data_wait_sec: 1.733291, train/step_wall_sec: 26.562357, train/data_wait_ratio: 0.065254, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20040.000000 2026-03-15 04:56:27,095 [Rank train_rank0] [Epoch 0 | Step 20060] train/loss: 0.465819, train/mse: 0.465819, train/data_wait_sec: 1.769666, train/step_wall_sec: 26.430363, train/data_wait_ratio: 0.066956, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20060.000000 2026-03-15 05:05:10,419 [Rank train_rank0] [Epoch 0 | Step 20080] train/loss: 0.436157, train/mse: 0.436157, train/data_wait_sec: 1.083266, train/step_wall_sec: 25.891300, train/data_wait_ratio: 0.041839, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20080.000000 2026-03-15 05:13:45,482 [Rank train_rank0] [Epoch 0 | Step 20100] train/loss: 0.434875, train/mse: 0.434875, train/data_wait_sec: 1.434314, train/step_wall_sec: 26.228178, train/data_wait_ratio: 0.054686, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20100.000000 2026-03-15 05:22:24,490 [Rank train_rank0] [Epoch 0 | Step 20120] train/loss: 0.431652, train/mse: 0.431652, train/data_wait_sec: 1.727742, train/step_wall_sec: 26.729073, train/data_wait_ratio: 0.064639, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20120.000000 2026-03-15 05:31:07,088 [Rank train_rank0] [Epoch 0 | Step 20140] train/loss: 0.461384, train/mse: 0.461384, train/data_wait_sec: 1.495980, train/step_wall_sec: 26.454365, train/data_wait_ratio: 0.056549, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20140.000000 2026-03-15 05:39:47,416 [Rank train_rank0] [Epoch 0 | Step 20160] train/loss: 0.444064, train/mse: 0.444064, train/data_wait_sec: 1.383344, train/step_wall_sec: 26.285511, train/data_wait_ratio: 0.052628, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20160.000000 2026-03-15 05:48:25,342 [Rank train_rank0] [Epoch 0 | Step 20180] train/loss: 0.455360, train/mse: 0.455360, train/data_wait_sec: 1.177086, train/step_wall_sec: 25.795681, train/data_wait_ratio: 0.045631, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20180.000000 2026-03-15 05:57:05,770 [Rank train_rank0] [Epoch 0 | Step 20200] train/loss: 0.438639, train/mse: 0.438639, train/data_wait_sec: 1.686183, train/step_wall_sec: 26.143225, train/data_wait_ratio: 0.064498, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20200.000000 2026-03-15 06:05:46,573 [Rank train_rank0] [Epoch 0 | Step 20220] train/loss: 0.463392, train/mse: 0.463392, train/data_wait_sec: 1.404079, train/step_wall_sec: 26.039753, train/data_wait_ratio: 0.053921, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20220.000000 2026-03-15 06:14:28,481 [Rank train_rank0] [Epoch 0 | Step 20240] train/loss: 0.477870, train/mse: 0.477870, train/data_wait_sec: 1.323851, train/step_wall_sec: 25.760201, train/data_wait_ratio: 0.051391, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20240.000000 2026-03-15 06:23:05,648 [Rank train_rank0] [Epoch 0 | Step 20260] train/loss: 0.446890, train/mse: 0.446890, train/data_wait_sec: 1.388374, train/step_wall_sec: 26.936291, train/data_wait_ratio: 0.051543, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20260.000000 2026-03-15 06:31:45,292 [Rank train_rank0] [Epoch 0 | Step 20280] train/loss: 0.468629, train/mse: 0.468629, train/data_wait_sec: 1.402862, train/step_wall_sec: 25.593922, train/data_wait_ratio: 0.054812, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20280.000000 2026-03-15 06:40:24,501 [Rank train_rank0] [Epoch 0 | Step 20300] train/loss: 0.463049, train/mse: 0.463049, train/data_wait_sec: 1.215038, train/step_wall_sec: 25.957819, train/data_wait_ratio: 0.046808, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20300.000000 2026-03-15 06:49:07,401 [Rank train_rank0] [Epoch 0 | Step 20320] train/loss: 0.463455, train/mse: 0.463455, train/data_wait_sec: 1.330330, train/step_wall_sec: 25.876461, train/data_wait_ratio: 0.051411, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20320.000000 2026-03-15 06:57:51,130 [Rank train_rank0] [Epoch 0 | Step 20340] train/loss: 0.456566, train/mse: 0.456566, train/data_wait_sec: 1.653535, train/step_wall_sec: 25.867126, train/data_wait_ratio: 0.063924, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20340.000000 2026-03-15 07:06:28,915 [Rank train_rank0] [Epoch 0 | Step 20360] train/loss: 0.430692, train/mse: 0.430692, train/data_wait_sec: 1.281077, train/step_wall_sec: 25.876835, train/data_wait_ratio: 0.049507, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20360.000000 2026-03-15 07:15:12,667 [Rank train_rank0] [Epoch 0 | Step 20380] train/loss: 0.448941, train/mse: 0.448941, train/data_wait_sec: 1.118727, train/step_wall_sec: 26.215816, train/data_wait_ratio: 0.042674, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20380.000000 2026-03-15 07:23:59,516 [Rank train_rank0] [Epoch 0 | Step 20400] train/loss: 0.451629, train/mse: 0.451629, train/data_wait_sec: 1.357000, train/step_wall_sec: 26.330200, train/data_wait_ratio: 0.051538, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20400.000000 2026-03-15 07:32:41,706 [Rank train_rank0] [Epoch 0 | Step 20420] train/loss: 0.455005, train/mse: 0.455005, train/data_wait_sec: 2.053962, train/step_wall_sec: 27.368053, train/data_wait_ratio: 0.075050, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20420.000000 2026-03-15 07:41:20,341 [Rank train_rank0] [Epoch 0 | Step 20440] train/loss: 0.414225, train/mse: 0.414225, train/data_wait_sec: 1.830817, train/step_wall_sec: 26.229071, train/data_wait_ratio: 0.069801, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20440.000000 2026-03-15 07:49:59,444 [Rank train_rank0] [Epoch 0 | Step 20460] train/loss: 0.455073, train/mse: 0.455073, train/data_wait_sec: 1.088567, train/step_wall_sec: 26.043943, train/data_wait_ratio: 0.041797, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20460.000000 2026-03-15 07:58:40,538 [Rank train_rank0] [Epoch 0 | Step 20480] train/loss: 0.452170, train/mse: 0.452170, train/data_wait_sec: 1.938679, train/step_wall_sec: 27.467371, train/data_wait_ratio: 0.070581, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20480.000000 2026-03-15 08:07:20,354 [Rank train_rank0] [Epoch 0 | Step 20500] train/loss: 0.446146, train/mse: 0.446146, train/data_wait_sec: 1.360492, train/step_wall_sec: 26.356554, train/data_wait_ratio: 0.051619, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20500.000000 2026-03-15 08:16:02,668 [Rank train_rank0] [Epoch 0 | Step 20520] train/loss: 0.458420, train/mse: 0.458420, train/data_wait_sec: 1.189381, train/step_wall_sec: 25.504856, train/data_wait_ratio: 0.046634, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20520.000000 2026-03-15 08:24:44,047 [Rank train_rank0] [Epoch 0 | Step 20540] train/loss: 0.448808, train/mse: 0.448808, train/data_wait_sec: 1.215823, train/step_wall_sec: 25.329611, train/data_wait_ratio: 0.048000, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20540.000000 2026-03-15 08:33:25,554 [Rank train_rank0] [Epoch 0 | Step 20560] train/loss: 0.436490, train/mse: 0.436490, train/data_wait_sec: 1.500374, train/step_wall_sec: 25.810034, train/data_wait_ratio: 0.058131, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20560.000000 2026-03-15 08:42:06,887 [Rank train_rank0] [Epoch 0 | Step 20580] train/loss: 0.465290, train/mse: 0.465290, train/data_wait_sec: 1.936060, train/step_wall_sec: 26.363407, train/data_wait_ratio: 0.073437, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20580.000000 2026-03-15 08:50:44,579 [Rank train_rank0] [Epoch 0 | Step 20600] train/loss: 0.484626, train/mse: 0.484626, train/data_wait_sec: 1.148141, train/step_wall_sec: 25.300417, train/data_wait_ratio: 0.045380, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20600.000000 2026-03-15 08:59:24,963 [Rank train_rank0] [Epoch 0 | Step 20620] train/loss: 0.472115, train/mse: 0.472115, train/data_wait_sec: 1.561985, train/step_wall_sec: 26.136185, train/data_wait_ratio: 0.059763, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20620.000000 2026-03-15 09:08:05,710 [Rank train_rank0] [Epoch 0 | Step 20640] train/loss: 0.456500, train/mse: 0.456500, train/data_wait_sec: 1.547686, train/step_wall_sec: 26.133532, train/data_wait_ratio: 0.059222, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20640.000000 2026-03-15 09:16:47,358 [Rank train_rank0] [Epoch 0 | Step 20660] train/loss: 0.477387, train/mse: 0.477387, train/data_wait_sec: 1.210352, train/step_wall_sec: 25.885246, train/data_wait_ratio: 0.046758, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20660.000000 2026-03-15 09:25:27,398 [Rank train_rank0] [Epoch 0 | Step 20680] train/loss: 0.437223, train/mse: 0.437223, train/data_wait_sec: 1.988872, train/step_wall_sec: 26.017038, train/data_wait_ratio: 0.076445, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20680.000000 2026-03-15 09:34:10,341 [Rank train_rank0] [Epoch 0 | Step 20700] train/loss: 0.452311, train/mse: 0.452311, train/data_wait_sec: 1.115344, train/step_wall_sec: 25.769794, train/data_wait_ratio: 0.043281, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20700.000000 2026-03-15 09:42:52,814 [Rank train_rank0] [Epoch 0 | Step 20720] train/loss: 0.442609, train/mse: 0.442609, train/data_wait_sec: 1.471590, train/step_wall_sec: 26.612743, train/data_wait_ratio: 0.055296, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20720.000000 2026-03-15 09:51:30,252 [Rank train_rank0] [Epoch 0 | Step 20740] train/loss: 0.436888, train/mse: 0.436888, train/data_wait_sec: 0.983704, train/step_wall_sec: 25.077774, train/data_wait_ratio: 0.039226, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20740.000000 2026-03-15 10:00:10,059 [Rank train_rank0] [Epoch 0 | Step 20760] train/loss: 0.466191, train/mse: 0.466191, train/data_wait_sec: 1.078664, train/step_wall_sec: 25.517328, train/data_wait_ratio: 0.042272, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20760.000000 2026-03-15 10:08:50,334 [Rank train_rank0] [Epoch 0 | Step 20780] train/loss: 0.450967, train/mse: 0.450967, train/data_wait_sec: 1.682792, train/step_wall_sec: 25.674423, train/data_wait_ratio: 0.065544, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20780.000000 2026-03-15 10:17:28,417 [Rank train_rank0] [Epoch 0 | Step 20800] train/loss: 0.481266, train/mse: 0.481266, train/data_wait_sec: 1.025492, train/step_wall_sec: 25.917080, train/data_wait_ratio: 0.039568, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20800.000000 2026-03-15 10:26:06,174 [Rank train_rank0] [Epoch 0 | Step 20820] train/loss: 0.454573, train/mse: 0.454573, train/data_wait_sec: 1.152386, train/step_wall_sec: 25.436554, train/data_wait_ratio: 0.045304, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20820.000000 2026-03-15 10:34:49,145 [Rank train_rank0] [Epoch 0 | Step 20840] train/loss: 0.434329, train/mse: 0.434329, train/data_wait_sec: 1.690341, train/step_wall_sec: 27.144108, train/data_wait_ratio: 0.062273, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20840.000000 2026-03-15 10:43:26,116 [Rank train_rank0] [Epoch 0 | Step 20860] train/loss: 0.452882, train/mse: 0.452882, train/data_wait_sec: 1.221496, train/step_wall_sec: 25.372200, train/data_wait_ratio: 0.048143, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20860.000000 2026-03-15 10:52:04,963 [Rank train_rank0] [Epoch 0 | Step 20880] train/loss: 0.471222, train/mse: 0.471222, train/data_wait_sec: 1.209225, train/step_wall_sec: 25.813892, train/data_wait_ratio: 0.046844, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20880.000000 2026-03-15 11:00:44,159 [Rank train_rank0] [Epoch 0 | Step 20900] train/loss: 0.431865, train/mse: 0.431865, train/data_wait_sec: 1.824140, train/step_wall_sec: 26.360435, train/data_wait_ratio: 0.069200, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20900.000000 2026-03-15 11:09:23,836 [Rank train_rank0] [Epoch 0 | Step 20920] train/loss: 0.467735, train/mse: 0.467735, train/data_wait_sec: 1.455586, train/step_wall_sec: 25.936708, train/data_wait_ratio: 0.056121, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20920.000000 2026-03-15 11:18:00,056 [Rank train_rank0] [Epoch 0 | Step 20940] train/loss: 0.457160, train/mse: 0.457160, train/data_wait_sec: 1.522683, train/step_wall_sec: 25.553825, train/data_wait_ratio: 0.059587, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20940.000000 2026-03-15 11:26:41,263 [Rank train_rank0] [Epoch 0 | Step 20960] train/loss: 0.442132, train/mse: 0.442132, train/data_wait_sec: 0.948268, train/step_wall_sec: 25.218096, train/data_wait_ratio: 0.037603, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20960.000000 2026-03-15 11:35:17,589 [Rank train_rank0] [Epoch 0 | Step 20980] train/loss: 0.469692, train/mse: 0.469692, train/data_wait_sec: 1.457630, train/step_wall_sec: 25.187023, train/data_wait_ratio: 0.057872, train/lr: 0.000200, train/epoch: 0.000000, train/step: 20980.000000 2026-03-15 11:46:59,508 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000021000.pt 2026-03-15 11:46:59,524 [Rank train_rank0] [Epoch 0 | Step 21000] train/loss: 0.474196, train/mse: 0.474196, train/data_wait_sec: 1.544677, train/step_wall_sec: 26.973934, train/data_wait_ratio: 0.057266, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21000.000000 2026-03-15 11:46:59,532 [Rank train_rank0] Generating samples for visualization... 2026-03-15 11:47:33,825 [Rank train_rank0] Sample generation done. 2026-03-15 11:56:19,019 [Rank train_rank0] [Epoch 0 | Step 21020] train/loss: 0.468665, train/mse: 0.468665, train/data_wait_sec: 1.368986, train/step_wall_sec: 26.166245, train/data_wait_ratio: 0.052319, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21020.000000 2026-03-15 12:05:07,459 [Rank train_rank0] [Epoch 0 | Step 21040] train/loss: 0.454446, train/mse: 0.454446, train/data_wait_sec: 1.515982, train/step_wall_sec: 26.454775, train/data_wait_ratio: 0.057305, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21040.000000 2026-03-15 12:13:49,068 [Rank train_rank0] [Epoch 0 | Step 21060] train/loss: 0.462126, train/mse: 0.462126, train/data_wait_sec: 1.195728, train/step_wall_sec: 27.291565, train/data_wait_ratio: 0.043813, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21060.000000 2026-03-15 12:22:34,083 [Rank train_rank0] [Epoch 0 | Step 21080] train/loss: 0.445723, train/mse: 0.445723, train/data_wait_sec: 1.602846, train/step_wall_sec: 25.955149, train/data_wait_ratio: 0.061754, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21080.000000 2026-03-15 12:31:20,812 [Rank train_rank0] [Epoch 0 | Step 21100] train/loss: 0.453240, train/mse: 0.453240, train/data_wait_sec: 1.685108, train/step_wall_sec: 26.717268, train/data_wait_ratio: 0.063072, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21100.000000 2026-03-15 12:40:08,439 [Rank train_rank0] [Epoch 0 | Step 21120] train/loss: 0.446618, train/mse: 0.446618, train/data_wait_sec: 1.442881, train/step_wall_sec: 25.921974, train/data_wait_ratio: 0.055662, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21120.000000 2026-03-15 12:48:58,500 [Rank train_rank0] [Epoch 0 | Step 21140] train/loss: 0.457742, train/mse: 0.457742, train/data_wait_sec: 1.720953, train/step_wall_sec: 26.960522, train/data_wait_ratio: 0.063832, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21140.000000 2026-03-15 12:55:33,150 [Rank train_rank0] [Epoch 0 | Step 21160] train/loss: 0.439002, train/mse: 0.439002, train/data_wait_sec: 0.220438, train/step_wall_sec: 5.757412, train/data_wait_ratio: 0.038288, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21160.000000 2026-03-15 12:57:30,485 [Rank train_rank0] [Epoch 0 | Step 21180] train/loss: 0.482144, train/mse: 0.482144, train/data_wait_sec: 0.363258, train/step_wall_sec: 5.826251, train/data_wait_ratio: 0.062348, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21180.000000 2026-03-15 12:59:27,466 [Rank train_rank0] [Epoch 0 | Step 21200] train/loss: 0.441404, train/mse: 0.441404, train/data_wait_sec: 0.217183, train/step_wall_sec: 5.708262, train/data_wait_ratio: 0.038047, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21200.000000 2026-03-15 13:01:24,830 [Rank train_rank0] [Epoch 0 | Step 21220] train/loss: 0.465219, train/mse: 0.465219, train/data_wait_sec: 0.377044, train/step_wall_sec: 5.827132, train/data_wait_ratio: 0.064705, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21220.000000 2026-03-15 13:03:22,073 [Rank train_rank0] [Epoch 0 | Step 21240] train/loss: 0.450981, train/mse: 0.450981, train/data_wait_sec: 0.256659, train/step_wall_sec: 5.840312, train/data_wait_ratio: 0.043946, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21240.000000 2026-03-15 13:05:18,750 [Rank train_rank0] [Epoch 0 | Step 21260] train/loss: 0.453819, train/mse: 0.453819, train/data_wait_sec: 0.273139, train/step_wall_sec: 5.811169, train/data_wait_ratio: 0.047002, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21260.000000 2026-03-15 13:07:15,869 [Rank train_rank0] [Epoch 0 | Step 21280] train/loss: 0.459173, train/mse: 0.459173, train/data_wait_sec: 0.332890, train/step_wall_sec: 5.923234, train/data_wait_ratio: 0.056201, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21280.000000 2026-03-15 13:09:13,508 [Rank train_rank0] [Epoch 0 | Step 21300] train/loss: 0.450555, train/mse: 0.450555, train/data_wait_sec: 0.388192, train/step_wall_sec: 6.020656, train/data_wait_ratio: 0.064477, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21300.000000 2026-03-15 13:11:10,649 [Rank train_rank0] [Epoch 0 | Step 21320] train/loss: 0.464617, train/mse: 0.464617, train/data_wait_sec: 0.275247, train/step_wall_sec: 5.756200, train/data_wait_ratio: 0.047817, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21320.000000 2026-03-15 13:13:07,215 [Rank train_rank0] [Epoch 0 | Step 21340] train/loss: 0.484115, train/mse: 0.484115, train/data_wait_sec: 0.214957, train/step_wall_sec: 5.709987, train/data_wait_ratio: 0.037646, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21340.000000 2026-03-15 13:15:03,592 [Rank train_rank0] [Epoch 0 | Step 21360] train/loss: 0.435560, train/mse: 0.435560, train/data_wait_sec: 0.317049, train/step_wall_sec: 5.817726, train/data_wait_ratio: 0.054497, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21360.000000 2026-03-15 13:17:00,724 [Rank train_rank0] [Epoch 0 | Step 21380] train/loss: 0.474774, train/mse: 0.474774, train/data_wait_sec: 0.339824, train/step_wall_sec: 5.832969, train/data_wait_ratio: 0.058259, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21380.000000 2026-03-15 13:18:57,195 [Rank train_rank0] [Epoch 0 | Step 21400] train/loss: 0.464271, train/mse: 0.464271, train/data_wait_sec: 0.344527, train/step_wall_sec: 5.853186, train/data_wait_ratio: 0.058861, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21400.000000 2026-03-15 13:20:54,087 [Rank train_rank0] [Epoch 0 | Step 21420] train/loss: 0.452201, train/mse: 0.452201, train/data_wait_sec: 0.250703, train/step_wall_sec: 5.710277, train/data_wait_ratio: 0.043904, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21420.000000 2026-03-15 13:22:51,076 [Rank train_rank0] [Epoch 0 | Step 21440] train/loss: 0.424364, train/mse: 0.424364, train/data_wait_sec: 0.300209, train/step_wall_sec: 5.827894, train/data_wait_ratio: 0.051512, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21440.000000 2026-03-15 13:24:47,283 [Rank train_rank0] [Epoch 0 | Step 21460] train/loss: 0.453535, train/mse: 0.453535, train/data_wait_sec: 0.342641, train/step_wall_sec: 5.813693, train/data_wait_ratio: 0.058937, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21460.000000 2026-03-15 13:26:44,217 [Rank train_rank0] [Epoch 0 | Step 21480] train/loss: 0.449920, train/mse: 0.449920, train/data_wait_sec: 0.384670, train/step_wall_sec: 5.906362, train/data_wait_ratio: 0.065128, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21480.000000 2026-03-15 13:28:40,741 [Rank train_rank0] [Epoch 0 | Step 21500] train/loss: 0.445914, train/mse: 0.445914, train/data_wait_sec: 0.246053, train/step_wall_sec: 5.701000, train/data_wait_ratio: 0.043160, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21500.000000 2026-03-15 13:30:37,541 [Rank train_rank0] [Epoch 0 | Step 21520] train/loss: 0.455255, train/mse: 0.455255, train/data_wait_sec: 0.256776, train/step_wall_sec: 5.798161, train/data_wait_ratio: 0.044286, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21520.000000 2026-03-15 13:32:35,281 [Rank train_rank0] [Epoch 0 | Step 21540] train/loss: 0.460267, train/mse: 0.460267, train/data_wait_sec: 0.318757, train/step_wall_sec: 5.843962, train/data_wait_ratio: 0.054545, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21540.000000 2026-03-15 13:34:33,784 [Rank train_rank0] [Epoch 0 | Step 21560] train/loss: 0.501787, train/mse: 0.501787, train/data_wait_sec: 0.345627, train/step_wall_sec: 5.923693, train/data_wait_ratio: 0.058347, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21560.000000 2026-03-15 13:36:32,687 [Rank train_rank0] [Epoch 0 | Step 21580] train/loss: 0.463148, train/mse: 0.463148, train/data_wait_sec: 0.426480, train/step_wall_sec: 5.844307, train/data_wait_ratio: 0.072974, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21580.000000 2026-03-15 13:38:30,812 [Rank train_rank0] [Epoch 0 | Step 21600] train/loss: 0.436610, train/mse: 0.436610, train/data_wait_sec: 0.284875, train/step_wall_sec: 5.860726, train/data_wait_ratio: 0.048607, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21600.000000 2026-03-15 13:40:29,183 [Rank train_rank0] [Epoch 0 | Step 21620] train/loss: 0.480591, train/mse: 0.480591, train/data_wait_sec: 0.372014, train/step_wall_sec: 5.992483, train/data_wait_ratio: 0.062080, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21620.000000 2026-03-15 13:42:27,877 [Rank train_rank0] [Epoch 0 | Step 21640] train/loss: 0.457354, train/mse: 0.457354, train/data_wait_sec: 0.289512, train/step_wall_sec: 5.846540, train/data_wait_ratio: 0.049519, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21640.000000 2026-03-15 13:44:25,896 [Rank train_rank0] [Epoch 0 | Step 21660] train/loss: 0.460740, train/mse: 0.460740, train/data_wait_sec: 0.288495, train/step_wall_sec: 5.814489, train/data_wait_ratio: 0.049617, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21660.000000 2026-03-15 13:46:24,373 [Rank train_rank0] [Epoch 0 | Step 21680] train/loss: 0.454889, train/mse: 0.454889, train/data_wait_sec: 0.265941, train/step_wall_sec: 5.797375, train/data_wait_ratio: 0.045873, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21680.000000 2026-03-15 13:48:21,940 [Rank train_rank0] [Epoch 0 | Step 21700] train/loss: 0.484180, train/mse: 0.484180, train/data_wait_sec: 0.201344, train/step_wall_sec: 5.597349, train/data_wait_ratio: 0.035971, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21700.000000 2026-03-15 13:50:19,078 [Rank train_rank0] [Epoch 0 | Step 21720] train/loss: 0.465083, train/mse: 0.465083, train/data_wait_sec: 0.325154, train/step_wall_sec: 5.773663, train/data_wait_ratio: 0.056317, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21720.000000 2026-03-15 13:52:16,070 [Rank train_rank0] [Epoch 0 | Step 21740] train/loss: 0.485914, train/mse: 0.485914, train/data_wait_sec: 0.349076, train/step_wall_sec: 5.790478, train/data_wait_ratio: 0.060284, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21740.000000 2026-03-15 13:54:12,863 [Rank train_rank0] [Epoch 0 | Step 21760] train/loss: 0.457219, train/mse: 0.457219, train/data_wait_sec: 0.283024, train/step_wall_sec: 5.712347, train/data_wait_ratio: 0.049546, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21760.000000 2026-03-15 13:56:09,732 [Rank train_rank0] [Epoch 0 | Step 21780] train/loss: 0.458735, train/mse: 0.458735, train/data_wait_sec: 0.397834, train/step_wall_sec: 5.776301, train/data_wait_ratio: 0.068873, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21780.000000 2026-03-15 13:58:06,912 [Rank train_rank0] [Epoch 0 | Step 21800] train/loss: 0.438223, train/mse: 0.438223, train/data_wait_sec: 0.359690, train/step_wall_sec: 5.909134, train/data_wait_ratio: 0.060870, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21800.000000 2026-03-15 14:00:04,153 [Rank train_rank0] [Epoch 0 | Step 21820] train/loss: 0.466310, train/mse: 0.466310, train/data_wait_sec: 0.297739, train/step_wall_sec: 5.895051, train/data_wait_ratio: 0.050507, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21820.000000 2026-03-15 14:02:00,986 [Rank train_rank0] [Epoch 0 | Step 21840] train/loss: 0.447402, train/mse: 0.447402, train/data_wait_sec: 0.346491, train/step_wall_sec: 5.876326, train/data_wait_ratio: 0.058964, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21840.000000 2026-03-15 14:03:58,046 [Rank train_rank0] [Epoch 0 | Step 21860] train/loss: 0.480244, train/mse: 0.480244, train/data_wait_sec: 0.339027, train/step_wall_sec: 5.804462, train/data_wait_ratio: 0.058408, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21860.000000 2026-03-15 14:05:55,609 [Rank train_rank0] [Epoch 0 | Step 21880] train/loss: 0.475228, train/mse: 0.475228, train/data_wait_sec: 0.256252, train/step_wall_sec: 5.795227, train/data_wait_ratio: 0.044218, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21880.000000 2026-03-15 14:07:52,886 [Rank train_rank0] [Epoch 0 | Step 21900] train/loss: 0.467541, train/mse: 0.467541, train/data_wait_sec: 0.329964, train/step_wall_sec: 5.810041, train/data_wait_ratio: 0.056792, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21900.000000 2026-03-15 14:09:49,450 [Rank train_rank0] [Epoch 0 | Step 21920] train/loss: 0.471346, train/mse: 0.471346, train/data_wait_sec: 0.265872, train/step_wall_sec: 5.720223, train/data_wait_ratio: 0.046479, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21920.000000 2026-03-15 14:11:46,710 [Rank train_rank0] [Epoch 0 | Step 21940] train/loss: 0.433159, train/mse: 0.433159, train/data_wait_sec: 0.366477, train/step_wall_sec: 5.890182, train/data_wait_ratio: 0.062218, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21940.000000 2026-03-15 14:13:43,239 [Rank train_rank0] [Epoch 0 | Step 21960] train/loss: 0.474317, train/mse: 0.474317, train/data_wait_sec: 0.374836, train/step_wall_sec: 5.827035, train/data_wait_ratio: 0.064327, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21960.000000 2026-03-15 14:15:40,473 [Rank train_rank0] [Epoch 0 | Step 21980] train/loss: 0.449856, train/mse: 0.449856, train/data_wait_sec: 0.238708, train/step_wall_sec: 5.753431, train/data_wait_ratio: 0.041490, train/lr: 0.000200, train/epoch: 0.000000, train/step: 21980.000000 2026-03-15 14:18:02,340 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000022000.pt 2026-03-15 14:18:02,343 [Rank train_rank0] [Epoch 0 | Step 22000] train/loss: 0.458153, train/mse: 0.458153, train/data_wait_sec: 0.297081, train/step_wall_sec: 5.831635, train/data_wait_ratio: 0.050943, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22000.000000 2026-03-15 14:18:02,344 [Rank train_rank0] Generating samples for visualization... 2026-03-15 14:18:05,508 [Rank train_rank0] Sample generation done. 2026-03-15 14:20:03,510 [Rank train_rank0] [Epoch 0 | Step 22020] train/loss: 0.460150, train/mse: 0.460150, train/data_wait_sec: 0.490921, train/step_wall_sec: 5.952590, train/data_wait_ratio: 0.082472, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22020.000000 2026-03-15 14:22:01,669 [Rank train_rank0] [Epoch 0 | Step 22040] train/loss: 0.447137, train/mse: 0.447137, train/data_wait_sec: 0.263180, train/step_wall_sec: 5.773470, train/data_wait_ratio: 0.045584, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22040.000000 2026-03-15 14:23:59,910 [Rank train_rank0] [Epoch 0 | Step 22060] train/loss: 0.442790, train/mse: 0.442790, train/data_wait_sec: 0.425039, train/step_wall_sec: 5.921500, train/data_wait_ratio: 0.071779, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22060.000000 2026-03-15 14:25:58,330 [Rank train_rank0] [Epoch 0 | Step 22080] train/loss: 0.453232, train/mse: 0.453232, train/data_wait_sec: 0.315196, train/step_wall_sec: 5.871181, train/data_wait_ratio: 0.053685, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22080.000000 2026-03-15 14:27:55,714 [Rank train_rank0] [Epoch 0 | Step 22100] train/loss: 0.436931, train/mse: 0.436931, train/data_wait_sec: 0.379890, train/step_wall_sec: 5.799342, train/data_wait_ratio: 0.065506, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22100.000000 2026-03-15 14:29:52,755 [Rank train_rank0] [Epoch 0 | Step 22120] train/loss: 0.466240, train/mse: 0.466240, train/data_wait_sec: 0.271857, train/step_wall_sec: 5.801104, train/data_wait_ratio: 0.046863, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22120.000000 2026-03-15 14:32:00,692 [Rank train_rank0] [Epoch 0 | Step 22140] train/loss: 0.445021, train/mse: 0.445021, train/data_wait_sec: 0.224375, train/step_wall_sec: 5.847314, train/data_wait_ratio: 0.038372, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22140.000000 2026-03-15 14:33:57,934 [Rank train_rank0] [Epoch 0 | Step 22160] train/loss: 0.442022, train/mse: 0.442022, train/data_wait_sec: 0.224141, train/step_wall_sec: 5.772955, train/data_wait_ratio: 0.038826, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22160.000000 2026-03-15 14:35:54,844 [Rank train_rank0] [Epoch 0 | Step 22180] train/loss: 0.444186, train/mse: 0.444186, train/data_wait_sec: 0.199928, train/step_wall_sec: 5.709883, train/data_wait_ratio: 0.035014, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22180.000000 2026-03-15 14:37:52,104 [Rank train_rank0] [Epoch 0 | Step 22200] train/loss: 0.448437, train/mse: 0.448437, train/data_wait_sec: 0.319599, train/step_wall_sec: 5.813649, train/data_wait_ratio: 0.054974, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22200.000000 2026-03-15 14:39:49,241 [Rank train_rank0] [Epoch 0 | Step 22220] train/loss: 0.450545, train/mse: 0.450545, train/data_wait_sec: 0.319615, train/step_wall_sec: 5.822935, train/data_wait_ratio: 0.054889, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22220.000000 2026-03-15 14:41:45,925 [Rank train_rank0] [Epoch 0 | Step 22240] train/loss: 0.444957, train/mse: 0.444957, train/data_wait_sec: 0.308937, train/step_wall_sec: 5.778561, train/data_wait_ratio: 0.053463, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22240.000000 2026-03-15 14:43:43,034 [Rank train_rank0] [Epoch 0 | Step 22260] train/loss: 0.481716, train/mse: 0.481716, train/data_wait_sec: 0.243449, train/step_wall_sec: 5.745966, train/data_wait_ratio: 0.042369, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22260.000000 2026-03-15 14:45:40,168 [Rank train_rank0] [Epoch 0 | Step 22280] train/loss: 0.462579, train/mse: 0.462579, train/data_wait_sec: 0.322949, train/step_wall_sec: 5.827100, train/data_wait_ratio: 0.055422, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22280.000000 2026-03-15 14:47:36,759 [Rank train_rank0] [Epoch 0 | Step 22300] train/loss: 0.463665, train/mse: 0.463665, train/data_wait_sec: 0.399099, train/step_wall_sec: 5.899162, train/data_wait_ratio: 0.067653, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22300.000000 2026-03-15 14:49:34,238 [Rank train_rank0] [Epoch 0 | Step 22320] train/loss: 0.455978, train/mse: 0.455978, train/data_wait_sec: 0.298741, train/step_wall_sec: 5.893682, train/data_wait_ratio: 0.050688, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22320.000000 2026-03-15 14:51:31,483 [Rank train_rank0] [Epoch 0 | Step 22340] train/loss: 0.463418, train/mse: 0.463418, train/data_wait_sec: 0.294409, train/step_wall_sec: 5.791976, train/data_wait_ratio: 0.050831, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22340.000000 2026-03-15 14:53:28,727 [Rank train_rank0] [Epoch 0 | Step 22360] train/loss: 0.479108, train/mse: 0.479108, train/data_wait_sec: 0.286285, train/step_wall_sec: 5.831192, train/data_wait_ratio: 0.049095, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22360.000000 2026-03-15 14:55:25,981 [Rank train_rank0] [Epoch 0 | Step 22380] train/loss: 0.464297, train/mse: 0.464297, train/data_wait_sec: 0.339276, train/step_wall_sec: 5.933116, train/data_wait_ratio: 0.057183, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22380.000000 2026-03-15 14:57:22,871 [Rank train_rank0] [Epoch 0 | Step 22400] train/loss: 0.460686, train/mse: 0.460686, train/data_wait_sec: 0.349416, train/step_wall_sec: 5.925962, train/data_wait_ratio: 0.058964, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22400.000000 2026-03-15 14:59:20,230 [Rank train_rank0] [Epoch 0 | Step 22420] train/loss: 0.507169, train/mse: 0.507169, train/data_wait_sec: 0.287346, train/step_wall_sec: 5.880183, train/data_wait_ratio: 0.048867, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22420.000000 2026-03-15 15:01:17,689 [Rank train_rank0] [Epoch 0 | Step 22440] train/loss: 0.448751, train/mse: 0.448751, train/data_wait_sec: 0.445568, train/step_wall_sec: 5.885944, train/data_wait_ratio: 0.075700, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22440.000000 2026-03-15 15:03:14,662 [Rank train_rank0] [Epoch 0 | Step 22460] train/loss: 0.448651, train/mse: 0.448651, train/data_wait_sec: 0.174525, train/step_wall_sec: 5.672478, train/data_wait_ratio: 0.030767, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22460.000000 2026-03-15 15:05:11,726 [Rank train_rank0] [Epoch 0 | Step 22480] train/loss: 0.463703, train/mse: 0.463703, train/data_wait_sec: 0.309703, train/step_wall_sec: 5.868611, train/data_wait_ratio: 0.052773, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22480.000000 2026-03-15 15:07:08,979 [Rank train_rank0] [Epoch 0 | Step 22500] train/loss: 0.481223, train/mse: 0.481223, train/data_wait_sec: 0.258835, train/step_wall_sec: 5.717172, train/data_wait_ratio: 0.045273, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22500.000000 2026-03-15 15:09:06,106 [Rank train_rank0] [Epoch 0 | Step 22520] train/loss: 0.470241, train/mse: 0.470241, train/data_wait_sec: 0.252916, train/step_wall_sec: 5.809129, train/data_wait_ratio: 0.043538, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22520.000000 2026-03-15 15:11:02,564 [Rank train_rank0] [Epoch 0 | Step 22540] train/loss: 0.459361, train/mse: 0.459361, train/data_wait_sec: 0.359821, train/step_wall_sec: 5.897598, train/data_wait_ratio: 0.061011, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22540.000000 2026-03-15 15:12:59,280 [Rank train_rank0] [Epoch 0 | Step 22560] train/loss: 0.480346, train/mse: 0.480346, train/data_wait_sec: 0.254139, train/step_wall_sec: 5.808011, train/data_wait_ratio: 0.043757, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22560.000000 2026-03-15 15:14:56,997 [Rank train_rank0] [Epoch 0 | Step 22580] train/loss: 0.462538, train/mse: 0.462538, train/data_wait_sec: 0.399732, train/step_wall_sec: 5.979396, train/data_wait_ratio: 0.066852, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22580.000000 2026-03-15 15:16:53,951 [Rank train_rank0] [Epoch 0 | Step 22600] train/loss: 0.453804, train/mse: 0.453804, train/data_wait_sec: 0.292021, train/step_wall_sec: 5.785103, train/data_wait_ratio: 0.050478, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22600.000000 2026-03-15 15:18:51,081 [Rank train_rank0] [Epoch 0 | Step 22620] train/loss: 0.446208, train/mse: 0.446208, train/data_wait_sec: 0.315024, train/step_wall_sec: 5.854989, train/data_wait_ratio: 0.053804, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22620.000000 2026-03-15 15:20:48,080 [Rank train_rank0] [Epoch 0 | Step 22640] train/loss: 0.438273, train/mse: 0.438273, train/data_wait_sec: 0.363183, train/step_wall_sec: 5.936741, train/data_wait_ratio: 0.061175, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22640.000000 2026-03-15 15:22:44,940 [Rank train_rank0] [Epoch 0 | Step 22660] train/loss: 0.425583, train/mse: 0.425583, train/data_wait_sec: 0.314293, train/step_wall_sec: 5.872816, train/data_wait_ratio: 0.053517, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22660.000000 2026-03-15 15:24:41,506 [Rank train_rank0] [Epoch 0 | Step 22680] train/loss: 0.458311, train/mse: 0.458311, train/data_wait_sec: 0.265694, train/step_wall_sec: 5.800819, train/data_wait_ratio: 0.045803, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22680.000000 2026-03-15 15:26:38,586 [Rank train_rank0] [Epoch 0 | Step 22700] train/loss: 0.430619, train/mse: 0.430619, train/data_wait_sec: 0.363165, train/step_wall_sec: 5.774464, train/data_wait_ratio: 0.062891, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22700.000000 2026-03-15 15:28:35,286 [Rank train_rank0] [Epoch 0 | Step 22720] train/loss: 0.459982, train/mse: 0.459982, train/data_wait_sec: 0.338402, train/step_wall_sec: 5.915091, train/data_wait_ratio: 0.057210, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22720.000000 2026-03-15 15:30:32,642 [Rank train_rank0] [Epoch 0 | Step 22740] train/loss: 0.467562, train/mse: 0.467562, train/data_wait_sec: 0.306733, train/step_wall_sec: 5.841620, train/data_wait_ratio: 0.052508, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22740.000000 2026-03-15 15:32:29,131 [Rank train_rank0] [Epoch 0 | Step 22760] train/loss: 0.468454, train/mse: 0.468454, train/data_wait_sec: 0.279916, train/step_wall_sec: 5.846229, train/data_wait_ratio: 0.047880, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22760.000000 2026-03-15 15:34:26,491 [Rank train_rank0] [Epoch 0 | Step 22780] train/loss: 0.449992, train/mse: 0.449992, train/data_wait_sec: 0.285293, train/step_wall_sec: 5.866541, train/data_wait_ratio: 0.048630, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22780.000000 2026-03-15 15:36:23,696 [Rank train_rank0] [Epoch 0 | Step 22800] train/loss: 0.431518, train/mse: 0.431518, train/data_wait_sec: 0.351521, train/step_wall_sec: 5.787490, train/data_wait_ratio: 0.060738, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22800.000000 2026-03-15 15:38:20,759 [Rank train_rank0] [Epoch 0 | Step 22820] train/loss: 0.459304, train/mse: 0.459304, train/data_wait_sec: 0.313036, train/step_wall_sec: 5.823141, train/data_wait_ratio: 0.053757, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22820.000000 2026-03-15 15:40:17,817 [Rank train_rank0] [Epoch 0 | Step 22840] train/loss: 0.453963, train/mse: 0.453963, train/data_wait_sec: 0.207518, train/step_wall_sec: 5.687497, train/data_wait_ratio: 0.036487, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22840.000000 2026-03-15 15:42:14,983 [Rank train_rank0] [Epoch 0 | Step 22860] train/loss: 0.479640, train/mse: 0.479640, train/data_wait_sec: 0.279742, train/step_wall_sec: 5.684388, train/data_wait_ratio: 0.049212, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22860.000000 2026-03-15 15:44:11,439 [Rank train_rank0] [Epoch 0 | Step 22880] train/loss: 0.443105, train/mse: 0.443105, train/data_wait_sec: 0.315128, train/step_wall_sec: 5.758140, train/data_wait_ratio: 0.054727, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22880.000000 2026-03-15 15:46:08,534 [Rank train_rank0] [Epoch 0 | Step 22900] train/loss: 0.457104, train/mse: 0.457104, train/data_wait_sec: 0.188056, train/step_wall_sec: 5.703820, train/data_wait_ratio: 0.032970, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22900.000000 2026-03-15 15:48:05,898 [Rank train_rank0] [Epoch 0 | Step 22920] train/loss: 0.431679, train/mse: 0.431679, train/data_wait_sec: 0.303494, train/step_wall_sec: 5.899948, train/data_wait_ratio: 0.051440, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22920.000000 2026-03-15 15:50:02,683 [Rank train_rank0] [Epoch 0 | Step 22940] train/loss: 0.454347, train/mse: 0.454347, train/data_wait_sec: 0.179518, train/step_wall_sec: 5.585513, train/data_wait_ratio: 0.032140, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22940.000000 2026-03-15 15:51:59,674 [Rank train_rank0] [Epoch 0 | Step 22960] train/loss: 0.445342, train/mse: 0.445342, train/data_wait_sec: 0.349362, train/step_wall_sec: 5.906818, train/data_wait_ratio: 0.059146, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22960.000000 2026-03-15 15:53:56,849 [Rank train_rank0] [Epoch 0 | Step 22980] train/loss: 0.440807, train/mse: 0.440807, train/data_wait_sec: 0.283132, train/step_wall_sec: 5.818016, train/data_wait_ratio: 0.048665, train/lr: 0.000200, train/epoch: 0.000000, train/step: 22980.000000 2026-03-15 15:56:19,640 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000023000.pt 2026-03-15 15:56:19,643 [Rank train_rank0] [Epoch 0 | Step 23000] train/loss: 0.461143, train/mse: 0.461143, train/data_wait_sec: 0.426697, train/step_wall_sec: 5.947573, train/data_wait_ratio: 0.071743, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23000.000000 2026-03-15 15:56:19,644 [Rank train_rank0] Generating samples for visualization... 2026-03-15 15:56:22,944 [Rank train_rank0] Sample generation done. 2026-03-15 15:58:21,059 [Rank train_rank0] [Epoch 0 | Step 23020] train/loss: 0.434611, train/mse: 0.434611, train/data_wait_sec: 0.300109, train/step_wall_sec: 5.804368, train/data_wait_ratio: 0.051704, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23020.000000 2026-03-15 16:00:18,297 [Rank train_rank0] [Epoch 0 | Step 23040] train/loss: 0.447093, train/mse: 0.447093, train/data_wait_sec: 0.301793, train/step_wall_sec: 5.785840, train/data_wait_ratio: 0.052161, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23040.000000 2026-03-15 16:02:15,590 [Rank train_rank0] [Epoch 0 | Step 23060] train/loss: 0.443856, train/mse: 0.443856, train/data_wait_sec: 0.267022, train/step_wall_sec: 5.806622, train/data_wait_ratio: 0.045986, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23060.000000 2026-03-15 16:04:12,188 [Rank train_rank0] [Epoch 0 | Step 23080] train/loss: 0.435435, train/mse: 0.435435, train/data_wait_sec: 0.358141, train/step_wall_sec: 5.813219, train/data_wait_ratio: 0.061608, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23080.000000 2026-03-15 16:06:09,148 [Rank train_rank0] [Epoch 0 | Step 23100] train/loss: 0.444925, train/mse: 0.444925, train/data_wait_sec: 0.353203, train/step_wall_sec: 5.885927, train/data_wait_ratio: 0.060008, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23100.000000 2026-03-15 16:08:06,252 [Rank train_rank0] [Epoch 0 | Step 23120] train/loss: 0.437368, train/mse: 0.437368, train/data_wait_sec: 0.272714, train/step_wall_sec: 5.798326, train/data_wait_ratio: 0.047033, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23120.000000 2026-03-15 16:10:03,295 [Rank train_rank0] [Epoch 0 | Step 23140] train/loss: 0.446666, train/mse: 0.446666, train/data_wait_sec: 0.333512, train/step_wall_sec: 5.897674, train/data_wait_ratio: 0.056550, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23140.000000 2026-03-15 16:12:00,496 [Rank train_rank0] [Epoch 0 | Step 23160] train/loss: 0.454172, train/mse: 0.454172, train/data_wait_sec: 0.237221, train/step_wall_sec: 5.671927, train/data_wait_ratio: 0.041824, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23160.000000 2026-03-15 16:13:57,644 [Rank train_rank0] [Epoch 0 | Step 23180] train/loss: 0.450993, train/mse: 0.450993, train/data_wait_sec: 0.343507, train/step_wall_sec: 5.835523, train/data_wait_ratio: 0.058865, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23180.000000 2026-03-15 16:15:54,385 [Rank train_rank0] [Epoch 0 | Step 23200] train/loss: 0.471511, train/mse: 0.471511, train/data_wait_sec: 0.365486, train/step_wall_sec: 5.822957, train/data_wait_ratio: 0.062766, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23200.000000 2026-03-15 16:17:51,234 [Rank train_rank0] [Epoch 0 | Step 23220] train/loss: 0.442172, train/mse: 0.442172, train/data_wait_sec: 0.358332, train/step_wall_sec: 5.817101, train/data_wait_ratio: 0.061600, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23220.000000 2026-03-15 16:19:47,686 [Rank train_rank0] [Epoch 0 | Step 23240] train/loss: 0.462922, train/mse: 0.462922, train/data_wait_sec: 0.248135, train/step_wall_sec: 5.651618, train/data_wait_ratio: 0.043905, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23240.000000 2026-03-15 16:21:44,378 [Rank train_rank0] [Epoch 0 | Step 23260] train/loss: 0.455996, train/mse: 0.455996, train/data_wait_sec: 0.326821, train/step_wall_sec: 5.939794, train/data_wait_ratio: 0.055022, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23260.000000 2026-03-15 16:23:41,288 [Rank train_rank0] [Epoch 0 | Step 23280] train/loss: 0.460096, train/mse: 0.460096, train/data_wait_sec: 0.296027, train/step_wall_sec: 5.769161, train/data_wait_ratio: 0.051312, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23280.000000 2026-03-15 16:25:37,443 [Rank train_rank0] [Epoch 0 | Step 23300] train/loss: 0.448473, train/mse: 0.448473, train/data_wait_sec: 0.314358, train/step_wall_sec: 5.757123, train/data_wait_ratio: 0.054603, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23300.000000 2026-03-15 16:27:34,750 [Rank train_rank0] [Epoch 0 | Step 23320] train/loss: 0.485628, train/mse: 0.485628, train/data_wait_sec: 0.365240, train/step_wall_sec: 5.872299, train/data_wait_ratio: 0.062197, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23320.000000 2026-03-15 16:29:31,772 [Rank train_rank0] [Epoch 0 | Step 23340] train/loss: 0.445518, train/mse: 0.445518, train/data_wait_sec: 0.302532, train/step_wall_sec: 5.784111, train/data_wait_ratio: 0.052304, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23340.000000 2026-03-15 16:31:28,320 [Rank train_rank0] [Epoch 0 | Step 23360] train/loss: 0.438802, train/mse: 0.438802, train/data_wait_sec: 0.378796, train/step_wall_sec: 5.954260, train/data_wait_ratio: 0.063618, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23360.000000 2026-03-15 16:33:24,327 [Rank train_rank0] [Epoch 0 | Step 23380] train/loss: 0.460194, train/mse: 0.460194, train/data_wait_sec: 0.240272, train/step_wall_sec: 5.702607, train/data_wait_ratio: 0.042134, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23380.000000 2026-03-15 16:35:21,208 [Rank train_rank0] [Epoch 0 | Step 23400] train/loss: 0.477916, train/mse: 0.477916, train/data_wait_sec: 0.399900, train/step_wall_sec: 5.946837, train/data_wait_ratio: 0.067246, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23400.000000 2026-03-15 16:37:18,317 [Rank train_rank0] [Epoch 0 | Step 23420] train/loss: 0.480727, train/mse: 0.480727, train/data_wait_sec: 0.281265, train/step_wall_sec: 5.790108, train/data_wait_ratio: 0.048577, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23420.000000 2026-03-15 16:39:15,605 [Rank train_rank0] [Epoch 0 | Step 23440] train/loss: 0.467186, train/mse: 0.467186, train/data_wait_sec: 0.436988, train/step_wall_sec: 5.992753, train/data_wait_ratio: 0.072919, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23440.000000 2026-03-15 16:41:12,280 [Rank train_rank0] [Epoch 0 | Step 23460] train/loss: 0.428032, train/mse: 0.428032, train/data_wait_sec: 0.254700, train/step_wall_sec: 5.735485, train/data_wait_ratio: 0.044408, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23460.000000 2026-03-15 16:43:08,856 [Rank train_rank0] [Epoch 0 | Step 23480] train/loss: 0.454223, train/mse: 0.454223, train/data_wait_sec: 0.280075, train/step_wall_sec: 5.836080, train/data_wait_ratio: 0.047990, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23480.000000 2026-03-15 16:45:05,460 [Rank train_rank0] [Epoch 0 | Step 23500] train/loss: 0.469176, train/mse: 0.469176, train/data_wait_sec: 0.270657, train/step_wall_sec: 5.825726, train/data_wait_ratio: 0.046459, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23500.000000 2026-03-15 16:47:02,923 [Rank train_rank0] [Epoch 0 | Step 23520] train/loss: 0.464360, train/mse: 0.464360, train/data_wait_sec: 0.322327, train/step_wall_sec: 5.746439, train/data_wait_ratio: 0.056092, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23520.000000 2026-03-15 16:48:59,573 [Rank train_rank0] [Epoch 0 | Step 23540] train/loss: 0.464748, train/mse: 0.464748, train/data_wait_sec: 0.193046, train/step_wall_sec: 5.700536, train/data_wait_ratio: 0.033865, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23540.000000 2026-03-15 16:50:57,228 [Rank train_rank0] [Epoch 0 | Step 23560] train/loss: 0.440903, train/mse: 0.440903, train/data_wait_sec: 0.290159, train/step_wall_sec: 5.822931, train/data_wait_ratio: 0.049830, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23560.000000 2026-03-15 16:52:54,578 [Rank train_rank0] [Epoch 0 | Step 23580] train/loss: 0.441906, train/mse: 0.441906, train/data_wait_sec: 0.370923, train/step_wall_sec: 5.855374, train/data_wait_ratio: 0.063347, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23580.000000 2026-03-15 16:54:51,672 [Rank train_rank0] [Epoch 0 | Step 23600] train/loss: 0.467466, train/mse: 0.467466, train/data_wait_sec: 0.275421, train/step_wall_sec: 5.779604, train/data_wait_ratio: 0.047654, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23600.000000 2026-03-15 16:56:48,486 [Rank train_rank0] [Epoch 0 | Step 23620] train/loss: 0.446924, train/mse: 0.446924, train/data_wait_sec: 0.299898, train/step_wall_sec: 5.804146, train/data_wait_ratio: 0.051670, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23620.000000 2026-03-15 16:58:45,650 [Rank train_rank0] [Epoch 0 | Step 23640] train/loss: 0.485470, train/mse: 0.485470, train/data_wait_sec: 0.303412, train/step_wall_sec: 5.774306, train/data_wait_ratio: 0.052545, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23640.000000 2026-03-15 17:00:42,402 [Rank train_rank0] [Epoch 0 | Step 23660] train/loss: 0.441374, train/mse: 0.441374, train/data_wait_sec: 0.382975, train/step_wall_sec: 5.875721, train/data_wait_ratio: 0.065179, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23660.000000 2026-03-15 17:02:39,943 [Rank train_rank0] [Epoch 0 | Step 23680] train/loss: 0.484777, train/mse: 0.484777, train/data_wait_sec: 0.340257, train/step_wall_sec: 5.779208, train/data_wait_ratio: 0.058876, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23680.000000 2026-03-15 17:04:37,323 [Rank train_rank0] [Epoch 0 | Step 23700] train/loss: 0.484825, train/mse: 0.484825, train/data_wait_sec: 0.373450, train/step_wall_sec: 5.941258, train/data_wait_ratio: 0.062857, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23700.000000 2026-03-15 17:06:33,608 [Rank train_rank0] [Epoch 0 | Step 23720] train/loss: 0.445972, train/mse: 0.445972, train/data_wait_sec: 0.241510, train/step_wall_sec: 5.794105, train/data_wait_ratio: 0.041682, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23720.000000 2026-03-15 17:08:29,880 [Rank train_rank0] [Epoch 0 | Step 23740] train/loss: 0.445512, train/mse: 0.445512, train/data_wait_sec: 0.211565, train/step_wall_sec: 5.733060, train/data_wait_ratio: 0.036903, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23740.000000 2026-03-15 17:10:26,792 [Rank train_rank0] [Epoch 0 | Step 23760] train/loss: 0.468279, train/mse: 0.468279, train/data_wait_sec: 0.344299, train/step_wall_sec: 5.837465, train/data_wait_ratio: 0.058981, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23760.000000 2026-03-15 17:12:23,742 [Rank train_rank0] [Epoch 0 | Step 23780] train/loss: 0.465847, train/mse: 0.465847, train/data_wait_sec: 0.337232, train/step_wall_sec: 5.823328, train/data_wait_ratio: 0.057911, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23780.000000 2026-03-15 17:14:21,183 [Rank train_rank0] [Epoch 0 | Step 23800] train/loss: 0.453851, train/mse: 0.453851, train/data_wait_sec: 0.261041, train/step_wall_sec: 5.832239, train/data_wait_ratio: 0.044758, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23800.000000 2026-03-15 17:16:17,816 [Rank train_rank0] [Epoch 0 | Step 23820] train/loss: 0.463299, train/mse: 0.463299, train/data_wait_sec: 0.231619, train/step_wall_sec: 5.676220, train/data_wait_ratio: 0.040805, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23820.000000 2026-03-15 17:18:14,847 [Rank train_rank0] [Epoch 0 | Step 23840] train/loss: 0.454184, train/mse: 0.454184, train/data_wait_sec: 0.278114, train/step_wall_sec: 5.723892, train/data_wait_ratio: 0.048588, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23840.000000 2026-03-15 17:20:11,805 [Rank train_rank0] [Epoch 0 | Step 23860] train/loss: 0.433939, train/mse: 0.433939, train/data_wait_sec: 0.269114, train/step_wall_sec: 5.817620, train/data_wait_ratio: 0.046258, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23860.000000 2026-03-15 17:22:09,479 [Rank train_rank0] [Epoch 0 | Step 23880] train/loss: 0.456930, train/mse: 0.456930, train/data_wait_sec: 0.239016, train/step_wall_sec: 5.766849, train/data_wait_ratio: 0.041447, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23880.000000 2026-03-15 17:24:06,244 [Rank train_rank0] [Epoch 0 | Step 23900] train/loss: 0.474057, train/mse: 0.474057, train/data_wait_sec: 0.344771, train/step_wall_sec: 5.854991, train/data_wait_ratio: 0.058885, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23900.000000 2026-03-15 17:26:03,450 [Rank train_rank0] [Epoch 0 | Step 23920] train/loss: 0.446191, train/mse: 0.446191, train/data_wait_sec: 0.309562, train/step_wall_sec: 5.836839, train/data_wait_ratio: 0.053036, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23920.000000 2026-03-15 17:28:00,483 [Rank train_rank0] [Epoch 0 | Step 23940] train/loss: 0.446856, train/mse: 0.446856, train/data_wait_sec: 0.358986, train/step_wall_sec: 5.841557, train/data_wait_ratio: 0.061454, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23940.000000 2026-03-15 17:29:57,926 [Rank train_rank0] [Epoch 0 | Step 23960] train/loss: 0.441848, train/mse: 0.441848, train/data_wait_sec: 0.289415, train/step_wall_sec: 5.897451, train/data_wait_ratio: 0.049075, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23960.000000 2026-03-15 17:31:54,891 [Rank train_rank0] [Epoch 0 | Step 23980] train/loss: 0.444868, train/mse: 0.444868, train/data_wait_sec: 0.205402, train/step_wall_sec: 5.696243, train/data_wait_ratio: 0.036059, train/lr: 0.000200, train/epoch: 0.000000, train/step: 23980.000000 2026-03-15 17:34:17,129 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000024000.pt 2026-03-15 17:34:17,132 [Rank train_rank0] [Epoch 0 | Step 24000] train/loss: 0.455169, train/mse: 0.455169, train/data_wait_sec: 0.277893, train/step_wall_sec: 5.762429, train/data_wait_ratio: 0.048225, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24000.000000 2026-03-15 17:34:17,133 [Rank train_rank0] Generating samples for visualization... 2026-03-15 17:34:20,324 [Rank train_rank0] Sample generation done. 2026-03-15 17:36:17,711 [Rank train_rank0] [Epoch 0 | Step 24020] train/loss: 0.455382, train/mse: 0.455382, train/data_wait_sec: 0.315765, train/step_wall_sec: 5.845310, train/data_wait_ratio: 0.054020, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24020.000000 2026-03-15 17:38:16,000 [Rank train_rank0] [Epoch 0 | Step 24040] train/loss: 0.469647, train/mse: 0.469647, train/data_wait_sec: 0.406253, train/step_wall_sec: 5.926491, train/data_wait_ratio: 0.068549, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24040.000000 2026-03-15 17:40:13,341 [Rank train_rank0] [Epoch 0 | Step 24060] train/loss: 0.452306, train/mse: 0.452306, train/data_wait_sec: 0.317705, train/step_wall_sec: 5.830758, train/data_wait_ratio: 0.054488, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24060.000000 2026-03-15 17:42:09,708 [Rank train_rank0] [Epoch 0 | Step 24080] train/loss: 0.453301, train/mse: 0.453301, train/data_wait_sec: 0.240516, train/step_wall_sec: 5.685434, train/data_wait_ratio: 0.042304, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24080.000000 2026-03-15 17:44:06,852 [Rank train_rank0] [Epoch 0 | Step 24100] train/loss: 0.451674, train/mse: 0.451674, train/data_wait_sec: 0.363132, train/step_wall_sec: 5.853188, train/data_wait_ratio: 0.062040, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24100.000000 2026-03-15 17:46:03,640 [Rank train_rank0] [Epoch 0 | Step 24120] train/loss: 0.439997, train/mse: 0.439997, train/data_wait_sec: 0.296418, train/step_wall_sec: 5.820080, train/data_wait_ratio: 0.050930, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24120.000000 2026-03-15 17:48:00,934 [Rank train_rank0] [Epoch 0 | Step 24140] train/loss: 0.473857, train/mse: 0.473857, train/data_wait_sec: 0.379299, train/step_wall_sec: 5.897859, train/data_wait_ratio: 0.064311, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24140.000000 2026-03-15 17:49:57,919 [Rank train_rank0] [Epoch 0 | Step 24160] train/loss: 0.448673, train/mse: 0.448673, train/data_wait_sec: 0.360396, train/step_wall_sec: 5.914645, train/data_wait_ratio: 0.060933, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24160.000000 2026-03-15 17:51:55,431 [Rank train_rank0] [Epoch 0 | Step 24180] train/loss: 0.461379, train/mse: 0.461379, train/data_wait_sec: 0.270336, train/step_wall_sec: 5.702191, train/data_wait_ratio: 0.047409, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24180.000000 2026-03-15 17:53:52,561 [Rank train_rank0] [Epoch 0 | Step 24200] train/loss: 0.458282, train/mse: 0.458282, train/data_wait_sec: 0.368005, train/step_wall_sec: 5.818288, train/data_wait_ratio: 0.063250, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24200.000000 2026-03-15 17:55:49,562 [Rank train_rank0] [Epoch 0 | Step 24220] train/loss: 0.440505, train/mse: 0.440505, train/data_wait_sec: 0.303624, train/step_wall_sec: 5.802644, train/data_wait_ratio: 0.052325, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24220.000000 2026-03-15 17:57:46,498 [Rank train_rank0] [Epoch 0 | Step 24240] train/loss: 0.439460, train/mse: 0.439460, train/data_wait_sec: 0.326774, train/step_wall_sec: 5.840907, train/data_wait_ratio: 0.055946, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24240.000000 2026-03-15 17:59:43,755 [Rank train_rank0] [Epoch 0 | Step 24260] train/loss: 0.488289, train/mse: 0.488289, train/data_wait_sec: 0.299267, train/step_wall_sec: 5.809623, train/data_wait_ratio: 0.051512, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24260.000000 2026-03-15 18:01:41,041 [Rank train_rank0] [Epoch 0 | Step 24280] train/loss: 0.467977, train/mse: 0.467977, train/data_wait_sec: 0.306025, train/step_wall_sec: 5.855211, train/data_wait_ratio: 0.052265, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24280.000000 2026-03-15 18:03:37,616 [Rank train_rank0] [Epoch 0 | Step 24300] train/loss: 0.461054, train/mse: 0.461054, train/data_wait_sec: 0.300996, train/step_wall_sec: 5.787594, train/data_wait_ratio: 0.052007, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24300.000000 2026-03-15 18:05:34,348 [Rank train_rank0] [Epoch 0 | Step 24320] train/loss: 0.461959, train/mse: 0.461959, train/data_wait_sec: 0.328227, train/step_wall_sec: 5.759462, train/data_wait_ratio: 0.056989, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24320.000000 2026-03-15 18:07:31,668 [Rank train_rank0] [Epoch 0 | Step 24340] train/loss: 0.455416, train/mse: 0.455416, train/data_wait_sec: 0.436483, train/step_wall_sec: 5.929138, train/data_wait_ratio: 0.073617, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24340.000000 2026-03-15 18:09:29,121 [Rank train_rank0] [Epoch 0 | Step 24360] train/loss: 0.455317, train/mse: 0.455317, train/data_wait_sec: 0.224171, train/step_wall_sec: 5.766428, train/data_wait_ratio: 0.038875, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24360.000000 2026-03-15 18:11:26,155 [Rank train_rank0] [Epoch 0 | Step 24380] train/loss: 0.455136, train/mse: 0.455136, train/data_wait_sec: 0.267676, train/step_wall_sec: 5.838825, train/data_wait_ratio: 0.045844, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24380.000000 2026-03-15 18:13:22,461 [Rank train_rank0] [Epoch 0 | Step 24400] train/loss: 0.481089, train/mse: 0.481089, train/data_wait_sec: 0.281580, train/step_wall_sec: 5.884577, train/data_wait_ratio: 0.047850, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24400.000000 2026-03-15 18:15:19,314 [Rank train_rank0] [Epoch 0 | Step 24420] train/loss: 0.454909, train/mse: 0.454909, train/data_wait_sec: 0.240347, train/step_wall_sec: 5.747726, train/data_wait_ratio: 0.041816, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24420.000000 2026-03-15 18:17:16,004 [Rank train_rank0] [Epoch 0 | Step 24440] train/loss: 0.450450, train/mse: 0.450450, train/data_wait_sec: 0.223880, train/step_wall_sec: 5.651071, train/data_wait_ratio: 0.039617, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24440.000000 2026-03-15 18:19:12,779 [Rank train_rank0] [Epoch 0 | Step 24460] train/loss: 0.463381, train/mse: 0.463381, train/data_wait_sec: 0.281648, train/step_wall_sec: 5.803354, train/data_wait_ratio: 0.048532, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24460.000000 2026-03-15 18:21:10,137 [Rank train_rank0] [Epoch 0 | Step 24480] train/loss: 0.469424, train/mse: 0.469424, train/data_wait_sec: 0.297732, train/step_wall_sec: 5.880813, train/data_wait_ratio: 0.050628, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24480.000000 2026-03-15 18:23:07,489 [Rank train_rank0] [Epoch 0 | Step 24500] train/loss: 0.461364, train/mse: 0.461364, train/data_wait_sec: 0.356788, train/step_wall_sec: 5.855424, train/data_wait_ratio: 0.060933, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24500.000000 2026-03-15 18:25:04,090 [Rank train_rank0] [Epoch 0 | Step 24520] train/loss: 0.443517, train/mse: 0.443517, train/data_wait_sec: 0.325937, train/step_wall_sec: 5.773688, train/data_wait_ratio: 0.056452, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24520.000000 2026-03-15 18:27:01,343 [Rank train_rank0] [Epoch 0 | Step 24540] train/loss: 0.485577, train/mse: 0.485577, train/data_wait_sec: 0.338461, train/step_wall_sec: 5.878234, train/data_wait_ratio: 0.057579, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24540.000000 2026-03-15 18:28:58,219 [Rank train_rank0] [Epoch 0 | Step 24560] train/loss: 0.452195, train/mse: 0.452195, train/data_wait_sec: 0.297242, train/step_wall_sec: 5.792748, train/data_wait_ratio: 0.051313, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24560.000000 2026-03-15 18:30:55,225 [Rank train_rank0] [Epoch 0 | Step 24580] train/loss: 0.463111, train/mse: 0.463111, train/data_wait_sec: 0.368821, train/step_wall_sec: 5.868595, train/data_wait_ratio: 0.062847, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24580.000000 2026-03-15 18:32:52,602 [Rank train_rank0] [Epoch 0 | Step 24600] train/loss: 0.445253, train/mse: 0.445253, train/data_wait_sec: 0.299103, train/step_wall_sec: 5.812462, train/data_wait_ratio: 0.051459, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24600.000000 2026-03-15 18:34:49,376 [Rank train_rank0] [Epoch 0 | Step 24620] train/loss: 0.451675, train/mse: 0.451675, train/data_wait_sec: 0.351848, train/step_wall_sec: 5.838173, train/data_wait_ratio: 0.060267, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24620.000000 2026-03-15 18:36:45,859 [Rank train_rank0] [Epoch 0 | Step 24640] train/loss: 0.459326, train/mse: 0.459326, train/data_wait_sec: 0.235685, train/step_wall_sec: 5.710382, train/data_wait_ratio: 0.041273, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24640.000000 2026-03-15 18:38:42,544 [Rank train_rank0] [Epoch 0 | Step 24660] train/loss: 0.431056, train/mse: 0.431056, train/data_wait_sec: 0.295009, train/step_wall_sec: 5.818727, train/data_wait_ratio: 0.050700, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24660.000000 2026-03-15 18:40:39,524 [Rank train_rank0] [Epoch 0 | Step 24680] train/loss: 0.448838, train/mse: 0.448838, train/data_wait_sec: 0.319865, train/step_wall_sec: 5.775524, train/data_wait_ratio: 0.055383, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24680.000000 2026-03-15 18:42:36,550 [Rank train_rank0] [Epoch 0 | Step 24700] train/loss: 0.456576, train/mse: 0.456576, train/data_wait_sec: 0.377078, train/step_wall_sec: 5.943742, train/data_wait_ratio: 0.063441, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24700.000000 2026-03-15 18:44:32,733 [Rank train_rank0] [Epoch 0 | Step 24720] train/loss: 0.445950, train/mse: 0.445950, train/data_wait_sec: 0.303745, train/step_wall_sec: 5.781788, train/data_wait_ratio: 0.052535, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24720.000000 2026-03-15 18:46:29,946 [Rank train_rank0] [Epoch 0 | Step 24740] train/loss: 0.455486, train/mse: 0.455486, train/data_wait_sec: 0.271302, train/step_wall_sec: 5.806707, train/data_wait_ratio: 0.046722, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24740.000000 2026-03-15 18:48:26,651 [Rank train_rank0] [Epoch 0 | Step 24760] train/loss: 0.461290, train/mse: 0.461290, train/data_wait_sec: 0.339271, train/step_wall_sec: 5.850086, train/data_wait_ratio: 0.057994, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24760.000000 2026-03-15 18:50:23,429 [Rank train_rank0] [Epoch 0 | Step 24780] train/loss: 0.450176, train/mse: 0.450176, train/data_wait_sec: 0.464532, train/step_wall_sec: 6.030426, train/data_wait_ratio: 0.077031, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24780.000000 2026-03-15 18:52:20,348 [Rank train_rank0] [Epoch 0 | Step 24800] train/loss: 0.485625, train/mse: 0.485625, train/data_wait_sec: 0.282826, train/step_wall_sec: 5.743875, train/data_wait_ratio: 0.049240, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24800.000000 2026-03-15 18:54:17,110 [Rank train_rank0] [Epoch 0 | Step 24820] train/loss: 0.467364, train/mse: 0.467364, train/data_wait_sec: 0.295877, train/step_wall_sec: 5.757586, train/data_wait_ratio: 0.051389, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24820.000000 2026-03-15 18:56:14,199 [Rank train_rank0] [Epoch 0 | Step 24840] train/loss: 0.488009, train/mse: 0.488009, train/data_wait_sec: 0.299465, train/step_wall_sec: 5.801499, train/data_wait_ratio: 0.051619, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24840.000000 2026-03-15 18:58:11,184 [Rank train_rank0] [Epoch 0 | Step 24860] train/loss: 0.460948, train/mse: 0.460948, train/data_wait_sec: 0.352819, train/step_wall_sec: 5.874456, train/data_wait_ratio: 0.060060, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24860.000000 2026-03-15 19:00:08,136 [Rank train_rank0] [Epoch 0 | Step 24880] train/loss: 0.449079, train/mse: 0.449079, train/data_wait_sec: 0.302343, train/step_wall_sec: 5.731267, train/data_wait_ratio: 0.052753, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24880.000000 2026-03-15 19:02:05,095 [Rank train_rank0] [Epoch 0 | Step 24900] train/loss: 0.473068, train/mse: 0.473068, train/data_wait_sec: 0.350712, train/step_wall_sec: 5.753857, train/data_wait_ratio: 0.060952, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24900.000000 2026-03-15 19:04:01,943 [Rank train_rank0] [Epoch 0 | Step 24920] train/loss: 0.450214, train/mse: 0.450214, train/data_wait_sec: 0.476048, train/step_wall_sec: 5.997140, train/data_wait_ratio: 0.079379, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24920.000000 2026-03-15 19:05:58,638 [Rank train_rank0] [Epoch 0 | Step 24940] train/loss: 0.424966, train/mse: 0.424966, train/data_wait_sec: 0.253388, train/step_wall_sec: 5.760059, train/data_wait_ratio: 0.043990, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24940.000000 2026-03-15 19:07:55,407 [Rank train_rank0] [Epoch 0 | Step 24960] train/loss: 0.448473, train/mse: 0.448473, train/data_wait_sec: 0.299121, train/step_wall_sec: 5.712544, train/data_wait_ratio: 0.052362, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24960.000000 2026-03-15 19:09:51,957 [Rank train_rank0] [Epoch 0 | Step 24980] train/loss: 0.445968, train/mse: 0.445968, train/data_wait_sec: 0.279108, train/step_wall_sec: 5.753108, train/data_wait_ratio: 0.048514, train/lr: 0.000200, train/epoch: 0.000000, train/step: 24980.000000 2026-03-15 19:12:13,767 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000025000.pt 2026-03-15 19:12:13,769 [Rank train_rank0] [Epoch 0 | Step 25000] train/loss: 0.450644, train/mse: 0.450644, train/data_wait_sec: 0.318994, train/step_wall_sec: 5.795450, train/data_wait_ratio: 0.055042, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25000.000000 2026-03-15 19:12:13,770 [Rank train_rank0] Generating samples for visualization... 2026-03-15 19:12:16,936 [Rank train_rank0] Sample generation done. 2026-03-15 19:14:14,026 [Rank train_rank0] [Epoch 0 | Step 25020] train/loss: 0.436503, train/mse: 0.436503, train/data_wait_sec: 0.347154, train/step_wall_sec: 5.843118, train/data_wait_ratio: 0.059412, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25020.000000 2026-03-15 19:16:11,191 [Rank train_rank0] [Epoch 0 | Step 25040] train/loss: 0.477025, train/mse: 0.477025, train/data_wait_sec: 0.343220, train/step_wall_sec: 5.826486, train/data_wait_ratio: 0.058907, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25040.000000 2026-03-15 19:18:08,603 [Rank train_rank0] [Epoch 0 | Step 25060] train/loss: 0.455973, train/mse: 0.455973, train/data_wait_sec: 0.370186, train/step_wall_sec: 5.999728, train/data_wait_ratio: 0.061701, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25060.000000 2026-03-15 19:20:05,194 [Rank train_rank0] [Epoch 0 | Step 25080] train/loss: 0.470241, train/mse: 0.470241, train/data_wait_sec: 0.254410, train/step_wall_sec: 5.746132, train/data_wait_ratio: 0.044275, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25080.000000 2026-03-15 19:22:02,266 [Rank train_rank0] [Epoch 0 | Step 25100] train/loss: 0.433187, train/mse: 0.433187, train/data_wait_sec: 0.313868, train/step_wall_sec: 5.855533, train/data_wait_ratio: 0.053602, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25100.000000 2026-03-15 19:23:59,063 [Rank train_rank0] [Epoch 0 | Step 25120] train/loss: 0.433456, train/mse: 0.433456, train/data_wait_sec: 0.211341, train/step_wall_sec: 5.685143, train/data_wait_ratio: 0.037174, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25120.000000 2026-03-15 19:25:55,736 [Rank train_rank0] [Epoch 0 | Step 25140] train/loss: 0.439991, train/mse: 0.439991, train/data_wait_sec: 0.267751, train/step_wall_sec: 5.799520, train/data_wait_ratio: 0.046168, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25140.000000 2026-03-15 19:27:52,430 [Rank train_rank0] [Epoch 0 | Step 25160] train/loss: 0.455305, train/mse: 0.455305, train/data_wait_sec: 0.346814, train/step_wall_sec: 5.760031, train/data_wait_ratio: 0.060210, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25160.000000 2026-03-15 19:29:49,567 [Rank train_rank0] [Epoch 0 | Step 25180] train/loss: 0.478393, train/mse: 0.478393, train/data_wait_sec: 0.305247, train/step_wall_sec: 5.712766, train/data_wait_ratio: 0.053432, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25180.000000 2026-03-15 19:31:46,707 [Rank train_rank0] [Epoch 0 | Step 25200] train/loss: 0.462438, train/mse: 0.462438, train/data_wait_sec: 0.299335, train/step_wall_sec: 5.808966, train/data_wait_ratio: 0.051530, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25200.000000 2026-03-15 19:33:44,452 [Rank train_rank0] [Epoch 0 | Step 25220] train/loss: 0.428475, train/mse: 0.428475, train/data_wait_sec: 0.365055, train/step_wall_sec: 5.942611, train/data_wait_ratio: 0.061430, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25220.000000 2026-03-15 19:35:40,573 [Rank train_rank0] [Epoch 0 | Step 25240] train/loss: 0.467729, train/mse: 0.467729, train/data_wait_sec: 0.339593, train/step_wall_sec: 5.814371, train/data_wait_ratio: 0.058406, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25240.000000 2026-03-15 19:37:37,290 [Rank train_rank0] [Epoch 0 | Step 25260] train/loss: 0.432123, train/mse: 0.432123, train/data_wait_sec: 0.205612, train/step_wall_sec: 5.626019, train/data_wait_ratio: 0.036547, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25260.000000 2026-03-15 19:39:34,457 [Rank train_rank0] [Epoch 0 | Step 25280] train/loss: 0.451288, train/mse: 0.451288, train/data_wait_sec: 0.294633, train/step_wall_sec: 5.822212, train/data_wait_ratio: 0.050605, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25280.000000 2026-03-15 19:41:31,149 [Rank train_rank0] [Epoch 0 | Step 25300] train/loss: 0.445712, train/mse: 0.445712, train/data_wait_sec: 0.322310, train/step_wall_sec: 5.946683, train/data_wait_ratio: 0.054200, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25300.000000 2026-03-15 19:43:28,021 [Rank train_rank0] [Epoch 0 | Step 25320] train/loss: 0.480173, train/mse: 0.480173, train/data_wait_sec: 0.329760, train/step_wall_sec: 5.796561, train/data_wait_ratio: 0.056889, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25320.000000 2026-03-15 19:45:24,242 [Rank train_rank0] [Epoch 0 | Step 25340] train/loss: 0.475150, train/mse: 0.475150, train/data_wait_sec: 0.238059, train/step_wall_sec: 5.665070, train/data_wait_ratio: 0.042022, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25340.000000 2026-03-15 19:47:21,248 [Rank train_rank0] [Epoch 0 | Step 25360] train/loss: 0.451869, train/mse: 0.451869, train/data_wait_sec: 0.266781, train/step_wall_sec: 5.764331, train/data_wait_ratio: 0.046281, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25360.000000 2026-03-15 19:49:18,226 [Rank train_rank0] [Epoch 0 | Step 25380] train/loss: 0.439429, train/mse: 0.439429, train/data_wait_sec: 0.301789, train/step_wall_sec: 5.860490, train/data_wait_ratio: 0.051495, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25380.000000 2026-03-15 19:51:15,470 [Rank train_rank0] [Epoch 0 | Step 25400] train/loss: 0.470534, train/mse: 0.470534, train/data_wait_sec: 0.422798, train/step_wall_sec: 6.026267, train/data_wait_ratio: 0.070159, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25400.000000 2026-03-15 19:53:12,986 [Rank train_rank0] [Epoch 0 | Step 25420] train/loss: 0.478418, train/mse: 0.478418, train/data_wait_sec: 0.285792, train/step_wall_sec: 5.823804, train/data_wait_ratio: 0.049073, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25420.000000 2026-03-15 19:55:09,424 [Rank train_rank0] [Epoch 0 | Step 25440] train/loss: 0.473806, train/mse: 0.473806, train/data_wait_sec: 0.248642, train/step_wall_sec: 5.793863, train/data_wait_ratio: 0.042915, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25440.000000 2026-03-15 19:57:06,278 [Rank train_rank0] [Epoch 0 | Step 25460] train/loss: 0.443065, train/mse: 0.443065, train/data_wait_sec: 0.268099, train/step_wall_sec: 5.915416, train/data_wait_ratio: 0.045322, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25460.000000 2026-03-15 19:59:03,733 [Rank train_rank0] [Epoch 0 | Step 25480] train/loss: 0.443266, train/mse: 0.443266, train/data_wait_sec: 0.303206, train/step_wall_sec: 5.899974, train/data_wait_ratio: 0.051391, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25480.000000 2026-03-15 20:01:00,349 [Rank train_rank0] [Epoch 0 | Step 25500] train/loss: 0.455930, train/mse: 0.455930, train/data_wait_sec: 0.233482, train/step_wall_sec: 5.728647, train/data_wait_ratio: 0.040757, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25500.000000 2026-03-15 20:02:57,329 [Rank train_rank0] [Epoch 0 | Step 25520] train/loss: 0.438786, train/mse: 0.438786, train/data_wait_sec: 0.301241, train/step_wall_sec: 5.758585, train/data_wait_ratio: 0.052312, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25520.000000 2026-03-15 20:04:54,237 [Rank train_rank0] [Epoch 0 | Step 25540] train/loss: 0.461927, train/mse: 0.461927, train/data_wait_sec: 0.286254, train/step_wall_sec: 5.689976, train/data_wait_ratio: 0.050308, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25540.000000 2026-03-15 20:06:51,028 [Rank train_rank0] [Epoch 0 | Step 25560] train/loss: 0.471374, train/mse: 0.471374, train/data_wait_sec: 0.228796, train/step_wall_sec: 5.746125, train/data_wait_ratio: 0.039817, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25560.000000 2026-03-15 20:08:47,775 [Rank train_rank0] [Epoch 0 | Step 25580] train/loss: 0.446698, train/mse: 0.446698, train/data_wait_sec: 0.318824, train/step_wall_sec: 5.841186, train/data_wait_ratio: 0.054582, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25580.000000 2026-03-15 20:10:44,495 [Rank train_rank0] [Epoch 0 | Step 25600] train/loss: 0.428767, train/mse: 0.428767, train/data_wait_sec: 0.333778, train/step_wall_sec: 5.804306, train/data_wait_ratio: 0.057505, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25600.000000 2026-03-15 20:12:41,181 [Rank train_rank0] [Epoch 0 | Step 25620] train/loss: 0.455792, train/mse: 0.455792, train/data_wait_sec: 0.266390, train/step_wall_sec: 5.743974, train/data_wait_ratio: 0.046377, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25620.000000 2026-03-15 20:14:38,314 [Rank train_rank0] [Epoch 0 | Step 25640] train/loss: 0.461555, train/mse: 0.461555, train/data_wait_sec: 0.348235, train/step_wall_sec: 5.898538, train/data_wait_ratio: 0.059037, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25640.000000 2026-03-15 20:16:34,594 [Rank train_rank0] [Epoch 0 | Step 25660] train/loss: 0.455574, train/mse: 0.455574, train/data_wait_sec: 0.207420, train/step_wall_sec: 5.721168, train/data_wait_ratio: 0.036255, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25660.000000 2026-03-15 20:18:31,377 [Rank train_rank0] [Epoch 0 | Step 25680] train/loss: 0.452198, train/mse: 0.452198, train/data_wait_sec: 0.291858, train/step_wall_sec: 5.881789, train/data_wait_ratio: 0.049621, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25680.000000 2026-03-15 20:20:28,565 [Rank train_rank0] [Epoch 0 | Step 25700] train/loss: 0.437453, train/mse: 0.437453, train/data_wait_sec: 0.240987, train/step_wall_sec: 5.746553, train/data_wait_ratio: 0.041936, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25700.000000 2026-03-15 20:22:25,239 [Rank train_rank0] [Epoch 0 | Step 25720] train/loss: 0.443309, train/mse: 0.443309, train/data_wait_sec: 0.190221, train/step_wall_sec: 5.692260, train/data_wait_ratio: 0.033418, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25720.000000 2026-03-15 20:24:22,654 [Rank train_rank0] [Epoch 0 | Step 25740] train/loss: 0.471850, train/mse: 0.471850, train/data_wait_sec: 0.285682, train/step_wall_sec: 5.858706, train/data_wait_ratio: 0.048762, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25740.000000 2026-03-15 20:26:18,796 [Rank train_rank0] [Epoch 0 | Step 25760] train/loss: 0.431688, train/mse: 0.431688, train/data_wait_sec: 0.256985, train/step_wall_sec: 5.770968, train/data_wait_ratio: 0.044531, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25760.000000 2026-03-15 20:28:15,297 [Rank train_rank0] [Epoch 0 | Step 25780] train/loss: 0.442479, train/mse: 0.442479, train/data_wait_sec: 0.244048, train/step_wall_sec: 5.729812, train/data_wait_ratio: 0.042593, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25780.000000 2026-03-15 20:30:12,062 [Rank train_rank0] [Epoch 0 | Step 25800] train/loss: 0.464524, train/mse: 0.464524, train/data_wait_sec: 0.280201, train/step_wall_sec: 5.895527, train/data_wait_ratio: 0.047528, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25800.000000 2026-03-15 20:32:08,987 [Rank train_rank0] [Epoch 0 | Step 25820] train/loss: 0.489540, train/mse: 0.489540, train/data_wait_sec: 0.214416, train/step_wall_sec: 5.829427, train/data_wait_ratio: 0.036782, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25820.000000 2026-03-15 20:34:06,267 [Rank train_rank0] [Epoch 0 | Step 25840] train/loss: 0.468352, train/mse: 0.468352, train/data_wait_sec: 0.245403, train/step_wall_sec: 5.685992, train/data_wait_ratio: 0.043159, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25840.000000 2026-03-15 20:36:03,813 [Rank train_rank0] [Epoch 0 | Step 25860] train/loss: 0.427318, train/mse: 0.427318, train/data_wait_sec: 0.369893, train/step_wall_sec: 5.991626, train/data_wait_ratio: 0.061735, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25860.000000 2026-03-15 20:38:00,582 [Rank train_rank0] [Epoch 0 | Step 25880] train/loss: 0.459412, train/mse: 0.459412, train/data_wait_sec: 0.273337, train/step_wall_sec: 5.848586, train/data_wait_ratio: 0.046736, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25880.000000 2026-03-15 20:39:57,384 [Rank train_rank0] [Epoch 0 | Step 25900] train/loss: 0.453354, train/mse: 0.453354, train/data_wait_sec: 0.244378, train/step_wall_sec: 5.747012, train/data_wait_ratio: 0.042523, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25900.000000 2026-03-15 20:41:54,185 [Rank train_rank0] [Epoch 0 | Step 25920] train/loss: 0.430937, train/mse: 0.430937, train/data_wait_sec: 0.242838, train/step_wall_sec: 5.755589, train/data_wait_ratio: 0.042192, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25920.000000 2026-03-15 20:43:50,642 [Rank train_rank0] [Epoch 0 | Step 25940] train/loss: 0.430768, train/mse: 0.430768, train/data_wait_sec: 0.289109, train/step_wall_sec: 5.810519, train/data_wait_ratio: 0.049756, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25940.000000 2026-03-15 20:45:47,769 [Rank train_rank0] [Epoch 0 | Step 25960] train/loss: 0.450828, train/mse: 0.450828, train/data_wait_sec: 0.347376, train/step_wall_sec: 5.889572, train/data_wait_ratio: 0.058982, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25960.000000 2026-03-15 20:47:44,451 [Rank train_rank0] [Epoch 0 | Step 25980] train/loss: 0.440001, train/mse: 0.440001, train/data_wait_sec: 0.176490, train/step_wall_sec: 5.802641, train/data_wait_ratio: 0.030415, train/lr: 0.000200, train/epoch: 0.000000, train/step: 25980.000000 2026-03-15 20:50:06,135 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000026000.pt 2026-03-15 20:50:06,139 [Rank train_rank0] [Epoch 0 | Step 26000] train/loss: 0.460404, train/mse: 0.460404, train/data_wait_sec: 0.340385, train/step_wall_sec: 5.830853, train/data_wait_ratio: 0.058377, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26000.000000 2026-03-15 20:50:06,140 [Rank train_rank0] Generating samples for visualization... 2026-03-15 20:50:09,246 [Rank train_rank0] Sample generation done. 2026-03-15 20:52:06,964 [Rank train_rank0] [Epoch 0 | Step 26020] train/loss: 0.462578, train/mse: 0.462578, train/data_wait_sec: 0.267731, train/step_wall_sec: 5.815724, train/data_wait_ratio: 0.046036, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26020.000000 2026-03-15 20:54:05,493 [Rank train_rank0] [Epoch 0 | Step 26040] train/loss: 0.433330, train/mse: 0.433330, train/data_wait_sec: 0.257060, train/step_wall_sec: 5.736524, train/data_wait_ratio: 0.044811, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26040.000000 2026-03-15 20:56:02,805 [Rank train_rank0] [Epoch 0 | Step 26060] train/loss: 0.463924, train/mse: 0.463924, train/data_wait_sec: 0.265011, train/step_wall_sec: 5.836874, train/data_wait_ratio: 0.045403, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26060.000000 2026-03-15 20:57:59,947 [Rank train_rank0] [Epoch 0 | Step 26080] train/loss: 0.453621, train/mse: 0.453621, train/data_wait_sec: 0.306777, train/step_wall_sec: 5.836075, train/data_wait_ratio: 0.052566, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26080.000000 2026-03-15 20:59:56,820 [Rank train_rank0] [Epoch 0 | Step 26100] train/loss: 0.461553, train/mse: 0.461553, train/data_wait_sec: 0.244413, train/step_wall_sec: 5.738658, train/data_wait_ratio: 0.042591, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26100.000000 2026-03-15 21:01:54,152 [Rank train_rank0] [Epoch 0 | Step 26120] train/loss: 0.469404, train/mse: 0.469404, train/data_wait_sec: 0.165510, train/step_wall_sec: 5.638773, train/data_wait_ratio: 0.029352, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26120.000000 2026-03-15 21:03:51,244 [Rank train_rank0] [Epoch 0 | Step 26140] train/loss: 0.483966, train/mse: 0.483966, train/data_wait_sec: 0.350456, train/step_wall_sec: 5.903718, train/data_wait_ratio: 0.059362, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26140.000000 2026-03-15 21:05:48,312 [Rank train_rank0] [Epoch 0 | Step 26160] train/loss: 0.454917, train/mse: 0.454917, train/data_wait_sec: 0.257137, train/step_wall_sec: 5.741939, train/data_wait_ratio: 0.044782, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26160.000000 2026-03-15 21:07:45,620 [Rank train_rank0] [Epoch 0 | Step 26180] train/loss: 0.455798, train/mse: 0.455798, train/data_wait_sec: 0.286430, train/step_wall_sec: 5.793778, train/data_wait_ratio: 0.049437, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26180.000000 2026-03-15 21:09:42,754 [Rank train_rank0] [Epoch 0 | Step 26200] train/loss: 0.475660, train/mse: 0.475660, train/data_wait_sec: 0.207777, train/step_wall_sec: 5.773478, train/data_wait_ratio: 0.035988, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26200.000000 2026-03-15 21:11:39,623 [Rank train_rank0] [Epoch 0 | Step 26220] train/loss: 0.462862, train/mse: 0.462862, train/data_wait_sec: 0.261649, train/step_wall_sec: 5.814820, train/data_wait_ratio: 0.044997, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26220.000000 2026-03-15 21:13:36,313 [Rank train_rank0] [Epoch 0 | Step 26240] train/loss: 0.446768, train/mse: 0.446768, train/data_wait_sec: 0.306623, train/step_wall_sec: 5.693216, train/data_wait_ratio: 0.053858, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26240.000000 2026-03-15 21:15:32,853 [Rank train_rank0] [Epoch 0 | Step 26260] train/loss: 0.453155, train/mse: 0.453155, train/data_wait_sec: 0.298561, train/step_wall_sec: 5.798636, train/data_wait_ratio: 0.051488, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26260.000000 2026-03-15 21:17:29,978 [Rank train_rank0] [Epoch 0 | Step 26280] train/loss: 0.450461, train/mse: 0.450461, train/data_wait_sec: 0.295720, train/step_wall_sec: 5.765457, train/data_wait_ratio: 0.051292, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26280.000000 2026-03-15 21:19:26,641 [Rank train_rank0] [Epoch 0 | Step 26300] train/loss: 0.455242, train/mse: 0.455242, train/data_wait_sec: 0.204781, train/step_wall_sec: 5.627204, train/data_wait_ratio: 0.036391, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26300.000000 2026-03-15 21:21:22,806 [Rank train_rank0] [Epoch 0 | Step 26320] train/loss: 0.446471, train/mse: 0.446471, train/data_wait_sec: 0.269465, train/step_wall_sec: 5.651729, train/data_wait_ratio: 0.047678, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26320.000000 2026-03-15 21:23:19,450 [Rank train_rank0] [Epoch 0 | Step 26340] train/loss: 0.471132, train/mse: 0.471132, train/data_wait_sec: 0.314124, train/step_wall_sec: 5.789768, train/data_wait_ratio: 0.054255, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26340.000000 2026-03-15 21:25:15,745 [Rank train_rank0] [Epoch 0 | Step 26360] train/loss: 0.442174, train/mse: 0.442174, train/data_wait_sec: 0.258484, train/step_wall_sec: 5.676264, train/data_wait_ratio: 0.045538, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26360.000000 2026-03-15 21:27:13,091 [Rank train_rank0] [Epoch 0 | Step 26380] train/loss: 0.459485, train/mse: 0.459485, train/data_wait_sec: 0.271178, train/step_wall_sec: 5.767741, train/data_wait_ratio: 0.047016, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26380.000000 2026-03-15 21:29:09,971 [Rank train_rank0] [Epoch 0 | Step 26400] train/loss: 0.453094, train/mse: 0.453094, train/data_wait_sec: 0.313419, train/step_wall_sec: 5.872287, train/data_wait_ratio: 0.053373, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26400.000000 2026-03-15 21:31:07,515 [Rank train_rank0] [Epoch 0 | Step 26420] train/loss: 0.460290, train/mse: 0.460290, train/data_wait_sec: 0.269770, train/step_wall_sec: 5.834890, train/data_wait_ratio: 0.046234, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26420.000000 2026-03-15 21:33:04,795 [Rank train_rank0] [Epoch 0 | Step 26440] train/loss: 0.452797, train/mse: 0.452797, train/data_wait_sec: 0.297433, train/step_wall_sec: 5.799037, train/data_wait_ratio: 0.051290, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26440.000000 2026-03-15 21:35:01,793 [Rank train_rank0] [Epoch 0 | Step 26460] train/loss: 0.445559, train/mse: 0.445559, train/data_wait_sec: 0.324349, train/step_wall_sec: 5.853178, train/data_wait_ratio: 0.055414, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26460.000000 2026-03-15 21:36:58,750 [Rank train_rank0] [Epoch 0 | Step 26480] train/loss: 0.486509, train/mse: 0.486509, train/data_wait_sec: 0.281876, train/step_wall_sec: 5.818788, train/data_wait_ratio: 0.048442, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26480.000000 2026-03-15 21:38:55,661 [Rank train_rank0] [Epoch 0 | Step 26500] train/loss: 0.457026, train/mse: 0.457026, train/data_wait_sec: 0.292580, train/step_wall_sec: 5.729231, train/data_wait_ratio: 0.051068, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26500.000000 2026-03-15 21:40:52,763 [Rank train_rank0] [Epoch 0 | Step 26520] train/loss: 0.471382, train/mse: 0.471382, train/data_wait_sec: 0.331267, train/step_wall_sec: 5.896098, train/data_wait_ratio: 0.056184, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26520.000000 2026-03-15 21:42:50,000 [Rank train_rank0] [Epoch 0 | Step 26540] train/loss: 0.448439, train/mse: 0.448439, train/data_wait_sec: 0.242071, train/step_wall_sec: 5.651573, train/data_wait_ratio: 0.042833, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26540.000000 2026-03-15 21:44:47,421 [Rank train_rank0] [Epoch 0 | Step 26560] train/loss: 0.465272, train/mse: 0.465272, train/data_wait_sec: 0.359545, train/step_wall_sec: 5.903256, train/data_wait_ratio: 0.060906, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26560.000000 2026-03-15 21:46:44,582 [Rank train_rank0] [Epoch 0 | Step 26580] train/loss: 0.445194, train/mse: 0.445194, train/data_wait_sec: 0.245603, train/step_wall_sec: 5.719762, train/data_wait_ratio: 0.042939, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26580.000000 2026-03-15 21:48:41,634 [Rank train_rank0] [Epoch 0 | Step 26600] train/loss: 0.456072, train/mse: 0.456072, train/data_wait_sec: 0.308966, train/step_wall_sec: 5.775484, train/data_wait_ratio: 0.053496, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26600.000000 2026-03-15 21:50:38,184 [Rank train_rank0] [Epoch 0 | Step 26620] train/loss: 0.456256, train/mse: 0.456256, train/data_wait_sec: 0.290332, train/step_wall_sec: 5.797533, train/data_wait_ratio: 0.050079, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26620.000000 2026-03-15 21:52:35,078 [Rank train_rank0] [Epoch 0 | Step 26640] train/loss: 0.431794, train/mse: 0.431794, train/data_wait_sec: 0.280498, train/step_wall_sec: 5.693399, train/data_wait_ratio: 0.049267, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26640.000000 2026-03-15 21:54:32,043 [Rank train_rank0] [Epoch 0 | Step 26660] train/loss: 0.465000, train/mse: 0.465000, train/data_wait_sec: 0.263848, train/step_wall_sec: 5.791335, train/data_wait_ratio: 0.045559, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26660.000000 2026-03-15 21:56:29,523 [Rank train_rank0] [Epoch 0 | Step 26680] train/loss: 0.433082, train/mse: 0.433082, train/data_wait_sec: 0.312051, train/step_wall_sec: 5.890121, train/data_wait_ratio: 0.052979, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26680.000000 2026-03-15 21:58:26,541 [Rank train_rank0] [Epoch 0 | Step 26700] train/loss: 0.453079, train/mse: 0.453079, train/data_wait_sec: 0.255363, train/step_wall_sec: 5.723122, train/data_wait_ratio: 0.044619, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26700.000000 2026-03-15 22:00:23,611 [Rank train_rank0] [Epoch 0 | Step 26720] train/loss: 0.486824, train/mse: 0.486824, train/data_wait_sec: 0.243350, train/step_wall_sec: 5.832826, train/data_wait_ratio: 0.041721, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26720.000000 2026-03-15 22:02:20,494 [Rank train_rank0] [Epoch 0 | Step 26740] train/loss: 0.457334, train/mse: 0.457334, train/data_wait_sec: 0.249663, train/step_wall_sec: 5.816938, train/data_wait_ratio: 0.042920, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26740.000000 2026-03-15 22:04:17,067 [Rank train_rank0] [Epoch 0 | Step 26760] train/loss: 0.449479, train/mse: 0.449479, train/data_wait_sec: 0.349914, train/step_wall_sec: 5.878113, train/data_wait_ratio: 0.059528, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26760.000000 2026-03-15 22:06:13,730 [Rank train_rank0] [Epoch 0 | Step 26780] train/loss: 0.462285, train/mse: 0.462285, train/data_wait_sec: 0.329565, train/step_wall_sec: 5.729727, train/data_wait_ratio: 0.057518, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26780.000000 2026-03-15 22:08:11,330 [Rank train_rank0] [Epoch 0 | Step 26800] train/loss: 0.468311, train/mse: 0.468311, train/data_wait_sec: 0.351253, train/step_wall_sec: 5.887397, train/data_wait_ratio: 0.059662, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26800.000000 2026-03-15 22:10:08,534 [Rank train_rank0] [Epoch 0 | Step 26820] train/loss: 0.435093, train/mse: 0.435093, train/data_wait_sec: 0.334360, train/step_wall_sec: 5.829995, train/data_wait_ratio: 0.057352, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26820.000000 2026-03-15 22:12:06,074 [Rank train_rank0] [Epoch 0 | Step 26840] train/loss: 0.425514, train/mse: 0.425514, train/data_wait_sec: 0.303648, train/step_wall_sec: 5.797170, train/data_wait_ratio: 0.052379, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26840.000000 2026-03-15 22:14:03,216 [Rank train_rank0] [Epoch 0 | Step 26860] train/loss: 0.450290, train/mse: 0.450290, train/data_wait_sec: 0.340844, train/step_wall_sec: 5.939732, train/data_wait_ratio: 0.057384, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26860.000000 2026-03-15 22:16:00,254 [Rank train_rank0] [Epoch 0 | Step 26880] train/loss: 0.466812, train/mse: 0.466812, train/data_wait_sec: 0.251219, train/step_wall_sec: 5.851138, train/data_wait_ratio: 0.042935, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26880.000000 2026-03-15 22:17:56,645 [Rank train_rank0] [Epoch 0 | Step 26900] train/loss: 0.467215, train/mse: 0.467215, train/data_wait_sec: 0.298280, train/step_wall_sec: 5.784657, train/data_wait_ratio: 0.051564, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26900.000000 2026-03-15 22:19:53,475 [Rank train_rank0] [Epoch 0 | Step 26920] train/loss: 0.465550, train/mse: 0.465550, train/data_wait_sec: 0.339076, train/step_wall_sec: 5.868026, train/data_wait_ratio: 0.057784, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26920.000000 2026-03-15 22:21:50,717 [Rank train_rank0] [Epoch 0 | Step 26940] train/loss: 0.470614, train/mse: 0.470614, train/data_wait_sec: 0.291124, train/step_wall_sec: 5.910674, train/data_wait_ratio: 0.049254, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26940.000000 2026-03-15 22:23:47,300 [Rank train_rank0] [Epoch 0 | Step 26960] train/loss: 0.442793, train/mse: 0.442793, train/data_wait_sec: 0.291655, train/step_wall_sec: 5.878501, train/data_wait_ratio: 0.049614, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26960.000000 2026-03-15 22:25:43,949 [Rank train_rank0] [Epoch 0 | Step 26980] train/loss: 0.416768, train/mse: 0.416768, train/data_wait_sec: 0.333866, train/step_wall_sec: 5.862792, train/data_wait_ratio: 0.056947, train/lr: 0.000200, train/epoch: 0.000000, train/step: 26980.000000 2026-03-15 22:28:06,827 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000027000.pt 2026-03-15 22:28:06,830 [Rank train_rank0] [Epoch 0 | Step 27000] train/loss: 0.468835, train/mse: 0.468835, train/data_wait_sec: 0.306383, train/step_wall_sec: 5.845388, train/data_wait_ratio: 0.052415, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27000.000000 2026-03-15 22:28:06,831 [Rank train_rank0] Generating samples for visualization... 2026-03-15 22:28:10,064 [Rank train_rank0] Sample generation done. 2026-03-15 22:30:08,403 [Rank train_rank0] [Epoch 0 | Step 27020] train/loss: 0.466990, train/mse: 0.466990, train/data_wait_sec: 0.332160, train/step_wall_sec: 5.888825, train/data_wait_ratio: 0.056405, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27020.000000 2026-03-15 22:32:05,702 [Rank train_rank0] [Epoch 0 | Step 27040] train/loss: 0.410445, train/mse: 0.410445, train/data_wait_sec: 0.288343, train/step_wall_sec: 5.730770, train/data_wait_ratio: 0.050315, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27040.000000 2026-03-15 22:34:02,639 [Rank train_rank0] [Epoch 0 | Step 27060] train/loss: 0.462210, train/mse: 0.462210, train/data_wait_sec: 0.334309, train/step_wall_sec: 5.731235, train/data_wait_ratio: 0.058331, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27060.000000 2026-03-15 22:35:59,482 [Rank train_rank0] [Epoch 0 | Step 27080] train/loss: 0.478261, train/mse: 0.478261, train/data_wait_sec: 0.232713, train/step_wall_sec: 5.693445, train/data_wait_ratio: 0.040874, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27080.000000 2026-03-15 22:37:56,268 [Rank train_rank0] [Epoch 0 | Step 27100] train/loss: 0.434151, train/mse: 0.434151, train/data_wait_sec: 0.253225, train/step_wall_sec: 5.720682, train/data_wait_ratio: 0.044265, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27100.000000 2026-03-15 22:39:53,054 [Rank train_rank0] [Epoch 0 | Step 27120] train/loss: 0.449835, train/mse: 0.449835, train/data_wait_sec: 0.312373, train/step_wall_sec: 5.810023, train/data_wait_ratio: 0.053764, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27120.000000 2026-03-15 22:41:49,933 [Rank train_rank0] [Epoch 0 | Step 27140] train/loss: 0.470458, train/mse: 0.470458, train/data_wait_sec: 0.227337, train/step_wall_sec: 5.750041, train/data_wait_ratio: 0.039537, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27140.000000 2026-03-15 22:43:47,180 [Rank train_rank0] [Epoch 0 | Step 27160] train/loss: 0.469369, train/mse: 0.469369, train/data_wait_sec: 0.283111, train/step_wall_sec: 5.774080, train/data_wait_ratio: 0.049031, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27160.000000 2026-03-15 22:45:44,187 [Rank train_rank0] [Epoch 0 | Step 27180] train/loss: 0.438095, train/mse: 0.438095, train/data_wait_sec: 0.254691, train/step_wall_sec: 5.768379, train/data_wait_ratio: 0.044153, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27180.000000 2026-03-15 22:47:41,422 [Rank train_rank0] [Epoch 0 | Step 27200] train/loss: 0.464034, train/mse: 0.464034, train/data_wait_sec: 0.333438, train/step_wall_sec: 5.846890, train/data_wait_ratio: 0.057028, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27200.000000 2026-03-15 22:49:38,576 [Rank train_rank0] [Epoch 0 | Step 27220] train/loss: 0.460020, train/mse: 0.460020, train/data_wait_sec: 0.342503, train/step_wall_sec: 5.811001, train/data_wait_ratio: 0.058940, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27220.000000 2026-03-15 22:51:35,591 [Rank train_rank0] [Epoch 0 | Step 27240] train/loss: 0.471435, train/mse: 0.471435, train/data_wait_sec: 0.298213, train/step_wall_sec: 5.869700, train/data_wait_ratio: 0.050806, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27240.000000 2026-03-15 22:53:32,676 [Rank train_rank0] [Epoch 0 | Step 27260] train/loss: 0.487854, train/mse: 0.487854, train/data_wait_sec: 0.391694, train/step_wall_sec: 5.865943, train/data_wait_ratio: 0.066774, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27260.000000 2026-03-15 22:55:29,656 [Rank train_rank0] [Epoch 0 | Step 27280] train/loss: 0.457091, train/mse: 0.457091, train/data_wait_sec: 0.296603, train/step_wall_sec: 5.787024, train/data_wait_ratio: 0.051253, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27280.000000 2026-03-15 22:57:26,594 [Rank train_rank0] [Epoch 0 | Step 27300] train/loss: 0.474892, train/mse: 0.474892, train/data_wait_sec: 0.261351, train/step_wall_sec: 5.718405, train/data_wait_ratio: 0.045704, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27300.000000 2026-03-15 22:59:24,005 [Rank train_rank0] [Epoch 0 | Step 27320] train/loss: 0.479226, train/mse: 0.479226, train/data_wait_sec: 0.313685, train/step_wall_sec: 5.845127, train/data_wait_ratio: 0.053666, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27320.000000 2026-03-15 23:01:21,103 [Rank train_rank0] [Epoch 0 | Step 27340] train/loss: 0.467297, train/mse: 0.467297, train/data_wait_sec: 0.367962, train/step_wall_sec: 5.882077, train/data_wait_ratio: 0.062557, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27340.000000 2026-03-15 23:03:18,039 [Rank train_rank0] [Epoch 0 | Step 27360] train/loss: 0.465819, train/mse: 0.465819, train/data_wait_sec: 0.292261, train/step_wall_sec: 5.714002, train/data_wait_ratio: 0.051148, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27360.000000 2026-03-15 23:05:14,998 [Rank train_rank0] [Epoch 0 | Step 27380] train/loss: 0.478710, train/mse: 0.478710, train/data_wait_sec: 0.247296, train/step_wall_sec: 5.735126, train/data_wait_ratio: 0.043119, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27380.000000 2026-03-15 23:07:11,066 [Rank train_rank0] [Epoch 0 | Step 27400] train/loss: 0.428291, train/mse: 0.428291, train/data_wait_sec: 0.291451, train/step_wall_sec: 5.680937, train/data_wait_ratio: 0.051303, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27400.000000 2026-03-15 23:09:08,246 [Rank train_rank0] [Epoch 0 | Step 27420] train/loss: 0.465675, train/mse: 0.465675, train/data_wait_sec: 0.326692, train/step_wall_sec: 5.917078, train/data_wait_ratio: 0.055212, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27420.000000 2026-03-15 23:11:05,311 [Rank train_rank0] [Epoch 0 | Step 27440] train/loss: 0.470247, train/mse: 0.470247, train/data_wait_sec: 0.436704, train/step_wall_sec: 5.958885, train/data_wait_ratio: 0.073286, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27440.000000 2026-03-15 23:13:02,245 [Rank train_rank0] [Epoch 0 | Step 27460] train/loss: 0.471035, train/mse: 0.471035, train/data_wait_sec: 0.311325, train/step_wall_sec: 5.879544, train/data_wait_ratio: 0.052951, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27460.000000 2026-03-15 23:14:59,245 [Rank train_rank0] [Epoch 0 | Step 27480] train/loss: 0.445503, train/mse: 0.445503, train/data_wait_sec: 0.247285, train/step_wall_sec: 5.729497, train/data_wait_ratio: 0.043160, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27480.000000 2026-03-15 23:16:55,962 [Rank train_rank0] [Epoch 0 | Step 27500] train/loss: 0.436697, train/mse: 0.436697, train/data_wait_sec: 0.293044, train/step_wall_sec: 5.751852, train/data_wait_ratio: 0.050948, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27500.000000 2026-03-15 23:18:52,543 [Rank train_rank0] [Epoch 0 | Step 27520] train/loss: 0.463182, train/mse: 0.463182, train/data_wait_sec: 0.356280, train/step_wall_sec: 5.901185, train/data_wait_ratio: 0.060374, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27520.000000 2026-03-15 23:20:48,895 [Rank train_rank0] [Epoch 0 | Step 27540] train/loss: 0.481577, train/mse: 0.481577, train/data_wait_sec: 0.178304, train/step_wall_sec: 5.712814, train/data_wait_ratio: 0.031211, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27540.000000 2026-03-15 23:22:45,901 [Rank train_rank0] [Epoch 0 | Step 27560] train/loss: 0.432804, train/mse: 0.432804, train/data_wait_sec: 0.330343, train/step_wall_sec: 5.834876, train/data_wait_ratio: 0.056615, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27560.000000 2026-03-15 23:24:42,955 [Rank train_rank0] [Epoch 0 | Step 27580] train/loss: 0.465882, train/mse: 0.465882, train/data_wait_sec: 0.266605, train/step_wall_sec: 5.840319, train/data_wait_ratio: 0.045649, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27580.000000 2026-03-15 23:26:39,907 [Rank train_rank0] [Epoch 0 | Step 27600] train/loss: 0.437239, train/mse: 0.437239, train/data_wait_sec: 0.390409, train/step_wall_sec: 5.922964, train/data_wait_ratio: 0.065914, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27600.000000 2026-03-15 23:28:36,286 [Rank train_rank0] [Epoch 0 | Step 27620] train/loss: 0.475973, train/mse: 0.475973, train/data_wait_sec: 0.300325, train/step_wall_sec: 5.874466, train/data_wait_ratio: 0.051124, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27620.000000 2026-03-15 23:30:32,757 [Rank train_rank0] [Epoch 0 | Step 27640] train/loss: 0.441276, train/mse: 0.441276, train/data_wait_sec: 0.284324, train/step_wall_sec: 5.762156, train/data_wait_ratio: 0.049343, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27640.000000 2026-03-15 23:32:29,807 [Rank train_rank0] [Epoch 0 | Step 27660] train/loss: 0.459131, train/mse: 0.459131, train/data_wait_sec: 0.244289, train/step_wall_sec: 5.699377, train/data_wait_ratio: 0.042862, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27660.000000 2026-03-15 23:34:26,738 [Rank train_rank0] [Epoch 0 | Step 27680] train/loss: 0.450421, train/mse: 0.450421, train/data_wait_sec: 0.307657, train/step_wall_sec: 5.787067, train/data_wait_ratio: 0.053163, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27680.000000 2026-03-15 23:36:23,096 [Rank train_rank0] [Epoch 0 | Step 27700] train/loss: 0.453580, train/mse: 0.453580, train/data_wait_sec: 0.346426, train/step_wall_sec: 5.828590, train/data_wait_ratio: 0.059436, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27700.000000 2026-03-15 23:38:19,298 [Rank train_rank0] [Epoch 0 | Step 27720] train/loss: 0.471136, train/mse: 0.471136, train/data_wait_sec: 0.351605, train/step_wall_sec: 5.896811, train/data_wait_ratio: 0.059626, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27720.000000 2026-03-15 23:40:15,957 [Rank train_rank0] [Epoch 0 | Step 27740] train/loss: 0.441750, train/mse: 0.441750, train/data_wait_sec: 0.357884, train/step_wall_sec: 5.877028, train/data_wait_ratio: 0.060895, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27740.000000 2026-03-15 23:42:12,803 [Rank train_rank0] [Epoch 0 | Step 27760] train/loss: 0.469753, train/mse: 0.469753, train/data_wait_sec: 0.234227, train/step_wall_sec: 5.751077, train/data_wait_ratio: 0.040728, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27760.000000 2026-03-15 23:44:09,455 [Rank train_rank0] [Epoch 0 | Step 27780] train/loss: 0.458443, train/mse: 0.458443, train/data_wait_sec: 0.290148, train/step_wall_sec: 5.729390, train/data_wait_ratio: 0.050642, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27780.000000 2026-03-15 23:46:06,573 [Rank train_rank0] [Epoch 0 | Step 27800] train/loss: 0.461987, train/mse: 0.461987, train/data_wait_sec: 0.327063, train/step_wall_sec: 5.789423, train/data_wait_ratio: 0.056493, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27800.000000 2026-03-15 23:48:03,184 [Rank train_rank0] [Epoch 0 | Step 27820] train/loss: 0.462377, train/mse: 0.462377, train/data_wait_sec: 0.285329, train/step_wall_sec: 5.678466, train/data_wait_ratio: 0.050248, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27820.000000 2026-03-15 23:50:00,469 [Rank train_rank0] [Epoch 0 | Step 27840] train/loss: 0.442991, train/mse: 0.442991, train/data_wait_sec: 0.243139, train/step_wall_sec: 5.732202, train/data_wait_ratio: 0.042416, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27840.000000 2026-03-15 23:51:56,789 [Rank train_rank0] [Epoch 0 | Step 27860] train/loss: 0.452212, train/mse: 0.452212, train/data_wait_sec: 0.277885, train/step_wall_sec: 5.759203, train/data_wait_ratio: 0.048251, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27860.000000 2026-03-15 23:53:53,207 [Rank train_rank0] [Epoch 0 | Step 27880] train/loss: 0.471817, train/mse: 0.471817, train/data_wait_sec: 0.292913, train/step_wall_sec: 5.856998, train/data_wait_ratio: 0.050011, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27880.000000 2026-03-15 23:55:50,399 [Rank train_rank0] [Epoch 0 | Step 27900] train/loss: 0.463865, train/mse: 0.463865, train/data_wait_sec: 0.325545, train/step_wall_sec: 5.911178, train/data_wait_ratio: 0.055073, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27900.000000 2026-03-15 23:57:47,516 [Rank train_rank0] [Epoch 0 | Step 27920] train/loss: 0.462956, train/mse: 0.462956, train/data_wait_sec: 0.368457, train/step_wall_sec: 5.876758, train/data_wait_ratio: 0.062697, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27920.000000 2026-03-15 23:59:44,223 [Rank train_rank0] [Epoch 0 | Step 27940] train/loss: 0.445490, train/mse: 0.445490, train/data_wait_sec: 0.198804, train/step_wall_sec: 5.718580, train/data_wait_ratio: 0.034765, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27940.000000 2026-03-16 00:01:40,938 [Rank train_rank0] [Epoch 0 | Step 27960] train/loss: 0.453440, train/mse: 0.453440, train/data_wait_sec: 0.176342, train/step_wall_sec: 5.640162, train/data_wait_ratio: 0.031265, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27960.000000 2026-03-16 00:03:37,609 [Rank train_rank0] [Epoch 0 | Step 27980] train/loss: 0.446605, train/mse: 0.446605, train/data_wait_sec: 0.293098, train/step_wall_sec: 5.735554, train/data_wait_ratio: 0.051102, train/lr: 0.000200, train/epoch: 0.000000, train/step: 27980.000000 2026-03-16 00:05:58,651 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000028000.pt 2026-03-16 00:05:58,654 [Rank train_rank0] [Epoch 0 | Step 28000] train/loss: 0.488759, train/mse: 0.488759, train/data_wait_sec: 0.231902, train/step_wall_sec: 5.691258, train/data_wait_ratio: 0.040747, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28000.000000 2026-03-16 00:05:58,656 [Rank train_rank0] Generating samples for visualization... 2026-03-16 00:06:01,848 [Rank train_rank0] Sample generation done. 2026-03-16 00:07:58,854 [Rank train_rank0] [Epoch 0 | Step 28020] train/loss: 0.487722, train/mse: 0.487722, train/data_wait_sec: 0.351361, train/step_wall_sec: 5.944394, train/data_wait_ratio: 0.059108, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28020.000000 2026-03-16 00:09:57,149 [Rank train_rank0] [Epoch 0 | Step 28040] train/loss: 0.461769, train/mse: 0.461769, train/data_wait_sec: 0.279021, train/step_wall_sec: 5.723235, train/data_wait_ratio: 0.048752, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28040.000000 2026-03-16 00:11:54,079 [Rank train_rank0] [Epoch 0 | Step 28060] train/loss: 0.463226, train/mse: 0.463226, train/data_wait_sec: 0.307715, train/step_wall_sec: 5.750069, train/data_wait_ratio: 0.053515, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28060.000000 2026-03-16 00:13:50,758 [Rank train_rank0] [Epoch 0 | Step 28080] train/loss: 0.465299, train/mse: 0.465299, train/data_wait_sec: 0.218462, train/step_wall_sec: 5.789234, train/data_wait_ratio: 0.037736, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28080.000000 2026-03-16 00:15:47,432 [Rank train_rank0] [Epoch 0 | Step 28100] train/loss: 0.422049, train/mse: 0.422049, train/data_wait_sec: 0.189735, train/step_wall_sec: 5.646527, train/data_wait_ratio: 0.033602, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28100.000000 2026-03-16 00:17:43,657 [Rank train_rank0] [Epoch 0 | Step 28120] train/loss: 0.447998, train/mse: 0.447998, train/data_wait_sec: 0.281971, train/step_wall_sec: 5.733839, train/data_wait_ratio: 0.049177, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28120.000000 2026-03-16 00:19:40,307 [Rank train_rank0] [Epoch 0 | Step 28140] train/loss: 0.440665, train/mse: 0.440665, train/data_wait_sec: 0.263882, train/step_wall_sec: 5.761473, train/data_wait_ratio: 0.045801, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28140.000000 2026-03-16 00:21:37,958 [Rank train_rank0] [Epoch 0 | Step 28160] train/loss: 0.441582, train/mse: 0.441582, train/data_wait_sec: 0.199044, train/step_wall_sec: 5.783377, train/data_wait_ratio: 0.034417, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28160.000000 2026-03-16 00:23:34,462 [Rank train_rank0] [Epoch 0 | Step 28180] train/loss: 0.435683, train/mse: 0.435683, train/data_wait_sec: 0.258979, train/step_wall_sec: 5.722073, train/data_wait_ratio: 0.045260, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28180.000000 2026-03-16 00:25:31,419 [Rank train_rank0] [Epoch 0 | Step 28200] train/loss: 0.446316, train/mse: 0.446316, train/data_wait_sec: 0.333464, train/step_wall_sec: 5.793880, train/data_wait_ratio: 0.057555, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28200.000000 2026-03-16 00:27:28,082 [Rank train_rank0] [Epoch 0 | Step 28220] train/loss: 0.471835, train/mse: 0.471835, train/data_wait_sec: 0.302988, train/step_wall_sec: 5.770910, train/data_wait_ratio: 0.052503, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28220.000000 2026-03-16 00:29:25,220 [Rank train_rank0] [Epoch 0 | Step 28240] train/loss: 0.477625, train/mse: 0.477625, train/data_wait_sec: 0.337143, train/step_wall_sec: 5.889938, train/data_wait_ratio: 0.057240, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28240.000000 2026-03-16 00:31:22,665 [Rank train_rank0] [Epoch 0 | Step 28260] train/loss: 0.441610, train/mse: 0.441610, train/data_wait_sec: 0.341887, train/step_wall_sec: 5.711538, train/data_wait_ratio: 0.059859, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28260.000000 2026-03-16 00:33:19,792 [Rank train_rank0] [Epoch 0 | Step 28280] train/loss: 0.474101, train/mse: 0.474101, train/data_wait_sec: 0.314222, train/step_wall_sec: 5.738489, train/data_wait_ratio: 0.054757, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28280.000000 2026-03-16 00:35:16,277 [Rank train_rank0] [Epoch 0 | Step 28300] train/loss: 0.474061, train/mse: 0.474061, train/data_wait_sec: 0.314396, train/step_wall_sec: 5.865728, train/data_wait_ratio: 0.053599, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28300.000000 2026-03-16 00:37:13,150 [Rank train_rank0] [Epoch 0 | Step 28320] train/loss: 0.472535, train/mse: 0.472535, train/data_wait_sec: 0.244754, train/step_wall_sec: 5.590109, train/data_wait_ratio: 0.043783, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28320.000000 2026-03-16 00:39:09,550 [Rank train_rank0] [Epoch 0 | Step 28340] train/loss: 0.444257, train/mse: 0.444257, train/data_wait_sec: 0.272028, train/step_wall_sec: 5.719886, train/data_wait_ratio: 0.047558, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28340.000000 2026-03-16 00:41:05,912 [Rank train_rank0] [Epoch 0 | Step 28360] train/loss: 0.452462, train/mse: 0.452462, train/data_wait_sec: 0.376107, train/step_wall_sec: 5.774862, train/data_wait_ratio: 0.065128, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28360.000000 2026-03-16 00:43:03,383 [Rank train_rank0] [Epoch 0 | Step 28380] train/loss: 0.445322, train/mse: 0.445322, train/data_wait_sec: 0.237370, train/step_wall_sec: 5.742372, train/data_wait_ratio: 0.041337, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28380.000000 2026-03-16 00:44:59,640 [Rank train_rank0] [Epoch 0 | Step 28400] train/loss: 0.434746, train/mse: 0.434746, train/data_wait_sec: 0.172492, train/step_wall_sec: 5.628039, train/data_wait_ratio: 0.030649, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28400.000000 2026-03-16 00:46:56,768 [Rank train_rank0] [Epoch 0 | Step 28420] train/loss: 0.418265, train/mse: 0.418265, train/data_wait_sec: 0.460417, train/step_wall_sec: 6.008247, train/data_wait_ratio: 0.076631, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28420.000000 2026-03-16 00:48:53,529 [Rank train_rank0] [Epoch 0 | Step 28440] train/loss: 0.440693, train/mse: 0.440693, train/data_wait_sec: 0.338523, train/step_wall_sec: 5.750975, train/data_wait_ratio: 0.058864, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28440.000000 2026-03-16 00:50:49,802 [Rank train_rank0] [Epoch 0 | Step 28460] train/loss: 0.448381, train/mse: 0.448381, train/data_wait_sec: 0.321971, train/step_wall_sec: 5.824877, train/data_wait_ratio: 0.055275, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28460.000000 2026-03-16 00:52:46,195 [Rank train_rank0] [Epoch 0 | Step 28480] train/loss: 0.442618, train/mse: 0.442618, train/data_wait_sec: 0.311685, train/step_wall_sec: 5.753737, train/data_wait_ratio: 0.054171, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28480.000000 2026-03-16 00:54:42,774 [Rank train_rank0] [Epoch 0 | Step 28500] train/loss: 0.499956, train/mse: 0.499956, train/data_wait_sec: 0.239204, train/step_wall_sec: 5.751124, train/data_wait_ratio: 0.041593, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28500.000000 2026-03-16 00:56:39,502 [Rank train_rank0] [Epoch 0 | Step 28520] train/loss: 0.459222, train/mse: 0.459222, train/data_wait_sec: 0.239320, train/step_wall_sec: 5.689197, train/data_wait_ratio: 0.042066, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28520.000000 2026-03-16 00:58:36,380 [Rank train_rank0] [Epoch 0 | Step 28540] train/loss: 0.453026, train/mse: 0.453026, train/data_wait_sec: 0.226685, train/step_wall_sec: 5.820922, train/data_wait_ratio: 0.038943, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28540.000000 2026-03-16 01:00:33,050 [Rank train_rank0] [Epoch 0 | Step 28560] train/loss: 0.433550, train/mse: 0.433550, train/data_wait_sec: 0.264641, train/step_wall_sec: 5.709847, train/data_wait_ratio: 0.046348, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28560.000000 2026-03-16 01:02:29,755 [Rank train_rank0] [Epoch 0 | Step 28580] train/loss: 0.427289, train/mse: 0.427289, train/data_wait_sec: 0.292374, train/step_wall_sec: 5.795175, train/data_wait_ratio: 0.050451, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28580.000000 2026-03-16 01:04:26,842 [Rank train_rank0] [Epoch 0 | Step 28600] train/loss: 0.444783, train/mse: 0.444783, train/data_wait_sec: 0.341493, train/step_wall_sec: 5.847427, train/data_wait_ratio: 0.058401, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28600.000000 2026-03-16 01:06:23,643 [Rank train_rank0] [Epoch 0 | Step 28620] train/loss: 0.435598, train/mse: 0.435598, train/data_wait_sec: 0.341293, train/step_wall_sec: 5.740753, train/data_wait_ratio: 0.059451, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28620.000000 2026-03-16 01:08:20,683 [Rank train_rank0] [Epoch 0 | Step 28640] train/loss: 0.456029, train/mse: 0.456029, train/data_wait_sec: 0.229027, train/step_wall_sec: 5.788351, train/data_wait_ratio: 0.039567, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28640.000000 2026-03-16 01:10:17,640 [Rank train_rank0] [Epoch 0 | Step 28660] train/loss: 0.463665, train/mse: 0.463665, train/data_wait_sec: 0.265097, train/step_wall_sec: 5.817121, train/data_wait_ratio: 0.045572, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28660.000000 2026-03-16 01:12:14,402 [Rank train_rank0] [Epoch 0 | Step 28680] train/loss: 0.477233, train/mse: 0.477233, train/data_wait_sec: 0.409345, train/step_wall_sec: 5.960596, train/data_wait_ratio: 0.068675, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28680.000000 2026-03-16 01:14:11,340 [Rank train_rank0] [Epoch 0 | Step 28700] train/loss: 0.456315, train/mse: 0.456315, train/data_wait_sec: 0.320324, train/step_wall_sec: 5.767278, train/data_wait_ratio: 0.055542, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28700.000000 2026-03-16 01:16:08,731 [Rank train_rank0] [Epoch 0 | Step 28720] train/loss: 0.463520, train/mse: 0.463520, train/data_wait_sec: 0.280075, train/step_wall_sec: 5.776525, train/data_wait_ratio: 0.048485, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28720.000000 2026-03-16 01:18:05,940 [Rank train_rank0] [Epoch 0 | Step 28740] train/loss: 0.466957, train/mse: 0.466957, train/data_wait_sec: 0.301069, train/step_wall_sec: 5.794487, train/data_wait_ratio: 0.051958, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28740.000000 2026-03-16 01:20:02,728 [Rank train_rank0] [Epoch 0 | Step 28760] train/loss: 0.458401, train/mse: 0.458401, train/data_wait_sec: 0.316789, train/step_wall_sec: 5.874290, train/data_wait_ratio: 0.053928, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28760.000000 2026-03-16 01:22:00,286 [Rank train_rank0] [Epoch 0 | Step 28780] train/loss: 0.450523, train/mse: 0.450523, train/data_wait_sec: 0.363024, train/step_wall_sec: 5.898684, train/data_wait_ratio: 0.061543, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28780.000000 2026-03-16 01:23:57,657 [Rank train_rank0] [Epoch 0 | Step 28800] train/loss: 0.492427, train/mse: 0.492427, train/data_wait_sec: 0.325204, train/step_wall_sec: 5.793281, train/data_wait_ratio: 0.056135, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28800.000000 2026-03-16 01:25:54,851 [Rank train_rank0] [Epoch 0 | Step 28820] train/loss: 0.412623, train/mse: 0.412623, train/data_wait_sec: 0.216061, train/step_wall_sec: 5.747739, train/data_wait_ratio: 0.037591, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28820.000000 2026-03-16 01:27:51,693 [Rank train_rank0] [Epoch 0 | Step 28840] train/loss: 0.503972, train/mse: 0.503972, train/data_wait_sec: 0.343496, train/step_wall_sec: 5.920527, train/data_wait_ratio: 0.058018, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28840.000000 2026-03-16 01:29:48,552 [Rank train_rank0] [Epoch 0 | Step 28860] train/loss: 0.450705, train/mse: 0.450705, train/data_wait_sec: 0.281200, train/step_wall_sec: 5.849848, train/data_wait_ratio: 0.048070, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28860.000000 2026-03-16 01:31:45,194 [Rank train_rank0] [Epoch 0 | Step 28880] train/loss: 0.448873, train/mse: 0.448873, train/data_wait_sec: 0.310053, train/step_wall_sec: 5.751000, train/data_wait_ratio: 0.053913, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28880.000000 2026-03-16 01:33:42,639 [Rank train_rank0] [Epoch 0 | Step 28900] train/loss: 0.444265, train/mse: 0.444265, train/data_wait_sec: 0.363061, train/step_wall_sec: 5.908266, train/data_wait_ratio: 0.061450, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28900.000000 2026-03-16 01:35:39,802 [Rank train_rank0] [Epoch 0 | Step 28920] train/loss: 0.471675, train/mse: 0.471675, train/data_wait_sec: 0.318485, train/step_wall_sec: 5.887496, train/data_wait_ratio: 0.054095, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28920.000000 2026-03-16 01:37:36,053 [Rank train_rank0] [Epoch 0 | Step 28940] train/loss: 0.437326, train/mse: 0.437326, train/data_wait_sec: 0.256271, train/step_wall_sec: 5.741942, train/data_wait_ratio: 0.044631, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28940.000000 2026-03-16 01:39:32,344 [Rank train_rank0] [Epoch 0 | Step 28960] train/loss: 0.447902, train/mse: 0.447902, train/data_wait_sec: 0.267283, train/step_wall_sec: 5.722464, train/data_wait_ratio: 0.046708, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28960.000000 2026-03-16 01:41:29,842 [Rank train_rank0] [Epoch 0 | Step 28980] train/loss: 0.477257, train/mse: 0.477257, train/data_wait_sec: 0.374203, train/step_wall_sec: 5.931319, train/data_wait_ratio: 0.063089, train/lr: 0.000200, train/epoch: 0.000000, train/step: 28980.000000 2026-03-16 01:43:51,531 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000029000.pt 2026-03-16 01:43:51,534 [Rank train_rank0] [Epoch 0 | Step 29000] train/loss: 0.467705, train/mse: 0.467705, train/data_wait_sec: 0.294164, train/step_wall_sec: 5.796681, train/data_wait_ratio: 0.050747, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29000.000000 2026-03-16 01:43:51,535 [Rank train_rank0] Generating samples for visualization... 2026-03-16 01:43:54,619 [Rank train_rank0] Sample generation done. 2026-03-16 01:45:52,127 [Rank train_rank0] [Epoch 0 | Step 29020] train/loss: 0.466681, train/mse: 0.466681, train/data_wait_sec: 0.357046, train/step_wall_sec: 5.848217, train/data_wait_ratio: 0.061052, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29020.000000 2026-03-16 01:47:49,848 [Rank train_rank0] [Epoch 0 | Step 29040] train/loss: 0.469077, train/mse: 0.469077, train/data_wait_sec: 0.312753, train/step_wall_sec: 5.869818, train/data_wait_ratio: 0.053282, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29040.000000 2026-03-16 01:49:46,313 [Rank train_rank0] [Epoch 0 | Step 29060] train/loss: 0.439717, train/mse: 0.439717, train/data_wait_sec: 0.257715, train/step_wall_sec: 5.743023, train/data_wait_ratio: 0.044875, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29060.000000 2026-03-16 01:51:43,432 [Rank train_rank0] [Epoch 0 | Step 29080] train/loss: 0.451298, train/mse: 0.451298, train/data_wait_sec: 0.309159, train/step_wall_sec: 5.876458, train/data_wait_ratio: 0.052610, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29080.000000 2026-03-16 01:53:40,027 [Rank train_rank0] [Epoch 0 | Step 29100] train/loss: 0.433700, train/mse: 0.433700, train/data_wait_sec: 0.231200, train/step_wall_sec: 5.725245, train/data_wait_ratio: 0.040383, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29100.000000 2026-03-16 01:55:36,474 [Rank train_rank0] [Epoch 0 | Step 29120] train/loss: 0.451274, train/mse: 0.451274, train/data_wait_sec: 0.338867, train/step_wall_sec: 5.822299, train/data_wait_ratio: 0.058202, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29120.000000 2026-03-16 01:57:33,017 [Rank train_rank0] [Epoch 0 | Step 29140] train/loss: 0.460736, train/mse: 0.460736, train/data_wait_sec: 0.284128, train/step_wall_sec: 5.803433, train/data_wait_ratio: 0.048959, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29140.000000 2026-03-16 01:59:30,168 [Rank train_rank0] [Epoch 0 | Step 29160] train/loss: 0.455233, train/mse: 0.455233, train/data_wait_sec: 0.283069, train/step_wall_sec: 5.845006, train/data_wait_ratio: 0.048429, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29160.000000 2026-03-16 02:01:27,113 [Rank train_rank0] [Epoch 0 | Step 29180] train/loss: 0.478086, train/mse: 0.478086, train/data_wait_sec: 0.312777, train/step_wall_sec: 5.800200, train/data_wait_ratio: 0.053925, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29180.000000 2026-03-16 02:03:23,795 [Rank train_rank0] [Epoch 0 | Step 29200] train/loss: 0.462856, train/mse: 0.462856, train/data_wait_sec: 0.373130, train/step_wall_sec: 5.930021, train/data_wait_ratio: 0.062922, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29200.000000 2026-03-16 02:05:20,391 [Rank train_rank0] [Epoch 0 | Step 29220] train/loss: 0.459647, train/mse: 0.459647, train/data_wait_sec: 0.298251, train/step_wall_sec: 5.763744, train/data_wait_ratio: 0.051746, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29220.000000 2026-03-16 02:07:17,382 [Rank train_rank0] [Epoch 0 | Step 29240] train/loss: 0.445878, train/mse: 0.445878, train/data_wait_sec: 0.347136, train/step_wall_sec: 5.777796, train/data_wait_ratio: 0.060081, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29240.000000 2026-03-16 02:09:13,377 [Rank train_rank0] [Epoch 0 | Step 29260] train/loss: 0.465771, train/mse: 0.465771, train/data_wait_sec: 0.321197, train/step_wall_sec: 5.889212, train/data_wait_ratio: 0.054540, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29260.000000 2026-03-16 02:11:10,152 [Rank train_rank0] [Epoch 0 | Step 29280] train/loss: 0.468005, train/mse: 0.468005, train/data_wait_sec: 0.293718, train/step_wall_sec: 5.785580, train/data_wait_ratio: 0.050767, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29280.000000 2026-03-16 02:13:06,230 [Rank train_rank0] [Epoch 0 | Step 29300] train/loss: 0.424754, train/mse: 0.424754, train/data_wait_sec: 0.345235, train/step_wall_sec: 5.845622, train/data_wait_ratio: 0.059059, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29300.000000 2026-03-16 02:15:03,261 [Rank train_rank0] [Epoch 0 | Step 29320] train/loss: 0.444104, train/mse: 0.444104, train/data_wait_sec: 0.288795, train/step_wall_sec: 5.754150, train/data_wait_ratio: 0.050189, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29320.000000 2026-03-16 02:17:00,282 [Rank train_rank0] [Epoch 0 | Step 29340] train/loss: 0.447602, train/mse: 0.447602, train/data_wait_sec: 0.338098, train/step_wall_sec: 5.786802, train/data_wait_ratio: 0.058426, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29340.000000 2026-03-16 02:18:57,359 [Rank train_rank0] [Epoch 0 | Step 29360] train/loss: 0.492633, train/mse: 0.492633, train/data_wait_sec: 0.279430, train/step_wall_sec: 5.799275, train/data_wait_ratio: 0.048184, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29360.000000 2026-03-16 02:20:53,802 [Rank train_rank0] [Epoch 0 | Step 29380] train/loss: 0.438551, train/mse: 0.438551, train/data_wait_sec: 0.232840, train/step_wall_sec: 5.630034, train/data_wait_ratio: 0.041357, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29380.000000 2026-03-16 02:22:50,786 [Rank train_rank0] [Epoch 0 | Step 29400] train/loss: 0.446723, train/mse: 0.446723, train/data_wait_sec: 0.222634, train/step_wall_sec: 5.706390, train/data_wait_ratio: 0.039015, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29400.000000 2026-03-16 02:24:47,544 [Rank train_rank0] [Epoch 0 | Step 29420] train/loss: 0.473141, train/mse: 0.473141, train/data_wait_sec: 0.265634, train/step_wall_sec: 5.733323, train/data_wait_ratio: 0.046332, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29420.000000 2026-03-16 02:26:43,444 [Rank train_rank0] [Epoch 0 | Step 29440] train/loss: 0.453911, train/mse: 0.453911, train/data_wait_sec: 0.278813, train/step_wall_sec: 5.769381, train/data_wait_ratio: 0.048326, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29440.000000 2026-03-16 02:28:40,111 [Rank train_rank0] [Epoch 0 | Step 29460] train/loss: 0.433507, train/mse: 0.433507, train/data_wait_sec: 0.365494, train/step_wall_sec: 5.904965, train/data_wait_ratio: 0.061896, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29460.000000 2026-03-16 02:30:37,171 [Rank train_rank0] [Epoch 0 | Step 29480] train/loss: 0.460432, train/mse: 0.460432, train/data_wait_sec: 0.332264, train/step_wall_sec: 5.870842, train/data_wait_ratio: 0.056596, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29480.000000 2026-03-16 02:32:34,171 [Rank train_rank0] [Epoch 0 | Step 29500] train/loss: 0.476661, train/mse: 0.476661, train/data_wait_sec: 0.326703, train/step_wall_sec: 5.902788, train/data_wait_ratio: 0.055347, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29500.000000 2026-03-16 02:34:31,169 [Rank train_rank0] [Epoch 0 | Step 29520] train/loss: 0.441081, train/mse: 0.441081, train/data_wait_sec: 0.264241, train/step_wall_sec: 5.763010, train/data_wait_ratio: 0.045851, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29520.000000 2026-03-16 02:36:27,864 [Rank train_rank0] [Epoch 0 | Step 29540] train/loss: 0.434030, train/mse: 0.434030, train/data_wait_sec: 0.342379, train/step_wall_sec: 5.766396, train/data_wait_ratio: 0.059375, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29540.000000 2026-03-16 02:38:23,579 [Rank train_rank0] [Epoch 0 | Step 29560] train/loss: 0.487154, train/mse: 0.487154, train/data_wait_sec: 0.293425, train/step_wall_sec: 5.882566, train/data_wait_ratio: 0.049880, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29560.000000 2026-03-16 02:40:20,800 [Rank train_rank0] [Epoch 0 | Step 29580] train/loss: 0.470818, train/mse: 0.470818, train/data_wait_sec: 0.270326, train/step_wall_sec: 5.817591, train/data_wait_ratio: 0.046467, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29580.000000 2026-03-16 02:42:17,500 [Rank train_rank0] [Epoch 0 | Step 29600] train/loss: 0.468605, train/mse: 0.468605, train/data_wait_sec: 0.254348, train/step_wall_sec: 5.671391, train/data_wait_ratio: 0.044848, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29600.000000 2026-03-16 02:44:13,870 [Rank train_rank0] [Epoch 0 | Step 29620] train/loss: 0.431206, train/mse: 0.431206, train/data_wait_sec: 0.259631, train/step_wall_sec: 5.802968, train/data_wait_ratio: 0.044741, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29620.000000 2026-03-16 02:46:10,715 [Rank train_rank0] [Epoch 0 | Step 29640] train/loss: 0.465724, train/mse: 0.465724, train/data_wait_sec: 0.198914, train/step_wall_sec: 5.708726, train/data_wait_ratio: 0.034844, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29640.000000 2026-03-16 02:48:06,979 [Rank train_rank0] [Epoch 0 | Step 29660] train/loss: 0.454699, train/mse: 0.454699, train/data_wait_sec: 0.305996, train/step_wall_sec: 5.772847, train/data_wait_ratio: 0.053006, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29660.000000 2026-03-16 02:50:03,682 [Rank train_rank0] [Epoch 0 | Step 29680] train/loss: 0.427142, train/mse: 0.427142, train/data_wait_sec: 0.338024, train/step_wall_sec: 5.864048, train/data_wait_ratio: 0.057644, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29680.000000 2026-03-16 02:52:00,342 [Rank train_rank0] [Epoch 0 | Step 29700] train/loss: 0.461362, train/mse: 0.461362, train/data_wait_sec: 0.267426, train/step_wall_sec: 5.790557, train/data_wait_ratio: 0.046183, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29700.000000 2026-03-16 02:53:56,502 [Rank train_rank0] [Epoch 0 | Step 29720] train/loss: 0.417203, train/mse: 0.417203, train/data_wait_sec: 0.239276, train/step_wall_sec: 5.782264, train/data_wait_ratio: 0.041381, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29720.000000 2026-03-16 02:55:53,238 [Rank train_rank0] [Epoch 0 | Step 29740] train/loss: 0.451402, train/mse: 0.451402, train/data_wait_sec: 0.245701, train/step_wall_sec: 5.813752, train/data_wait_ratio: 0.042262, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29740.000000 2026-03-16 02:57:50,378 [Rank train_rank0] [Epoch 0 | Step 29760] train/loss: 0.456851, train/mse: 0.456851, train/data_wait_sec: 0.276603, train/step_wall_sec: 5.817828, train/data_wait_ratio: 0.047544, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29760.000000 2026-03-16 02:59:47,280 [Rank train_rank0] [Epoch 0 | Step 29780] train/loss: 0.445760, train/mse: 0.445760, train/data_wait_sec: 0.285854, train/step_wall_sec: 5.796845, train/data_wait_ratio: 0.049312, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29780.000000 2026-03-16 03:01:44,659 [Rank train_rank0] [Epoch 0 | Step 29800] train/loss: 0.465078, train/mse: 0.465078, train/data_wait_sec: 0.452359, train/step_wall_sec: 5.986399, train/data_wait_ratio: 0.075565, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29800.000000 2026-03-16 03:03:41,872 [Rank train_rank0] [Epoch 0 | Step 29820] train/loss: 0.467944, train/mse: 0.467944, train/data_wait_sec: 0.237495, train/step_wall_sec: 5.705504, train/data_wait_ratio: 0.041626, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29820.000000 2026-03-16 03:05:38,473 [Rank train_rank0] [Epoch 0 | Step 29840] train/loss: 0.451949, train/mse: 0.451949, train/data_wait_sec: 0.332363, train/step_wall_sec: 5.773385, train/data_wait_ratio: 0.057568, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29840.000000 2026-03-16 03:07:34,861 [Rank train_rank0] [Epoch 0 | Step 29860] train/loss: 0.437564, train/mse: 0.437564, train/data_wait_sec: 0.267730, train/step_wall_sec: 5.814426, train/data_wait_ratio: 0.046046, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29860.000000 2026-03-16 03:09:31,518 [Rank train_rank0] [Epoch 0 | Step 29880] train/loss: 0.462336, train/mse: 0.462336, train/data_wait_sec: 0.282107, train/step_wall_sec: 5.805713, train/data_wait_ratio: 0.048591, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29880.000000 2026-03-16 03:11:28,591 [Rank train_rank0] [Epoch 0 | Step 29900] train/loss: 0.416159, train/mse: 0.416159, train/data_wait_sec: 0.247761, train/step_wall_sec: 5.744666, train/data_wait_ratio: 0.043129, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29900.000000 2026-03-16 03:13:25,815 [Rank train_rank0] [Epoch 0 | Step 29920] train/loss: 0.447906, train/mse: 0.447906, train/data_wait_sec: 0.347619, train/step_wall_sec: 5.831588, train/data_wait_ratio: 0.059610, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29920.000000 2026-03-16 03:15:22,012 [Rank train_rank0] [Epoch 0 | Step 29940] train/loss: 0.469557, train/mse: 0.469557, train/data_wait_sec: 0.282296, train/step_wall_sec: 5.745823, train/data_wait_ratio: 0.049131, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29940.000000 2026-03-16 03:17:18,507 [Rank train_rank0] [Epoch 0 | Step 29960] train/loss: 0.416238, train/mse: 0.416238, train/data_wait_sec: 0.185869, train/step_wall_sec: 5.599286, train/data_wait_ratio: 0.033195, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29960.000000 2026-03-16 03:19:14,939 [Rank train_rank0] [Epoch 0 | Step 29980] train/loss: 0.455460, train/mse: 0.455460, train/data_wait_sec: 0.251044, train/step_wall_sec: 5.786136, train/data_wait_ratio: 0.043387, train/lr: 0.000200, train/epoch: 0.000000, train/step: 29980.000000 2026-03-16 03:21:35,576 [Rank train_rank0] Saved checkpoint: /scratch/e1539128/ckpt-4gpu-new/dit_training/checkpoints/step-0000030000.pt 2026-03-16 03:21:35,579 [Rank train_rank0] [Epoch 0 | Step 30000] train/loss: 0.475730, train/mse: 0.475730, train/data_wait_sec: 0.270468, train/step_wall_sec: 5.709002, train/data_wait_ratio: 0.047376, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30000.000000 2026-03-16 03:21:35,581 [Rank train_rank0] Generating samples for visualization... 2026-03-16 03:21:38,624 [Rank train_rank0] Sample generation done. 2026-03-16 03:23:35,674 [Rank train_rank0] [Epoch 0 | Step 30020] train/loss: 0.468988, train/mse: 0.468988, train/data_wait_sec: 0.320709, train/step_wall_sec: 5.752861, train/data_wait_ratio: 0.055748, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30020.000000 2026-03-16 03:25:32,576 [Rank train_rank0] [Epoch 0 | Step 30040] train/loss: 0.465485, train/mse: 0.465485, train/data_wait_sec: 0.358399, train/step_wall_sec: 5.838418, train/data_wait_ratio: 0.061386, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30040.000000 2026-03-16 03:27:29,057 [Rank train_rank0] [Epoch 0 | Step 30060] train/loss: 0.464427, train/mse: 0.464427, train/data_wait_sec: 0.281974, train/step_wall_sec: 5.868649, train/data_wait_ratio: 0.048048, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30060.000000 2026-03-16 03:29:25,997 [Rank train_rank0] [Epoch 0 | Step 30080] train/loss: 0.456114, train/mse: 0.456114, train/data_wait_sec: 0.227492, train/step_wall_sec: 5.725672, train/data_wait_ratio: 0.039732, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30080.000000 2026-03-16 03:31:22,351 [Rank train_rank0] [Epoch 0 | Step 30100] train/loss: 0.465765, train/mse: 0.465765, train/data_wait_sec: 0.207137, train/step_wall_sec: 5.679121, train/data_wait_ratio: 0.036473, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30100.000000 2026-03-16 03:33:19,457 [Rank train_rank0] [Epoch 0 | Step 30120] train/loss: 0.448180, train/mse: 0.448180, train/data_wait_sec: 0.278958, train/step_wall_sec: 5.834232, train/data_wait_ratio: 0.047814, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30120.000000 2026-03-16 03:35:16,026 [Rank train_rank0] [Epoch 0 | Step 30140] train/loss: 0.415246, train/mse: 0.415246, train/data_wait_sec: 0.350910, train/step_wall_sec: 5.830762, train/data_wait_ratio: 0.060183, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30140.000000 2026-03-16 03:37:12,821 [Rank train_rank0] [Epoch 0 | Step 30160] train/loss: 0.456987, train/mse: 0.456987, train/data_wait_sec: 0.288072, train/step_wall_sec: 5.794818, train/data_wait_ratio: 0.049712, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30160.000000 2026-03-16 03:39:09,468 [Rank train_rank0] [Epoch 0 | Step 30180] train/loss: 0.454401, train/mse: 0.454401, train/data_wait_sec: 0.262074, train/step_wall_sec: 5.853097, train/data_wait_ratio: 0.044775, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30180.000000 2026-03-16 03:41:06,208 [Rank train_rank0] [Epoch 0 | Step 30200] train/loss: 0.447216, train/mse: 0.447216, train/data_wait_sec: 0.221337, train/step_wall_sec: 5.799309, train/data_wait_ratio: 0.038166, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30200.000000 2026-03-16 03:43:03,125 [Rank train_rank0] [Epoch 0 | Step 30220] train/loss: 0.442365, train/mse: 0.442365, train/data_wait_sec: 0.319517, train/step_wall_sec: 5.871043, train/data_wait_ratio: 0.054422, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30220.000000 2026-03-16 03:44:59,819 [Rank train_rank0] [Epoch 0 | Step 30240] train/loss: 0.481152, train/mse: 0.481152, train/data_wait_sec: 0.359601, train/step_wall_sec: 5.919464, train/data_wait_ratio: 0.060749, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30240.000000 2026-03-16 03:46:56,117 [Rank train_rank0] [Epoch 0 | Step 30260] train/loss: 0.419779, train/mse: 0.419779, train/data_wait_sec: 0.349565, train/step_wall_sec: 5.818233, train/data_wait_ratio: 0.060081, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30260.000000 2026-03-16 03:48:52,595 [Rank train_rank0] [Epoch 0 | Step 30280] train/loss: 0.437921, train/mse: 0.437921, train/data_wait_sec: 0.230218, train/step_wall_sec: 5.808334, train/data_wait_ratio: 0.039636, train/lr: 0.000200, train/epoch: 0.000000, train/step: 30280.000000