Add files using upload-large-folder tool

Abs_6D + DeltaEE/Baseline.interrupted_reboot_20260719T051924/wandb/debug.log

@@ -0,0 +1,19 @@
+2026-07-17 15:43:31,292 INFO    MainThread:499783 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-17 15:43:31,293 INFO    MainThread:499783 [wandb_setup.py:_flush():81] Configure stats pid to 499783
+2026-07-17 15:43:31,293 INFO    MainThread:499783 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-17 15:43:31,293 INFO    MainThread:499783 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/Baseline/wandb/run-20260717_154331-qi6m7u1i/logs/debug.log
+2026-07-17 15:43:31,293 INFO    MainThread:499783 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/Baseline/wandb/run-20260717_154331-qi6m7u1i/logs/debug-internal.log
+2026-07-17 15:43:31,293 INFO    MainThread:499783 [wandb_init.py:init():766] calling init triggers
+2026-07-17 15:43:31,293 INFO    MainThread:499783 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/Baseline', 'job_name': 'INSIGHT_RD_baseline', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 500, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-17 15:43:31,293 INFO    MainThread:499783 [wandb_init.py:init():814] starting backend
+2026-07-17 15:43:31,517 INFO    MainThread:499783 [wandb_init.py:init():829] sending inform_init request
+2026-07-17 15:43:32,100 INFO    MainThread:499783 [wandb_init.py:init():834] backend started and connected
+2026-07-17 15:43:32,103 INFO    MainThread:499783 [wandb_init.py:init():904] updated telemetry
+2026-07-17 15:43:32,111 INFO    MainThread:499783 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-17 15:43:32,605 INFO    MainThread:499783 [wandb_init.py:init():972] starting run threads in backend
+2026-07-17 15:43:32,696 INFO    MainThread:499783 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-17 15:43:32,696 INFO    MainThread:499783 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-17 15:43:32,696 INFO    MainThread:499783 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-17 15:43:32,696 INFO    MainThread:499783 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-17 15:43:32,698 INFO    MainThread:499783 [wandb_init.py:init():1010] run started, returning control to user process

Abs_6D + DeltaEE/Baseline.interrupted_reboot_20260719T051924/wandb/run-20260717_154331-qi6m7u1i/files/output.log

@@ -0,0 +1,217 @@
+INFO 2026-07-17 15:43:32 db_utils.py:121 Logs will be synced with wandb.
+INFO 2026-07-17 15:43:32 db_utils.py:122 Track this run --> https://wandb.ai/minje227_hyu-hanyang-university/lerobot/runs/qi6m7u1i
+INFO 2026-07-17 15:43:32 ot_train.py:298 Creating dataset
+INFO 2026-07-17 15:43:36 ot_train.py:332 Creating policy
+Fetching 27 files: 100%|██████████| 27/27 [00:00<00:00, 2714.43it/s]
+`torch_dtype` is deprecated! Use `dtype` instead!
+Loading weights: 100%|██████████| 1031/1031 [00:00<00:00, 1971.57it/s]
+INFO 2026-07-17 15:43:42 ot_train.py:405 Creating optimizer and scheduler
+INFO 2026-07-17 15:43:42 ot_train.py:437 Output dir: /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/Baseline
+INFO 2026-07-17 15:43:42 ot_train.py:444 cfg.steps=60000 (60K)
+INFO 2026-07-17 15:43:42 ot_train.py:445 dataset.num_frames=218367 (218K)
+INFO 2026-07-17 15:43:42 ot_train.py:446 dataset.num_episodes=4930
+INFO 2026-07-17 15:43:42 ot_train.py:449 Effective batch size: 128 x 1 = 128
+INFO 2026-07-17 15:43:42 ot_train.py:450 num_learnable_params=1620515968 (2B)
+INFO 2026-07-17 15:43:42 ot_train.py:451 num_total_params=3144016000 (3B)
+Training:   0%|          | 0/60000 [00:00<?, ?step/s]INFO 2026-07-17 15:43:43 ot_train.py:602 Start offline training on a fixed dataset, with effective batch size: 128
+Training:   0%|          | 200/60000 [11:10<51:48:52,  3.12s/step]INFO 2026-07-17 15:54:53 ot_train.py:646 step:200 smpl:26K ep:578 epch:0.12 loss:0.805 grdn:1.040 lr:2.0e-05 updt_s:1.623 data_s:1.726 smp/s:38 mem_gb:52.75
+Training:   1%|          | 400/60000 [22:06<52:01:56,  3.14s/step]INFO 2026-07-17 16:05:49 ot_train.py:646 step:400 smpl:51K ep:1K epch:0.23 loss:0.181 grdn:1.631 lr:6.0e-05 updt_s:1.635 data_s:1.641 smp/s:39 mem_gb:52.81
+Training:   1%|          | 600/60000 [33:02<53:49:06,  3.26s/step]INFO 2026-07-17 16:16:45 ot_train.py:646 step:600 smpl:77K ep:2K epch:0.35 loss:0.140 grdn:1.147 lr:9.5e-05 updt_s:1.646 data_s:1.631 smp/s:39 mem_gb:52.81
+Training:   1%|▏         | 800/60000 [43:54<54:28:01,  3.31s/step]INFO 2026-07-17 16:27:37 ot_train.py:646 step:800 smpl:102K ep:2K epch:0.47 loss:0.110 grdn:0.772 lr:1.0e-04 updt_s:1.643 data_s:1.615 smp/s:39 mem_gb:52.81
+Training:   2%|▏         | 1000/60000 [54:37<56:26:58,  3.44s/step]INFO 2026-07-17 16:38:20 ot_train.py:646 step:1K smpl:128K ep:3K epch:0.59 loss:0.096 grdn:0.643 lr:1.0e-04 updt_s:1.592 data_s:1.620 smp/s:40 mem_gb:52.81
+Training:   2%|▏         | 1200/60000 [1:06:23<64:23:07,  3.94s/step]INFO 2026-07-17 16:50:06 ot_train.py:646 step:1K smpl:154K ep:3K epch:0.70 loss:0.088 grdn:0.590 lr:1.0e-04 updt_s:1.814 data_s:1.716 smp/s:36 mem_gb:52.73
+Training:   2%|▏         | 1400/60000 [1:17:38<57:27:28,  3.53s/step]INFO 2026-07-17 17:01:21 ot_train.py:646 step:1K smpl:179K ep:4K epch:0.82 loss:0.082 grdn:0.526 lr:1.0e-04 updt_s:1.678 data_s:1.693 smp/s:38 mem_gb:52.83
+Training:   3%|▎         | 1600/60000 [1:29:35<52:35:00,  3.24s/step]INFO 2026-07-17 17:13:18 ot_train.py:646 step:2K smpl:205K ep:5K epch:0.94 loss:0.077 grdn:0.490 lr:1.0e-04 updt_s:1.834 data_s:1.748 smp/s:36 mem_gb:52.83
+Training:   3%|▎         | 1800/60000 [1:40:42<58:36:12,  3.62s/step]INFO 2026-07-17 17:24:25 ot_train.py:646 step:2K smpl:230K ep:5K epch:1.06 loss:0.074 grdn:0.472 lr:1.0e-04 updt_s:1.642 data_s:1.687 smp/s:38 mem_gb:52.83
+Training:   3%|▎         | 2000/60000 [1:51:19<51:09:43,  3.18s/step]INFO 2026-07-17 17:35:02 ot_train.py:646 step:2K smpl:256K ep:6K epch:1.17 loss:0.072 grdn:0.451 lr:1.0e-04 updt_s:1.574 data_s:1.608 smp/s:40 mem_gb:52.83
+Training:   4%|▎         | 2200/60000 [2:01:46<51:18:04,  3.20s/step]INFO 2026-07-17 17:45:29 ot_train.py:646 step:2K smpl:282K ep:6K epch:1.29 loss:0.069 grdn:0.426 lr:1.0e-04 updt_s:1.524 data_s:1.609 smp/s:41 mem_gb:52.83
+Training:   4%|▍         | 2400/60000 [2:12:12<50:16:20,  3.14s/step]INFO 2026-07-17 17:55:55 ot_train.py:646 step:2K smpl:307K ep:7K epch:1.41 loss:0.066 grdn:0.388 lr:1.0e-04 updt_s:1.523 data_s:1.605 smp/s:41 mem_gb:52.75
+Training:   4%|▍         | 2600/60000 [2:22:47<57:17:39,  3.59s/step]INFO 2026-07-17 18:06:30 ot_train.py:646 step:3K smpl:333K ep:8K epch:1.52 loss:0.064 grdn:0.378 lr:1.0e-04 updt_s:1.553 data_s:1.616 smp/s:40 mem_gb:52.83
+Training:   5%|▍         | 2800/60000 [2:34:21<59:51:58,  3.77s/step]INFO 2026-07-17 18:18:04 ot_train.py:646 step:3K smpl:358K ep:8K epch:1.64 loss:0.061 grdn:0.361 lr:1.0e-04 updt_s:1.689 data_s:1.777 smp/s:37 mem_gb:52.83
+Training:   5%|▌         | 3000/60000 [2:45:39<50:35:42,  3.20s/step]INFO 2026-07-17 18:29:22 ot_train.py:646 step:3K smpl:384K ep:9K epch:1.76 loss:0.060 grdn:0.349 lr:1.0e-04 updt_s:1.667 data_s:1.720 smp/s:38 mem_gb:52.83
+Training:   5%|▌         | 3200/60000 [2:57:40<58:01:20,  3.68s/step]INFO 2026-07-17 18:41:23 ot_train.py:646 step:3K smpl:410K ep:9K epch:1.88 loss:0.059 grdn:0.338 lr:1.0e-04 updt_s:1.688 data_s:1.915 smp/s:36 mem_gb:52.83
+Training:   6%|▌         | 3400/60000 [3:09:21<49:59:39,  3.18s/step]INFO 2026-07-17 18:53:04 ot_train.py:646 step:3K smpl:435K ep:10K epch:1.99 loss:0.059 grdn:0.332 lr:9.9e-05 updt_s:1.725 data_s:1.780 smp/s:37 mem_gb:52.75
+Training:   6%|▌         | 3600/60000 [3:20:43<54:13:35,  3.46s/step]INFO 2026-07-17 19:04:26 ot_train.py:646 step:4K smpl:461K ep:10K epch:2.11 loss:0.056 grdn:0.323 lr:9.9e-05 updt_s:1.667 data_s:1.739 smp/s:38 mem_gb:52.83
+Training:   6%|▋         | 3800/60000 [3:32:07<50:04:04,  3.21s/step]INFO 2026-07-17 19:15:50 ot_train.py:646 step:4K smpl:486K ep:11K epch:2.23 loss:0.055 grdn:0.309 lr:9.9e-05 updt_s:1.634 data_s:1.781 smp/s:37 mem_gb:52.83
+Training:   7%|▋         | 4000/60000 [3:43:46<59:06:51,  3.80s/step]INFO 2026-07-17 19:27:29 ot_train.py:646 step:4K smpl:512K ep:12K epch:2.34 loss:0.055 grdn:0.303 lr:9.9e-05 updt_s:1.713 data_s:1.779 smp/s:37 mem_gb:52.83
+Training:   7%|▋         | 4200/60000 [3:56:17<52:11:20,  3.37s/step]INFO 2026-07-17 19:40:00 ot_train.py:646 step:4K smpl:538K ep:12K epch:2.46 loss:0.054 grdn:0.294 lr:9.9e-05 updt_s:1.692 data_s:2.059 smp/s:34 mem_gb:52.83
+Training:   7%|▋         | 4400/60000 [4:08:04<54:08:30,  3.51s/step]INFO 2026-07-17 19:51:47 ot_train.py:646 step:4K smpl:563K ep:13K epch:2.58 loss:0.054 grdn:0.290 lr:9.9e-05 updt_s:1.759 data_s:1.775 smp/s:36 mem_gb:52.83
+Training:   8%|▊         | 4600/60000 [4:20:19<57:58:34,  3.77s/step]INFO 2026-07-17 20:04:02 ot_train.py:646 step:5K smpl:589K ep:13K epch:2.70 loss:0.052 grdn:0.289 lr:9.9e-05 updt_s:1.800 data_s:1.871 smp/s:35 mem_gb:52.75
+Training:   8%|▊         | 4800/60000 [4:32:49<54:16:28,  3.54s/step]INFO 2026-07-17 20:16:32 ot_train.py:646 step:5K smpl:614K ep:14K epch:2.81 loss:0.053 grdn:0.286 lr:9.9e-05 updt_s:1.792 data_s:1.956 smp/s:34 mem_gb:52.83
+Training:   8%|▊         | 5000/60000 [4:44:21<59:08:15,  3.87s/step]INFO 2026-07-17 20:28:04 ot_train.py:646 step:5K smpl:640K ep:14K epch:2.93 loss:0.051 grdn:0.272 lr:9.9e-05 updt_s:1.808 data_s:1.645 smp/s:37 mem_gb:52.83
+Training:   9%|▊         | 5200/60000 [4:55:45<53:55:52,  3.54s/step]INFO 2026-07-17 20:39:28 ot_train.py:646 step:5K smpl:666K ep:15K epch:3.05 loss:0.050 grdn:0.263 lr:9.9e-05 updt_s:1.802 data_s:1.618 smp/s:37 mem_gb:52.83
+Training:   9%|▉         | 5400/60000 [5:08:17<52:59:04,  3.49s/step]INFO 2026-07-17 20:52:00 ot_train.py:646 step:5K smpl:691K ep:16K epch:3.17 loss:0.049 grdn:0.256 lr:9.8e-05 updt_s:1.817 data_s:1.937 smp/s:34 mem_gb:52.83
+Training:   9%|▉         | 5600/60000 [5:19:41<57:14:27,  3.79s/step]INFO 2026-07-17 21:03:24 ot_train.py:646 step:6K smpl:717K ep:16K epch:3.28 loss:0.048 grdn:0.259 lr:9.8e-05 updt_s:1.763 data_s:1.654 smp/s:37 mem_gb:52.83
+Training:  10%|▉         | 5800/60000 [5:30:51<51:03:54,  3.39s/step]INFO 2026-07-17 21:14:34 ot_train.py:646 step:6K smpl:742K ep:17K epch:3.40 loss:0.049 grdn:0.276 lr:9.8e-05 updt_s:1.675 data_s:1.674 smp/s:38 mem_gb:52.75
+Training:  10%|█         | 6000/60000 [5:42:04<52:12:50,  3.48s/step]INFO 2026-07-17 21:25:47 ot_train.py:646 step:6K smpl:768K ep:17K epch:3.52 loss:0.050 grdn:0.266 lr:9.8e-05 updt_s:1.774 data_s:1.587 smp/s:38 mem_gb:52.83
+Training:  10%|█         | 6200/60000 [5:52:48<48:37:16,  3.25s/step]INFO 2026-07-17 21:36:31 ot_train.py:646 step:6K smpl:794K ep:18K epch:3.63 loss:0.051 grdn:0.292 lr:9.8e-05 updt_s:1.711 data_s:1.505 smp/s:40 mem_gb:52.83
+Training:  11%|█         | 6400/60000 [6:03:20<45:53:15,  3.08s/step]INFO 2026-07-17 21:47:03 ot_train.py:646 step:6K smpl:819K ep:18K epch:3.75 loss:0.047 grdn:0.253 lr:9.8e-05 updt_s:1.789 data_s:1.367 smp/s:41 mem_gb:52.83
+Training:  11%|█         | 6600/60000 [6:13:54<45:52:26,  3.09s/step]INFO 2026-07-17 21:57:37 ot_train.py:646 step:7K smpl:845K ep:19K epch:3.87 loss:0.047 grdn:0.245 lr:9.8e-05 updt_s:1.722 data_s:1.447 smp/s:40 mem_gb:52.83
+Training:  11%|█▏        | 6800/60000 [6:24:23<44:47:44,  3.03s/step]INFO 2026-07-17 22:08:06 ot_train.py:646 step:7K smpl:870K ep:20K epch:3.99 loss:0.047 grdn:0.254 lr:9.7e-05 updt_s:1.571 data_s:1.572 smp/s:41 mem_gb:52.75
+Training:  12%|█▏        | 7000/60000 [6:34:46<40:43:07,  2.77s/step]INFO 2026-07-17 22:18:29 ot_train.py:646 step:7K smpl:896K ep:20K epch:4.10 loss:0.046 grdn:0.242 lr:9.7e-05 updt_s:1.624 data_s:1.488 smp/s:41 mem_gb:52.83
+Training:  12%|█▏        | 7200/60000 [6:45:10<45:55:55,  3.13s/step]INFO 2026-07-17 22:28:53 ot_train.py:646 step:7K smpl:922K ep:21K epch:4.22 loss:0.045 grdn:0.236 lr:9.7e-05 updt_s:1.644 data_s:1.473 smp/s:41 mem_gb:52.83
+Training:  12%|█▏        | 7400/60000 [6:55:38<46:11:17,  3.16s/step]INFO 2026-07-17 22:39:21 ot_train.py:646 step:7K smpl:947K ep:21K epch:4.34 loss:0.046 grdn:0.247 lr:9.7e-05 updt_s:1.667 data_s:1.470 smp/s:41 mem_gb:52.83
+Training:  13%|█▎        | 7600/60000 [7:06:22<58:40:49,  4.03s/step]INFO 2026-07-17 22:50:06 ot_train.py:646 step:8K smpl:973K ep:22K epch:4.45 loss:0.045 grdn:0.238 lr:9.7e-05 updt_s:1.692 data_s:1.527 smp/s:40 mem_gb:52.83
+Training:  13%|█▎        | 7800/60000 [7:17:00<46:49:55,  3.23s/step]INFO 2026-07-17 23:00:44 ot_train.py:646 step:8K smpl:998K ep:23K epch:4.57 loss:0.044 grdn:0.234 lr:9.6e-05 updt_s:1.640 data_s:1.547 smp/s:40 mem_gb:52.83
+Training:  13%|█▎        | 8000/60000 [7:27:44<51:29:17,  3.56s/step]INFO 2026-07-17 23:11:27 ot_train.py:646 step:8K smpl:1M ep:23K epch:4.69 loss:0.044 grdn:0.246 lr:9.6e-05 updt_s:1.701 data_s:1.511 smp/s:40 mem_gb:52.75
+Training:  14%|█▎        | 8200/60000 [7:38:17<44:58:08,  3.13s/step]INFO 2026-07-17 23:22:00 ot_train.py:646 step:8K smpl:1M ep:24K epch:4.81 loss:0.043 grdn:0.236 lr:9.6e-05 updt_s:1.634 data_s:1.530 smp/s:40 mem_gb:52.83
+Training:  14%|█▍        | 8400/60000 [7:48:38<42:01:13,  2.93s/step]INFO 2026-07-17 23:32:21 ot_train.py:646 step:8K smpl:1M ep:24K epch:4.92 loss:0.043 grdn:0.225 lr:9.6e-05 updt_s:1.577 data_s:1.524 smp/s:41 mem_gb:52.83
+Training:  14%|█▍        | 8600/60000 [7:58:57<44:40:14,  3.13s/step]INFO 2026-07-17 23:42:40 ot_train.py:646 step:9K smpl:1M ep:25K epch:5.04 loss:0.046 grdn:0.249 lr:9.6e-05 updt_s:1.581 data_s:1.512 smp/s:41 mem_gb:52.83
+Training:  15%|█▍        | 8800/60000 [8:09:19<43:07:47,  3.03s/step]INFO 2026-07-17 23:53:02 ot_train.py:646 step:9K smpl:1M ep:25K epch:5.16 loss:0.043 grdn:0.236 lr:9.5e-05 updt_s:1.695 data_s:1.410 smp/s:41 mem_gb:52.83
+Training:  15%|█▌        | 9000/60000 [8:19:29<44:04:22,  3.11s/step]INFO 2026-07-18 00:03:12 ot_train.py:646 step:9K smpl:1M ep:26K epch:5.28 loss:0.042 grdn:0.225 lr:9.5e-05 updt_s:1.719 data_s:1.328 smp/s:42 mem_gb:52.83
+Training:  15%|█▌        | 9200/60000 [8:30:04<43:53:00,  3.11s/step]INFO 2026-07-18 00:13:47 ot_train.py:646 step:9K smpl:1M ep:27K epch:5.39 loss:0.041 grdn:0.216 lr:9.5e-05 updt_s:1.649 data_s:1.523 smp/s:40 mem_gb:52.75
+Training:  16%|█▌        | 9400/60000 [8:40:56<50:30:09,  3.59s/step]INFO 2026-07-18 00:24:39 ot_train.py:646 step:9K smpl:1M ep:27K epch:5.51 loss:0.040 grdn:0.219 lr:9.5e-05 updt_s:1.681 data_s:1.578 smp/s:39 mem_gb:52.83
+Training:  16%|█▌        | 9600/60000 [8:52:03<42:30:00,  3.04s/step]INFO 2026-07-18 00:35:46 ot_train.py:646 step:10K smpl:1M ep:28K epch:5.63 loss:0.041 grdn:0.216 lr:9.4e-05 updt_s:1.715 data_s:1.614 smp/s:38 mem_gb:52.83
+Training:  16%|█▋        | 9800/60000 [9:02:17<42:35:52,  3.05s/step]INFO 2026-07-18 00:46:00 ot_train.py:646 step:10K smpl:1M ep:28K epch:5.74 loss:0.041 grdn:0.225 lr:9.4e-05 updt_s:1.634 data_s:1.432 smp/s:42 mem_gb:52.83
+Training:  17%|█▋        | 10000/60000 [9:12:14<38:44:24,  2.79s/step]INFO 2026-07-18 00:55:57 ot_train.py:646 step:10K smpl:1M ep:29K epch:5.86 loss:0.041 grdn:0.215 lr:9.4e-05 updt_s:1.601 data_s:1.385 smp/s:43 mem_gb:52.83
+Training:  17%|█▋        | 10200/60000 [9:22:10<41:03:00,  2.97s/step]INFO 2026-07-18 01:05:53 ot_train.py:646 step:10K smpl:1M ep:29K epch:5.98 loss:0.040 grdn:0.214 lr:9.4e-05 updt_s:1.638 data_s:1.338 smp/s:43 mem_gb:52.75
+Training:  17%|█▋        | 10400/60000 [9:32:03<38:45:00,  2.81s/step]INFO 2026-07-18 01:15:46 ot_train.py:646 step:10K smpl:1M ep:30K epch:6.10 loss:0.039 grdn:0.214 lr:9.3e-05 updt_s:1.623 data_s:1.340 smp/s:43 mem_gb:52.83
+Training:  18%|█▊        | 10600/60000 [9:41:58<39:38:58,  2.89s/step]INFO 2026-07-18 01:25:41 ot_train.py:646 step:11K smpl:1M ep:31K epch:6.21 loss:0.039 grdn:0.212 lr:9.3e-05 updt_s:1.620 data_s:1.348 smp/s:43 mem_gb:52.83
+Training:  18%|█▊        | 10800/60000 [9:51:55<41:24:11,  3.03s/step]INFO 2026-07-18 01:35:39 ot_train.py:646 step:11K smpl:1M ep:31K epch:6.33 loss:0.039 grdn:0.218 lr:9.3e-05 updt_s:1.640 data_s:1.346 smp/s:43 mem_gb:52.83
+Training:  18%|█▊        | 11000/60000 [10:02:11<49:33:32,  3.64s/step]INFO 2026-07-18 01:45:54 ot_train.py:646 step:11K smpl:1M ep:32K epch:6.45 loss:0.039 grdn:0.222 lr:9.3e-05 updt_s:1.749 data_s:1.326 smp/s:42 mem_gb:52.83
+Training:  19%|█▊        | 11200/60000 [10:12:07<41:30:05,  3.06s/step]INFO 2026-07-18 01:55:50 ot_train.py:646 step:11K smpl:1M ep:32K epch:6.57 loss:0.039 grdn:0.213 lr:9.2e-05 updt_s:1.637 data_s:1.340 smp/s:43 mem_gb:52.83
+Training:  19%|█▉        | 11400/60000 [10:22:09<38:17:43,  2.84s/step]INFO 2026-07-18 02:05:52 ot_train.py:646 step:11K smpl:1M ep:33K epch:6.68 loss:0.041 grdn:0.222 lr:9.2e-05 updt_s:1.624 data_s:1.381 smp/s:43 mem_gb:52.75
+Training:  19%|█▉        | 11600/60000 [10:31:59<40:14:51,  2.99s/step]INFO 2026-07-18 02:15:42 ot_train.py:646 step:12K smpl:1M ep:34K epch:6.80 loss:0.039 grdn:0.231 lr:9.2e-05 updt_s:1.699 data_s:1.249 smp/s:43 mem_gb:52.83
+Training:  20%|█▉        | 11800/60000 [10:42:44<42:45:55,  3.19s/step]INFO 2026-07-18 02:26:27 ot_train.py:646 step:12K smpl:2M ep:34K epch:6.92 loss:0.037 grdn:0.200 lr:9.2e-05 updt_s:1.734 data_s:1.491 smp/s:40 mem_gb:52.83
+Training:  20%|██        | 12000/60000 [10:53:11<41:35:57,  3.12s/step]INFO 2026-07-18 02:36:54 ot_train.py:646 step:12K smpl:2M ep:35K epch:7.03 loss:0.037 grdn:0.207 lr:9.1e-05 updt_s:1.544 data_s:1.589 smp/s:41 mem_gb:52.83
+Training:  20%|██        | 12200/60000 [11:03:27<40:31:52,  3.05s/step]INFO 2026-07-18 02:47:10 ot_train.py:646 step:12K smpl:2M ep:35K epch:7.15 loss:0.037 grdn:0.199 lr:9.1e-05 updt_s:1.595 data_s:1.481 smp/s:42 mem_gb:52.83
+Training:  21%|██        | 12400/60000 [11:12:57<36:57:43,  2.80s/step]INFO 2026-07-18 02:56:40 ot_train.py:646 step:12K smpl:2M ep:36K epch:7.27 loss:0.037 grdn:0.201 lr:9.1e-05 updt_s:1.582 data_s:1.263 smp/s:45 mem_gb:52.83
+Training:  21%|██        | 12600/60000 [11:22:17<39:05:41,  2.97s/step]INFO 2026-07-18 03:06:00 ot_train.py:646 step:13K smpl:2M ep:36K epch:7.39 loss:0.036 grdn:0.208 lr:9.0e-05 updt_s:1.568 data_s:1.227 smp/s:46 mem_gb:52.75
+Training:  21%|██▏       | 12800/60000 [11:31:39<37:40:03,  2.87s/step]INFO 2026-07-18 03:15:22 ot_train.py:646 step:13K smpl:2M ep:37K epch:7.50 loss:0.036 grdn:0.204 lr:9.0e-05 updt_s:1.587 data_s:1.222 smp/s:46 mem_gb:52.83
+Training:  22%|██▏       | 13000/60000 [11:41:02<35:57:14,  2.75s/step]INFO 2026-07-18 03:24:45 ot_train.py:646 step:13K smpl:2M ep:38K epch:7.62 loss:0.036 grdn:0.207 lr:9.0e-05 updt_s:1.577 data_s:1.235 smp/s:46 mem_gb:52.83
+Training:  22%|██▏       | 13200/60000 [11:50:31<37:31:02,  2.89s/step]INFO 2026-07-18 03:34:14 ot_train.py:646 step:13K smpl:2M ep:38K epch:7.74 loss:0.036 grdn:0.202 lr:8.9e-05 updt_s:1.589 data_s:1.253 smp/s:45 mem_gb:52.83
+Training:  22%|██▏       | 13400/60000 [11:59:49<34:47:20,  2.69s/step]INFO 2026-07-18 03:43:32 ot_train.py:646 step:13K smpl:2M ep:39K epch:7.85 loss:0.037 grdn:0.216 lr:8.9e-05 updt_s:1.570 data_s:1.218 smp/s:46 mem_gb:52.83
+Training:  23%|██▎       | 13600/60000 [12:09:20<35:41:22,  2.77s/step]INFO 2026-07-18 03:53:03 ot_train.py:646 step:14K smpl:2M ep:39K epch:7.97 loss:0.037 grdn:0.218 lr:8.9e-05 updt_s:1.596 data_s:1.256 smp/s:45 mem_gb:52.75
+Training:  23%|██▎       | 13800/60000 [12:18:43<34:08:25,  2.66s/step]INFO 2026-07-18 04:02:26 ot_train.py:646 step:14K smpl:2M ep:40K epch:8.09 loss:0.035 grdn:0.202 lr:8.8e-05 updt_s:1.565 data_s:1.247 smp/s:46 mem_gb:52.83
+Training:  23%|██▎       | 14000/60000 [12:28:19<36:38:04,  2.87s/step]INFO 2026-07-18 04:12:02 ot_train.py:646 step:14K smpl:2M ep:40K epch:8.21 loss:0.034 grdn:0.203 lr:8.8e-05 updt_s:1.579 data_s:1.296 smp/s:45 mem_gb:52.83
+Training:  24%|██▎       | 14200/60000 [12:37:55<36:49:40,  2.89s/step]INFO 2026-07-18 04:21:38 ot_train.py:646 step:14K smpl:2M ep:41K epch:8.32 loss:0.034 grdn:0.195 lr:8.8e-05 updt_s:1.593 data_s:1.284 smp/s:44 mem_gb:52.83
+Training:  24%|██▍       | 14400/60000 [12:47:51<38:16:46,  3.02s/step]INFO 2026-07-18 04:31:34 ot_train.py:646 step:14K smpl:2M ep:42K epch:8.44 loss:0.034 grdn:0.202 lr:8.7e-05 updt_s:1.617 data_s:1.361 smp/s:43 mem_gb:52.83
+Training:  24%|██▍       | 14600/60000 [12:57:27<41:26:22,  3.29s/step]INFO 2026-07-18 04:41:10 ot_train.py:646 step:15K smpl:2M ep:42K epch:8.56 loss:0.035 grdn:0.203 lr:8.7e-05 updt_s:1.605 data_s:1.270 smp/s:45 mem_gb:52.83
+Training:  25%|██▍       | 14800/60000 [13:07:04<32:54:05,  2.62s/step]INFO 2026-07-18 04:50:47 ot_train.py:646 step:15K smpl:2M ep:43K epch:8.68 loss:0.034 grdn:0.203 lr:8.7e-05 updt_s:1.583 data_s:1.301 smp/s:44 mem_gb:52.75
+Training:  25%|██▌       | 15000/60000 [13:16:35<31:20:36,  2.51s/step]INFO 2026-07-18 05:00:18 ot_train.py:646 step:15K smpl:2M ep:43K epch:8.79 loss:0.035 grdn:0.216 lr:8.6e-05 updt_s:1.589 data_s:1.261 smp/s:45 mem_gb:52.83
+Training:  25%|██▌       | 15200/60000 [13:26:09<36:35:35,  2.94s/step]INFO 2026-07-18 05:09:52 ot_train.py:646 step:15K smpl:2M ep:44K epch:8.91 loss:0.033 grdn:0.194 lr:8.6e-05 updt_s:1.592 data_s:1.276 smp/s:45 mem_gb:52.83
+Training:  26%|██▌       | 15400/60000 [13:35:51<37:21:49,  3.02s/step]INFO 2026-07-18 05:19:34 ot_train.py:646 step:15K smpl:2M ep:45K epch:9.03 loss:0.033 grdn:0.194 lr:8.5e-05 updt_s:1.617 data_s:1.292 smp/s:44 mem_gb:52.83
+Training:  26%|██▌       | 15600/60000 [13:45:44<35:25:20,  2.87s/step]INFO 2026-07-18 05:29:27 ot_train.py:646 step:16K smpl:2M ep:45K epch:9.14 loss:0.033 grdn:0.191 lr:8.5e-05 updt_s:1.613 data_s:1.347 smp/s:43 mem_gb:52.83
+Training:  26%|██▋       | 15800/60000 [13:55:41<37:28:43,  3.05s/step]INFO 2026-07-18 05:39:24 ot_train.py:646 step:16K smpl:2M ep:46K epch:9.26 loss:0.033 grdn:0.200 lr:8.5e-05 updt_s:1.636 data_s:1.347 smp/s:43 mem_gb:52.83
+Training:  27%|██▋       | 16000/60000 [14:05:41<39:27:34,  3.23s/step]INFO 2026-07-18 05:49:24 ot_train.py:646 step:16K smpl:2M ep:46K epch:9.38 loss:0.031 grdn:0.189 lr:8.4e-05 updt_s:1.610 data_s:1.387 smp/s:43 mem_gb:52.75
+Training:  27%|██▋       | 16200/60000 [14:15:43<34:57:06,  2.87s/step]INFO 2026-07-18 05:59:26 ot_train.py:646 step:16K smpl:2M ep:47K epch:9.50 loss:0.031 grdn:0.189 lr:8.4e-05 updt_s:1.632 data_s:1.373 smp/s:43 mem_gb:52.83
+Training:  27%|██▋       | 16400/60000 [14:25:52<43:27:29,  3.59s/step]INFO 2026-07-18 06:09:35 ot_train.py:646 step:16K smpl:2M ep:47K epch:9.61 loss:0.031 grdn:0.189 lr:8.4e-05 updt_s:1.654 data_s:1.388 smp/s:42 mem_gb:52.83
+Training:  28%|██▊       | 16600/60000 [14:36:00<36:28:07,  3.03s/step]INFO 2026-07-18 06:19:43 ot_train.py:646 step:17K smpl:2M ep:48K epch:9.73 loss:0.031 grdn:0.192 lr:8.3e-05 updt_s:1.661 data_s:1.376 smp/s:42 mem_gb:52.83
+Training:  28%|██▊       | 16800/60000 [14:46:01<35:24:07,  2.95s/step]INFO 2026-07-18 06:29:44 ot_train.py:646 step:17K smpl:2M ep:49K epch:9.85 loss:0.032 grdn:0.192 lr:8.3e-05 updt_s:1.649 data_s:1.354 smp/s:43 mem_gb:52.83
+Training:  28%|██▊       | 17000/60000 [14:55:55<35:36:34,  2.98s/step]INFO 2026-07-18 06:39:38 ot_train.py:646 step:17K smpl:2M ep:49K epch:9.96 loss:0.031 grdn:0.183 lr:8.2e-05 updt_s:1.594 data_s:1.377 smp/s:43 mem_gb:52.75
+Training:  29%|██▊       | 17200/60000 [15:05:56<34:07:52,  2.87s/step]INFO 2026-07-18 06:49:39 ot_train.py:646 step:17K smpl:2M ep:50K epch:10.08 loss:0.030 grdn:0.190 lr:8.2e-05 updt_s:1.608 data_s:1.394 smp/s:43 mem_gb:52.83
+Training:  29%|██▉       | 17400/60000 [15:15:37<34:43:03,  2.93s/step]INFO 2026-07-18 06:59:20 ot_train.py:646 step:17K smpl:2M ep:50K epch:10.20 loss:0.031 grdn:0.189 lr:8.2e-05 updt_s:1.609 data_s:1.290 smp/s:44 mem_gb:52.83
+Training:  29%|██▉       | 17600/60000 [15:25:05<32:00:28,  2.72s/step]INFO 2026-07-18 07:08:48 ot_train.py:646 step:18K smpl:2M ep:51K epch:10.32 loss:0.030 grdn:0.188 lr:8.1e-05 updt_s:1.591 data_s:1.246 smp/s:45 mem_gb:52.83
+Training:  30%|██▉       | 17800/60000 [15:34:34<37:35:53,  3.21s/step]INFO 2026-07-18 07:18:17 ot_train.py:646 step:18K smpl:2M ep:51K epch:10.43 loss:0.030 grdn:0.192 lr:8.1e-05 updt_s:1.619 data_s:1.223 smp/s:45 mem_gb:52.83
+Training:  30%|███       | 18000/60000 [15:43:45<31:32:54,  2.70s/step]INFO 2026-07-18 07:27:28 ot_train.py:646 step:18K smpl:2M ep:52K epch:10.55 loss:0.030 grdn:0.185 lr:8.0e-05 updt_s:1.563 data_s:1.189 smp/s:47 mem_gb:52.83
+Training:  30%|███       | 18200/60000 [15:53:01<30:16:41,  2.61s/step]INFO 2026-07-18 07:36:44 ot_train.py:646 step:18K smpl:2M ep:53K epch:10.67 loss:0.029 grdn:0.187 lr:8.0e-05 updt_s:1.566 data_s:1.215 smp/s:46 mem_gb:52.75
+Training:  31%|███       | 18400/60000 [16:02:26<32:40:44,  2.83s/step]INFO 2026-07-18 07:46:09 ot_train.py:646 step:18K smpl:2M ep:53K epch:10.79 loss:0.029 grdn:0.186 lr:7.9e-05 updt_s:1.605 data_s:1.215 smp/s:45 mem_gb:52.83
+Training:  31%|███       | 18600/60000 [16:11:46<30:33:21,  2.66s/step]INFO 2026-07-18 07:55:29 ot_train.py:646 step:19K smpl:2M ep:54K epch:10.90 loss:0.029 grdn:0.185 lr:7.9e-05 updt_s:1.591 data_s:1.208 smp/s:46 mem_gb:52.83
+Training:  31%|███▏      | 18800/60000 [16:20:57<34:02:40,  2.97s/step]INFO 2026-07-18 08:04:40 ot_train.py:646 step:19K smpl:2M ep:54K epch:11.02 loss:0.031 grdn:0.202 lr:7.9e-05 updt_s:1.556 data_s:1.197 smp/s:47 mem_gb:52.83
+Training:  32%|███▏      | 19000/60000 [16:30:25<34:42:37,  3.05s/step]INFO 2026-07-18 08:14:08 ot_train.py:646 step:19K smpl:2M ep:55K epch:11.14 loss:0.029 grdn:0.184 lr:7.8e-05 updt_s:1.582 data_s:1.253 smp/s:45 mem_gb:52.83
+Training:  32%|███▏      | 19200/60000 [16:40:21<34:58:25,  3.09s/step]INFO 2026-07-18 08:24:04 ot_train.py:646 step:19K smpl:2M ep:55K epch:11.25 loss:0.028 grdn:0.186 lr:7.8e-05 updt_s:1.637 data_s:1.342 smp/s:43 mem_gb:52.75
+Training:  32%|███▏      | 19400/60000 [16:50:03<30:38:55,  2.72s/step]INFO 2026-07-18 08:33:46 ot_train.py:646 step:19K smpl:2M ep:56K epch:11.37 loss:0.028 grdn:0.182 lr:7.7e-05 updt_s:1.607 data_s:1.302 smp/s:44 mem_gb:52.83
+Training:  33%|███▎      | 19600/60000 [16:59:39<30:01:49,  2.68s/step]INFO 2026-07-18 08:43:22 ot_train.py:646 step:20K smpl:3M ep:57K epch:11.49 loss:0.028 grdn:0.191 lr:7.7e-05 updt_s:1.611 data_s:1.265 smp/s:45 mem_gb:52.83
+Training:  33%|███▎      | 19800/60000 [17:09:08<31:02:52,  2.78s/step]INFO 2026-07-18 08:52:51 ot_train.py:646 step:20K smpl:3M ep:57K epch:11.61 loss:0.028 grdn:0.182 lr:7.6e-05 updt_s:1.588 data_s:1.250 smp/s:45 mem_gb:52.83
+Training:  33%|███▎      | 20000/60000 [17:18:58<34:03:44,  3.07s/step]INFO 2026-07-18 09:02:41 ot_train.py:646 step:20K smpl:3M ep:58K epch:11.72 loss:0.027 grdn:0.176 lr:7.6e-05 updt_s:1.617 data_s:1.331 smp/s:43 mem_gb:52.83
+Training:  34%|███▎      | 20200/60000 [17:28:52<33:52:58,  3.06s/step]INFO 2026-07-18 09:12:35 ot_train.py:646 step:20K smpl:3M ep:58K epch:11.84 loss:0.028 grdn:0.187 lr:7.6e-05 updt_s:1.665 data_s:1.303 smp/s:43 mem_gb:52.83
+Training:  34%|███▍      | 20400/60000 [17:38:42<33:41:58,  3.06s/step]INFO 2026-07-18 09:22:25 ot_train.py:646 step:20K smpl:3M ep:59K epch:11.96 loss:0.027 grdn:0.178 lr:7.5e-05 updt_s:1.618 data_s:1.329 smp/s:43 mem_gb:52.75
+Training:  34%|███▍      | 20600/60000 [17:48:35<33:16:12,  3.04s/step]INFO 2026-07-18 09:32:18 ot_train.py:646 step:21K smpl:3M ep:60K epch:12.08 loss:0.026 grdn:0.180 lr:7.5e-05 updt_s:1.633 data_s:1.329 smp/s:43 mem_gb:52.83
+Training:  35%|███▍      | 20800/60000 [17:58:45<35:42:30,  3.28s/step]INFO 2026-07-18 09:42:28 ot_train.py:646 step:21K smpl:3M ep:60K epch:12.19 loss:0.028 grdn:0.190 lr:7.4e-05 updt_s:1.741 data_s:1.307 smp/s:42 mem_gb:52.83
+Training:  35%|███▌      | 21000/60000 [18:09:20<34:57:50,  3.23s/step]INFO 2026-07-18 09:53:03 ot_train.py:646 step:21K smpl:3M ep:61K epch:12.31 loss:0.027 grdn:0.183 lr:7.4e-05 updt_s:1.812 data_s:1.362 smp/s:40 mem_gb:52.83
+Training:  35%|███▌      | 21200/60000 [18:19:54<33:18:17,  3.09s/step]INFO 2026-07-18 10:03:37 ot_train.py:646 step:21K smpl:3M ep:61K epch:12.43 loss:0.026 grdn:0.185 lr:7.3e-05 updt_s:1.821 data_s:1.347 smp/s:40 mem_gb:52.83
+Training:  36%|███▌      | 21400/60000 [18:30:06<32:50:17,  3.06s/step]INFO 2026-07-18 10:13:49 ot_train.py:646 step:21K smpl:3M ep:62K epch:12.54 loss:0.026 grdn:0.176 lr:7.3e-05 updt_s:1.780 data_s:1.277 smp/s:42 mem_gb:52.83
+Training:  36%|███▌      | 21600/60000 [18:40:44<34:53:12,  3.27s/step]INFO 2026-07-18 10:24:27 ot_train.py:646 step:22K smpl:3M ep:62K epch:12.66 loss:0.025 grdn:0.171 lr:7.2e-05 updt_s:1.831 data_s:1.354 smp/s:40 mem_gb:52.75
+Training:  36%|███▋      | 21800/60000 [18:51:12<34:07:48,  3.22s/step]INFO 2026-07-18 10:34:55 ot_train.py:646 step:22K smpl:3M ep:63K epch:12.78 loss:0.025 grdn:0.171 lr:7.2e-05 updt_s:1.816 data_s:1.321 smp/s:41 mem_gb:52.83
+Training:  37%|███▋      | 22000/60000 [19:01:48<33:13:39,  3.15s/step]INFO 2026-07-18 10:45:31 ot_train.py:646 step:22K smpl:3M ep:64K epch:12.90 loss:0.025 grdn:0.179 lr:7.1e-05 updt_s:1.779 data_s:1.399 smp/s:40 mem_gb:52.83
+Training:  37%|███▋      | 22200/60000 [19:12:18<32:53:21,  3.13s/step]INFO 2026-07-18 10:56:01 ot_train.py:646 step:22K smpl:3M ep:64K epch:13.01 loss:0.026 grdn:0.174 lr:7.1e-05 updt_s:1.556 data_s:1.591 smp/s:41 mem_gb:52.83
+Training:  37%|███▋      | 22400/60000 [19:23:16<32:25:58,  3.11s/step]INFO 2026-07-18 11:06:59 ot_train.py:646 step:22K smpl:3M ep:65K epch:13.13 loss:0.025 grdn:0.171 lr:7.0e-05 updt_s:1.658 data_s:1.627 smp/s:39 mem_gb:52.83
+Training:  38%|███▊      | 22600/60000 [19:33:49<32:44:37,  3.15s/step]INFO 2026-07-18 11:17:32 ot_train.py:646 step:23K smpl:3M ep:65K epch:13.25 loss:0.025 grdn:0.176 lr:7.0e-05 updt_s:1.530 data_s:1.634 smp/s:40 mem_gb:52.75
+Training:  38%|███▊      | 22800/60000 [19:44:20<33:41:51,  3.26s/step]INFO 2026-07-18 11:28:03 ot_train.py:646 step:23K smpl:3M ep:66K epch:13.36 loss:0.024 grdn:0.173 lr:6.9e-05 updt_s:1.565 data_s:1.584 smp/s:41 mem_gb:52.83
+Training:  38%|███▊      | 23000/60000 [19:54:43<32:04:27,  3.12s/step]INFO 2026-07-18 11:38:26 ot_train.py:646 step:23K smpl:3M ep:66K epch:13.48 loss:0.024 grdn:0.162 lr:6.9e-05 updt_s:1.556 data_s:1.556 smp/s:41 mem_gb:52.83
+Training:  39%|███▊      | 23200/60000 [20:05:11<31:25:41,  3.07s/step]INFO 2026-07-18 11:48:54 ot_train.py:646 step:23K smpl:3M ep:67K epch:13.60 loss:0.024 grdn:0.179 lr:6.8e-05 updt_s:1.546 data_s:1.591 smp/s:41 mem_gb:52.83
+Training:  39%|███▉      | 23400/60000 [20:15:35<32:36:05,  3.21s/step]INFO 2026-07-18 11:59:18 ot_train.py:646 step:23K smpl:3M ep:68K epch:13.72 loss:0.024 grdn:0.179 lr:6.8e-05 updt_s:1.530 data_s:1.587 smp/s:41 mem_gb:52.83
+Training:  39%|███▉      | 23600/60000 [20:25:55<31:00:03,  3.07s/step]INFO 2026-07-18 12:09:38 ot_train.py:646 step:24K smpl:3M ep:68K epch:13.83 loss:0.024 grdn:0.172 lr:6.7e-05 updt_s:1.491 data_s:1.606 smp/s:41 mem_gb:52.83
+Training:  40%|███▉      | 23800/60000 [20:36:21<31:15:51,  3.11s/step]INFO 2026-07-18 12:20:04 ot_train.py:646 step:24K smpl:3M ep:69K epch:13.95 loss:0.024 grdn:0.173 lr:6.7e-05 updt_s:1.524 data_s:1.600 smp/s:41 mem_gb:52.75
+Training:  40%|████      | 24000/60000 [20:46:44<30:08:33,  3.01s/step]INFO 2026-07-18 12:30:27 ot_train.py:646 step:24K smpl:3M ep:69K epch:14.07 loss:0.024 grdn:0.169 lr:6.6e-05 updt_s:1.524 data_s:1.591 smp/s:41 mem_gb:52.83
+Training:  40%|████      | 24200/60000 [20:57:06<29:55:24,  3.01s/step]INFO 2026-07-18 12:40:50 ot_train.py:646 step:24K smpl:3M ep:70K epch:14.19 loss:0.023 grdn:0.168 lr:6.6e-05 updt_s:1.526 data_s:1.582 smp/s:41 mem_gb:52.83
+Training:  41%|████      | 24400/60000 [21:07:28<30:56:57,  3.13s/step]INFO 2026-07-18 12:51:12 ot_train.py:646 step:24K smpl:3M ep:71K epch:14.30 loss:0.024 grdn:0.177 lr:6.5e-05 updt_s:1.529 data_s:1.578 smp/s:41 mem_gb:52.83
+Training:  41%|████      | 24600/60000 [21:17:51<30:16:18,  3.08s/step]INFO 2026-07-18 13:01:34 ot_train.py:646 step:25K smpl:3M ep:71K epch:14.42 loss:0.023 grdn:0.172 lr:6.5e-05 updt_s:1.549 data_s:1.558 smp/s:41 mem_gb:52.83
+Training:  41%|████▏     | 24800/60000 [21:28:15<30:13:48,  3.09s/step]INFO 2026-07-18 13:11:58 ot_train.py:646 step:25K smpl:3M ep:72K epch:14.54 loss:0.023 grdn:0.164 lr:6.4e-05 updt_s:1.549 data_s:1.571 smp/s:41 mem_gb:52.83
+Training:  42%|████▏     | 25000/60000 [21:38:37<30:17:32,  3.12s/step]INFO 2026-07-18 13:22:20 ot_train.py:646 step:25K smpl:3M ep:72K epch:14.65 loss:0.022 grdn:0.168 lr:6.4e-05 updt_s:1.498 data_s:1.610 smp/s:41 mem_gb:52.75
+Training:  42%|████▏     | 25200/60000 [21:49:01<29:47:07,  3.08s/step]INFO 2026-07-18 13:32:45 ot_train.py:646 step:25K smpl:3M ep:73K epch:14.77 loss:0.022 grdn:0.169 lr:6.3e-05 updt_s:1.601 data_s:1.515 smp/s:41 mem_gb:52.83
+Training:  42%|████▏     | 25400/60000 [21:59:24<29:36:14,  3.08s/step]INFO 2026-07-18 13:43:07 ot_train.py:646 step:25K smpl:3M ep:73K epch:14.89 loss:0.023 grdn:0.172 lr:6.3e-05 updt_s:1.554 data_s:1.555 smp/s:41 mem_gb:52.83
+Training:  43%|████▎     | 25600/60000 [22:09:39<28:25:09,  2.97s/step]INFO 2026-07-18 13:53:22 ot_train.py:646 step:26K smpl:3M ep:74K epch:15.01 loss:0.022 grdn:0.164 lr:6.2e-05 updt_s:1.563 data_s:1.508 smp/s:42 mem_gb:52.83
+Training:  43%|████▎     | 25800/60000 [22:19:47<28:17:07,  2.98s/step]INFO 2026-07-18 14:03:30 ot_train.py:646 step:26K smpl:3M ep:75K epch:15.12 loss:0.022 grdn:0.165 lr:6.2e-05 updt_s:1.597 data_s:1.440 smp/s:42 mem_gb:52.83
+Training:  43%|████▎     | 26000/60000 [22:29:57<28:31:16,  3.02s/step]INFO 2026-07-18 14:13:40 ot_train.py:646 step:26K smpl:3M ep:75K epch:15.24 loss:0.022 grdn:0.166 lr:6.1e-05 updt_s:1.618 data_s:1.429 smp/s:42 mem_gb:52.75
+Training:  44%|████▎     | 26200/60000 [22:40:00<28:26:48,  3.03s/step]INFO 2026-07-18 14:23:43 ot_train.py:646 step:26K smpl:3M ep:76K epch:15.36 loss:0.020 grdn:0.159 lr:6.1e-05 updt_s:1.598 data_s:1.416 smp/s:42 mem_gb:52.83
+Training:  44%|████▍     | 26400/60000 [22:50:13<27:24:11,  2.94s/step]INFO 2026-07-18 14:33:56 ot_train.py:646 step:26K smpl:3M ep:76K epch:15.47 loss:0.021 grdn:0.159 lr:6.0e-05 updt_s:1.632 data_s:1.429 smp/s:42 mem_gb:52.83
+Training:  44%|████▍     | 26600/60000 [23:00:26<27:33:04,  2.97s/step]INFO 2026-07-18 14:44:09 ot_train.py:646 step:27K smpl:3M ep:77K epch:15.59 loss:0.020 grdn:0.163 lr:6.0e-05 updt_s:1.572 data_s:1.490 smp/s:42 mem_gb:52.83
+Training:  45%|████▍     | 26800/60000 [23:10:42<29:24:36,  3.19s/step]INFO 2026-07-18 14:54:25 ot_train.py:646 step:27K smpl:3M ep:77K epch:15.71 loss:0.021 grdn:0.157 lr:5.9e-05 updt_s:1.554 data_s:1.521 smp/s:42 mem_gb:52.83
+Training:  45%|████▌     | 27000/60000 [23:21:04<28:23:16,  3.10s/step]INFO 2026-07-18 15:04:47 ot_train.py:646 step:27K smpl:3M ep:78K epch:15.83 loss:0.021 grdn:0.160 lr:5.9e-05 updt_s:1.585 data_s:1.523 smp/s:41 mem_gb:52.83
+Training:  45%|████▌     | 27200/60000 [23:31:32<27:51:49,  3.06s/step]INFO 2026-07-18 15:15:15 ot_train.py:646 step:27K smpl:3M ep:79K epch:15.94 loss:0.020 grdn:0.159 lr:5.8e-05 updt_s:1.595 data_s:1.543 smp/s:41 mem_gb:52.75
+Training:  46%|████▌     | 27400/60000 [23:41:48<27:06:43,  2.99s/step]INFO 2026-07-18 15:25:31 ot_train.py:646 step:27K smpl:4M ep:79K epch:16.06 loss:0.020 grdn:0.156 lr:5.8e-05 updt_s:1.606 data_s:1.470 smp/s:42 mem_gb:52.83
+Training:  46%|████▌     | 27600/60000 [23:51:59<27:23:21,  3.04s/step]INFO 2026-07-18 15:35:42 ot_train.py:646 step:28K smpl:4M ep:80K epch:16.18 loss:0.020 grdn:0.155 lr:5.7e-05 updt_s:1.570 data_s:1.482 smp/s:42 mem_gb:52.83
+Training:  46%|████▋     | 27800/60000 [24:01:58<27:58:44,  3.13s/step]INFO 2026-07-18 15:45:41 ot_train.py:646 step:28K smpl:4M ep:80K epch:16.30 loss:0.020 grdn:0.165 lr:5.7e-05 updt_s:1.592 data_s:1.401 smp/s:43 mem_gb:52.83
+Training:  47%|████▋     | 28000/60000 [24:11:53<27:09:18,  3.05s/step]INFO 2026-07-18 15:55:36 ot_train.py:646 step:28K smpl:4M ep:81K epch:16.41 loss:0.020 grdn:0.149 lr:5.6e-05 updt_s:1.579 data_s:1.395 smp/s:43 mem_gb:52.83
+Training:  47%|████▋     | 28200/60000 [24:21:49<26:50:08,  3.04s/step]INFO 2026-07-18 16:05:32 ot_train.py:646 step:28K smpl:4M ep:81K epch:16.53 loss:0.020 grdn:0.152 lr:5.6e-05 updt_s:1.612 data_s:1.365 smp/s:43 mem_gb:52.83
+Training:  47%|████▋     | 28400/60000 [24:31:55<26:44:21,  3.05s/step]INFO 2026-07-18 16:15:38 ot_train.py:646 step:28K smpl:4M ep:82K epch:16.65 loss:0.019 grdn:0.157 lr:5.5e-05 updt_s:1.631 data_s:1.393 smp/s:42 mem_gb:52.75
+Training:  48%|████▊     | 28600/60000 [24:41:58<26:27:42,  3.03s/step]INFO 2026-07-18 16:25:41 ot_train.py:646 step:29K smpl:4M ep:83K epch:16.76 loss:0.019 grdn:0.158 lr:5.5e-05 updt_s:1.618 data_s:1.398 smp/s:42 mem_gb:52.83
+Training:  48%|████▊     | 28800/60000 [24:52:11<25:52:25,  2.99s/step]INFO 2026-07-18 16:35:54 ot_train.py:646 step:29K smpl:4M ep:83K epch:16.88 loss:0.019 grdn:0.151 lr:5.4e-05 updt_s:1.617 data_s:1.445 smp/s:42 mem_gb:52.83
+Training:  48%|████▊     | 29000/60000 [25:02:08<25:15:46,  2.93s/step]INFO 2026-07-18 16:45:51 ot_train.py:646 step:29K smpl:4M ep:84K epch:17.00 loss:0.019 grdn:0.152 lr:5.4e-05 updt_s:1.607 data_s:1.376 smp/s:43 mem_gb:52.83
+Training:  49%|████▊     | 29200/60000 [25:12:03<26:51:59,  3.14s/step]INFO 2026-07-18 16:55:46 ot_train.py:646 step:29K smpl:4M ep:84K epch:17.12 loss:0.019 grdn:0.155 lr:5.3e-05 updt_s:1.675 data_s:1.297 smp/s:43 mem_gb:52.83
+Training:  49%|████▉     | 29400/60000 [25:22:19<25:46:27,  3.03s/step]INFO 2026-07-18 17:06:02 ot_train.py:646 step:29K smpl:4M ep:85K epch:17.23 loss:0.018 grdn:0.152 lr:5.3e-05 updt_s:1.723 data_s:1.353 smp/s:42 mem_gb:52.75
+Training:  49%|████▉     | 29600/60000 [25:32:35<23:19:08,  2.76s/step]INFO 2026-07-18 17:16:18 ot_train.py:646 step:30K smpl:4M ep:86K epch:17.35 loss:0.018 grdn:0.146 lr:5.2e-05 updt_s:1.609 data_s:1.466 smp/s:42 mem_gb:52.83
+Training:  50%|████▉     | 29800/60000 [25:42:51<26:24:09,  3.15s/step]INFO 2026-07-18 17:26:35 ot_train.py:646 step:30K smpl:4M ep:86K epch:17.47 loss:0.019 grdn:0.153 lr:5.1e-05 updt_s:1.611 data_s:1.471 smp/s:42 mem_gb:52.83
+Training:  50%|█████     | 30000/60000 [25:52:51<24:14:09,  2.91s/step]INFO 2026-07-18 17:36:34 ot_train.py:646 step:30K smpl:4M ep:87K epch:17.59 loss:0.019 grdn:0.164 lr:5.1e-05 updt_s:1.621 data_s:1.375 smp/s:43 mem_gb:52.83
+Training:  50%|█████     | 30200/60000 [26:03:00<24:21:46,  2.94s/step]INFO 2026-07-18 17:46:43 ot_train.py:646 step:30K smpl:4M ep:87K epch:17.70 loss:0.018 grdn:0.142 lr:5.0e-05 updt_s:1.599 data_s:1.440 smp/s:42 mem_gb:52.83
+Training:  51%|█████     | 30400/60000 [26:13:11<25:41:02,  3.12s/step]INFO 2026-07-18 17:56:54 ot_train.py:646 step:30K smpl:4M ep:88K epch:17.82 loss:0.018 grdn:0.148 lr:5.0e-05 updt_s:1.634 data_s:1.421 smp/s:42 mem_gb:52.83
+Training:  51%|█████     | 30600/60000 [26:23:30<24:00:34,  2.94s/step]INFO 2026-07-18 18:07:13 ot_train.py:646 step:31K smpl:4M ep:88K epch:17.94 loss:0.017 grdn:0.145 lr:4.9e-05 updt_s:1.635 data_s:1.457 smp/s:41 mem_gb:52.75
+Training:  51%|█████▏    | 30800/60000 [26:33:35<25:19:19,  3.12s/step]INFO 2026-07-18 18:17:18 ot_train.py:646 step:31K smpl:4M ep:89K epch:18.05 loss:0.017 grdn:0.154 lr:4.9e-05 updt_s:1.589 data_s:1.432 smp/s:42 mem_gb:52.83
+Training:  52%|█████▏    | 31000/60000 [26:43:50<28:22:10,  3.52s/step]INFO 2026-07-18 18:27:33 ot_train.py:646 step:31K smpl:4M ep:90K epch:18.17 loss:0.018 grdn:0.149 lr:4.8e-05 updt_s:1.587 data_s:1.485 smp/s:42 mem_gb:52.83
+Training:  52%|█████▏    | 31200/60000 [26:53:56<24:05:08,  3.01s/step]INFO 2026-07-18 18:37:39 ot_train.py:646 step:31K smpl:4M ep:90K epch:18.29 loss:0.017 grdn:0.138 lr:4.8e-05 updt_s:1.548 data_s:1.477 smp/s:42 mem_gb:52.83
+Training:  52%|█████▏    | 31400/60000 [27:04:10<24:54:49,  3.14s/step]INFO 2026-07-18 18:47:53 ot_train.py:646 step:31K smpl:4M ep:91K epch:18.41 loss:0.017 grdn:0.145 lr:4.7e-05 updt_s:1.620 data_s:1.449 smp/s:42 mem_gb:52.83
+Training:  53%|█████▎    | 31600/60000 [27:14:17<23:10:14,  2.94s/step]INFO 2026-07-18 18:58:00 ot_train.py:646 step:32K smpl:4M ep:91K epch:18.52 loss:0.018 grdn:0.148 lr:4.7e-05 updt_s:1.606 data_s:1.427 smp/s:42 mem_gb:52.83
+Training:  53%|█████▎    | 31800/60000 [27:24:10<23:42:56,  3.03s/step]INFO 2026-07-18 19:07:53 ot_train.py:646 step:32K smpl:4M ep:92K epch:18.64 loss:0.016 grdn:0.143 lr:4.6e-05 updt_s:1.608 data_s:1.350 smp/s:43 mem_gb:52.75
+Training:  53%|█████▎    | 32000/60000 [27:34:03<23:45:35,  3.05s/step]INFO 2026-07-18 19:17:46 ot_train.py:646 step:32K smpl:4M ep:92K epch:18.76 loss:0.016 grdn:0.141 lr:4.6e-05 updt_s:1.597 data_s:1.366 smp/s:43 mem_gb:52.83
+Training:  54%|█████▎    | 32200/60000 [27:44:02<23:35:07,  3.05s/step]INFO 2026-07-18 19:27:45 ot_train.py:646 step:32K smpl:4M ep:93K epch:18.87 loss:0.016 grdn:0.147 lr:4.5e-05 updt_s:1.600 data_s:1.395 smp/s:43 mem_gb:52.83
+Training:  54%|█████▍    | 32400/60000 [27:54:08<23:37:23,  3.08s/step]INFO 2026-07-18 19:37:51 ot_train.py:646 step:32K smpl:4M ep:94K epch:18.99 loss:0.016 grdn:0.133 lr:4.5e-05 updt_s:1.619 data_s:1.405 smp/s:42 mem_gb:52.83
+Training:  54%|█████▍    | 32600/60000 [28:04:05<22:41:55,  2.98s/step]INFO 2026-07-18 19:47:48 ot_train.py:646 step:33K smpl:4M ep:94K epch:19.11 loss:0.016 grdn:0.133 lr:4.4e-05 updt_s:1.639 data_s:1.346 smp/s:43 mem_gb:52.83
+Training:  55%|█████▍    | 32800/60000 [28:14:05<22:52:01,  3.03s/step]INFO 2026-07-18 19:57:48 ot_train.py:646 step:33K smpl:4M ep:95K epch:19.23 loss:0.016 grdn:0.147 lr:4.4e-05 updt_s:1.628 data_s:1.369 smp/s:43 mem_gb:52.75
+Training:  55%|█████▌    | 33000/60000 [28:23:57<21:09:09,  2.82s/step]INFO 2026-07-18 20:07:40 ot_train.py:646 step:33K smpl:4M ep:95K epch:19.34 loss:0.015 grdn:0.136 lr:4.3e-05 updt_s:1.544 data_s:1.411 smp/s:43 mem_gb:52.83
+Training:  55%|█████▌    | 33200/60000 [28:33:55<23:02:40,  3.10s/step]INFO 2026-07-18 20:17:38 ot_train.py:646 step:33K smpl:4M ep:96K epch:19.46 loss:0.015 grdn:0.141 lr:4.3e-05 updt_s:1.580 data_s:1.409 smp/s:43 mem_gb:52.83
+Training:  56%|█████▌    | 33400/60000 [28:44:00<25:25:19,  3.44s/step]INFO 2026-07-18 20:27:43 ot_train.py:646 step:33K smpl:4M ep:97K epch:19.58 loss:0.015 grdn:0.128 lr:4.2e-05 updt_s:1.621 data_s:1.401 smp/s:42 mem_gb:52.83
+Training:  56%|█████▌    | 33600/60000 [28:54:14<24:50:47,  3.39s/step]INFO 2026-07-18 20:37:57 ot_train.py:646 step:34K smpl:4M ep:97K epch:19.70 loss:0.015 grdn:0.139 lr:4.1e-05 updt_s:1.686 data_s:1.382 smp/s:42 mem_gb:52.83
+Training:  56%|█████▋    | 33800/60000 [29:04:24<25:55:11,  3.56s/step]INFO 2026-07-18 20:48:07 ot_train.py:646 step:34K smpl:4M ep:98K epch:19.81 loss:0.015 grdn:0.135 lr:4.1e-05 updt_s:1.626 data_s:1.419 smp/s:42 mem_gb:52.83
+Training:  57%|█████▋    | 34000/60000 [29:14:33<22:25:18,  3.10s/step]INFO 2026-07-18 20:58:16 ot_train.py:646 step:34K smpl:4M ep:98K epch:19.93 loss:0.015 grdn:0.138 lr:4.0e-05 updt_s:1.605 data_s:1.436 smp/s:42 mem_gb:52.75
+Training:  57%|█████▋    | 34200/60000 [29:24:36<21:49:15,  3.04s/step]INFO 2026-07-18 21:08:19 ot_train.py:646 step:34K smpl:4M ep:99K epch:20.05 loss:0.015 grdn:0.136 lr:4.0e-05 updt_s:1.622 data_s:1.393 smp/s:42 mem_gb:52.83
+Training:  57%|█████▋    | 34400/60000 [29:34:39<23:29:28,  3.30s/step]INFO 2026-07-18 21:18:22 ot_train.py:646 step:34K smpl:4M ep:99K epch:20.16 loss:0.015 grdn:0.131 lr:3.9e-05 updt_s:1.565 data_s:1.446 smp/s:43 mem_gb:52.83
+Training:  58%|█████▊    | 34600/60000 [29:45:07<23:21:23,  3.31s/step]INFO 2026-07-18 21:28:50 ot_train.py:646 step:35K smpl:4M ep:100K epch:20.28 loss:0.015 grdn:0.130 lr:3.9e-05 updt_s:1.584 data_s:1.552 smp/s:41 mem_gb:52.83
+Training:  58%|█████▊    | 34800/60000 [29:55:51<24:54:19,  3.56s/step]INFO 2026-07-18 21:39:34 ot_train.py:646 step:35K smpl:4M ep:101K epch:20.40 loss:0.015 grdn:0.129 lr:3.8e-05 updt_s:1.659 data_s:1.562 smp/s:40 mem_gb:52.83
+Training:  58%|█████▊    | 35000/60000 [30:06:11<25:18:51,  3.65s/step]INFO 2026-07-18 21:49:54 ot_train.py:646 step:35K smpl:4M ep:101K epch:20.52 loss:0.014 grdn:0.124 lr:3.8e-05 updt_s:1.558 data_s:1.536 smp/s:41 mem_gb:52.75
+Training:  59%|█████▊    | 35200/60000 [30:16:36<21:50:20,  3.17s/step]INFO 2026-07-18 22:00:19 ot_train.py:646 step:35K smpl:5M ep:102K epch:20.63 loss:0.014 grdn:0.120 lr:3.7e-05 updt_s:1.563 data_s:1.558 smp/s:41 mem_gb:52.83
+Training:  59%|█████▉    | 35400/60000 [30:27:03<20:29:20,  3.00s/step]INFO 2026-07-18 22:10:46 ot_train.py:646 step:35K smpl:5M ep:102K epch:20.75 loss:0.014 grdn:0.133 lr:3.7e-05 updt_s:1.583 data_s:1.554 smp/s:41 mem_gb:52.83
+Training:  59%|█████▉    | 35600/60000 [30:37:32<21:07:26,  3.12s/step]INFO 2026-07-18 22:21:15 ot_train.py:646 step:36K smpl:5M ep:103K epch:20.87 loss:0.014 grdn:0.127 lr:3.6e-05 updt_s:1.568 data_s:1.573 smp/s:41 mem_gb:52.83
+Training:  60%|█████▉    | 35800/60000 [30:48:08<20:54:56,  3.11s/step]INFO 2026-07-18 22:31:51 ot_train.py:646 step:36K smpl:5M ep:103K epch:20.98 loss:0.014 grdn:0.131 lr:3.6e-05 updt_s:1.606 data_s:1.570 smp/s:40 mem_gb:52.83
+Training:  60%|██████    | 36000/60000 [30:58:34<20:41:29,  3.10s/step]INFO 2026-07-18 22:42:17 ot_train.py:646 step:36K smpl:5M ep:104K epch:21.10 loss:0.014 grdn:0.135 lr:3.5e-05 updt_s:1.583 data_s:1.545 smp/s:41 mem_gb:52.83
+Training:  60%|██████    | 36200/60000 [31:09:06<20:54:44,  3.16s/step]INFO 2026-07-18 22:52:49 ot_train.py:646 step:36K smpl:5M ep:105K epch:21.22 loss:0.014 grdn:0.138 lr:3.5e-05 updt_s:1.563 data_s:1.595 smp/s:41 mem_gb:52.75
+Training:  61%|██████    | 36400/60000 [31:19:47<21:11:01,  3.23s/step]INFO 2026-07-18 23:03:30 ot_train.py:646 step:36K smpl:5M ep:105K epch:21.34 loss:0.013 grdn:0.127 lr:3.4e-05 updt_s:1.614 data_s:1.590 smp/s:40 mem_gb:52.83
+Training:  61%|██████    | 36600/60000 [31:30:32<21:30:17,  3.31s/step]INFO 2026-07-18 23:14:15 ot_train.py:646 step:37K smpl:5M ep:106K epch:21.45 loss:0.013 grdn:0.123 lr:3.4e-05 updt_s:1.651 data_s:1.568 smp/s:40 mem_gb:52.83
+Training:  61%|██████▏   | 36800/60000 [31:41:32<20:07:34,  3.12s/step]INFO 2026-07-18 23:25:15 ot_train.py:646 step:37K smpl:5M ep:106K epch:21.57 loss:0.013 grdn:0.123 lr:3.3e-05 updt_s:1.712 data_s:1.587 smp/s:39 mem_gb:52.83
+Training:  62%|██████▏   | 37000/60000 [31:52:02<19:49:48,  3.10s/step]INFO 2026-07-18 23:35:45 ot_train.py:646 step:37K smpl:5M ep:107K epch:21.69 loss:0.013 grdn:0.121 lr:3.3e-05 updt_s:1.546 data_s:1.601 smp/s:41 mem_gb:52.83
+Training:  62%|██████▏   | 37200/60000 [32:02:26<19:43:20,  3.11s/step]INFO 2026-07-18 23:46:10 ot_train.py:646 step:37K smpl:5M ep:108K epch:21.81 loss:0.013 grdn:0.124 lr:3.2e-05 updt_s:1.535 data_s:1.583 smp/s:41 mem_gb:52.83
+Training:  62%|██████▏   | 37400/60000 [32:12:58<21:46:14,  3.47s/step]INFO 2026-07-18 23:56:41 ot_train.py:646 step:37K smpl:5M ep:108K epch:21.92 loss:0.013 grdn:0.118 lr:3.2e-05 updt_s:1.565 data_s:1.590 smp/s:41 mem_gb:52.75
+Training:  63%|██████▎   | 37600/60000 [32:23:16<17:56:33,  2.88s/step]INFO 2026-07-19 00:06:59 ot_train.py:646 step:38K smpl:5M ep:109K epch:22.04 loss:0.012 grdn:0.118 lr:3.1e-05 updt_s:1.564 data_s:1.524 smp/s:41 mem_gb:52.83
+Training:  63%|██████▎   | 37800/60000 [32:33:34<17:27:34,  2.83s/step]INFO 2026-07-19 00:17:17 ot_train.py:646 step:38K smpl:5M ep:109K epch:22.16 loss:0.012 grdn:0.114 lr:3.1e-05 updt_s:1.600 data_s:1.484 smp/s:42 mem_gb:52.83
+Training:  63%|██████▎   | 38000/60000 [32:43:41<19:02:31,  3.12s/step]INFO 2026-07-19 00:27:24 ot_train.py:646 step:38K smpl:5M ep:110K epch:22.27 loss:0.013 grdn:0.121 lr:3.0e-05 updt_s:1.573 data_s:1.462 smp/s:42 mem_gb:52.83
+Training:  64%|██████▎   | 38200/60000 [32:54:01<22:56:12,  3.79s/step]INFO 2026-07-19 00:37:44 ot_train.py:646 step:38K smpl:5M ep:110K epch:22.39 loss:0.013 grdn:0.122 lr:3.0e-05 updt_s:1.591 data_s:1.506 smp/s:41 mem_gb:52.83
+Training:  64%|██████▍   | 38400/60000 [33:04:09<18:11:07,  3.03s/step]INFO 2026-07-19 00:47:52 ot_train.py:646 step:38K smpl:5M ep:111K epch:22.51 loss:0.012 grdn:0.121 lr:2.9e-05 updt_s:1.585 data_s:1.448 smp/s:42 mem_gb:52.75
+Training:  64%|██████▍   | 38600/60000 [33:14:07<16:50:01,  2.83s/step]INFO 2026-07-19 00:57:50 ot_train.py:646 step:39K smpl:5M ep:112K epch:22.63 loss:0.012 grdn:0.119 lr:2.9e-05 updt_s:1.573 data_s:1.414 smp/s:43 mem_gb:52.83
+Training:  65%|██████▍   | 38800/60000 [33:24:18<17:40:57,  3.00s/step]INFO 2026-07-19 01:08:01 ot_train.py:646 step:39K smpl:5M ep:112K epch:22.74 loss:0.012 grdn:0.114 lr:2.8e-05 updt_s:1.571 data_s:1.482 smp/s:42 mem_gb:52.83
+Training:  65%|██████▌   | 39000/60000 [33:34:32<15:24:10,  2.64s/step]INFO 2026-07-19 01:18:15 ot_train.py:646 step:39K smpl:5M ep:113K epch:22.86 loss:0.012 grdn:0.114 lr:2.8e-05 updt_s:1.636 data_s:1.433 smp/s:42 mem_gb:52.83
+Training:  65%|██████▌   | 39200/60000 [33:44:36<17:45:44,  3.07s/step]INFO 2026-07-19 01:28:19 ot_train.py:646 step:39K smpl:5M ep:113K epch:22.98 loss:0.012 grdn:0.121 lr:2.7e-05 updt_s:1.598 data_s:1.417 smp/s:42 mem_gb:52.83
+Training:  66%|██████▌   | 39400/60000 [33:54:50<18:46:50,  3.28s/step]INFO 2026-07-19 01:38:33 ot_train.py:646 step:39K smpl:5M ep:114K epch:23.10 loss:0.011 grdn:0.113 lr:2.7e-05 updt_s:1.633 data_s:1.434 smp/s:42 mem_gb:52.83
+Training:  66%|██████▌   | 39600/60000 [34:05:06<17:13:24,  3.04s/step]INFO 2026-07-19 01:48:49 ot_train.py:646 step:40K smpl:5M ep:114K epch:23.21 loss:0.012 grdn:0.118 lr:2.7e-05 updt_s:1.571 data_s:1.507 smp/s:42 mem_gb:52.75
+Training:  66%|██████▋   | 39800/60000 [34:15:06<17:03:02,  3.04s/step]INFO 2026-07-19 01:58:49 ot_train.py:646 step:40K smpl:5M ep:115K epch:23.33 loss:0.011 grdn:0.115 lr:2.6e-05 updt_s:1.559 data_s:1.436 smp/s:43 mem_gb:52.83
+Training:  67%|██████▋   | 40000/60000 [34:25:03<16:01:55,  2.89s/step]INFO 2026-07-19 02:08:46 ot_train.py:646 step:40K smpl:5M ep:116K epch:23.45 loss:0.012 grdn:0.112 lr:2.6e-05 updt_s:1.531 data_s:1.448 smp/s:43 mem_gb:52.83
+Training:  67%|██████▋   | 40197/60000 [34:34:51<15:45:14,  2.86s/step]

Abs_6D + DeltaEE/Baseline.interrupted_reboot_20260719T051924/wandb/run-20260717_154331-qi6m7u1i/files/requirements.txt

@@ -0,0 +1,117 @@
+pyarrow==25.0.0
+importlib_metadata==9.0.0
+wheel==0.47.0
+cloudpickle==3.1.2
+nvidia-cusolver-cu12==11.7.3.90
+multiprocess==0.70.19
+torch==2.11.0+cu128
+nvidia-nvjitlink-cu12==12.8.93
+fsspec==2026.2.0
+pytz==2026.2
+hf-xet==1.5.1
+tokenizers==0.22.2
+opencv-python-headless==4.13.0.92
+cuda-pathfinder==1.2.2
+frozenlist==1.8.0
+smmap==5.0.3
+httpcore==1.0.9
+nvidia-nvshmem-cu12==3.4.5
+nvidia-cuda-nvrtc-cu12==12.8.93
+annotated-types==0.7.0
+platformdirs==4.10.0
+click==8.4.2
+pyyaml-include==1.4.1
+MarkupSafe==3.0.3
+Jinja2==3.1.6
+urllib3==2.7.0
+nvidia-curand-cu12==10.3.9.90
+certifi==2026.6.17
+nvidia-cuda-runtime-cu12==12.8.90
+torchcodec==0.11.1
+toml==0.10.2
+av==15.1.0
+nvidia-cufft-cu12==11.3.3.83
+attrs==26.1.0
+lerobot==0.6.0
+PyYAML==6.0.3
+safetensors==0.8.0
+diffusers==0.35.2
+requests==2.34.2
+nvidia-nvtx-cu12==12.8.90
+charset-normalizer==3.4.9
+networkx==3.6.1
+cmake==4.1.3
+pydantic_core==2.46.4
+timm==1.0.28
+dm-tree==0.1.10
+nvidia-cufile-cu12==1.13.1.3
+termcolor==3.3.0
+multidict==6.7.1
+filelock==3.29.0
+tzdata==2026.3
+aiohappyeyeballs==2.7.1
+httpx==0.28.1
+nvidia-nccl-cu12==2.28.9
+Pygments==2.20.0
+einops==0.8.2
+nvidia-cusparse-cu12==12.5.8.93
+torchvision==0.26.0+cu128
+dill==0.4.1
+GitPython==3.1.51
+nvidia-cuda-cupti-cu12==12.8.90
+decord==0.6.0
+sympy==1.14.0
+cuda-toolkit==12.8.1
+python-dateutil==2.9.0.post0
+shellingham==1.5.4
+propcache==0.5.2
+Farama-Notifications==0.0.6
+pillow==12.2.0
+cuda-bindings==12.9.4
+huggingface_hub==1.23.0
+pandas==2.3.3
+typing-inspect==0.9.0
+packaging==25.0
+rich==15.0.0
+nvidia-cusparselt-cu12==0.7.1
+setuptools==78.1.0
+wrapt==2.2.2
+mypy_extensions==1.1.0
+tqdm==4.68.4
+draccus==0.10.0
+aiosignal==1.4.0
+mpmath==1.3.0
+numpy==2.2.6
+typing_extensions==4.15.0
+psutil==7.2.2
+mdurl==0.1.2
+nvidia-cublas-cu12==12.8.4.1
+absl-py==2.5.0
+gitdb==4.0.12
+transformers==5.5.4
+accelerate==1.14.0
+pydantic==2.13.4
+pip==26.1.2
+markdown-it-py==4.2.0
+typer==0.26.8
+triton==3.6.0
+wandb==0.27.2
+regex==2026.7.10
+yarl==1.24.2
+jsonlines==4.0.0
+mergedeep==1.3.4
+aiohttp==3.14.1
+datasets==4.8.5
+gymnasium==1.3.0
+peft==0.19.1
+zipp==4.1.0
+idna==3.18
+nvidia-cudnn-cu12==9.19.0.56
+h11==0.16.0
+sentry-sdk==2.65.0
+annotated-doc==0.0.4
+protobuf==7.35.1
+typing-inspection==0.4.2
+anyio==4.14.2
+xxhash==3.8.1
+six==1.17.0

Abs_6D + DeltaEE/Baseline/wandb/debug.log

@@ -0,0 +1,45 @@
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_setup.py:_flush():81] Configure stats pid to 227242
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/Baseline/wandb/run-20260719_051936-811mctc7/logs/debug.log
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/Baseline/wandb/run-20260719_051936-811mctc7/logs/debug-internal.log
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_init.py:init():766] calling init triggers
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/Baseline', 'job_name': 'INSIGHT_RD_baseline', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 500, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_init.py:init():814] starting backend
+2026-07-19 05:19:36,575 INFO    MainThread:227242 [wandb_init.py:init():829] sending inform_init request
+2026-07-19 05:19:37,069 INFO    MainThread:227242 [wandb_init.py:init():834] backend started and connected
+2026-07-19 05:19:37,073 INFO    MainThread:227242 [wandb_init.py:init():904] updated telemetry
+2026-07-19 05:19:37,079 INFO    MainThread:227242 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-19 05:19:37,488 INFO    MainThread:227242 [wandb_init.py:init():972] starting run threads in backend
+2026-07-19 05:19:37,540 INFO    MainThread:227242 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-19 05:19:37,540 INFO    MainThread:227242 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-19 05:19:37,540 INFO    MainThread:227242 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-19 05:19:37,540 INFO    MainThread:227242 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-19 05:19:37,543 INFO    MainThread:227242 [wandb_init.py:init():1010] run started, returning control to user process
+2026-07-19 15:29:54,199 INFO    wandb-AsyncioManager-main:227242 [service_client.py:_forward_responses():122] Reached EOF.
+2026-07-19 15:29:54,200 INFO    wandb-AsyncioManager-main:227242 [mailbox.py:close():154] Closing mailbox, abandoning 2 handles.
+2026-07-19 15:29:54,200 ERROR   wandb-AsyncioManager-main:227242 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback.
+Traceback (most recent call last):
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions
+    await fn()
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop
+    await asyncio_compat.race(
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 278, in race
+    async with open_task_group(race=True) as tg:
+               ^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/contextlib.py", line 217, in __aexit__
+    await anext(self.gen)
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 240, in open_task_group
+    await task_group._wait_all(race=race, timeout=exit_timeout)
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 180, in _wait_all
+    raise exc
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all
+    result = await handle.wait_async(timeout=None)
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async
+    response = await self._handle.wait_async(timeout=timeout)
+               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async
+    raise HandleAbandonedError()
+wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError

Abs_6D + DeltaEE/Baseline/wandb/run-20260719_051936-811mctc7/logs/debug.log

@@ -0,0 +1,45 @@
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_setup.py:_flush():81] Configure stats pid to 227242
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/Baseline/wandb/run-20260719_051936-811mctc7/logs/debug.log
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/Baseline/wandb/run-20260719_051936-811mctc7/logs/debug-internal.log
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_init.py:init():766] calling init triggers
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/Baseline', 'job_name': 'INSIGHT_RD_baseline', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 500, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-19 05:19:36,354 INFO    MainThread:227242 [wandb_init.py:init():814] starting backend
+2026-07-19 05:19:36,575 INFO    MainThread:227242 [wandb_init.py:init():829] sending inform_init request
+2026-07-19 05:19:37,069 INFO    MainThread:227242 [wandb_init.py:init():834] backend started and connected
+2026-07-19 05:19:37,073 INFO    MainThread:227242 [wandb_init.py:init():904] updated telemetry
+2026-07-19 05:19:37,079 INFO    MainThread:227242 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-19 05:19:37,488 INFO    MainThread:227242 [wandb_init.py:init():972] starting run threads in backend
+2026-07-19 05:19:37,540 INFO    MainThread:227242 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-19 05:19:37,540 INFO    MainThread:227242 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-19 05:19:37,540 INFO    MainThread:227242 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-19 05:19:37,540 INFO    MainThread:227242 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-19 05:19:37,543 INFO    MainThread:227242 [wandb_init.py:init():1010] run started, returning control to user process
+2026-07-19 15:29:54,199 INFO    wandb-AsyncioManager-main:227242 [service_client.py:_forward_responses():122] Reached EOF.
+2026-07-19 15:29:54,200 INFO    wandb-AsyncioManager-main:227242 [mailbox.py:close():154] Closing mailbox, abandoning 2 handles.
+2026-07-19 15:29:54,200 ERROR   wandb-AsyncioManager-main:227242 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback.
+Traceback (most recent call last):
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions
+    await fn()
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop
+    await asyncio_compat.race(
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 278, in race
+    async with open_task_group(race=True) as tg:
+               ^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/contextlib.py", line 217, in __aexit__
+    await anext(self.gen)
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 240, in open_task_group
+    await task_group._wait_all(race=race, timeout=exit_timeout)
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 180, in _wait_all
+    raise exc
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all
+    result = await handle.wait_async(timeout=None)
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async
+    response = await self._handle.wait_async(timeout=timeout)
+               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async
+    raise HandleAbandonedError()
+wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError

Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K.interrupted_reboot_20260719T051924/wandb/run-20260717_154330-ro94ou5y/files/wandb-metadata.json

@@ -0,0 +1,92 @@
+{
+  "os": "Linux-6.8.0-107-generic-x86_64-with-glibc2.39",
+  "python": "CPython 3.12.13",
+  "startedAt": "2026-07-17T06:43:30.634832Z",
+  "args": [
+    "--config_path=/tmp/insight_lerobot_cfg_1zujjhv4/LAPstyle_RKD_linear_decay_10K.yaml",
+    "--batch_size=128"
+  ],
+  "program": "/home/ext_minje/clvla/benchmarks/INSIGHT/my_scripts/lerobot_train_yaml.py",
+  "codePath": "benchmarks/INSIGHT/my_scripts/lerobot_train_yaml.py",
+  "codePathLocal": "my_scripts/lerobot_train_yaml.py",
+  "git": {
+    "remote": "git@github.com:paragon7060/clvla.git",
+    "commit": "a970d9f98f189faccce13cf2fb61ccf5d55921f3"
+  },
+  "root": "/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K",
+  "host": "DGX-H200-01",
+  "executable": "/home/ext_minje/miniforge3/envs/lerobot060_groot/bin/python",
+  "cpu_count": 112,
+  "cpu_count_logical": 224,
+  "gpu": "NVIDIA H200",
+  "gpu_count": 8,
+  "disk": {
+    "/": {
+      "total": "1888425144320",
+      "used": "1493418987520"
+    }
+  },
+  "memory": {
+    "total": "2164177248256"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-1e531a45-480e-3c1a-e9a0-2ba2cc27b0a3"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-5d3d5ae2-ffde-89d9-8818-dc19e11ad2fb"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-305c0dae-dcf5-d7bb-7dd3-5477d2194275"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-ed8b7920-8881-4e8f-3e88-be1bbecafa21"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-a91176db-55f2-b284-bbbf-56626596947d"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-922ff6d5-a948-205a-0426-dbee495fc3dd"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-1fde7c75-18fe-e3db-a4c1-85e67522f55f"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-a9b1c49b-d527-97d8-ce91-7e50bd02154c"
+    }
+  ],
+  "cudaVersion": "12.9",
+  "writerId": "2u0quob3djgdefor1a7sv2pinqcipnhx"
+}

Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K/wandb/debug.log

@@ -0,0 +1,45 @@
+2026-07-19 05:19:36,520 INFO    MainThread:227257 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-19 05:19:36,520 INFO    MainThread:227257 [wandb_setup.py:_flush():81] Configure stats pid to 227257
+2026-07-19 05:19:36,520 INFO    MainThread:227257 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-19 05:19:36,521 INFO    MainThread:227257 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K/wandb/run-20260719_051936-kdbkadzu/logs/debug.log
+2026-07-19 05:19:36,521 INFO    MainThread:227257 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K/wandb/run-20260719_051936-kdbkadzu/logs/debug-internal.log
+2026-07-19 05:19:36,521 INFO    MainThread:227257 [wandb_init.py:init():766] calling init triggers
+2026-07-19 05:19:36,521 INFO    MainThread:227257 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot_rkd', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_steps': 6000, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False, 'rkd_enabled': True, 'rkd_vae_checkpoint': '/home/ext_minje/groot_insight/TimewarpVAE_ckpt/multispace_gripper_dedup_batch_ablation_v1/b512/RD/K08_z16_g4_h256/checkpoint-step-20000-epoch-46.84', 'rkd_student_pooling': 'attention', 'rkd_student_attention_hidden_dim': 512, 'rkd_student_attention_output_dim': 512, 'rkd_fm_loss_weight': 1.0, 'rkd_loss_weight': 1.0, 'rkd_distance_weight': 1.0, 'rkd_angle_weight': 2.0}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K', 'job_name': 'INSIGHT_RD_RKD_TimewarpVAE_LAPstyle_10K', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 6000, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-19 05:19:36,521 INFO    MainThread:227257 [wandb_init.py:init():814] starting backend
+2026-07-19 05:19:36,736 INFO    MainThread:227257 [wandb_init.py:init():829] sending inform_init request
+2026-07-19 05:19:37,421 INFO    MainThread:227257 [wandb_init.py:init():834] backend started and connected
+2026-07-19 05:19:37,424 INFO    MainThread:227257 [wandb_init.py:init():904] updated telemetry
+2026-07-19 05:19:37,431 INFO    MainThread:227257 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-19 05:19:37,840 INFO    MainThread:227257 [wandb_init.py:init():972] starting run threads in backend
+2026-07-19 05:19:37,892 INFO    MainThread:227257 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-19 05:19:37,892 INFO    MainThread:227257 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-19 05:19:37,892 INFO    MainThread:227257 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-19 05:19:37,892 INFO    MainThread:227257 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-19 05:19:37,895 INFO    MainThread:227257 [wandb_init.py:init():1010] run started, returning control to user process
+2026-07-19 15:29:54,215 INFO    wandb-AsyncioManager-main:227257 [service_client.py:_forward_responses():122] Reached EOF.
+2026-07-19 15:29:54,215 INFO    wandb-AsyncioManager-main:227257 [mailbox.py:close():154] Closing mailbox, abandoning 2 handles.
+2026-07-19 15:29:54,215 ERROR   wandb-AsyncioManager-main:227257 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback.
+Traceback (most recent call last):
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions
+    await fn()
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop
+    await asyncio_compat.race(
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 278, in race
+    async with open_task_group(race=True) as tg:
+               ^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/contextlib.py", line 217, in __aexit__
+    await anext(self.gen)
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 240, in open_task_group
+    await task_group._wait_all(race=race, timeout=exit_timeout)
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 180, in _wait_all
+    raise exc
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all
+    result = await handle.wait_async(timeout=None)
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async
+    response = await self._handle.wait_async(timeout=timeout)
+               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async
+    raise HandleAbandonedError()
+wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError

Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K/wandb/run-20260719_051936-kdbkadzu/files/requirements.txt

@@ -0,0 +1,117 @@
+pyarrow==25.0.0
+importlib_metadata==9.0.0
+wheel==0.47.0
+cloudpickle==3.1.2
+nvidia-cusolver-cu12==11.7.3.90
+multiprocess==0.70.19
+torch==2.11.0+cu128
+nvidia-nvjitlink-cu12==12.8.93
+fsspec==2026.2.0
+pytz==2026.2
+hf-xet==1.5.1
+tokenizers==0.22.2
+opencv-python-headless==4.13.0.92
+cuda-pathfinder==1.2.2
+frozenlist==1.8.0
+smmap==5.0.3
+httpcore==1.0.9
+nvidia-nvshmem-cu12==3.4.5
+nvidia-cuda-nvrtc-cu12==12.8.93
+annotated-types==0.7.0
+platformdirs==4.10.0
+click==8.4.2
+pyyaml-include==1.4.1
+MarkupSafe==3.0.3
+Jinja2==3.1.6
+urllib3==2.7.0
+nvidia-curand-cu12==10.3.9.90
+certifi==2026.6.17
+nvidia-cuda-runtime-cu12==12.8.90
+torchcodec==0.11.1
+toml==0.10.2
+av==15.1.0
+nvidia-cufft-cu12==11.3.3.83
+attrs==26.1.0
+lerobot==0.6.0
+PyYAML==6.0.3
+safetensors==0.8.0
+diffusers==0.35.2
+requests==2.34.2
+nvidia-nvtx-cu12==12.8.90
+charset-normalizer==3.4.9
+networkx==3.6.1
+cmake==4.1.3
+pydantic_core==2.46.4
+timm==1.0.28
+dm-tree==0.1.10
+nvidia-cufile-cu12==1.13.1.3
+termcolor==3.3.0
+multidict==6.7.1
+filelock==3.29.0
+tzdata==2026.3
+aiohappyeyeballs==2.7.1
+httpx==0.28.1
+nvidia-nccl-cu12==2.28.9
+Pygments==2.20.0
+einops==0.8.2
+nvidia-cusparse-cu12==12.5.8.93
+torchvision==0.26.0+cu128
+dill==0.4.1
+GitPython==3.1.51
+nvidia-cuda-cupti-cu12==12.8.90
+decord==0.6.0
+sympy==1.14.0
+cuda-toolkit==12.8.1
+python-dateutil==2.9.0.post0
+shellingham==1.5.4
+propcache==0.5.2
+Farama-Notifications==0.0.6
+pillow==12.2.0
+cuda-bindings==12.9.4
+huggingface_hub==1.23.0
+pandas==2.3.3
+typing-inspect==0.9.0
+packaging==25.0
+rich==15.0.0
+nvidia-cusparselt-cu12==0.7.1
+setuptools==78.1.0
+wrapt==2.2.2
+mypy_extensions==1.1.0
+tqdm==4.68.4
+draccus==0.10.0
+aiosignal==1.4.0
+mpmath==1.3.0
+numpy==2.2.6
+typing_extensions==4.15.0
+psutil==7.2.2
+mdurl==0.1.2
+nvidia-cublas-cu12==12.8.4.1
+absl-py==2.5.0
+gitdb==4.0.12
+transformers==5.5.4
+accelerate==1.14.0
+pydantic==2.13.4
+pip==26.1.2
+markdown-it-py==4.2.0
+typer==0.26.8
+triton==3.6.0
+wandb==0.27.2
+regex==2026.7.10
+yarl==1.24.2
+jsonlines==4.0.0
+mergedeep==1.3.4
+aiohttp==3.14.1
+datasets==4.8.5
+gymnasium==1.3.0
+peft==0.19.1
+zipp==4.1.0
+idna==3.18
+nvidia-cudnn-cu12==9.19.0.56
+h11==0.16.0
+sentry-sdk==2.65.0
+annotated-doc==0.0.4
+protobuf==7.35.1
+typing-inspection==0.4.2
+anyio==4.14.2
+xxhash==3.8.1
+six==1.17.0

Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K/wandb/run-20260719_051936-kdbkadzu/files/wandb-metadata.json

@@ -0,0 +1,92 @@
+{
+  "os": "Linux-6.8.0-107-generic-x86_64-with-glibc2.39",
+  "python": "CPython 3.12.13",
+  "startedAt": "2026-07-18T20:19:36.520173Z",
+  "args": [
+    "--config_path=/tmp/insight_lerobot_cfg_6cmjxg89/LAPstyle_RKD_linear_decay_10K.yaml",
+    "--batch_size=128"
+  ],
+  "program": "/home/ext_minje/clvla/benchmarks/INSIGHT/my_scripts/lerobot_train_yaml.py",
+  "codePath": "benchmarks/INSIGHT/my_scripts/lerobot_train_yaml.py",
+  "codePathLocal": "my_scripts/lerobot_train_yaml.py",
+  "git": {
+    "remote": "git@github.com:paragon7060/clvla.git",
+    "commit": "a9df08f7da2d8eb1fa7f28708b4eb6f6dc81b1f9"
+  },
+  "root": "/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K",
+  "host": "DGX-H200-01",
+  "executable": "/home/ext_minje/miniforge3/envs/lerobot060_groot/bin/python",
+  "cpu_count": 112,
+  "cpu_count_logical": 224,
+  "gpu": "NVIDIA H200",
+  "gpu_count": 8,
+  "disk": {
+    "/": {
+      "total": "1888425144320",
+      "used": "1505621217280"
+    }
+  },
+  "memory": {
+    "total": "2164177309696"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-1e531a45-480e-3c1a-e9a0-2ba2cc27b0a3"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-5d3d5ae2-ffde-89d9-8818-dc19e11ad2fb"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-305c0dae-dcf5-d7bb-7dd3-5477d2194275"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-ed8b7920-8881-4e8f-3e88-be1bbecafa21"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-a91176db-55f2-b284-bbbf-56626596947d"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-922ff6d5-a948-205a-0426-dbee495fc3dd"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-1fde7c75-18fe-e3db-a4c1-85e67522f55f"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-a9b1c49b-d527-97d8-ce91-7e50bd02154c"
+    }
+  ],
+  "cudaVersion": "12.9",
+  "writerId": "350yg0ang5xgl2imkgjcpqcv5pbcsct6"
+}

Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K/wandb/run-20260719_051936-kdbkadzu/logs/debug.log

@@ -0,0 +1,45 @@
+2026-07-19 05:19:36,520 INFO    MainThread:227257 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-19 05:19:36,520 INFO    MainThread:227257 [wandb_setup.py:_flush():81] Configure stats pid to 227257
+2026-07-19 05:19:36,520 INFO    MainThread:227257 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-19 05:19:36,521 INFO    MainThread:227257 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K/wandb/run-20260719_051936-kdbkadzu/logs/debug.log
+2026-07-19 05:19:36,521 INFO    MainThread:227257 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K/wandb/run-20260719_051936-kdbkadzu/logs/debug-internal.log
+2026-07-19 05:19:36,521 INFO    MainThread:227257 [wandb_init.py:init():766] calling init triggers
+2026-07-19 05:19:36,521 INFO    MainThread:227257 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot_rkd', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_steps': 6000, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False, 'rkd_enabled': True, 'rkd_vae_checkpoint': '/home/ext_minje/groot_insight/TimewarpVAE_ckpt/multispace_gripper_dedup_batch_ablation_v1/b512/RD/K08_z16_g4_h256/checkpoint-step-20000-epoch-46.84', 'rkd_student_pooling': 'attention', 'rkd_student_attention_hidden_dim': 512, 'rkd_student_attention_output_dim': 512, 'rkd_fm_loss_weight': 1.0, 'rkd_loss_weight': 1.0, 'rkd_distance_weight': 1.0, 'rkd_angle_weight': 2.0}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_10K', 'job_name': 'INSIGHT_RD_RKD_TimewarpVAE_LAPstyle_10K', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 6000, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-19 05:19:36,521 INFO    MainThread:227257 [wandb_init.py:init():814] starting backend
+2026-07-19 05:19:36,736 INFO    MainThread:227257 [wandb_init.py:init():829] sending inform_init request
+2026-07-19 05:19:37,421 INFO    MainThread:227257 [wandb_init.py:init():834] backend started and connected
+2026-07-19 05:19:37,424 INFO    MainThread:227257 [wandb_init.py:init():904] updated telemetry
+2026-07-19 05:19:37,431 INFO    MainThread:227257 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-19 05:19:37,840 INFO    MainThread:227257 [wandb_init.py:init():972] starting run threads in backend
+2026-07-19 05:19:37,892 INFO    MainThread:227257 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-19 05:19:37,892 INFO    MainThread:227257 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-19 05:19:37,892 INFO    MainThread:227257 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-19 05:19:37,892 INFO    MainThread:227257 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-19 05:19:37,895 INFO    MainThread:227257 [wandb_init.py:init():1010] run started, returning control to user process
+2026-07-19 15:29:54,215 INFO    wandb-AsyncioManager-main:227257 [service_client.py:_forward_responses():122] Reached EOF.
+2026-07-19 15:29:54,215 INFO    wandb-AsyncioManager-main:227257 [mailbox.py:close():154] Closing mailbox, abandoning 2 handles.
+2026-07-19 15:29:54,215 ERROR   wandb-AsyncioManager-main:227257 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback.
+Traceback (most recent call last):
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions
+    await fn()
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop
+    await asyncio_compat.race(
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 278, in race
+    async with open_task_group(race=True) as tg:
+               ^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/contextlib.py", line 217, in __aexit__
+    await anext(self.gen)
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 240, in open_task_group
+    await task_group._wait_all(race=race, timeout=exit_timeout)
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 180, in _wait_all
+    raise exc
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all
+    result = await handle.wait_async(timeout=None)
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async
+    response = await self._handle.wait_async(timeout=timeout)
+               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async
+    raise HandleAbandonedError()
+wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError

Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K.interrupted_reboot_20260719T051924/wandb/debug.log

@@ -0,0 +1,19 @@
+2026-07-17 15:43:26,119 INFO    MainThread:499778 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-17 15:43:26,119 INFO    MainThread:499778 [wandb_setup.py:_flush():81] Configure stats pid to 499778
+2026-07-17 15:43:26,119 INFO    MainThread:499778 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-17 15:43:26,119 INFO    MainThread:499778 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K/wandb/run-20260717_154326-sz25gl12/logs/debug.log
+2026-07-17 15:43:26,120 INFO    MainThread:499778 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K/wandb/run-20260717_154326-sz25gl12/logs/debug-internal.log
+2026-07-17 15:43:26,120 INFO    MainThread:499778 [wandb_init.py:init():766] calling init triggers
+2026-07-17 15:43:26,120 INFO    MainThread:499778 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot_rkd', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_steps': 6000, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False, 'rkd_enabled': True, 'rkd_vae_checkpoint': '/home/ext_minje/groot_insight/TimewarpVAE_ckpt/multispace_gripper_dedup_batch_ablation_v1/b512/RD/K08_z16_g4_h256/checkpoint-step-20000-epoch-46.84', 'rkd_student_pooling': 'attention', 'rkd_student_attention_hidden_dim': 512, 'rkd_student_attention_output_dim': 512, 'rkd_fm_loss_weight': 1.0, 'rkd_loss_weight': 1.0, 'rkd_distance_weight': 1.0, 'rkd_angle_weight': 2.0}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K', 'job_name': 'INSIGHT_RD_RKD_TimewarpVAE_LAPstyle_6K', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 6000, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-17 15:43:26,120 INFO    MainThread:499778 [wandb_init.py:init():814] starting backend
+2026-07-17 15:43:26,346 INFO    MainThread:499778 [wandb_init.py:init():829] sending inform_init request
+2026-07-17 15:43:26,990 INFO    MainThread:499778 [wandb_init.py:init():834] backend started and connected
+2026-07-17 15:43:26,993 INFO    MainThread:499778 [wandb_init.py:init():904] updated telemetry
+2026-07-17 15:43:27,003 INFO    MainThread:499778 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-17 15:43:27,554 INFO    MainThread:499778 [wandb_init.py:init():972] starting run threads in backend
+2026-07-17 15:43:27,750 INFO    MainThread:499778 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-17 15:43:27,750 INFO    MainThread:499778 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-17 15:43:27,750 INFO    MainThread:499778 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-17 15:43:27,750 INFO    MainThread:499778 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-17 15:43:27,754 INFO    MainThread:499778 [wandb_init.py:init():1010] run started, returning control to user process

Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K.interrupted_reboot_20260719T051924/wandb/run-20260717_154326-sz25gl12/files/output.log

@@ -0,0 +1,218 @@
+INFO 2026-07-17 15:43:27 db_utils.py:121 Logs will be synced with wandb.
+INFO 2026-07-17 15:43:27 db_utils.py:122 Track this run --> https://wandb.ai/minje227_hyu-hanyang-university/lerobot/runs/sz25gl12
+INFO 2026-07-17 15:43:27 ot_train.py:298 Creating dataset
+INFO 2026-07-17 15:43:31 ot_train.py:332 Creating policy
+Fetching 27 files: 100%|██████████| 27/27 [00:00<00:00, 1175.90it/s]
+`torch_dtype` is deprecated! Use `dtype` instead!
+Loading weights: 100%|██████████| 1031/1031 [00:00<00:00, 1943.98it/s]
+INFO 2026-07-17 15:43:36 ot_train.py:405 Creating optimizer and scheduler
+INFO 2026-07-17 15:43:36 ot_train.py:439 Output dir: /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K
+INFO 2026-07-17 15:43:36 ot_train.py:446 cfg.steps=60000 (60K)
+INFO 2026-07-17 15:43:36 ot_train.py:447 dataset.num_frames=218367 (218K)
+INFO 2026-07-17 15:43:36 ot_train.py:448 dataset.num_episodes=4930
+INFO 2026-07-17 15:43:36 ot_train.py:451 Effective batch size: 128 x 1 = 128
+INFO 2026-07-17 15:43:36 ot_train.py:452 num_learnable_params=1622090881 (2B)
+INFO 2026-07-17 15:43:36 ot_train.py:453 num_total_params=3145712033 (3B)
+Training:   0%|          | 0/60000 [00:00<?, ?step/s]INFO 2026-07-17 15:43:36 ot_train.py:604 Start offline training on a fixed dataset, with effective batch size: 128
+Training:   0%|          | 200/60000 [17:46<57:11:40,  3.44s/step]INFO 2026-07-17 16:01:22 ot_train.py:649 step:200 smpl:26K ep:578 epch:0.12 loss:1.233 grdn:0.465 lr:1.7e-06 updt_s:1.774 data_s:3.552 smp/s:24 mem_gb:52.89
+Training:   1%|          | 400/60000 [29:13<54:37:54,  3.30s/step]INFO 2026-07-17 16:12:49 ot_train.py:649 step:400 smpl:51K ep:1K epch:0.23 loss:0.859 grdn:0.963 lr:5.0e-06 updt_s:1.780 data_s:1.649 smp/s:37 mem_gb:52.95
+Training:   1%|          | 600/60000 [40:39<61:05:01,  3.70s/step]INFO 2026-07-17 16:24:15 ot_train.py:649 step:600 smpl:77K ep:2K epch:0.35 loss:0.336 grdn:1.622 lr:8.3e-06 updt_s:1.787 data_s:1.636 smp/s:37 mem_gb:52.95
+Training:   1%|▏         | 800/60000 [51:55<53:43:10,  3.27s/step]INFO 2026-07-17 16:35:31 ot_train.py:649 step:800 smpl:102K ep:2K epch:0.47 loss:0.212 grdn:2.051 lr:1.2e-05 updt_s:1.774 data_s:1.601 smp/s:38 mem_gb:52.95
+Training:   2%|▏         | 1000/60000 [1:03:15<51:44:23,  3.16s/step]INFO 2026-07-17 16:46:51 ot_train.py:649 step:1K smpl:128K ep:3K epch:0.59 loss:0.173 grdn:2.122 lr:1.5e-05 updt_s:1.807 data_s:1.590 smp/s:38 mem_gb:52.95
+Training:   2%|▏         | 1200/60000 [1:14:32<61:06:37,  3.74s/step]INFO 2026-07-17 16:58:08 ot_train.py:649 step:1K smpl:154K ep:3K epch:0.70 loss:0.151 grdn:2.006 lr:1.8e-05 updt_s:1.813 data_s:1.567 smp/s:38 mem_gb:52.88
+Training:   2%|▏         | 1400/60000 [1:25:28<58:59:01,  3.62s/step]INFO 2026-07-17 17:09:05 ot_train.py:649 step:1K smpl:179K ep:4K epch:0.82 loss:0.136 grdn:1.855 lr:2.2e-05 updt_s:1.776 data_s:1.500 smp/s:39 mem_gb:52.98
+Training:   3%|▎         | 1600/60000 [1:36:15<51:24:54,  3.17s/step]INFO 2026-07-17 17:19:51 ot_train.py:649 step:2K smpl:205K ep:5K epch:0.94 loss:0.125 grdn:1.691 lr:2.5e-05 updt_s:1.773 data_s:1.454 smp/s:40 mem_gb:52.98
+Training:   3%|▎         | 1800/60000 [1:46:54<50:53:11,  3.15s/step]INFO 2026-07-17 17:30:30 ot_train.py:649 step:2K smpl:230K ep:5K epch:1.06 loss:0.118 grdn:1.530 lr:2.8e-05 updt_s:1.733 data_s:1.460 smp/s:40 mem_gb:52.98
+Training:   3%|▎         | 2000/60000 [1:57:30<46:27:57,  2.88s/step]INFO 2026-07-17 17:41:06 ot_train.py:649 step:2K smpl:256K ep:6K epch:1.17 loss:0.111 grdn:1.401 lr:3.2e-05 updt_s:1.780 data_s:1.396 smp/s:40 mem_gb:52.98
+Training:   4%|▎         | 2200/60000 [2:08:00<50:46:50,  3.16s/step]INFO 2026-07-17 17:51:36 ot_train.py:649 step:2K smpl:282K ep:6K epch:1.29 loss:0.105 grdn:1.245 lr:3.5e-05 updt_s:1.726 data_s:1.418 smp/s:41 mem_gb:52.98
+Training:   4%|▍         | 2400/60000 [2:18:23<50:59:59,  3.19s/step]INFO 2026-07-17 18:01:59 ot_train.py:649 step:2K smpl:307K ep:7K epch:1.41 loss:0.100 grdn:1.116 lr:3.8e-05 updt_s:1.703 data_s:1.406 smp/s:41 mem_gb:52.90
+Training:   4%|▍         | 2600/60000 [2:29:57<52:04:27,  3.27s/step]INFO 2026-07-17 18:13:33 ot_train.py:649 step:3K smpl:333K ep:8K epch:1.52 loss:0.096 grdn:1.001 lr:4.2e-05 updt_s:1.737 data_s:1.727 smp/s:37 mem_gb:52.98
+Training:   5%|▍         | 2800/60000 [2:40:46<48:19:29,  3.04s/step]INFO 2026-07-17 18:24:22 ot_train.py:649 step:3K smpl:358K ep:8K epch:1.64 loss:0.091 grdn:0.939 lr:4.5e-05 updt_s:1.772 data_s:1.470 smp/s:39 mem_gb:52.98
+Training:   5%|▌         | 3000/60000 [2:51:38<51:35:36,  3.26s/step]INFO 2026-07-17 18:35:14 ot_train.py:649 step:3K smpl:384K ep:9K epch:1.76 loss:0.087 grdn:0.832 lr:4.8e-05 updt_s:1.766 data_s:1.487 smp/s:39 mem_gb:52.98
+Training:   5%|▌         | 3200/60000 [3:02:51<51:44:14,  3.28s/step]INFO 2026-07-17 18:46:27 ot_train.py:649 step:3K smpl:410K ep:9K epch:1.88 loss:0.083 grdn:0.773 lr:5.2e-05 updt_s:1.703 data_s:1.657 smp/s:38 mem_gb:52.98
+Training:   6%|▌         | 3400/60000 [3:13:36<50:03:54,  3.18s/step]INFO 2026-07-17 18:57:12 ot_train.py:649 step:3K smpl:435K ep:10K epch:1.99 loss:0.081 grdn:0.742 lr:5.5e-05 updt_s:1.678 data_s:1.544 smp/s:40 mem_gb:52.90
+Training:   6%|▌         | 3600/60000 [3:24:36<52:33:54,  3.36s/step]INFO 2026-07-17 19:08:12 ot_train.py:649 step:4K smpl:461K ep:10K epch:2.11 loss:0.077 grdn:0.684 lr:5.8e-05 updt_s:1.674 data_s:1.620 smp/s:39 mem_gb:52.98
+Training:   6%|▋         | 3800/60000 [3:35:47<49:35:34,  3.18s/step]INFO 2026-07-17 19:19:23 ot_train.py:649 step:4K smpl:486K ep:11K epch:2.23 loss:0.074 grdn:0.639 lr:6.2e-05 updt_s:1.620 data_s:1.733 smp/s:38 mem_gb:52.98
+Training:   7%|▋         | 4000/60000 [3:47:37<58:10:28,  3.74s/step]INFO 2026-07-17 19:31:13 ot_train.py:649 step:4K smpl:512K ep:12K epch:2.34 loss:0.072 grdn:0.614 lr:6.5e-05 updt_s:1.678 data_s:1.865 smp/s:36 mem_gb:52.98
+Training:   7%|▋         | 4200/60000 [4:00:12<54:10:31,  3.50s/step]INFO 2026-07-17 19:43:48 ot_train.py:649 step:4K smpl:538K ep:12K epch:2.46 loss:0.070 grdn:0.567 lr:6.8e-05 updt_s:1.619 data_s:2.151 smp/s:34 mem_gb:52.98
+Training:   7%|▋         | 4400/60000 [4:12:06<53:47:33,  3.48s/step]INFO 2026-07-17 19:55:42 ot_train.py:649 step:4K smpl:563K ep:13K epch:2.58 loss:0.068 grdn:0.549 lr:7.2e-05 updt_s:1.680 data_s:1.886 smp/s:36 mem_gb:52.98
+Training:   8%|▊         | 4600/60000 [4:23:59<53:08:05,  3.45s/step]INFO 2026-07-17 20:07:35 ot_train.py:649 step:5K smpl:589K ep:13K epch:2.70 loss:0.067 grdn:0.518 lr:7.5e-05 updt_s:1.736 data_s:1.825 smp/s:36 mem_gb:52.90
+Training:   8%|▊         | 4800/60000 [4:35:32<47:32:41,  3.10s/step]INFO 2026-07-17 20:19:08 ot_train.py:649 step:5K smpl:614K ep:14K epch:2.81 loss:0.064 grdn:0.520 lr:7.8e-05 updt_s:1.801 data_s:1.659 smp/s:37 mem_gb:52.98
+Training:   8%|▊         | 5000/60000 [4:46:16<48:21:38,  3.17s/step]INFO 2026-07-17 20:29:52 ot_train.py:649 step:5K smpl:640K ep:14K epch:2.93 loss:0.063 grdn:0.483 lr:8.2e-05 updt_s:1.752 data_s:1.464 smp/s:40 mem_gb:52.98
+Training:   9%|▊         | 5200/60000 [4:57:13<50:14:42,  3.30s/step]INFO 2026-07-17 20:40:49 ot_train.py:649 step:5K smpl:666K ep:15K epch:3.05 loss:0.061 grdn:0.458 lr:8.5e-05 updt_s:1.756 data_s:1.525 smp/s:39 mem_gb:52.98
+Training:   9%|▉         | 5400/60000 [5:09:24<58:35:42,  3.86s/step]INFO 2026-07-17 20:53:00 ot_train.py:649 step:5K smpl:691K ep:16K epch:3.17 loss:0.059 grdn:0.434 lr:8.8e-05 updt_s:1.694 data_s:1.953 smp/s:35 mem_gb:52.98
+Training:   9%|▉         | 5600/60000 [5:20:27<46:25:15,  3.07s/step]INFO 2026-07-17 21:04:03 ot_train.py:649 step:6K smpl:717K ep:16K epch:3.28 loss:0.058 grdn:0.423 lr:9.2e-05 updt_s:1.640 data_s:1.671 smp/s:39 mem_gb:52.98
+Training:  10%|▉         | 5800/60000 [5:31:10<50:29:01,  3.35s/step]INFO 2026-07-17 21:14:46 ot_train.py:649 step:6K smpl:742K ep:17K epch:3.40 loss:0.057 grdn:0.414 lr:9.5e-05 updt_s:1.632 data_s:1.575 smp/s:40 mem_gb:52.90
+Training:  10%|█         | 6000/60000 [5:41:54<48:49:09,  3.25s/step]INFO 2026-07-17 21:25:30 ot_train.py:649 step:6K smpl:768K ep:17K epch:3.52 loss:0.057 grdn:0.406 lr:9.8e-05 updt_s:1.605 data_s:1.611 smp/s:40 mem_gb:52.98
+INFO 2026-07-17 21:25:30 ot_train.py:103 INSIGHT phase transition: index=1 name=fm_only range=[6000,60000) trainable_params=1620515968
+Training:  10%|█         | 6200/60000 [5:52:29<46:16:41,  3.10s/step]INFO 2026-07-17 21:36:05 ot_train.py:649 step:6K smpl:794K ep:18K epch:3.63 loss:0.056 grdn:0.392 lr:1.0e-04 updt_s:1.586 data_s:1.583 smp/s:40 mem_gb:52.85
+Training:  11%|█         | 6400/60000 [6:03:06<46:02:29,  3.09s/step]INFO 2026-07-17 21:46:42 ot_train.py:649 step:6K smpl:819K ep:18K epch:3.75 loss:0.054 grdn:0.365 lr:1.0e-04 updt_s:1.605 data_s:1.572 smp/s:40 mem_gb:52.85
+Training:  11%|█         | 6600/60000 [6:13:43<47:06:11,  3.18s/step]INFO 2026-07-17 21:57:19 ot_train.py:649 step:7K smpl:845K ep:19K epch:3.87 loss:0.054 grdn:0.357 lr:1.0e-04 updt_s:1.610 data_s:1.571 smp/s:40 mem_gb:52.85
+Training:  11%|█▏        | 6800/60000 [6:24:09<46:44:09,  3.16s/step]INFO 2026-07-17 22:07:45 ot_train.py:649 step:7K smpl:870K ep:20K epch:3.99 loss:0.053 grdn:0.349 lr:1.0e-04 updt_s:1.552 data_s:1.570 smp/s:41 mem_gb:52.77
+Training:  12%|█▏        | 7000/60000 [6:34:31<43:13:43,  2.94s/step]INFO 2026-07-17 22:18:07 ot_train.py:649 step:7K smpl:896K ep:20K epch:4.10 loss:0.052 grdn:0.340 lr:1.0e-04 updt_s:1.640 data_s:1.466 smp/s:41 mem_gb:52.85
+Training:  12%|█▏        | 7200/60000 [6:44:48<44:04:39,  3.01s/step]INFO 2026-07-17 22:28:25 ot_train.py:649 step:7K smpl:922K ep:21K epch:4.22 loss:0.051 grdn:0.331 lr:1.0e-04 updt_s:1.618 data_s:1.466 smp/s:42 mem_gb:52.85
+Training:  12%|█▏        | 7400/60000 [6:55:03<44:10:37,  3.02s/step]INFO 2026-07-17 22:38:40 ot_train.py:649 step:7K smpl:947K ep:21K epch:4.34 loss:0.052 grdn:0.348 lr:1.0e-04 updt_s:1.626 data_s:1.444 smp/s:42 mem_gb:52.85
+Training:  13%|█▎        | 7600/60000 [7:05:22<45:24:06,  3.12s/step]INFO 2026-07-17 22:48:58 ot_train.py:649 step:8K smpl:973K ep:22K epch:4.45 loss:0.052 grdn:0.342 lr:1.0e-04 updt_s:1.606 data_s:1.480 smp/s:41 mem_gb:52.85
+Training:  13%|█���        | 7800/60000 [7:15:48<45:55:41,  3.17s/step]INFO 2026-07-17 22:59:24 ot_train.py:649 step:8K smpl:998K ep:23K epch:4.57 loss:0.051 grdn:0.314 lr:1.0e-04 updt_s:1.608 data_s:1.517 smp/s:41 mem_gb:52.85
+Training:  13%|█▎        | 8000/60000 [7:26:09<51:11:15,  3.54s/step]INFO 2026-07-17 23:09:45 ot_train.py:649 step:8K smpl:1M ep:23K epch:4.69 loss:0.049 grdn:0.313 lr:1.0e-04 updt_s:1.568 data_s:1.530 smp/s:41 mem_gb:52.77
+Training:  14%|█▎        | 8200/60000 [7:36:30<45:59:57,  3.20s/step]INFO 2026-07-17 23:20:06 ot_train.py:649 step:8K smpl:1M ep:24K epch:4.81 loss:0.048 grdn:0.300 lr:1.0e-04 updt_s:1.586 data_s:1.515 smp/s:41 mem_gb:52.85
+Training:  14%|█▍        | 8400/60000 [7:46:48<44:33:07,  3.11s/step]INFO 2026-07-17 23:30:24 ot_train.py:649 step:8K smpl:1M ep:24K epch:4.92 loss:0.048 grdn:0.307 lr:1.0e-04 updt_s:1.581 data_s:1.501 smp/s:42 mem_gb:52.85
+Training:  14%|█▍        | 8600/60000 [7:57:14<44:40:43,  3.13s/step]INFO 2026-07-17 23:40:50 ot_train.py:649 step:9K smpl:1M ep:25K epch:5.04 loss:0.047 grdn:0.300 lr:9.9e-05 updt_s:1.645 data_s:1.479 smp/s:41 mem_gb:52.85
+Training:  15%|█▍        | 8800/60000 [8:07:35<43:40:53,  3.07s/step]INFO 2026-07-17 23:51:11 ot_train.py:649 step:9K smpl:1M ep:25K epch:5.16 loss:0.047 grdn:0.291 lr:9.9e-05 updt_s:1.594 data_s:1.506 smp/s:41 mem_gb:52.85
+Training:  15%|█▌        | 9000/60000 [8:17:59<44:14:47,  3.12s/step]INFO 2026-07-18 00:01:35 ot_train.py:649 step:9K smpl:1M ep:26K epch:5.28 loss:0.050 grdn:0.314 lr:9.9e-05 updt_s:1.553 data_s:1.562 smp/s:41 mem_gb:52.85
+Training:  15%|█▌        | 9200/60000 [8:28:26<44:07:30,  3.13s/step]INFO 2026-07-18 00:12:03 ot_train.py:649 step:9K smpl:1M ep:27K epch:5.39 loss:0.045 grdn:0.276 lr:9.9e-05 updt_s:1.596 data_s:1.536 smp/s:41 mem_gb:52.77
+Training:  16%|█▌        | 9400/60000 [8:38:51<42:34:53,  3.03s/step]INFO 2026-07-18 00:22:27 ot_train.py:649 step:9K smpl:1M ep:27K epch:5.51 loss:0.046 grdn:0.298 lr:9.9e-05 updt_s:1.617 data_s:1.499 smp/s:41 mem_gb:52.85
+Training:  16%|█▌        | 9600/60000 [8:49:15<43:35:40,  3.11s/step]INFO 2026-07-18 00:32:52 ot_train.py:649 step:10K smpl:1M ep:28K epch:5.63 loss:0.046 grdn:0.285 lr:9.9e-05 updt_s:1.633 data_s:1.484 smp/s:41 mem_gb:52.85
+Training:  16%|█▋        | 9800/60000 [8:59:25<42:32:33,  3.05s/step]INFO 2026-07-18 00:43:02 ot_train.py:649 step:10K smpl:1M ep:28K epch:5.74 loss:0.045 grdn:0.279 lr:9.9e-05 updt_s:1.608 data_s:1.437 smp/s:42 mem_gb:52.85
+Training:  17%|█▋        | 10000/60000 [9:09:29<39:33:39,  2.85s/step]INFO 2026-07-18 00:53:05 ot_train.py:649 step:10K smpl:1M ep:29K epch:5.86 loss:0.045 grdn:0.266 lr:9.9e-05 updt_s:1.615 data_s:1.396 smp/s:43 mem_gb:52.85
+Training:  17%|█▋        | 10200/60000 [9:19:37<45:29:41,  3.29s/step]INFO 2026-07-18 01:03:13 ot_train.py:649 step:10K smpl:1M ep:29K epch:5.98 loss:0.045 grdn:0.268 lr:9.9e-05 updt_s:1.639 data_s:1.394 smp/s:42 mem_gb:52.77
+Training:  17%|█▋        | 10400/60000 [9:29:49<41:05:39,  2.98s/step]INFO 2026-07-18 01:13:25 ot_train.py:649 step:10K smpl:1M ep:30K epch:6.10 loss:0.044 grdn:0.266 lr:9.8e-05 updt_s:1.660 data_s:1.399 smp/s:42 mem_gb:52.85
+Training:  18%|█▊        | 10600/60000 [9:39:57<43:06:12,  3.14s/step]INFO 2026-07-18 01:23:33 ot_train.py:649 step:11K smpl:1M ep:31K epch:6.21 loss:0.044 grdn:0.274 lr:9.8e-05 updt_s:1.666 data_s:1.369 smp/s:42 mem_gb:52.85
+Training:  18%|█▊        | 10800/60000 [9:50:14<46:36:08,  3.41s/step]INFO 2026-07-18 01:33:50 ot_train.py:649 step:11K smpl:1M ep:31K epch:6.33 loss:0.045 grdn:0.272 lr:9.8e-05 updt_s:1.588 data_s:1.489 smp/s:42 mem_gb:52.85
+Training:  18%|█▊        | 11000/60000 [10:00:32<42:20:18,  3.11s/step]INFO 2026-07-18 01:44:09 ot_train.py:649 step:11K smpl:1M ep:32K epch:6.45 loss:0.043 grdn:0.261 lr:9.8e-05 updt_s:1.576 data_s:1.512 smp/s:41 mem_gb:52.85
+Training:  19%|█▊        | 11200/60000 [10:10:42<43:28:02,  3.21s/step]INFO 2026-07-18 01:54:18 ot_train.py:649 step:11K smpl:1M ep:32K epch:6.57 loss:0.044 grdn:0.259 lr:9.8e-05 updt_s:1.615 data_s:1.428 smp/s:42 mem_gb:52.85
+Training:  19%|█▉        | 11400/60000 [10:20:58<46:57:06,  3.48s/step]INFO 2026-07-18 02:04:35 ot_train.py:649 step:11K smpl:1M ep:33K epch:6.68 loss:0.042 grdn:0.268 lr:9.8e-05 updt_s:1.640 data_s:1.436 smp/s:42 mem_gb:52.77
+Training:  19%|█▉        | 11600/60000 [10:31:06<40:11:20,  2.99s/step]INFO 2026-07-18 02:14:42 ot_train.py:649 step:12K smpl:1M ep:34K epch:6.80 loss:0.042 grdn:0.253 lr:9.7e-05 updt_s:1.604 data_s:1.427 smp/s:42 mem_gb:52.85
+Training:  20%|█▉        | 11800/60000 [10:41:23<41:45:51,  3.12s/step]INFO 2026-07-18 02:25:00 ot_train.py:649 step:12K smpl:2M ep:34K epch:6.92 loss:0.041 grdn:0.252 lr:9.7e-05 updt_s:1.577 data_s:1.506 smp/s:42 mem_gb:52.85
+Training:  20%|██        | 12000/60000 [10:51:45<41:52:17,  3.14s/step]INFO 2026-07-18 02:35:21 ot_train.py:649 step:12K smpl:2M ep:35K epch:7.03 loss:0.042 grdn:0.257 lr:9.7e-05 updt_s:1.517 data_s:1.587 smp/s:41 mem_gb:52.85
+Training:  20%|██        | 12200/60000 [11:02:00<41:48:14,  3.15s/step]INFO 2026-07-18 02:45:36 ot_train.py:649 step:12K smpl:2M ep:35K epch:7.15 loss:0.041 grdn:0.250 lr:9.7e-05 updt_s:1.586 data_s:1.481 smp/s:42 mem_gb:52.85
+Training:  21%|██        | 12400/60000 [11:12:04<41:00:36,  3.10s/step]INFO 2026-07-18 02:55:40 ot_train.py:649 step:12K smpl:2M ep:36K epch:7.27 loss:0.041 grdn:0.248 lr:9.7e-05 updt_s:1.780 data_s:1.237 smp/s:42 mem_gb:52.85
+Training:  21%|██        | 12600/60000 [11:22:06<39:20:54,  2.99s/step]INFO 2026-07-18 03:05:42 ot_train.py:649 step:13K smpl:2M ep:36K epch:7.39 loss:0.040 grdn:0.246 lr:9.6e-05 updt_s:1.808 data_s:1.196 smp/s:43 mem_gb:52.77
+Training:  21%|██▏       | 12800/60000 [11:32:04<39:12:32,  2.99s/step]INFO 2026-07-18 03:15:40 ot_train.py:649 step:13K smpl:2M ep:37K epch:7.50 loss:0.040 grdn:0.241 lr:9.6e-05 updt_s:1.785 data_s:1.199 smp/s:43 mem_gb:52.85
+Training:  22%|██▏       | 13000/60000 [11:42:07<37:37:38,  2.88s/step]INFO 2026-07-18 03:25:43 ot_train.py:649 step:13K smpl:2M ep:38K epch:7.62 loss:0.040 grdn:0.244 lr:9.6e-05 updt_s:1.784 data_s:1.226 smp/s:43 mem_gb:52.85
+Training:  22%|██▏       | 13200/60000 [11:52:02<39:58:36,  3.08s/step]INFO 2026-07-18 03:35:38 ot_train.py:649 step:13K smpl:2M ep:38K epch:7.74 loss:0.039 grdn:0.241 lr:9.6e-05 updt_s:1.753 data_s:1.217 smp/s:43 mem_gb:52.85
+Training:  22%|██▏       | 13400/60000 [12:01:54<37:13:39,  2.88s/step]INFO 2026-07-18 03:45:31 ot_train.py:649 step:13K smpl:2M ep:39K epch:7.85 loss:0.039 grdn:0.247 lr:9.6e-05 updt_s:1.744 data_s:1.215 smp/s:43 mem_gb:52.85
+Training:  23%|██▎       | 13600/60000 [12:11:58<39:35:23,  3.07s/step]INFO 2026-07-18 03:55:34 ot_train.py:649 step:14K smpl:2M ep:39K epch:7.97 loss:0.038 grdn:0.228 lr:9.5e-05 updt_s:1.779 data_s:1.235 smp/s:42 mem_gb:52.77
+Training:  23%|██▎       | 13800/60000 [12:22:00<39:56:20,  3.11s/step]INFO 2026-07-18 04:05:36 ot_train.py:649 step:14K smpl:2M ep:40K epch:8.09 loss:0.037 grdn:0.239 lr:9.5e-05 updt_s:1.788 data_s:1.217 smp/s:43 mem_gb:52.85
+Training:  23%|██▎       | 14000/60000 [12:32:06<36:37:45,  2.87s/step]INFO 2026-07-18 04:15:43 ot_train.py:649 step:14K smpl:2M ep:40K epch:8.21 loss:0.037 grdn:0.237 lr:9.5e-05 updt_s:1.760 data_s:1.268 smp/s:42 mem_gb:52.85
+Training:  24%|██▎       | 14200/60000 [12:42:15<35:55:22,  2.82s/step]INFO 2026-07-18 04:25:51 ot_train.py:649 step:14K smpl:2M ep:41K epch:8.32 loss:0.039 grdn:0.240 lr:9.5e-05 updt_s:1.738 data_s:1.300 smp/s:42 mem_gb:52.85
+Training:  24%|██▍       | 14400/60000 [12:52:19<37:39:36,  2.97s/step]INFO 2026-07-18 04:35:55 ot_train.py:649 step:14K smpl:2M ep:42K epch:8.44 loss:0.041 grdn:0.267 lr:9.4e-05 updt_s:1.735 data_s:1.282 smp/s:42 mem_gb:52.85
+Training:  24%|██▍       | 14600/60000 [13:02:24<37:31:44,  2.98s/step]INFO 2026-07-18 04:46:00 ot_train.py:649 step:15K smpl:2M ep:42K epch:8.56 loss:0.038 grdn:0.242 lr:9.4e-05 updt_s:1.771 data_s:1.250 smp/s:42 mem_gb:52.85
+Training:  25%|██▍       | 14800/60000 [13:12:26<37:31:13,  2.99s/step]INFO 2026-07-18 04:56:03 ot_train.py:649 step:15K smpl:2M ep:43K epch:8.68 loss:0.037 grdn:0.237 lr:9.4e-05 updt_s:1.727 data_s:1.280 smp/s:43 mem_gb:52.77
+Training:  25%|██▌       | 15000/60000 [13:22:35<39:05:06,  3.13s/step]INFO 2026-07-18 05:06:11 ot_train.py:649 step:15K smpl:2M ep:43K epch:8.79 loss:0.037 grdn:0.237 lr:9.3e-05 updt_s:1.814 data_s:1.225 smp/s:42 mem_gb:52.85
+Training:  25%|██▌       | 15200/60000 [13:32:53<38:56:20,  3.13s/step]INFO 2026-07-18 05:16:29 ot_train.py:649 step:15K smpl:2M ep:44K epch:8.91 loss:0.037 grdn:0.230 lr:9.3e-05 updt_s:1.748 data_s:1.335 smp/s:42 mem_gb:52.85
+Training:  26%|██▌       | 15400/60000 [13:43:11<36:18:46,  2.93s/step]INFO 2026-07-18 05:26:47 ot_train.py:649 step:15K smpl:2M ep:45K epch:9.03 loss:0.037 grdn:0.229 lr:9.3e-05 updt_s:1.615 data_s:1.472 smp/s:41 mem_gb:52.85
+Training:  26%|██▌       | 15600/60000 [13:53:30<37:33:18,  3.05s/step]INFO 2026-07-18 05:37:06 ot_train.py:649 step:16K smpl:2M ep:45K epch:9.14 loss:0.035 grdn:0.223 lr:9.3e-05 updt_s:1.582 data_s:1.505 smp/s:41 mem_gb:52.85
+Training:  26%|██▋       | 15800/60000 [14:03:50<37:32:14,  3.06s/step]INFO 2026-07-18 05:47:26 ot_train.py:649 step:16K smpl:2M ep:46K epch:9.26 loss:0.036 grdn:0.225 lr:9.2e-05 updt_s:1.586 data_s:1.510 smp/s:41 mem_gb:52.85
+Training:  27%|██▋       | 16000/60000 [14:14:09<37:20:22,  3.06s/step]INFO 2026-07-18 05:57:45 ot_train.py:649 step:16K smpl:2M ep:46K epch:9.38 loss:0.034 grdn:0.225 lr:9.2e-05 updt_s:1.580 data_s:1.509 smp/s:41 mem_gb:52.77
+Training:  27%|██▋       | 16200/60000 [14:24:26<38:41:45,  3.18s/step]INFO 2026-07-18 06:08:02 ot_train.py:649 step:16K smpl:2M ep:47K epch:9.50 loss:0.035 grdn:0.229 lr:9.2e-05 updt_s:1.573 data_s:1.510 smp/s:42 mem_gb:52.85
+Training:  27%|██▋       | 16400/60000 [14:34:41<37:48:53,  3.12s/step]INFO 2026-07-18 06:18:17 ot_train.py:649 step:16K smpl:2M ep:47K epch:9.61 loss:0.035 grdn:0.227 lr:9.1e-05 updt_s:1.577 data_s:1.492 smp/s:42 mem_gb:52.85
+Training:  28%|██▊       | 16600/60000 [14:45:03<37:13:50,  3.09s/step]INFO 2026-07-18 06:28:40 ot_train.py:649 step:17K smpl:2M ep:48K epch:9.73 loss:0.035 grdn:0.220 lr:9.1e-05 updt_s:1.583 data_s:1.523 smp/s:41 mem_gb:52.85
+Training:  28%|██▊       | 16800/60000 [14:55:19<38:22:40,  3.20s/step]INFO 2026-07-18 06:38:55 ot_train.py:649 step:17K smpl:2M ep:49K epch:9.85 loss:0.035 grdn:0.222 lr:9.1e-05 updt_s:1.537 data_s:1.534 smp/s:42 mem_gb:52.85
+Training:  28%|██▊       | 17000/60000 [15:05:32<36:47:49,  3.08s/step]INFO 2026-07-18 06:49:08 ot_train.py:649 step:17K smpl:2M ep:49K epch:9.96 loss:0.036 grdn:0.234 lr:9.0e-05 updt_s:1.583 data_s:1.477 smp/s:42 mem_gb:52.77
+Training:  29%|██▊       | 17200/60000 [15:16:20<44:41:29,  3.76s/step]INFO 2026-07-18 06:59:56 ot_train.py:649 step:17K smpl:2M ep:50K epch:10.08 loss:0.033 grdn:0.224 lr:9.0e-05 updt_s:1.657 data_s:1.577 smp/s:40 mem_gb:52.85
+Training:  29%|██▉       | 17400/60000 [15:26:57<37:16:33,  3.15s/step]INFO 2026-07-18 07:10:33 ot_train.py:649 step:17K smpl:2M ep:50K epch:10.20 loss:0.033 grdn:0.224 lr:9.0e-05 updt_s:1.607 data_s:1.576 smp/s:40 mem_gb:52.85
+Training:  29%|██▉       | 17600/60000 [15:37:32<35:39:29,  3.03s/step]INFO 2026-07-18 07:21:08 ot_train.py:649 step:18K smpl:2M ep:51K epch:10.32 loss:0.033 grdn:0.207 lr:8.9e-05 updt_s:1.590 data_s:1.580 smp/s:40 mem_gb:52.85
+Training:  30%|██▉       | 17800/60000 [15:48:13<36:24:02,  3.11s/step]INFO 2026-07-18 07:31:49 ot_train.py:649 step:18K smpl:2M ep:51K epch:10.43 loss:0.033 grdn:0.221 lr:8.9e-05 updt_s:1.630 data_s:1.569 smp/s:40 mem_gb:52.85
+Training:  30%|███       | 18000/60000 [15:58:52<39:11:44,  3.36s/step]INFO 2026-07-18 07:42:29 ot_train.py:649 step:18K smpl:2M ep:52K epch:10.55 loss:0.033 grdn:0.224 lr:8.8e-05 updt_s:1.622 data_s:1.573 smp/s:40 mem_gb:52.85
+Training:  30%|███       | 18200/60000 [16:09:33<37:27:13,  3.23s/step]INFO 2026-07-18 07:53:09 ot_train.py:649 step:18K smpl:2M ep:53K epch:10.67 loss:0.034 grdn:0.221 lr:8.8e-05 updt_s:1.610 data_s:1.588 smp/s:40 mem_gb:52.77
+Training:  31%|███       | 18400/60000 [16:20:24<36:01:54,  3.12s/step]INFO 2026-07-18 08:04:00 ot_train.py:649 step:18K smpl:2M ep:53K epch:10.79 loss:0.033 grdn:0.229 lr:8.8e-05 updt_s:1.648 data_s:1.601 smp/s:39 mem_gb:52.85
+Training:  31%|███       | 18600/60000 [16:30:51<35:56:39,  3.13s/step]INFO 2026-07-18 08:14:27 ot_train.py:649 step:19K smpl:2M ep:54K epch:10.90 loss:0.032 grdn:0.211 lr:8.7e-05 updt_s:1.541 data_s:1.591 smp/s:41 mem_gb:52.85
+Training:  31%|███▏      | 18800/60000 [16:41:06<36:03:46,  3.15s/step]INFO 2026-07-18 08:24:42 ot_train.py:649 step:19K smpl:2M ep:54K epch:11.02 loss:0.032 grdn:0.210 lr:8.7e-05 updt_s:1.538 data_s:1.532 smp/s:42 mem_gb:52.85
+Training:  32%|███▏      | 19000/60000 [16:51:44<40:26:02,  3.55s/step]INFO 2026-07-18 08:35:20 ot_train.py:649 step:19K smpl:2M ep:55K epch:11.14 loss:0.031 grdn:0.205 lr:8.7e-05 updt_s:1.623 data_s:1.561 smp/s:40 mem_gb:52.85
+Training:  32%|███▏      | 19200/60000 [17:02:16<36:02:44,  3.18s/step]INFO 2026-07-18 08:45:52 ot_train.py:649 step:19K smpl:2M ep:55K epch:11.25 loss:0.033 grdn:0.226 lr:8.6e-05 updt_s:1.583 data_s:1.575 smp/s:41 mem_gb:52.77
+Training:  32%|███▏      | 19400/60000 [17:12:44<31:35:48,  2.80s/step]INFO 2026-07-18 08:56:21 ot_train.py:649 step:19K smpl:2M ep:56K epch:11.37 loss:0.031 grdn:0.224 lr:8.6e-05 updt_s:1.591 data_s:1.546 smp/s:41 mem_gb:52.85
+Training:  33%|███▎      | 19600/60000 [17:22:38<35:12:48,  3.14s/step]INFO 2026-07-18 09:06:14 ot_train.py:649 step:20K smpl:3M ep:57K epch:11.49 loss:0.031 grdn:0.207 lr:8.5e-05 updt_s:1.633 data_s:1.329 smp/s:43 mem_gb:52.85
+Training:  33%|███▎      | 19800/60000 [17:32:21<30:56:45,  2.77s/step]INFO 2026-07-18 09:15:57 ot_train.py:649 step:20K smpl:3M ep:57K epch:11.61 loss:0.030 grdn:0.211 lr:8.5e-05 updt_s:1.603 data_s:1.304 smp/s:44 mem_gb:52.85
+Training:  33%|███▎      | 20000/60000 [17:42:10<32:17:24,  2.91s/step]INFO 2026-07-18 09:25:46 ot_train.py:649 step:20K smpl:3M ep:58K epch:11.72 loss:0.031 grdn:0.209 lr:8.5e-05 updt_s:1.604 data_s:1.336 smp/s:44 mem_gb:52.85
+Training:  34%|███▎      | 20200/60000 [17:51:57<30:58:34,  2.80s/step]INFO 2026-07-18 09:35:33 ot_train.py:649 step:20K smpl:3M ep:58K epch:11.84 loss:0.030 grdn:0.215 lr:8.4e-05 updt_s:1.624 data_s:1.307 smp/s:44 mem_gb:52.85
+Training:  34%|███▍      | 20400/60000 [18:01:49<33:31:13,  3.05s/step]INFO 2026-07-18 09:45:25 ot_train.py:649 step:20K smpl:3M ep:59K epch:11.96 loss:0.030 grdn:0.208 lr:8.4e-05 updt_s:1.624 data_s:1.329 smp/s:43 mem_gb:52.77
+Training:  34%|███▍      | 20600/60000 [18:11:53<31:59:43,  2.92s/step]INFO 2026-07-18 09:55:29 ot_train.py:649 step:21K smpl:3M ep:60K epch:12.08 loss:0.029 grdn:0.200 lr:8.3e-05 updt_s:1.652 data_s:1.363 smp/s:42 mem_gb:52.85
+Training:  35%|███▍      | 20800/60000 [18:21:47<30:39:15,  2.82s/step]INFO 2026-07-18 10:05:23 ot_train.py:649 step:21K smpl:3M ep:60K epch:12.19 loss:0.029 grdn:0.208 lr:8.3e-05 updt_s:1.633 data_s:1.334 smp/s:43 mem_gb:52.85
+Training:  35%|███▌      | 21000/60000 [18:31:43<33:55:54,  3.13s/step]INFO 2026-07-18 10:15:19 ot_train.py:649 step:21K smpl:3M ep:61K epch:12.31 loss:0.029 grdn:0.202 lr:8.2e-05 updt_s:1.639 data_s:1.335 smp/s:43 mem_gb:52.85
+Training:  35%|███▌      | 21200/60000 [18:41:34<31:44:25,  2.94s/step]INFO 2026-07-18 10:25:11 ot_train.py:649 step:21K smpl:3M ep:61K epch:12.43 loss:0.030 grdn:0.208 lr:8.2e-05 updt_s:1.616 data_s:1.336 smp/s:43 mem_gb:52.85
+Training:  36%|███▌      | 21400/60000 [18:51:24<31:42:04,  2.96s/step]INFO 2026-07-18 10:35:00 ot_train.py:649 step:21K smpl:3M ep:62K epch:12.54 loss:0.029 grdn:0.200 lr:8.1e-05 updt_s:1.584 data_s:1.359 smp/s:43 mem_gb:52.85
+Training:  36%|███▌      | 21600/60000 [19:00:59<31:09:33,  2.92s/step]INFO 2026-07-18 10:44:35 ot_train.py:649 step:22K smpl:3M ep:62K epch:12.66 loss:0.030 grdn:0.222 lr:8.1e-05 updt_s:1.582 data_s:1.286 smp/s:45 mem_gb:52.77
+Training:  36%|███▋      | 21800/60000 [19:10:29<33:28:47,  3.16s/step]INFO 2026-07-18 10:54:06 ot_train.py:649 step:22K smpl:3M ep:63K epch:12.78 loss:0.028 grdn:0.211 lr:8.1e-05 updt_s:1.574 data_s:1.275 smp/s:45 mem_gb:52.85
+Training:  37%|███▋      | 22000/60000 [19:20:02<30:10:33,  2.86s/step]INFO 2026-07-18 11:03:38 ot_train.py:649 step:22K smpl:3M ep:64K epch:12.90 loss:0.028 grdn:0.204 lr:8.0e-05 updt_s:1.591 data_s:1.269 smp/s:45 mem_gb:52.85
+Training:  37%|███▋      | 22200/60000 [19:29:28<31:03:57,  2.96s/step]INFO 2026-07-18 11:13:04 ot_train.py:649 step:22K smpl:3M ep:64K epch:13.01 loss:0.029 grdn:0.210 lr:8.0e-05 updt_s:1.536 data_s:1.288 smp/s:45 mem_gb:52.85
+Training:  37%|███▋      | 22400/60000 [19:38:57<28:50:38,  2.76s/step]INFO 2026-07-18 11:22:33 ot_train.py:649 step:22K smpl:3M ep:65K epch:13.13 loss:0.027 grdn:0.194 lr:7.9e-05 updt_s:1.556 data_s:1.284 smp/s:45 mem_gb:52.85
+Training:  38%|███▊      | 22600/60000 [19:49:05<31:44:04,  3.05s/step]INFO 2026-07-18 11:32:41 ot_train.py:649 step:23K smpl:3M ep:65K epch:13.25 loss:0.028 grdn:0.205 lr:7.9e-05 updt_s:1.743 data_s:1.294 smp/s:42 mem_gb:52.77
+Training:  38%|███▊      | 22800/60000 [19:59:25<32:26:54,  3.14s/step]INFO 2026-07-18 11:43:01 ot_train.py:649 step:23K smpl:3M ep:66K epch:13.36 loss:0.027 grdn:0.202 lr:7.8e-05 updt_s:1.811 data_s:1.285 smp/s:41 mem_gb:52.85
+Training:  38%|███▊      | 23000/60000 [20:09:40<31:42:25,  3.09s/step]INFO 2026-07-18 11:53:16 ot_train.py:649 step:23K smpl:3M ep:66K epch:13.48 loss:0.027 grdn:0.203 lr:7.8e-05 updt_s:1.762 data_s:1.305 smp/s:42 mem_gb:52.85
+Training:  39%|███▊      | 23200/60000 [20:19:53<31:15:38,  3.06s/step]INFO 2026-07-18 12:03:30 ot_train.py:649 step:23K smpl:3M ep:67K epch:13.60 loss:0.027 grdn:0.195 lr:7.7e-05 updt_s:1.804 data_s:1.260 smp/s:42 mem_gb:52.85
+Training:  39%|███▉      | 23400/60000 [20:29:57<31:13:50,  3.07s/step]INFO 2026-07-18 12:13:34 ot_train.py:649 step:23K smpl:3M ep:68K epch:13.72 loss:0.026 grdn:0.199 lr:7.7e-05 updt_s:1.764 data_s:1.251 smp/s:42 mem_gb:52.85
+Training:  39%|███▉      | 23600/60000 [20:40:09<31:42:13,  3.14s/step]INFO 2026-07-18 12:23:46 ot_train.py:649 step:24K smpl:3M ep:68K epch:13.83 loss:0.026 grdn:0.198 lr:7.6e-05 updt_s:1.784 data_s:1.272 smp/s:42 mem_gb:52.85
+Training:  40%|███▉      | 23800/60000 [20:50:20<30:35:00,  3.04s/step]INFO 2026-07-18 12:33:57 ot_train.py:649 step:24K smpl:3M ep:69K epch:13.95 loss:0.026 grdn:0.193 lr:7.6e-05 updt_s:1.778 data_s:1.272 smp/s:42 mem_gb:52.77
+Training:  40%|████      | 24000/60000 [21:00:34<31:31:05,  3.15s/step]INFO 2026-07-18 12:44:10 ot_train.py:649 step:24K smpl:3M ep:69K epch:14.07 loss:0.025 grdn:0.195 lr:7.5e-05 updt_s:1.735 data_s:1.326 smp/s:42 mem_gb:52.85
+Training:  40%|████      | 24200/60000 [21:11:01<30:26:56,  3.06s/step]INFO 2026-07-18 12:54:37 ot_train.py:649 step:24K smpl:3M ep:70K epch:14.19 loss:0.025 grdn:0.191 lr:7.5e-05 updt_s:1.560 data_s:1.570 smp/s:41 mem_gb:52.85
+Training:  41%|████      | 24400/60000 [21:21:15<30:34:01,  3.09s/step]INFO 2026-07-18 13:04:51 ot_train.py:649 step:24K smpl:3M ep:71K epch:14.30 loss:0.025 grdn:0.188 lr:7.4e-05 updt_s:1.808 data_s:1.258 smp/s:42 mem_gb:52.85
+Training:  41%|████      | 24600/60000 [21:31:36<31:01:47,  3.16s/step]INFO 2026-07-18 13:15:12 ot_train.py:649 step:25K smpl:3M ep:71K epch:14.42 loss:0.027 grdn:0.208 lr:7.4e-05 updt_s:1.733 data_s:1.366 smp/s:41 mem_gb:52.85
+Training:  41%|████▏     | 24800/60000 [21:42:05<30:13:56,  3.09s/step]INFO 2026-07-18 13:25:41 ot_train.py:649 step:25K smpl:3M ep:72K epch:14.54 loss:0.026 grdn:0.197 lr:7.3e-05 updt_s:1.554 data_s:1.586 smp/s:41 mem_gb:52.85
+Training:  42%|████▏     | 25000/60000 [21:52:31<30:16:53,  3.11s/step]INFO 2026-07-18 13:36:07 ot_train.py:649 step:25K smpl:3M ep:72K epch:14.65 loss:0.024 grdn:0.187 lr:7.3e-05 updt_s:1.586 data_s:1.539 smp/s:41 mem_gb:52.77
+Training:  42%|████▏     | 25200/60000 [22:02:54<29:34:37,  3.06s/step]INFO 2026-07-18 13:46:30 ot_train.py:649 step:25K smpl:3M ep:73K epch:14.77 loss:0.024 grdn:0.190 lr:7.2e-05 updt_s:1.650 data_s:1.458 smp/s:41 mem_gb:52.85
+Training:  42%|████▏     | 25400/60000 [22:13:12<28:09:50,  2.93s/step]INFO 2026-07-18 13:56:48 ot_train.py:649 step:25K smpl:3M ep:73K epch:14.89 loss:0.025 grdn:0.193 lr:7.2e-05 updt_s:1.623 data_s:1.464 smp/s:41 mem_gb:52.85
+Training:  43%|████▎     | 25600/60000 [22:23:19<29:49:19,  3.12s/step]INFO 2026-07-18 14:06:55 ot_train.py:649 step:26K smpl:3M ep:74K epch:15.01 loss:0.026 grdn:0.200 lr:7.1e-05 updt_s:1.582 data_s:1.444 smp/s:42 mem_gb:52.85
+Training:  43%|████▎     | 25800/60000 [22:33:35<27:45:28,  2.92s/step]INFO 2026-07-18 14:17:11 ot_train.py:649 step:26K smpl:3M ep:75K epch:15.12 loss:0.024 grdn:0.185 lr:7.1e-05 updt_s:1.649 data_s:1.426 smp/s:42 mem_gb:52.85
+Training:  43%|████▎     | 26000/60000 [22:43:57<30:43:07,  3.25s/step]INFO 2026-07-18 14:27:33 ot_train.py:649 step:26K smpl:3M ep:75K epch:15.24 loss:0.024 grdn:0.185 lr:7.0e-05 updt_s:1.673 data_s:1.434 smp/s:41 mem_gb:52.77
+Training:  44%|████▎     | 26200/60000 [22:54:09<29:37:02,  3.15s/step]INFO 2026-07-18 14:37:46 ot_train.py:649 step:26K smpl:3M ep:76K epch:15.36 loss:0.023 grdn:0.185 lr:7.0e-05 updt_s:1.590 data_s:1.467 smp/s:42 mem_gb:52.85
+Training:  44%|████▍     | 26400/60000 [23:04:18<28:10:49,  3.02s/step]INFO 2026-07-18 14:47:54 ot_train.py:649 step:26K smpl:3M ep:76K epch:15.47 loss:0.023 grdn:0.187 lr:6.9e-05 updt_s:1.549 data_s:1.489 smp/s:42 mem_gb:52.85
+Training:  44%|████▍     | 26600/60000 [23:14:43<28:29:42,  3.07s/step]INFO 2026-07-18 14:58:19 ot_train.py:649 step:27K smpl:3M ep:77K epch:15.59 loss:0.022 grdn:0.179 lr:6.8e-05 updt_s:1.564 data_s:1.554 smp/s:41 mem_gb:52.85
+Training:  45%|████▍     | 26800/60000 [23:25:00<30:34:26,  3.32s/step]INFO 2026-07-18 15:08:36 ot_train.py:649 step:27K smpl:3M ep:77K epch:15.71 loss:0.023 grdn:0.181 lr:6.8e-05 updt_s:1.570 data_s:1.509 smp/s:42 mem_gb:52.85
+Training:  45%|████▌     | 27000/60000 [23:35:14<27:21:48,  2.99s/step]INFO 2026-07-18 15:18:50 ot_train.py:649 step:27K smpl:3M ep:78K epch:15.83 loss:0.023 grdn:0.185 lr:6.7e-05 updt_s:1.567 data_s:1.501 smp/s:42 mem_gb:52.85
+Training:  45%|████▌     | 27200/60000 [23:45:38<27:56:54,  3.07s/step]INFO 2026-07-18 15:29:15 ot_train.py:649 step:27K smpl:3M ep:79K epch:15.94 loss:0.022 grdn:0.173 lr:6.7e-05 updt_s:1.606 data_s:1.509 smp/s:41 mem_gb:52.77
+Training:  46%|████▌     | 27400/60000 [23:55:46<27:19:33,  3.02s/step]INFO 2026-07-18 15:39:23 ot_train.py:649 step:27K smpl:4M ep:79K epch:16.06 loss:0.022 grdn:0.179 lr:6.6e-05 updt_s:1.612 data_s:1.423 smp/s:42 mem_gb:52.85
+Training:  46%|████▌     | 27600/60000 [24:05:54<27:44:18,  3.08s/step]INFO 2026-07-18 15:49:30 ot_train.py:649 step:28K smpl:4M ep:80K epch:16.18 loss:0.022 grdn:0.175 lr:6.6e-05 updt_s:1.612 data_s:1.419 smp/s:42 mem_gb:52.85
+Training:  46%|████▋     | 27800/60000 [24:16:04<27:02:33,  3.02s/step]INFO 2026-07-18 15:59:40 ot_train.py:649 step:28K smpl:4M ep:80K epch:16.30 loss:0.023 grdn:0.183 lr:6.5e-05 updt_s:1.670 data_s:1.376 smp/s:42 mem_gb:52.85
+Training:  47%|████▋     | 28000/60000 [24:25:52<25:10:49,  2.83s/step]INFO 2026-07-18 16:09:28 ot_train.py:649 step:28K smpl:4M ep:81K epch:16.41 loss:0.023 grdn:0.186 lr:6.5e-05 updt_s:1.594 data_s:1.339 smp/s:44 mem_gb:52.85
+Training:  47%|████▋     | 28200/60000 [24:36:03<26:43:19,  3.03s/step]INFO 2026-07-18 16:19:39 ot_train.py:649 step:28K smpl:4M ep:81K epch:16.53 loss:0.022 grdn:0.174 lr:6.4e-05 updt_s:1.649 data_s:1.403 smp/s:42 mem_gb:52.85
+Training:  47%|████▋     | 28400/60000 [24:46:18<27:00:46,  3.08s/step]INFO 2026-07-18 16:29:54 ot_train.py:649 step:28K smpl:4M ep:82K epch:16.65 loss:0.020 grdn:0.170 lr:6.4e-05 updt_s:1.564 data_s:1.504 smp/s:42 mem_gb:52.77
+Training:  48%|████▊     | 28600/60000 [24:56:28<25:58:21,  2.98s/step]INFO 2026-07-18 16:40:05 ot_train.py:649 step:29K smpl:4M ep:83K epch:16.76 loss:0.021 grdn:0.183 lr:6.3e-05 updt_s:1.589 data_s:1.458 smp/s:42 mem_gb:52.85
+Training:  48%|████▊     | 28800/60000 [25:06:32<27:14:29,  3.14s/step]INFO 2026-07-18 16:50:08 ot_train.py:649 step:29K smpl:4M ep:83K epch:16.88 loss:0.022 grdn:0.185 lr:6.2e-05 updt_s:1.637 data_s:1.376 smp/s:42 mem_gb:52.85
+Training:  48%|████▊     | 29000/60000 [25:16:50<25:42:42,  2.99s/step]INFO 2026-07-18 17:00:26 ot_train.py:649 step:29K smpl:4M ep:84K epch:17.00 loss:0.021 grdn:0.174 lr:6.2e-05 updt_s:1.614 data_s:1.468 smp/s:42 mem_gb:52.85
+Training:  49%|████▊     | 29200/60000 [25:27:06<26:22:50,  3.08s/step]INFO 2026-07-18 17:10:42 ot_train.py:649 step:29K smpl:4M ep:84K epch:17.12 loss:0.021 grdn:0.177 lr:6.1e-05 updt_s:1.583 data_s:1.492 smp/s:42 mem_gb:52.85
+Training:  49%|████▉     | 29400/60000 [25:37:14<26:39:06,  3.14s/step]INFO 2026-07-18 17:20:51 ot_train.py:649 step:29K smpl:4M ep:85K epch:17.23 loss:0.021 grdn:0.176 lr:6.1e-05 updt_s:1.605 data_s:1.433 smp/s:42 mem_gb:52.77
+Training:  49%|████▉     | 29600/60000 [25:47:26<25:24:04,  3.01s/step]INFO 2026-07-18 17:31:02 ot_train.py:649 step:30K smpl:4M ep:86K epch:17.35 loss:0.020 grdn:0.178 lr:6.0e-05 updt_s:1.600 data_s:1.451 smp/s:42 mem_gb:52.85
+Training:  50%|████▉     | 29800/60000 [25:57:33<25:21:40,  3.02s/step]INFO 2026-07-18 17:41:09 ot_train.py:649 step:30K smpl:4M ep:86K epch:17.47 loss:0.020 grdn:0.173 lr:6.0e-05 updt_s:1.685 data_s:1.342 smp/s:42 mem_gb:52.85
+Training:  50%|█████     | 30000/60000 [26:07:28<24:39:55,  2.96s/step]INFO 2026-07-18 17:51:04 ot_train.py:649 step:30K smpl:4M ep:87K epch:17.59 loss:0.020 grdn:0.171 lr:5.9e-05 updt_s:1.640 data_s:1.333 smp/s:43 mem_gb:52.85
+Training:  50%|█████     | 30200/60000 [26:17:37<24:15:16,  2.93s/step]INFO 2026-07-18 18:01:13 ot_train.py:649 step:30K smpl:4M ep:87K epch:17.70 loss:0.020 grdn:0.163 lr:5.8e-05 updt_s:1.603 data_s:1.436 smp/s:42 mem_gb:52.85
+Training:  51%|█████     | 30400/60000 [26:27:46<25:17:59,  3.08s/step]INFO 2026-07-18 18:11:22 ot_train.py:649 step:30K smpl:4M ep:88K epch:17.82 loss:0.020 grdn:0.166 lr:5.8e-05 updt_s:1.630 data_s:1.412 smp/s:42 mem_gb:52.85
+Training:  51%|█████     | 30600/60000 [26:38:00<23:53:26,  2.93s/step]INFO 2026-07-18 18:21:36 ot_train.py:649 step:31K smpl:4M ep:88K epch:17.94 loss:0.019 grdn:0.164 lr:5.7e-05 updt_s:1.576 data_s:1.489 smp/s:42 mem_gb:52.77
+Training:  51%|█████▏    | 30800/60000 [26:48:09<24:05:11,  2.97s/step]INFO 2026-07-18 18:31:45 ot_train.py:649 step:31K smpl:4M ep:89K epch:18.05 loss:0.018 grdn:0.160 lr:5.7e-05 updt_s:1.624 data_s:1.416 smp/s:42 mem_gb:52.85
+Training:  52%|█████▏    | 31000/60000 [26:58:15<24:23:44,  3.03s/step]INFO 2026-07-18 18:41:51 ot_train.py:649 step:31K smpl:4M ep:90K epch:18.17 loss:0.020 grdn:0.170 lr:5.6e-05 updt_s:1.608 data_s:1.414 smp/s:42 mem_gb:52.85
+Training:  52%|█████▏    | 31200/60000 [27:08:33<23:36:26,  2.95s/step]INFO 2026-07-18 18:52:09 ot_train.py:649 step:31K smpl:4M ep:90K epch:18.29 loss:0.019 grdn:0.165 lr:5.6e-05 updt_s:1.672 data_s:1.415 smp/s:41 mem_gb:52.85
+Training:  52%|█████▏    | 31400/60000 [27:18:34<22:09:17,  2.79s/step]INFO 2026-07-18 19:02:10 ot_train.py:649 step:31K smpl:4M ep:91K epch:18.41 loss:0.019 grdn:0.164 lr:5.5e-05 updt_s:1.596 data_s:1.402 smp/s:43 mem_gb:52.85
+Training:  53%|█████▎    | 31600/60000 [27:28:43<24:50:40,  3.15s/step]INFO 2026-07-18 19:12:19 ot_train.py:649 step:32K smpl:4M ep:91K epch:18.52 loss:0.019 grdn:0.167 lr:5.4e-05 updt_s:1.624 data_s:1.416 smp/s:42 mem_gb:52.85
+Training:  53%|█████▎    | 31800/60000 [27:38:36<26:20:54,  3.36s/step]INFO 2026-07-18 19:22:12 ot_train.py:649 step:32K smpl:4M ep:92K epch:18.64 loss:0.018 grdn:0.168 lr:5.4e-05 updt_s:1.610 data_s:1.350 smp/s:43 mem_gb:52.77
+Training:  53%|█████▎    | 32000/60000 [27:48:42<23:44:24,  3.05s/step]INFO 2026-07-18 19:32:18 ot_train.py:649 step:32K smpl:4M ep:92K epch:18.76 loss:0.018 grdn:0.180 lr:5.3e-05 updt_s:1.595 data_s:1.431 smp/s:42 mem_gb:52.85
+Training:  54%|█████▎    | 32200/60000 [27:58:40<26:04:51,  3.38s/step]INFO 2026-07-18 19:42:16 ot_train.py:649 step:32K smpl:4M ep:93K epch:18.87 loss:0.019 grdn:0.171 lr:5.3e-05 updt_s:1.586 data_s:1.397 smp/s:43 mem_gb:52.85
+Training:  54%|█████▍    | 32400/60000 [28:08:34<21:34:22,  2.81s/step]INFO 2026-07-18 19:52:10 ot_train.py:649 step:32K smpl:4M ep:94K epch:18.99 loss:0.018 grdn:0.162 lr:5.2e-05 updt_s:1.614 data_s:1.351 smp/s:43 mem_gb:52.85
+Training:  54%|█████▍    | 32600/60000 [28:18:33<22:50:50,  3.00s/step]INFO 2026-07-18 20:02:09 ot_train.py:649 step:33K smpl:4M ep:94K epch:19.11 loss:0.018 grdn:0.170 lr:5.1e-05 updt_s:1.616 data_s:1.376 smp/s:43 mem_gb:52.85
+Training:  55%|█████▍    | 32800/60000 [28:28:41<23:37:20,  3.13s/step]INFO 2026-07-18 20:12:17 ot_train.py:649 step:33K smpl:4M ep:95K epch:19.23 loss:0.018 grdn:0.166 lr:5.1e-05 updt_s:1.644 data_s:1.388 smp/s:42 mem_gb:52.77
+Training:  55%|█████▌    | 33000/60000 [28:38:43<22:09:47,  2.96s/step]INFO 2026-07-18 20:22:19 ot_train.py:649 step:33K smpl:4M ep:95K epch:19.34 loss:0.017 grdn:0.150 lr:5.0e-05 updt_s:1.623 data_s:1.382 smp/s:43 mem_gb:52.85
+Training:  55%|█████▌    | 33200/60000 [28:48:45<24:19:20,  3.27s/step]INFO 2026-07-18 20:32:21 ot_train.py:649 step:33K smpl:4M ep:96K epch:19.46 loss:0.017 grdn:0.165 lr:5.0e-05 updt_s:1.617 data_s:1.391 smp/s:43 mem_gb:52.85
+Training:  56%|█████▌    | 33400/60000 [28:58:58<23:03:55,  3.12s/step]INFO 2026-07-18 20:42:34 ot_train.py:649 step:33K smpl:4M ep:97K epch:19.58 loss:0.017 grdn:0.150 lr:4.9e-05 updt_s:1.607 data_s:1.451 smp/s:42 mem_gb:52.85
+Training:  56%|█████▌    | 33600/60000 [29:09:11<23:05:53,  3.15s/step]INFO 2026-07-18 20:52:47 ot_train.py:649 step:34K smpl:4M ep:97K epch:19.70 loss:0.017 grdn:0.157 lr:4.9e-05 updt_s:1.619 data_s:1.439 smp/s:42 mem_gb:52.85
+Training:  56%|█████▋    | 33800/60000 [29:19:25<23:22:37,  3.21s/step]INFO 2026-07-18 21:03:01 ot_train.py:649 step:34K smpl:4M ep:98K epch:19.81 loss:0.017 grdn:0.151 lr:4.8e-05 updt_s:1.617 data_s:1.448 smp/s:42 mem_gb:52.85
+Training:  57%|█████▋    | 34000/60000 [29:29:37<21:52:02,  3.03s/step]INFO 2026-07-18 21:13:13 ot_train.py:649 step:34K smpl:4M ep:98K epch:19.93 loss:0.016 grdn:0.153 lr:4.7e-05 updt_s:1.582 data_s:1.474 smp/s:42 mem_gb:52.77
+Training:  57%|█████▋    | 34200/60000 [29:39:49<19:11:36,  2.68s/step]INFO 2026-07-18 21:23:26 ot_train.py:649 step:34K smpl:4M ep:99K epch:20.05 loss:0.016 grdn:0.156 lr:4.7e-05 updt_s:1.564 data_s:1.495 smp/s:42 mem_gb:52.85
+Training:  57%|█████▋    | 34400/60000 [29:49:22<19:36:52,  2.76s/step]INFO 2026-07-18 21:32:58 ot_train.py:649 step:34K smpl:4M ep:99K epch:20.16 loss:0.016 grdn:0.157 lr:4.6e-05 updt_s:1.608 data_s:1.250 smp/s:45 mem_gb:52.85
+Training:  58%|█████▊    | 34600/60000 [29:58:47<19:36:45,  2.78s/step]INFO 2026-07-18 21:42:23 ot_train.py:649 step:35K smpl:4M ep:100K epch:20.28 loss:0.017 grdn:0.165 lr:4.6e-05 updt_s:1.588 data_s:1.232 smp/s:45 mem_gb:52.85
+Training:  58%|█████▊    | 34800/60000 [30:08:20<20:17:25,  2.90s/step]INFO 2026-07-18 21:51:56 ot_train.py:649 step:35K smpl:4M ep:101K epch:20.40 loss:0.016 grdn:0.149 lr:4.5e-05 updt_s:1.602 data_s:1.257 smp/s:45 mem_gb:52.85
+Training:  58%|█████▊    | 35000/60000 [30:17:43<22:50:19,  3.29s/step]INFO 2026-07-18 22:01:19 ot_train.py:649 step:35K smpl:4M ep:101K epch:20.52 loss:0.016 grdn:0.156 lr:4.4e-05 updt_s:1.547 data_s:1.263 smp/s:46 mem_gb:52.77
+Training:  59%|█████▊    | 35200/60000 [30:27:09<17:43:14,  2.57s/step]INFO 2026-07-18 22:10:45 ot_train.py:649 step:35K smpl:5M ep:102K epch:20.63 loss:0.015 grdn:0.150 lr:4.4e-05 updt_s:1.594 data_s:1.230 smp/s:45 mem_gb:52.85
+Training:  59%|█████▉    | 35400/60000 [30:36:38<19:18:31,  2.83s/step]INFO 2026-07-18 22:20:14 ot_train.py:649 step:35K smpl:5M ep:102K epch:20.75 loss:0.015 grdn:0.139 lr:4.3e-05 updt_s:1.587 data_s:1.255 smp/s:45 mem_gb:52.85
+Training:  59%|█████▉    | 35600/60000 [30:46:05<18:27:03,  2.72s/step]INFO 2026-07-18 22:29:41 ot_train.py:649 step:36K smpl:5M ep:103K epch:20.87 loss:0.015 grdn:0.149 lr:4.3e-05 updt_s:1.637 data_s:1.190 smp/s:45 mem_gb:52.85
+Training:  60%|█████▉    | 35800/60000 [30:55:28<17:46:57,  2.65s/step]INFO 2026-07-18 22:39:04 ot_train.py:649 step:36K smpl:5M ep:103K epch:20.98 loss:0.015 grdn:0.143 lr:4.2e-05 updt_s:1.592 data_s:1.221 smp/s:45 mem_gb:52.85
+Training:  60%|██████    | 36000/60000 [31:04:46<19:31:07,  2.93s/step]INFO 2026-07-18 22:48:22 ot_train.py:649 step:36K smpl:5M ep:104K epch:21.10 loss:0.015 grdn:0.144 lr:4.2e-05 updt_s:1.578 data_s:1.205 smp/s:46 mem_gb:52.85
+Training:  60%|██████    | 36200/60000 [31:14:48<20:23:12,  3.08s/step]INFO 2026-07-18 22:58:24 ot_train.py:649 step:36K smpl:5M ep:105K epch:21.22 loss:0.015 grdn:0.157 lr:4.1e-05 updt_s:1.611 data_s:1.396 smp/s:43 mem_gb:52.77
+Training:  61%|██████    | 36400/60000 [31:24:40<20:51:24,  3.18s/step]INFO 2026-07-18 23:08:16 ot_train.py:649 step:36K smpl:5M ep:105K epch:21.34 loss:0.014 grdn:0.141 lr:4.0e-05 updt_s:1.556 data_s:1.399 smp/s:43 mem_gb:52.85
+Training:  61%|██████    | 36600/60000 [31:35:05<24:13:28,  3.73s/step]INFO 2026-07-18 23:18:41 ot_train.py:649 step:37K smpl:5M ep:106K epch:21.45 loss:0.015 grdn:0.150 lr:4.0e-05 updt_s:1.659 data_s:1.459 smp/s:41 mem_gb:52.85
+Training:  61%|██████▏   | 36800/60000 [31:45:51<22:01:31,  3.42s/step]INFO 2026-07-18 23:29:27 ot_train.py:649 step:37K smpl:5M ep:106K epch:21.57 loss:0.015 grdn:0.144 lr:3.9e-05 updt_s:1.639 data_s:1.585 smp/s:40 mem_gb:52.85
+Training:  62%|██████▏   | 37000/60000 [31:56:35<20:17:38,  3.18s/step]INFO 2026-07-18 23:40:11 ot_train.py:649 step:37K smpl:5M ep:107K epch:21.69 loss:0.015 grdn:0.149 lr:3.9e-05 updt_s:1.582 data_s:1.634 smp/s:40 mem_gb:52.85
+Training:  62%|██████▏   | 37200/60000 [32:07:14<20:12:06,  3.19s/step]INFO 2026-07-18 23:50:51 ot_train.py:649 step:37K smpl:5M ep:108K epch:21.81 loss:0.014 grdn:0.140 lr:3.8e-05 updt_s:1.578 data_s:1.612 smp/s:40 mem_gb:52.85
+Training:  62%|██████▏   | 37400/60000 [32:17:58<19:34:44,  3.12s/step]INFO 2026-07-19 00:01:35 ot_train.py:649 step:37K smpl:5M ep:108K epch:21.92 loss:0.014 grdn:0.135 lr:3.8e-05 updt_s:1.622 data_s:1.592 smp/s:40 mem_gb:52.77
+Training:  63%|██████▎   | 37600/60000 [32:28:15<19:32:49,  3.14s/step]INFO 2026-07-19 00:11:52 ot_train.py:649 step:38K smpl:5M ep:109K epch:22.04 loss:0.014 grdn:0.133 lr:3.7e-05 updt_s:1.525 data_s:1.554 smp/s:42 mem_gb:52.85
+Training:  63%|██████▎   | 37800/60000 [32:38:31<18:54:05,  3.07s/step]INFO 2026-07-19 00:22:07 ot_train.py:649 step:38K smpl:5M ep:109K epch:22.16 loss:0.014 grdn:0.139 lr:3.6e-05 updt_s:1.601 data_s:1.472 smp/s:42 mem_gb:52.85
+Training:  63%|██████▎   | 38000/60000 [32:48:48<18:30:21,  3.03s/step]INFO 2026-07-19 00:32:24 ot_train.py:649 step:38K smpl:5M ep:110K epch:22.27 loss:0.013 grdn:0.135 lr:3.6e-05 updt_s:1.610 data_s:1.470 smp/s:42 mem_gb:52.85
+Training:  64%|██████▎   | 38200/60000 [32:59:03<18:26:15,  3.04s/step]INFO 2026-07-19 00:42:39 ot_train.py:649 step:38K smpl:5M ep:110K epch:22.39 loss:0.013 grdn:0.133 lr:3.5e-05 updt_s:1.560 data_s:1.506 smp/s:42 mem_gb:52.85
+Training:  64%|██████▍   | 38400/60000 [33:09:08<18:26:14,  3.07s/step]INFO 2026-07-19 00:52:44 ot_train.py:649 step:38K smpl:5M ep:111K epch:22.51 loss:0.014 grdn:0.146 lr:3.5e-05 updt_s:1.596 data_s:1.426 smp/s:42 mem_gb:52.77
+Training:  64%|██████▍   | 38600/60000 [33:19:24<17:44:30,  2.98s/step]INFO 2026-07-19 01:03:00 ot_train.py:649 step:39K smpl:5M ep:112K epch:22.63 loss:0.013 grdn:0.129 lr:3.4e-05 updt_s:1.649 data_s:1.424 smp/s:42 mem_gb:52.85
+Training:  65%|██████▍   | 38800/60000 [33:29:38<17:53:09,  3.04s/step]INFO 2026-07-19 01:13:14 ot_train.py:649 step:39K smpl:5M ep:112K epch:22.74 loss:0.013 grdn:0.141 lr:3.4e-05 updt_s:1.646 data_s:1.422 smp/s:42 mem_gb:52.85
+Training:  65%|██████▌   | 39000/60000 [33:39:46<17:57:58,  3.08s/step]INFO 2026-07-19 01:23:23 ot_train.py:649 step:39K smpl:5M ep:113K epch:22.86 loss:0.013 grdn:0.133 lr:3.3e-05 updt_s:1.633 data_s:1.403 smp/s:42 mem_gb:52.85
+Training:  65%|██████▌   | 39200/60000 [33:49:57<18:14:43,  3.16s/step]INFO 2026-07-19 01:33:33 ot_train.py:649 step:39K smpl:5M ep:113K epch:22.98 loss:0.013 grdn:0.133 lr:3.3e-05 updt_s:1.589 data_s:1.456 smp/s:42 mem_gb:52.85
+Training:  66%|██████▌   | 39400/60000 [34:00:19<16:01:33,  2.80s/step]INFO 2026-07-19 01:43:56 ot_train.py:649 step:39K smpl:5M ep:114K epch:23.10 loss:0.013 grdn:0.139 lr:3.2e-05 updt_s:1.587 data_s:1.522 smp/s:41 mem_gb:52.85
+Training:  66%|██████▌   | 39600/60000 [34:10:03<15:31:18,  2.74s/step]INFO 2026-07-19 01:53:39 ot_train.py:649 step:40K smpl:5M ep:114K epch:23.21 loss:0.013 grdn:0.129 lr:3.2e-05 updt_s:1.506 data_s:1.407 smp/s:44 mem_gb:52.77
+Training:  66%|██████▋   | 39800/60000 [34:19:50<15:45:11,  2.81s/step]INFO 2026-07-19 02:03:26 ot_train.py:649 step:40K smpl:5M ep:115K epch:23.33 loss:0.012 grdn:0.141 lr:3.1e-05 updt_s:1.520 data_s:1.409 smp/s:44 mem_gb:52.85
+Training:  67%|██████▋   | 40000/60000 [34:29:21<17:31:13,  3.15s/step]INFO 2026-07-19 02:12:57 ot_train.py:649 step:40K smpl:5M ep:116K epch:23.45 loss:0.012 grdn:0.134 lr:3.0e-05 updt_s:1.489 data_s:1.359 smp/s:45 mem_gb:52.85
+Training:  67%|██████▋   | 40111/60000 [34:34:56<16:35:21,  3.00s/step]

Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K.interrupted_reboot_20260719T051924/wandb/run-20260717_154326-sz25gl12/files/requirements.txt

@@ -0,0 +1,117 @@
+pyarrow==25.0.0
+importlib_metadata==9.0.0
+wheel==0.47.0
+cloudpickle==3.1.2
+nvidia-cusolver-cu12==11.7.3.90
+multiprocess==0.70.19
+torch==2.11.0+cu128
+nvidia-nvjitlink-cu12==12.8.93
+fsspec==2026.2.0
+pytz==2026.2
+hf-xet==1.5.1
+tokenizers==0.22.2
+opencv-python-headless==4.13.0.92
+cuda-pathfinder==1.2.2
+frozenlist==1.8.0
+smmap==5.0.3
+httpcore==1.0.9
+nvidia-nvshmem-cu12==3.4.5
+nvidia-cuda-nvrtc-cu12==12.8.93
+annotated-types==0.7.0
+platformdirs==4.10.0
+click==8.4.2
+pyyaml-include==1.4.1
+MarkupSafe==3.0.3
+Jinja2==3.1.6
+urllib3==2.7.0
+nvidia-curand-cu12==10.3.9.90
+certifi==2026.6.17
+nvidia-cuda-runtime-cu12==12.8.90
+torchcodec==0.11.1
+toml==0.10.2
+av==15.1.0
+nvidia-cufft-cu12==11.3.3.83
+attrs==26.1.0
+lerobot==0.6.0
+PyYAML==6.0.3
+safetensors==0.8.0
+diffusers==0.35.2
+requests==2.34.2
+nvidia-nvtx-cu12==12.8.90
+charset-normalizer==3.4.9
+networkx==3.6.1
+cmake==4.1.3
+pydantic_core==2.46.4
+timm==1.0.28
+dm-tree==0.1.10
+nvidia-cufile-cu12==1.13.1.3
+termcolor==3.3.0
+multidict==6.7.1
+filelock==3.29.0
+tzdata==2026.3
+aiohappyeyeballs==2.7.1
+httpx==0.28.1
+nvidia-nccl-cu12==2.28.9
+Pygments==2.20.0
+einops==0.8.2
+nvidia-cusparse-cu12==12.5.8.93
+torchvision==0.26.0+cu128
+dill==0.4.1
+GitPython==3.1.51
+nvidia-cuda-cupti-cu12==12.8.90
+decord==0.6.0
+sympy==1.14.0
+cuda-toolkit==12.8.1
+python-dateutil==2.9.0.post0
+shellingham==1.5.4
+propcache==0.5.2
+Farama-Notifications==0.0.6
+pillow==12.2.0
+cuda-bindings==12.9.4
+huggingface_hub==1.23.0
+pandas==2.3.3
+typing-inspect==0.9.0
+packaging==25.0
+rich==15.0.0
+nvidia-cusparselt-cu12==0.7.1
+setuptools==78.1.0
+wrapt==2.2.2
+mypy_extensions==1.1.0
+tqdm==4.68.4
+draccus==0.10.0
+aiosignal==1.4.0
+mpmath==1.3.0
+numpy==2.2.6
+typing_extensions==4.15.0
+psutil==7.2.2
+mdurl==0.1.2
+nvidia-cublas-cu12==12.8.4.1
+absl-py==2.5.0
+gitdb==4.0.12
+transformers==5.5.4
+accelerate==1.14.0
+pydantic==2.13.4
+pip==26.1.2
+markdown-it-py==4.2.0
+typer==0.26.8
+triton==3.6.0
+wandb==0.27.2
+regex==2026.7.10
+yarl==1.24.2
+jsonlines==4.0.0
+mergedeep==1.3.4
+aiohttp==3.14.1
+datasets==4.8.5
+gymnasium==1.3.0
+peft==0.19.1
+zipp==4.1.0
+idna==3.18
+nvidia-cudnn-cu12==9.19.0.56
+h11==0.16.0
+sentry-sdk==2.65.0
+annotated-doc==0.0.4
+protobuf==7.35.1
+typing-inspection==0.4.2
+anyio==4.14.2
+xxhash==3.8.1
+six==1.17.0

Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K.interrupted_reboot_20260719T051924/wandb/run-20260717_154326-sz25gl12/files/wandb-metadata.json

@@ -0,0 +1,92 @@
+{
+  "os": "Linux-6.8.0-107-generic-x86_64-with-glibc2.39",
+  "python": "CPython 3.12.13",
+  "startedAt": "2026-07-17T06:43:26.118344Z",
+  "args": [
+    "--config_path=/tmp/insight_lerobot_cfg_yp0hihif/LAPstyle_RKD_linear_decay_6K.yaml",
+    "--batch_size=128"
+  ],
+  "program": "/home/ext_minje/clvla/benchmarks/INSIGHT/my_scripts/lerobot_train_yaml.py",
+  "codePath": "benchmarks/INSIGHT/my_scripts/lerobot_train_yaml.py",
+  "codePathLocal": "my_scripts/lerobot_train_yaml.py",
+  "git": {
+    "remote": "git@github.com:paragon7060/clvla.git",
+    "commit": "a970d9f98f189faccce13cf2fb61ccf5d55921f3"
+  },
+  "root": "/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K",
+  "host": "DGX-H200-01",
+  "executable": "/home/ext_minje/miniforge3/envs/lerobot060_groot/bin/python",
+  "cpu_count": 112,
+  "cpu_count_logical": 224,
+  "gpu": "NVIDIA H200",
+  "gpu_count": 8,
+  "disk": {
+    "/": {
+      "total": "1888425144320",
+      "used": "1493416677376"
+    }
+  },
+  "memory": {
+    "total": "2164177248256"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-1e531a45-480e-3c1a-e9a0-2ba2cc27b0a3"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-5d3d5ae2-ffde-89d9-8818-dc19e11ad2fb"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-305c0dae-dcf5-d7bb-7dd3-5477d2194275"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-ed8b7920-8881-4e8f-3e88-be1bbecafa21"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-a91176db-55f2-b284-bbbf-56626596947d"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-922ff6d5-a948-205a-0426-dbee495fc3dd"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-1fde7c75-18fe-e3db-a4c1-85e67522f55f"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-a9b1c49b-d527-97d8-ce91-7e50bd02154c"
+    }
+  ],
+  "cudaVersion": "12.9",
+  "writerId": "18e0dkydbeh9kbgur2u0ri0dyzmcya11"
+}

Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K.interrupted_reboot_20260719T051924/wandb/run-20260717_154326-sz25gl12/logs/debug.log

@@ -0,0 +1,19 @@
+2026-07-17 15:43:26,119 INFO    MainThread:499778 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-17 15:43:26,119 INFO    MainThread:499778 [wandb_setup.py:_flush():81] Configure stats pid to 499778
+2026-07-17 15:43:26,119 INFO    MainThread:499778 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-17 15:43:26,119 INFO    MainThread:499778 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K/wandb/run-20260717_154326-sz25gl12/logs/debug.log
+2026-07-17 15:43:26,120 INFO    MainThread:499778 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K/wandb/run-20260717_154326-sz25gl12/logs/debug-internal.log
+2026-07-17 15:43:26,120 INFO    MainThread:499778 [wandb_init.py:init():766] calling init triggers
+2026-07-17 15:43:26,120 INFO    MainThread:499778 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot_rkd', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_steps': 6000, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False, 'rkd_enabled': True, 'rkd_vae_checkpoint': '/home/ext_minje/groot_insight/TimewarpVAE_ckpt/multispace_gripper_dedup_batch_ablation_v1/b512/RD/K08_z16_g4_h256/checkpoint-step-20000-epoch-46.84', 'rkd_student_pooling': 'attention', 'rkd_student_attention_hidden_dim': 512, 'rkd_student_attention_output_dim': 512, 'rkd_fm_loss_weight': 1.0, 'rkd_loss_weight': 1.0, 'rkd_distance_weight': 1.0, 'rkd_angle_weight': 2.0}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K', 'job_name': 'INSIGHT_RD_RKD_TimewarpVAE_LAPstyle_6K', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 6000, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-17 15:43:26,120 INFO    MainThread:499778 [wandb_init.py:init():814] starting backend
+2026-07-17 15:43:26,346 INFO    MainThread:499778 [wandb_init.py:init():829] sending inform_init request
+2026-07-17 15:43:26,990 INFO    MainThread:499778 [wandb_init.py:init():834] backend started and connected
+2026-07-17 15:43:26,993 INFO    MainThread:499778 [wandb_init.py:init():904] updated telemetry
+2026-07-17 15:43:27,003 INFO    MainThread:499778 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-17 15:43:27,554 INFO    MainThread:499778 [wandb_init.py:init():972] starting run threads in backend
+2026-07-17 15:43:27,750 INFO    MainThread:499778 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-17 15:43:27,750 INFO    MainThread:499778 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-17 15:43:27,750 INFO    MainThread:499778 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-17 15:43:27,750 INFO    MainThread:499778 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-17 15:43:27,754 INFO    MainThread:499778 [wandb_init.py:init():1010] run started, returning control to user process

Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K/wandb/run-20260719_051936-rnsbujva/files/output.log

@@ -0,0 +1,76 @@
+INFO 2026-07-19 05:19:37 db_utils.py:121 Logs will be synced with wandb.
+INFO 2026-07-19 05:19:37 db_utils.py:122 Track this run --> https://wandb.ai/minje227_hyu-hanyang-university/lerobot/runs/rnsbujva
+INFO 2026-07-19 05:19:37 ot_train.py:298 Creating dataset
+INFO 2026-07-19 05:19:39 ot_train.py:332 Creating policy
+Fetching 27 files: 100%|██████████| 27/27 [00:00<00:00, 1548.07it/s]
+`torch_dtype` is deprecated! Use `dtype` instead!
+Loading weights: 100%|██████████| 1031/1031 [00:01<00:00, 550.23it/s]
+INFO 2026-07-19 05:19:50 ot_train.py:405 Creating optimizer and scheduler
+INFO 2026-07-19 05:19:50 ot_train.py:439 Output dir: /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K
+INFO 2026-07-19 05:19:50 ot_train.py:446 cfg.steps=60000 (60K)
+INFO 2026-07-19 05:19:50 ot_train.py:447 dataset.num_frames=218367 (218K)
+INFO 2026-07-19 05:19:50 ot_train.py:448 dataset.num_episodes=4930
+INFO 2026-07-19 05:19:50 ot_train.py:451 Effective batch size: 128 x 1 = 128
+INFO 2026-07-19 05:19:50 ot_train.py:452 num_learnable_params=1622090881 (2B)
+INFO 2026-07-19 05:19:50 ot_train.py:453 num_total_params=3145712033 (3B)
+Training:   0%|          | 0/60000 [00:00<?, ?step/s]INFO 2026-07-19 05:19:50 ot_train.py:604 Start offline training on a fixed dataset, with effective batch size: 128
+Training:   0%|          | 200/60000 [09:35<48:43:12,  2.93s/step]INFO 2026-07-19 05:29:25 ot_train.py:649 step:200 smpl:26K ep:578 epch:0.12 loss:1.234 grdn:0.482 lr:1.7e-06 updt_s:1.645 data_s:1.226 smp/s:45 mem_gb:52.89
+Training:   1%|          | 400/60000 [19:25<51:27:15,  3.11s/step]INFO 2026-07-19 05:39:16 ot_train.py:649 step:400 smpl:51K ep:1K epch:0.23 loss:0.864 grdn:0.955 lr:5.0e-06 updt_s:1.641 data_s:1.305 smp/s:43 mem_gb:52.95
+Training:   1%|          | 600/60000 [29:21<48:14:21,  2.92s/step]INFO 2026-07-19 05:49:12 ot_train.py:649 step:600 smpl:77K ep:2K epch:0.35 loss:0.339 grdn:1.616 lr:8.3e-06 updt_s:1.646 data_s:1.331 smp/s:43 mem_gb:52.95
+Training:   1%|▏         | 800/60000 [39:18<51:38:34,  3.14s/step]INFO 2026-07-19 05:59:09 ot_train.py:649 step:800 smpl:102K ep:2K epch:0.47 loss:0.213 grdn:2.064 lr:1.2e-05 updt_s:1.629 data_s:1.352 smp/s:43 mem_gb:52.95
+Training:   2%|▏         | 1000/60000 [49:20<49:48:47,  3.04s/step]INFO 2026-07-19 06:09:11 ot_train.py:649 step:1K smpl:128K ep:3K epch:0.59 loss:0.174 grdn:2.131 lr:1.5e-05 updt_s:1.594 data_s:1.409 smp/s:43 mem_gb:52.95
+Training:   2%|▏         | 1200/60000 [59:34<49:29:09,  3.03s/step]INFO 2026-07-19 06:19:25 ot_train.py:649 step:1K smpl:154K ep:3K epch:0.70 loss:0.152 grdn:1.998 lr:1.8e-05 updt_s:1.696 data_s:1.372 smp/s:42 mem_gb:52.88
+Training:   2%|▏         | 1400/60000 [1:09:42<50:28:30,  3.10s/step]INFO 2026-07-19 06:29:33 ot_train.py:649 step:1K smpl:179K ep:4K epch:0.82 loss:0.137 grdn:1.849 lr:2.2e-05 updt_s:1.621 data_s:1.412 smp/s:42 mem_gb:52.98
+Training:   3%|▎         | 1600/60000 [1:19:45<52:11:12,  3.22s/step]INFO 2026-07-19 06:39:36 ot_train.py:649 step:2K smpl:205K ep:5K epch:0.94 loss:0.125 grdn:1.681 lr:2.5e-05 updt_s:1.626 data_s:1.384 smp/s:43 mem_gb:52.98
+Training:   3%|▎         | 1800/60000 [1:29:45<48:00:54,  2.97s/step]INFO 2026-07-19 06:49:36 ot_train.py:649 step:2K smpl:230K ep:5K epch:1.06 loss:0.117 grdn:1.544 lr:2.8e-05 updt_s:1.621 data_s:1.376 smp/s:43 mem_gb:52.98
+Training:   3%|▎         | 2000/60000 [1:39:46<49:46:39,  3.09s/step]INFO 2026-07-19 06:59:37 ot_train.py:649 step:2K smpl:256K ep:6K epch:1.17 loss:0.111 grdn:1.381 lr:3.2e-05 updt_s:1.618 data_s:1.382 smp/s:43 mem_gb:52.98
+Training:   4%|▎         | 2200/60000 [1:49:52<47:41:18,  2.97s/step]INFO 2026-07-19 07:09:43 ot_train.py:649 step:2K smpl:282K ep:6K epch:1.29 loss:0.105 grdn:1.281 lr:3.5e-05 updt_s:1.603 data_s:1.423 smp/s:42 mem_gb:52.98
+Training:   4%|▍         | 2400/60000 [1:59:54<48:01:23,  3.00s/step]INFO 2026-07-19 07:19:45 ot_train.py:649 step:2K smpl:307K ep:7K epch:1.41 loss:0.100 grdn:1.131 lr:3.8e-05 updt_s:1.588 data_s:1.417 smp/s:43 mem_gb:52.90
+Training:   4%|▍         | 2600/60000 [2:09:53<46:26:40,  2.91s/step]INFO 2026-07-19 07:29:44 ot_train.py:649 step:3K smpl:333K ep:8K epch:1.52 loss:0.096 grdn:1.003 lr:4.2e-05 updt_s:1.603 data_s:1.388 smp/s:43 mem_gb:52.98
+Training:   5%|▍         | 2800/60000 [2:19:56<49:26:52,  3.11s/step]INFO 2026-07-19 07:39:47 ot_train.py:649 step:3K smpl:358K ep:8K epch:1.64 loss:0.091 grdn:0.930 lr:4.5e-05 updt_s:1.644 data_s:1.365 smp/s:43 mem_gb:52.98
+Training:   5%|▌         | 3000/60000 [2:30:01<47:37:02,  3.01s/step]INFO 2026-07-19 07:49:52 ot_train.py:649 step:3K smpl:384K ep:9K epch:1.76 loss:0.087 grdn:0.857 lr:4.8e-05 updt_s:1.619 data_s:1.402 smp/s:42 mem_gb:52.98
+Training:   5%|▌         | 3200/60000 [2:40:00<48:33:51,  3.08s/step]INFO 2026-07-19 07:59:51 ot_train.py:649 step:3K smpl:410K ep:9K epch:1.88 loss:0.083 grdn:0.783 lr:5.2e-05 updt_s:1.620 data_s:1.371 smp/s:43 mem_gb:52.98
+Training:   6%|▌         | 3400/60000 [2:50:08<53:08:03,  3.38s/step]INFO 2026-07-19 08:09:59 ot_train.py:649 step:3K smpl:435K ep:10K epch:1.99 loss:0.080 grdn:0.736 lr:5.5e-05 updt_s:1.665 data_s:1.372 smp/s:42 mem_gb:52.90
+Training:   6%|▌         | 3600/60000 [3:00:08<49:40:31,  3.17s/step]INFO 2026-07-19 08:19:59 ot_train.py:649 step:4K smpl:461K ep:10K epch:2.11 loss:0.076 grdn:0.687 lr:5.8e-05 updt_s:1.620 data_s:1.374 smp/s:43 mem_gb:52.98
+Training:   6%|▋         | 3800/60000 [3:10:07<48:08:59,  3.08s/step]INFO 2026-07-19 08:29:57 ot_train.py:649 step:4K smpl:486K ep:11K epch:2.23 loss:0.075 grdn:0.647 lr:6.2e-05 updt_s:1.601 data_s:1.386 smp/s:43 mem_gb:52.98
+Training:   7%|▋         | 4000/60000 [3:20:05<47:33:36,  3.06s/step]INFO 2026-07-19 08:39:56 ot_train.py:649 step:4K smpl:512K ep:12K epch:2.34 loss:0.072 grdn:0.613 lr:6.5e-05 updt_s:1.595 data_s:1.392 smp/s:43 mem_gb:52.98
+Training:   7%|▋         | 4200/60000 [3:30:07<45:31:37,  2.94s/step]INFO 2026-07-19 08:49:58 ot_train.py:649 step:4K smpl:538K ep:12K epch:2.46 loss:0.071 grdn:0.570 lr:6.8e-05 updt_s:1.649 data_s:1.357 smp/s:43 mem_gb:52.98
+Training:   7%|▋         | 4400/60000 [3:40:09<45:57:52,  2.98s/step]INFO 2026-07-19 09:00:00 ot_train.py:649 step:4K smpl:563K ep:13K epch:2.58 loss:0.069 grdn:0.554 lr:7.2e-05 updt_s:1.606 data_s:1.398 smp/s:43 mem_gb:52.98
+Training:   8%|▊         | 4600/60000 [3:50:09<45:57:46,  2.99s/step]INFO 2026-07-19 09:10:00 ot_train.py:649 step:5K smpl:589K ep:13K epch:2.70 loss:0.067 grdn:0.521 lr:7.5e-05 updt_s:1.621 data_s:1.376 smp/s:43 mem_gb:52.90
+Training:   8%|▊         | 4800/60000 [4:00:07<46:57:00,  3.06s/step]INFO 2026-07-19 09:19:57 ot_train.py:649 step:5K smpl:614K ep:14K epch:2.81 loss:0.063 grdn:0.491 lr:7.8e-05 updt_s:1.623 data_s:1.360 smp/s:43 mem_gb:52.98
+Training:   8%|▊         | 5000/60000 [4:10:11<45:56:21,  3.01s/step]INFO 2026-07-19 09:30:02 ot_train.py:649 step:5K smpl:640K ep:14K epch:2.93 loss:0.062 grdn:0.478 lr:8.2e-05 updt_s:1.578 data_s:1.439 smp/s:42 mem_gb:52.98
+Training:   9%|▊         | 5200/60000 [4:20:15<47:11:34,  3.10s/step]INFO 2026-07-19 09:40:06 ot_train.py:649 step:5K smpl:666K ep:15K epch:3.05 loss:0.062 grdn:0.467 lr:8.5e-05 updt_s:1.567 data_s:1.448 smp/s:42 mem_gb:52.98
+Training:   9%|▉         | 5400/60000 [4:30:26<45:09:05,  2.98s/step]INFO 2026-07-19 09:50:17 ot_train.py:649 step:5K smpl:691K ep:16K epch:3.17 loss:0.059 grdn:0.422 lr:8.8e-05 updt_s:1.623 data_s:1.426 smp/s:42 mem_gb:52.98
+Training:   9%|▉         | 5600/60000 [4:40:31<47:43:44,  3.16s/step]INFO 2026-07-19 10:00:21 ot_train.py:649 step:6K smpl:717K ep:16K epch:3.28 loss:0.058 grdn:0.425 lr:9.2e-05 updt_s:1.597 data_s:1.422 smp/s:42 mem_gb:52.98
+Training:  10%|▉         | 5800/60000 [4:50:36<43:43:45,  2.90s/step]INFO 2026-07-19 10:10:27 ot_train.py:649 step:6K smpl:742K ep:17K epch:3.40 loss:0.057 grdn:0.411 lr:9.5e-05 updt_s:1.603 data_s:1.418 smp/s:42 mem_gb:52.90
+Training:  10%|█         | 6000/60000 [5:00:47<49:13:24,  3.28s/step]INFO 2026-07-19 10:20:38 ot_train.py:649 step:6K smpl:768K ep:17K epch:3.52 loss:0.057 grdn:0.406 lr:9.8e-05 updt_s:1.602 data_s:1.450 smp/s:42 mem_gb:52.98
+INFO 2026-07-19 10:20:38 ot_train.py:103 INSIGHT phase transition: index=1 name=fm_only range=[6000,60000) trainable_params=1620515968
+Training:  10%|█         | 6200/60000 [5:10:58<44:32:35,  2.98s/step]INFO 2026-07-19 10:30:49 ot_train.py:649 step:6K smpl:794K ep:18K epch:3.63 loss:0.056 grdn:0.394 lr:1.0e-04 updt_s:1.601 data_s:1.451 smp/s:42 mem_gb:52.85
+Training:  11%|█         | 6400/60000 [5:20:59<46:22:00,  3.11s/step]INFO 2026-07-19 10:40:50 ot_train.py:649 step:6K smpl:819K ep:18K epch:3.75 loss:0.054 grdn:0.365 lr:1.0e-04 updt_s:1.577 data_s:1.423 smp/s:43 mem_gb:52.85
+Training:  11%|█         | 6600/60000 [5:31:02<43:17:29,  2.92s/step]INFO 2026-07-19 10:50:53 ot_train.py:649 step:7K smpl:845K ep:19K epch:3.87 loss:0.053 grdn:0.353 lr:1.0e-04 updt_s:1.569 data_s:1.441 smp/s:43 mem_gb:52.85
+Training:  11%|█▏        | 6800/60000 [5:41:16<42:01:24,  2.84s/step]INFO 2026-07-19 11:01:07 ot_train.py:649 step:7K smpl:870K ep:20K epch:3.99 loss:0.053 grdn:0.355 lr:1.0e-04 updt_s:1.572 data_s:1.494 smp/s:42 mem_gb:52.77
+Training:  12%|█▏        | 7000/60000 [5:51:22<44:32:03,  3.02s/step]INFO 2026-07-19 11:11:12 ot_train.py:649 step:7K smpl:896K ep:20K epch:4.10 loss:0.052 grdn:0.350 lr:1.0e-04 updt_s:1.591 data_s:1.430 smp/s:42 mem_gb:52.85
+Training:  12%|█▏        | 7200/60000 [6:01:29<43:41:36,  2.98s/step]INFO 2026-07-19 11:21:20 ot_train.py:649 step:7K smpl:922K ep:21K epch:4.22 loss:0.051 grdn:0.337 lr:1.0e-04 updt_s:1.558 data_s:1.473 smp/s:42 mem_gb:52.85
+Training:  12%|█▏        | 7400/60000 [6:11:43<44:44:55,  3.06s/step]INFO 2026-07-19 11:31:34 ot_train.py:649 step:7K smpl:947K ep:21K epch:4.34 loss:0.053 grdn:0.349 lr:1.0e-04 updt_s:1.618 data_s:1.447 smp/s:42 mem_gb:52.85
+Training:  13%|█▎        | 7600/60000 [6:22:55<51:51:17,  3.56s/step]INFO 2026-07-19 11:42:46 ot_train.py:649 step:8K smpl:973K ep:22K epch:4.45 loss:0.051 grdn:0.331 lr:1.0e-04 updt_s:1.733 data_s:1.622 smp/s:38 mem_gb:52.85
+Training:  13%|█▎        | 7800/60000 [6:34:53<52:37:51,  3.63s/step]INFO 2026-07-19 11:54:44 ot_train.py:649 step:8K smpl:998K ep:23K epch:4.57 loss:0.051 grdn:0.317 lr:1.0e-04 updt_s:1.799 data_s:1.783 smp/s:36 mem_gb:52.85
+Training:  13%|█▎        | 8000/60000 [6:45:08<52:54:48,  3.66s/step]INFO 2026-07-19 12:04:59 ot_train.py:649 step:8K smpl:1M ep:23K epch:4.69 loss:0.048 grdn:0.315 lr:1.0e-04 updt_s:1.650 data_s:1.422 smp/s:42 mem_gb:52.77
+Training:  14%|█▎        | 8200/60000 [6:57:10<50:01:31,  3.48s/step]INFO 2026-07-19 12:17:01 ot_train.py:649 step:8K smpl:1M ep:24K epch:4.81 loss:0.047 grdn:0.299 lr:1.0e-04 updt_s:1.805 data_s:1.797 smp/s:36 mem_gb:52.85
+Training:  14%|█▍        | 8400/60000 [7:07:33<35:31:15,  2.48s/step]INFO 2026-07-19 12:27:24 ot_train.py:649 step:8K smpl:1M ep:24K epch:4.92 loss:0.049 grdn:0.311 lr:1.0e-04 updt_s:1.717 data_s:1.393 smp/s:41 mem_gb:52.85
+Training:  14%|█▍        | 8600/60000 [7:18:57<52:56:11,  3.71s/step]INFO 2026-07-19 12:38:48 ot_train.py:649 step:9K smpl:1M ep:25K epch:5.04 loss:0.048 grdn:0.304 lr:9.9e-05 updt_s:1.731 data_s:1.682 smp/s:38 mem_gb:52.85
+Training:  15%|█▍        | 8800/60000 [7:30:30<35:32:26,  2.50s/step]INFO 2026-07-19 12:50:21 ot_train.py:649 step:9K smpl:1M ep:25K epch:5.16 loss:0.048 grdn:0.307 lr:9.9e-05 updt_s:1.814 data_s:1.643 smp/s:37 mem_gb:52.85
+Training:  15%|█▌        | 9000/60000 [7:41:07<52:30:13,  3.71s/step]INFO 2026-07-19 13:00:58 ot_train.py:649 step:9K smpl:1M ep:26K epch:5.28 loss:0.048 grdn:0.296 lr:9.9e-05 updt_s:1.669 data_s:1.512 smp/s:40 mem_gb:52.85
+Training:  15%|█▌        | 9200/60000 [7:53:23<51:52:36,  3.68s/step]INFO 2026-07-19 13:13:14 ot_train.py:649 step:9K smpl:1M ep:27K epch:5.39 loss:0.046 grdn:0.285 lr:9.9e-05 updt_s:1.891 data_s:1.779 smp/s:35 mem_gb:52.77
+Training:  16%|█▌        | 9400/60000 [8:03:53<50:10:50,  3.57s/step]INFO 2026-07-19 13:23:44 ot_train.py:649 step:9K smpl:1M ep:27K epch:5.51 loss:0.047 grdn:0.296 lr:9.9e-05 updt_s:1.673 data_s:1.473 smp/s:41 mem_gb:52.85
+Training:  16%|█▌        | 9600/60000 [8:15:51<50:21:53,  3.60s/step]INFO 2026-07-19 13:35:42 ot_train.py:649 step:10K smpl:1M ep:28K epch:5.63 loss:0.046 grdn:0.285 lr:9.9e-05 updt_s:1.782 data_s:1.798 smp/s:36 mem_gb:52.85
+Training:  16%|█▋        | 9800/60000 [8:25:45<43:10:36,  3.10s/step]INFO 2026-07-19 13:45:36 ot_train.py:649 step:10K smpl:1M ep:28K epch:5.74 loss:0.045 grdn:0.280 lr:9.9e-05 updt_s:1.627 data_s:1.340 smp/s:43 mem_gb:52.85
+Training:  17%|█▋        | 10000/60000 [8:37:29<49:34:25,  3.57s/step]INFO 2026-07-19 13:57:20 ot_train.py:649 step:10K smpl:1M ep:29K epch:5.86 loss:0.045 grdn:0.268 lr:9.9e-05 updt_s:1.759 data_s:1.753 smp/s:36 mem_gb:52.85
+Training:  17%|█▋        | 10200/60000 [8:48:27<34:20:49,  2.48s/step]INFO 2026-07-19 14:08:18 ot_train.py:649 step:10K smpl:1M ep:29K epch:5.98 loss:0.045 grdn:0.273 lr:9.9e-05 updt_s:1.765 data_s:1.518 smp/s:39 mem_gb:52.77
+Training:  17%|█▋        | 10400/60000 [8:59:07<49:17:34,  3.58s/step]INFO 2026-07-19 14:18:58 ot_train.py:649 step:10K smpl:1M ep:30K epch:6.10 loss:0.044 grdn:0.263 lr:9.8e-05 updt_s:1.655 data_s:1.538 smp/s:40 mem_gb:52.85
+Training:  18%|█▊        | 10600/60000 [9:10:57<44:48:14,  3.27s/step]INFO 2026-07-19 14:30:48 ot_train.py:649 step:11K smpl:1M ep:31K epch:6.21 loss:0.043 grdn:0.262 lr:9.8e-05 updt_s:1.848 data_s:1.697 smp/s:36 mem_gb:52.85
+Training:  18%|█▊        | 10800/60000 [9:20:25<47:00:44,  3.44s/step]INFO 2026-07-19 14:40:16 ot_train.py:649 step:11K smpl:1M ep:31K epch:6.33 loss:0.046 grdn:0.288 lr:9.8e-05 updt_s:1.519 data_s:1.314 smp/s:45 mem_gb:52.85
+Training:  18%|█▊        | 11000/60000 [9:32:19<48:14:56,  3.54s/step]INFO 2026-07-19 14:52:10 ot_train.py:649 step:11K smpl:1M ep:32K epch:6.45 loss:0.044 grdn:0.270 lr:9.8e-05 updt_s:1.777 data_s:1.785 smp/s:36 mem_gb:52.85
+Training:  19%|█▊        | 11200/60000 [9:42:31<44:09:19,  3.26s/step]INFO 2026-07-19 15:02:22 ot_train.py:649 step:11K smpl:1M ep:32K epch:6.57 loss:0.043 grdn:0.249 lr:9.8e-05 updt_s:1.671 data_s:1.385 smp/s:42 mem_gb:52.85
+Training:  19%|█▉        | 11400/60000 [9:54:14<48:29:03,  3.59s/step]INFO 2026-07-19 15:14:05 ot_train.py:649 step:11K smpl:1M ep:33K epch:6.68 loss:0.042 grdn:0.258 lr:9.8e-05 updt_s:1.750 data_s:1.756 smp/s:37 mem_gb:52.77
+Training:  19%|█▉        | 11600/60000 [10:05:07<31:52:24,  2.37s/step]INFO 2026-07-19 15:24:58 ot_train.py:649 step:12K smpl:1M ep:34K epch:6.80 loss:0.042 grdn:0.256 lr:9.7e-05 updt_s:1.731 data_s:1.529 smp/s:39 mem_gb:52.85
+Training:  20%|█▉        | 11705/60000 [10:10:02<44:49:47,  3.34s/step]

Abs_6D + DeltaEE/RKD_TimewarpVAE/LAPstyle_linear_6K/wandb/run-20260719_051936-rnsbujva/files/requirements.txt

@@ -0,0 +1,117 @@
+pyarrow==25.0.0
+importlib_metadata==9.0.0
+wheel==0.47.0
+cloudpickle==3.1.2
+nvidia-cusolver-cu12==11.7.3.90
+multiprocess==0.70.19
+torch==2.11.0+cu128
+nvidia-nvjitlink-cu12==12.8.93
+fsspec==2026.2.0
+pytz==2026.2
+hf-xet==1.5.1
+tokenizers==0.22.2
+opencv-python-headless==4.13.0.92
+cuda-pathfinder==1.2.2
+frozenlist==1.8.0
+smmap==5.0.3
+httpcore==1.0.9
+nvidia-nvshmem-cu12==3.4.5
+nvidia-cuda-nvrtc-cu12==12.8.93
+annotated-types==0.7.0
+platformdirs==4.10.0
+click==8.4.2
+pyyaml-include==1.4.1
+MarkupSafe==3.0.3
+Jinja2==3.1.6
+urllib3==2.7.0
+nvidia-curand-cu12==10.3.9.90
+certifi==2026.6.17
+nvidia-cuda-runtime-cu12==12.8.90
+torchcodec==0.11.1
+toml==0.10.2
+av==15.1.0
+nvidia-cufft-cu12==11.3.3.83
+attrs==26.1.0
+lerobot==0.6.0
+PyYAML==6.0.3
+safetensors==0.8.0
+diffusers==0.35.2
+requests==2.34.2
+nvidia-nvtx-cu12==12.8.90
+charset-normalizer==3.4.9
+networkx==3.6.1
+cmake==4.1.3
+pydantic_core==2.46.4
+timm==1.0.28
+dm-tree==0.1.10
+nvidia-cufile-cu12==1.13.1.3
+termcolor==3.3.0
+multidict==6.7.1
+filelock==3.29.0
+tzdata==2026.3
+aiohappyeyeballs==2.7.1
+httpx==0.28.1
+nvidia-nccl-cu12==2.28.9
+Pygments==2.20.0
+einops==0.8.2
+nvidia-cusparse-cu12==12.5.8.93
+torchvision==0.26.0+cu128
+dill==0.4.1
+GitPython==3.1.51
+nvidia-cuda-cupti-cu12==12.8.90
+decord==0.6.0
+sympy==1.14.0
+cuda-toolkit==12.8.1
+python-dateutil==2.9.0.post0
+shellingham==1.5.4
+propcache==0.5.2
+Farama-Notifications==0.0.6
+pillow==12.2.0
+cuda-bindings==12.9.4
+huggingface_hub==1.23.0
+pandas==2.3.3
+typing-inspect==0.9.0
+packaging==25.0
+rich==15.0.0
+nvidia-cusparselt-cu12==0.7.1
+setuptools==78.1.0
+wrapt==2.2.2
+mypy_extensions==1.1.0
+tqdm==4.68.4
+draccus==0.10.0
+aiosignal==1.4.0
+mpmath==1.3.0
+numpy==2.2.6
+typing_extensions==4.15.0
+psutil==7.2.2
+mdurl==0.1.2
+nvidia-cublas-cu12==12.8.4.1
+absl-py==2.5.0
+gitdb==4.0.12
+transformers==5.5.4
+accelerate==1.14.0
+pydantic==2.13.4
+pip==26.1.2
+markdown-it-py==4.2.0
+typer==0.26.8
+triton==3.6.0
+wandb==0.27.2
+regex==2026.7.10
+yarl==1.24.2
+jsonlines==4.0.0
+mergedeep==1.3.4
+aiohttp==3.14.1
+datasets==4.8.5
+gymnasium==1.3.0
+peft==0.19.1
+zipp==4.1.0
+idna==3.18
+nvidia-cudnn-cu12==9.19.0.56
+h11==0.16.0
+sentry-sdk==2.65.0
+annotated-doc==0.0.4
+protobuf==7.35.1
+typing-inspection==0.4.2
+anyio==4.14.2
+xxhash==3.8.1
+six==1.17.0

Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K.interrupted_reboot_20260719T051924/wandb/run-20260717_154331-e5avxg6w/files/wandb-metadata.json

@@ -0,0 +1,92 @@
+{
+  "os": "Linux-6.8.0-107-generic-x86_64-with-glibc2.39",
+  "python": "CPython 3.12.13",
+  "startedAt": "2026-07-17T06:43:31.452096Z",
+  "args": [
+    "--config_path=/tmp/insight_lerobot_cfg_9olxbi19/RSCLstyle_RKD_cosine_decay_60K.yaml",
+    "--batch_size=128"
+  ],
+  "program": "/home/ext_minje/clvla/benchmarks/INSIGHT/my_scripts/lerobot_train_yaml.py",
+  "codePath": "benchmarks/INSIGHT/my_scripts/lerobot_train_yaml.py",
+  "codePathLocal": "my_scripts/lerobot_train_yaml.py",
+  "git": {
+    "remote": "git@github.com:paragon7060/clvla.git",
+    "commit": "a970d9f98f189faccce13cf2fb61ccf5d55921f3"
+  },
+  "root": "/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K",
+  "host": "DGX-H200-01",
+  "executable": "/home/ext_minje/miniforge3/envs/lerobot060_groot/bin/python",
+  "cpu_count": 112,
+  "cpu_count_logical": 224,
+  "gpu": "NVIDIA H200",
+  "gpu_count": 8,
+  "disk": {
+    "/": {
+      "total": "1888425144320",
+      "used": "1493419024384"
+    }
+  },
+  "memory": {
+    "total": "2164177248256"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-1e531a45-480e-3c1a-e9a0-2ba2cc27b0a3"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-5d3d5ae2-ffde-89d9-8818-dc19e11ad2fb"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-305c0dae-dcf5-d7bb-7dd3-5477d2194275"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-ed8b7920-8881-4e8f-3e88-be1bbecafa21"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-a91176db-55f2-b284-bbbf-56626596947d"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-922ff6d5-a948-205a-0426-dbee495fc3dd"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-1fde7c75-18fe-e3db-a4c1-85e67522f55f"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-a9b1c49b-d527-97d8-ce91-7e50bd02154c"
+    }
+  ],
+  "cudaVersion": "12.9",
+  "writerId": "dkxiphgeck5cens85cj82zktobu10jie"
+}

Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K.interrupted_reboot_20260719T051924/wandb/run-20260717_154331-e5avxg6w/logs/debug.log

@@ -0,0 +1,19 @@
+2026-07-17 15:43:31,453 INFO    MainThread:499773 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-17 15:43:31,453 INFO    MainThread:499773 [wandb_setup.py:_flush():81] Configure stats pid to 499773
+2026-07-17 15:43:31,453 INFO    MainThread:499773 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-17 15:43:31,453 INFO    MainThread:499773 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K/wandb/run-20260717_154331-e5avxg6w/logs/debug.log
+2026-07-17 15:43:31,453 INFO    MainThread:499773 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K/wandb/run-20260717_154331-e5avxg6w/logs/debug-internal.log
+2026-07-17 15:43:31,453 INFO    MainThread:499773 [wandb_init.py:init():766] calling init triggers
+2026-07-17 15:43:31,453 INFO    MainThread:499773 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot_rkd', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_steps': 3000, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False, 'rkd_enabled': True, 'rkd_vae_checkpoint': '/home/ext_minje/groot_insight/TimewarpVAE_ckpt/multispace_gripper_dedup_batch_ablation_v1/b512/RD/K08_z16_g4_h256/checkpoint-step-20000-epoch-46.84', 'rkd_student_pooling': 'attention', 'rkd_student_attention_hidden_dim': 512, 'rkd_student_attention_output_dim': 512, 'rkd_fm_loss_weight': 1.0, 'rkd_loss_weight': 1.0, 'rkd_distance_weight': 1.0, 'rkd_angle_weight': 2.0}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K', 'job_name': 'INSIGHT_RD_RKD_TimewarpVAE_RSCLstyle_C60', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 3000, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-17 15:43:31,453 INFO    MainThread:499773 [wandb_init.py:init():814] starting backend
+2026-07-17 15:43:31,678 INFO    MainThread:499773 [wandb_init.py:init():829] sending inform_init request
+2026-07-17 15:43:32,380 INFO    MainThread:499773 [wandb_init.py:init():834] backend started and connected
+2026-07-17 15:43:32,384 INFO    MainThread:499773 [wandb_init.py:init():904] updated telemetry
+2026-07-17 15:43:32,392 INFO    MainThread:499773 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-17 15:43:32,831 INFO    MainThread:499773 [wandb_init.py:init():972] starting run threads in backend
+2026-07-17 15:43:33,011 INFO    MainThread:499773 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-17 15:43:33,011 INFO    MainThread:499773 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-17 15:43:33,011 INFO    MainThread:499773 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-17 15:43:33,011 INFO    MainThread:499773 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-17 15:43:33,016 INFO    MainThread:499773 [wandb_init.py:init():1010] run started, returning control to user process

Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K/wandb/debug.log

@@ -0,0 +1,45 @@
+2026-07-19 05:19:36,517 INFO    MainThread:227246 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-19 05:19:36,517 INFO    MainThread:227246 [wandb_setup.py:_flush():81] Configure stats pid to 227246
+2026-07-19 05:19:36,517 INFO    MainThread:227246 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-19 05:19:36,517 INFO    MainThread:227246 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K/wandb/run-20260719_051936-2mdgc559/logs/debug.log
+2026-07-19 05:19:36,518 INFO    MainThread:227246 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K/wandb/run-20260719_051936-2mdgc559/logs/debug-internal.log
+2026-07-19 05:19:36,518 INFO    MainThread:227246 [wandb_init.py:init():766] calling init triggers
+2026-07-19 05:19:36,518 INFO    MainThread:227246 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot_rkd', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_steps': 3000, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False, 'rkd_enabled': True, 'rkd_vae_checkpoint': '/home/ext_minje/groot_insight/TimewarpVAE_ckpt/multispace_gripper_dedup_batch_ablation_v1/b512/RD/K08_z16_g4_h256/checkpoint-step-20000-epoch-46.84', 'rkd_student_pooling': 'attention', 'rkd_student_attention_hidden_dim': 512, 'rkd_student_attention_output_dim': 512, 'rkd_fm_loss_weight': 1.0, 'rkd_loss_weight': 1.0, 'rkd_distance_weight': 1.0, 'rkd_angle_weight': 2.0}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K', 'job_name': 'INSIGHT_RD_RKD_TimewarpVAE_RSCLstyle_C60', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 3000, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-19 05:19:36,518 INFO    MainThread:227246 [wandb_init.py:init():814] starting backend
+2026-07-19 05:19:36,733 INFO    MainThread:227246 [wandb_init.py:init():829] sending inform_init request
+2026-07-19 05:19:37,420 INFO    MainThread:227246 [wandb_init.py:init():834] backend started and connected
+2026-07-19 05:19:37,424 INFO    MainThread:227246 [wandb_init.py:init():904] updated telemetry
+2026-07-19 05:19:37,431 INFO    MainThread:227246 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-19 05:19:37,856 INFO    MainThread:227246 [wandb_init.py:init():972] starting run threads in backend
+2026-07-19 05:19:37,908 INFO    MainThread:227246 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-19 05:19:37,908 INFO    MainThread:227246 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-19 05:19:37,908 INFO    MainThread:227246 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-19 05:19:37,908 INFO    MainThread:227246 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-19 05:19:37,911 INFO    MainThread:227246 [wandb_init.py:init():1010] run started, returning control to user process
+2026-07-19 15:29:54,199 INFO    wandb-AsyncioManager-main:227246 [service_client.py:_forward_responses():122] Reached EOF.
+2026-07-19 15:29:54,199 INFO    wandb-AsyncioManager-main:227246 [mailbox.py:close():154] Closing mailbox, abandoning 2 handles.
+2026-07-19 15:29:54,200 ERROR   wandb-AsyncioManager-main:227246 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback.
+Traceback (most recent call last):
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions
+    await fn()
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop
+    await asyncio_compat.race(
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 278, in race
+    async with open_task_group(race=True) as tg:
+               ^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/contextlib.py", line 217, in __aexit__
+    await anext(self.gen)
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 240, in open_task_group
+    await task_group._wait_all(race=race, timeout=exit_timeout)
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 180, in _wait_all
+    raise exc
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all
+    result = await handle.wait_async(timeout=None)
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async
+    response = await self._handle.wait_async(timeout=timeout)
+               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async
+    raise HandleAbandonedError()
+wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError

Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K/wandb/run-20260719_051936-2mdgc559/files/output.log

@@ -0,0 +1,76 @@
+INFO 2026-07-19 05:19:37 db_utils.py:121 Logs will be synced with wandb.
+INFO 2026-07-19 05:19:37 db_utils.py:122 Track this run --> https://wandb.ai/minje227_hyu-hanyang-university/lerobot/runs/2mdgc559
+INFO 2026-07-19 05:19:37 ot_train.py:298 Creating dataset
+INFO 2026-07-19 05:19:40 ot_train.py:332 Creating policy
+Fetching 27 files: 100%|██████████| 27/27 [00:00<00:00, 3026.27it/s]
+`torch_dtype` is deprecated! Use `dtype` instead!
+Loading weights: 100%|██████████| 1031/1031 [00:00<00:00, 1242.59it/s]
+INFO 2026-07-19 05:19:51 ot_train.py:405 Creating optimizer and scheduler
+INFO 2026-07-19 05:19:51 ot_train.py:439 Output dir: /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K
+INFO 2026-07-19 05:19:51 ot_train.py:446 cfg.steps=60000 (60K)
+INFO 2026-07-19 05:19:51 ot_train.py:447 dataset.num_frames=218367 (218K)
+INFO 2026-07-19 05:19:51 ot_train.py:448 dataset.num_episodes=4930
+INFO 2026-07-19 05:19:51 ot_train.py:451 Effective batch size: 128 x 1 = 128
+INFO 2026-07-19 05:19:51 ot_train.py:452 num_learnable_params=1622090881 (2B)
+INFO 2026-07-19 05:19:51 ot_train.py:453 num_total_params=3145712033 (3B)
+Training:   0%|          | 0/60000 [00:00<?, ?step/s]INFO 2026-07-19 05:19:51 ot_train.py:604 Start offline training on a fixed dataset, with effective batch size: 128
+Training:   0%|          | 200/60000 [09:48<47:22:46,  2.85s/step]INFO 2026-07-19 05:29:40 ot_train.py:649 step:200 smpl:26K ep:578 epch:0.12 loss:1.200 grdn:0.507 lr:3.3e-06 updt_s:1.685 data_s:1.254 smp/s:44 mem_gb:52.89
+Training:   1%|          | 400/60000 [19:41<49:26:49,  2.99s/step]INFO 2026-07-19 05:39:33 ot_train.py:649 step:400 smpl:51K ep:1K epch:0.23 loss:0.548 grdn:1.415 lr:1.0e-05 updt_s:1.649 data_s:1.311 smp/s:43 mem_gb:52.95
+Training:   1%|          | 600/60000 [29:39<52:01:24,  3.15s/step]INFO 2026-07-19 05:49:31 ot_train.py:649 step:600 smpl:77K ep:2K epch:0.35 loss:0.225 grdn:1.984 lr:1.7e-05 updt_s:1.630 data_s:1.357 smp/s:43 mem_gb:52.95
+Training:   1%|▏         | 800/60000 [39:36<49:24:25,  3.00s/step]INFO 2026-07-19 05:59:28 ot_train.py:649 step:800 smpl:102K ep:2K epch:0.47 loss:0.174 grdn:2.023 lr:2.3e-05 updt_s:1.584 data_s:1.397 smp/s:43 mem_gb:52.95
+Training:   2%|▏         | 1000/60000 [49:28<49:44:38,  3.04s/step]INFO 2026-07-19 06:09:20 ot_train.py:649 step:1K smpl:128K ep:3K epch:0.59 loss:0.149 grdn:1.842 lr:3.0e-05 updt_s:1.579 data_s:1.374 smp/s:43 mem_gb:52.95
+Training:   2%|▏         | 1200/60000 [59:27<48:17:18,  2.96s/step]INFO 2026-07-19 06:19:19 ot_train.py:649 step:1K smpl:154K ep:3K epch:0.70 loss:0.136 grdn:1.578 lr:3.7e-05 updt_s:1.630 data_s:1.362 smp/s:43 mem_gb:52.88
+Training:   2%|▏         | 1400/60000 [1:09:35<50:55:13,  3.13s/step]INFO 2026-07-19 06:29:27 ot_train.py:649 step:1K smpl:179K ep:4K epch:0.82 loss:0.127 grdn:1.386 lr:4.3e-05 updt_s:1.646 data_s:1.389 smp/s:42 mem_gb:52.98
+Training:   3%|▎         | 1600/60000 [1:19:30<49:01:04,  3.02s/step]INFO 2026-07-19 06:39:21 ot_train.py:649 step:2K smpl:205K ep:5K epch:0.94 loss:0.120 grdn:1.159 lr:5.0e-05 updt_s:1.612 data_s:1.356 smp/s:43 mem_gb:52.98
+Training:   3%|▎         | 1800/60000 [1:29:17<45:54:00,  2.84s/step]INFO 2026-07-19 06:49:09 ot_train.py:649 step:2K smpl:230K ep:5K epch:1.06 loss:0.113 grdn:1.004 lr:5.7e-05 updt_s:1.599 data_s:1.334 smp/s:44 mem_gb:52.98
+Training:   3%|▎         | 2000/60000 [1:39:17<48:48:54,  3.03s/step]INFO 2026-07-19 06:59:08 ot_train.py:649 step:2K smpl:256K ep:6K epch:1.17 loss:0.107 grdn:0.882 lr:6.3e-05 updt_s:1.588 data_s:1.403 smp/s:43 mem_gb:52.98
+Training:   4%|▎         | 2200/60000 [1:49:14<46:16:36,  2.88s/step]INFO 2026-07-19 07:09:05 ot_train.py:649 step:2K smpl:282K ep:6K epch:1.29 loss:0.100 grdn:0.751 lr:7.0e-05 updt_s:1.543 data_s:1.437 smp/s:43 mem_gb:52.98
+Training:   4%|▍         | 2400/60000 [1:59:16<46:11:14,  2.89s/step]INFO 2026-07-19 07:19:08 ot_train.py:649 step:2K smpl:307K ep:7K epch:1.41 loss:0.095 grdn:0.676 lr:7.7e-05 updt_s:1.573 data_s:1.435 smp/s:43 mem_gb:52.90
+Training:   4%|▍         | 2600/60000 [2:09:13<46:52:46,  2.94s/step]INFO 2026-07-19 07:29:05 ot_train.py:649 step:3K smpl:333K ep:8K epch:1.52 loss:0.092 grdn:0.628 lr:8.3e-05 updt_s:1.614 data_s:1.367 smp/s:43 mem_gb:52.98
+Training:   5%|▍         | 2800/60000 [2:19:06<48:17:15,  3.04s/step]INFO 2026-07-19 07:38:57 ot_train.py:649 step:3K smpl:358K ep:8K epch:1.64 loss:0.087 grdn:0.590 lr:9.0e-05 updt_s:1.602 data_s:1.357 smp/s:43 mem_gb:52.98
+Training:   5%|▌         | 3000/60000 [2:29:08<46:01:17,  2.91s/step]INFO 2026-07-19 07:48:59 ot_train.py:649 step:3K smpl:384K ep:9K epch:1.76 loss:0.086 grdn:0.565 lr:9.7e-05 updt_s:1.565 data_s:1.442 smp/s:43 mem_gb:52.98
+Training:   5%|▌         | 3200/60000 [2:39:11<48:11:47,  3.05s/step]INFO 2026-07-19 07:59:02 ot_train.py:649 step:3K smpl:410K ep:9K epch:1.88 loss:0.082 grdn:0.511 lr:1.0e-04 updt_s:1.627 data_s:1.382 smp/s:43 mem_gb:52.98
+Training:   6%|▌         | 3400/60000 [2:49:08<44:49:17,  2.85s/step]INFO 2026-07-19 08:09:00 ot_train.py:649 step:3K smpl:435K ep:10K epch:1.99 loss:0.079 grdn:0.466 lr:1.0e-04 updt_s:1.603 data_s:1.379 smp/s:43 mem_gb:52.90
+Training:   6%|▌         | 3600/60000 [2:59:07<46:49:48,  2.99s/step]INFO 2026-07-19 08:18:59 ot_train.py:649 step:4K smpl:461K ep:10K epch:2.11 loss:0.076 grdn:0.445 lr:1.0e-04 updt_s:1.647 data_s:1.344 smp/s:43 mem_gb:52.98
+Training:   6%|▋         | 3800/60000 [3:09:09<44:53:53,  2.88s/step]INFO 2026-07-19 08:29:00 ot_train.py:649 step:4K smpl:486K ep:11K epch:2.23 loss:0.075 grdn:0.432 lr:1.0e-04 updt_s:1.571 data_s:1.432 smp/s:43 mem_gb:52.98
+Training:   7%|▋         | 4000/60000 [3:19:05<43:39:10,  2.81s/step]INFO 2026-07-19 08:38:56 ot_train.py:649 step:4K smpl:512K ep:12K epch:2.34 loss:0.072 grdn:0.404 lr:1.0e-04 updt_s:1.576 data_s:1.400 smp/s:43 mem_gb:52.98
+Training:   7%|▋         | 4200/60000 [3:29:06<47:18:43,  3.05s/step]INFO 2026-07-19 08:48:57 ot_train.py:649 step:4K smpl:538K ep:12K epch:2.46 loss:0.071 grdn:0.383 lr:1.0e-04 updt_s:1.640 data_s:1.361 smp/s:43 mem_gb:52.98
+Training:   7%|▋         | 4400/60000 [3:39:08<45:34:31,  2.95s/step]INFO 2026-07-19 08:58:59 ot_train.py:649 step:4K smpl:563K ep:13K epch:2.58 loss:0.069 grdn:0.384 lr:1.0e-04 updt_s:1.600 data_s:1.406 smp/s:43 mem_gb:52.98
+Training:   8%|▊         | 4600/60000 [3:49:05<44:12:36,  2.87s/step]INFO 2026-07-19 09:08:57 ot_train.py:649 step:5K smpl:589K ep:13K epch:2.70 loss:0.068 grdn:0.372 lr:1.0e-04 updt_s:1.602 data_s:1.381 smp/s:43 mem_gb:52.90
+Training:   8%|▊         | 4800/60000 [3:58:58<46:45:07,  3.05s/step]INFO 2026-07-19 09:18:50 ot_train.py:649 step:5K smpl:614K ep:14K epch:2.81 loss:0.066 grdn:0.377 lr:1.0e-04 updt_s:1.615 data_s:1.348 smp/s:43 mem_gb:52.98
+Training:   8%|▊         | 5000/60000 [4:08:59<45:54:55,  3.01s/step]INFO 2026-07-19 09:28:50 ot_train.py:649 step:5K smpl:640K ep:14K epch:2.93 loss:0.064 grdn:0.352 lr:1.0e-04 updt_s:1.576 data_s:1.421 smp/s:43 mem_gb:52.98
+Training:   9%|▊         | 5200/60000 [4:19:00<46:19:41,  3.04s/step]INFO 2026-07-19 09:38:52 ot_train.py:649 step:5K smpl:666K ep:15K epch:3.05 loss:0.064 grdn:0.354 lr:1.0e-04 updt_s:1.576 data_s:1.427 smp/s:43 mem_gb:52.98
+Training:   9%|▉         | 5400/60000 [4:28:59<44:55:56,  2.96s/step]INFO 2026-07-19 09:48:50 ot_train.py:649 step:5K smpl:691K ep:16K epch:3.17 loss:0.061 grdn:0.323 lr:1.0e-04 updt_s:1.576 data_s:1.413 smp/s:43 mem_gb:52.98
+Training:   9%|▉         | 5600/60000 [4:39:02<47:23:21,  3.14s/step]INFO 2026-07-19 09:58:54 ot_train.py:649 step:6K smpl:717K ep:16K epch:3.28 loss:0.061 grdn:0.336 lr:1.0e-04 updt_s:1.567 data_s:1.447 smp/s:42 mem_gb:52.98
+Training:  10%|▉         | 5800/60000 [4:48:59<43:01:12,  2.86s/step]INFO 2026-07-19 10:08:51 ot_train.py:649 step:6K smpl:742K ep:17K epch:3.40 loss:0.059 grdn:0.338 lr:9.9e-05 updt_s:1.589 data_s:1.389 smp/s:43 mem_gb:52.90
+Training:  10%|█         | 6000/60000 [4:58:50<43:42:38,  2.91s/step]INFO 2026-07-19 10:18:42 ot_train.py:649 step:6K smpl:768K ep:17K epch:3.52 loss:0.061 grdn:0.333 lr:9.9e-05 updt_s:1.526 data_s:1.425 smp/s:43 mem_gb:52.98
+Training:  10%|█         | 6200/60000 [5:08:49<45:06:45,  3.02s/step]INFO 2026-07-19 10:28:41 ot_train.py:649 step:6K smpl:794K ep:18K epch:3.63 loss:0.059 grdn:0.315 lr:9.9e-05 updt_s:1.520 data_s:1.472 smp/s:43 mem_gb:52.98
+Training:  11%|█         | 6400/60000 [5:18:48<43:42:05,  2.94s/step]INFO 2026-07-19 10:38:40 ot_train.py:649 step:6K smpl:819K ep:18K epch:3.75 loss:0.057 grdn:0.316 lr:9.9e-05 updt_s:1.595 data_s:1.395 smp/s:43 mem_gb:52.98
+Training:  11%|█         | 6600/60000 [5:28:51<42:18:38,  2.85s/step]INFO 2026-07-19 10:48:43 ot_train.py:649 step:7K smpl:845K ep:19K epch:3.87 loss:0.058 grdn:0.314 lr:9.9e-05 updt_s:1.564 data_s:1.445 smp/s:43 mem_gb:52.98
+Training:  11%|█▏        | 6800/60000 [5:39:00<47:05:59,  3.19s/step]INFO 2026-07-19 10:58:51 ot_train.py:649 step:7K smpl:870K ep:20K epch:3.99 loss:0.057 grdn:0.294 lr:9.9e-05 updt_s:1.552 data_s:1.486 smp/s:42 mem_gb:52.90
+Training:  12%|█▏        | 7000/60000 [5:48:58<44:34:33,  3.03s/step]INFO 2026-07-19 11:08:50 ot_train.py:649 step:7K smpl:896K ep:20K epch:4.10 loss:0.055 grdn:0.307 lr:9.9e-05 updt_s:1.562 data_s:1.426 smp/s:43 mem_gb:52.98
+Training:  12%|█▏        | 7200/60000 [5:59:00<44:27:27,  3.03s/step]INFO 2026-07-19 11:18:51 ot_train.py:649 step:7K smpl:922K ep:21K epch:4.22 loss:0.056 grdn:0.320 lr:9.9e-05 updt_s:1.535 data_s:1.469 smp/s:43 mem_gb:52.98
+Training:  12%|█▏        | 7400/60000 [6:09:02<43:55:04,  3.01s/step]INFO 2026-07-19 11:28:54 ot_train.py:649 step:7K smpl:947K ep:21K epch:4.34 loss:0.053 grdn:0.298 lr:9.9e-05 updt_s:1.530 data_s:1.478 smp/s:43 mem_gb:52.98
+Training:  13%|█▎        | 7600/60000 [6:19:10<48:51:17,  3.36s/step]INFO 2026-07-19 11:39:01 ot_train.py:649 step:8K smpl:973K ep:22K epch:4.45 loss:0.054 grdn:0.293 lr:9.8e-05 updt_s:1.561 data_s:1.471 smp/s:42 mem_gb:52.98
+Training:  13%|█▎        | 7800/60000 [6:31:15<51:57:42,  3.58s/step]INFO 2026-07-19 11:51:06 ot_train.py:649 step:8K smpl:998K ep:23K epch:4.57 loss:0.054 grdn:0.292 lr:9.8e-05 updt_s:1.798 data_s:1.820 smp/s:35 mem_gb:52.98
+Training:  13%|█▎        | 8000/60000 [6:41:52<39:17:51,  2.72s/step]INFO 2026-07-19 12:01:44 ot_train.py:649 step:8K smpl:1M ep:23K epch:4.69 loss:0.052 grdn:0.298 lr:9.8e-05 updt_s:1.690 data_s:1.493 smp/s:40 mem_gb:52.90
+Training:  14%|█▎        | 8200/60000 [6:53:23<51:35:45,  3.59s/step]INFO 2026-07-19 12:13:15 ot_train.py:649 step:8K smpl:1M ep:24K epch:4.81 loss:0.051 grdn:0.287 lr:9.8e-05 updt_s:1.739 data_s:1.708 smp/s:37 mem_gb:52.98
+Training:  14%|█▍        | 8400/60000 [7:04:29<41:23:41,  2.89s/step]INFO 2026-07-19 12:24:21 ot_train.py:649 step:8K smpl:1M ep:24K epch:4.92 loss:0.051 grdn:0.287 lr:9.8e-05 updt_s:1.702 data_s:1.623 smp/s:38 mem_gb:52.98
+Training:  14%|█▍        | 8600/60000 [7:14:41<51:30:06,  3.61s/step]INFO 2026-07-19 12:34:33 ot_train.py:649 step:9K smpl:1M ep:25K epch:5.04 loss:0.051 grdn:0.288 lr:9.8e-05 updt_s:1.640 data_s:1.412 smp/s:42 mem_gb:52.98
+Training:  15%|█▍        | 8800/60000 [7:26:31<46:37:46,  3.28s/step]INFO 2026-07-19 12:46:23 ot_train.py:649 step:9K smpl:1M ep:25K epch:5.16 loss:0.050 grdn:0.287 lr:9.8e-05 updt_s:1.779 data_s:1.765 smp/s:36 mem_gb:52.98
+Training:  15%|█▌        | 9000/60000 [7:36:19<52:49:40,  3.73s/step]INFO 2026-07-19 12:56:11 ot_train.py:649 step:9K smpl:1M ep:26K epch:5.28 loss:0.049 grdn:0.272 lr:9.7e-05 updt_s:1.565 data_s:1.370 smp/s:44 mem_gb:52.98
+Training:  15%|█▌        | 9200/60000 [7:48:18<52:29:25,  3.72s/step]INFO 2026-07-19 13:08:09 ot_train.py:649 step:9K smpl:1M ep:27K epch:5.39 loss:0.048 grdn:0.279 lr:9.7e-05 updt_s:1.773 data_s:1.810 smp/s:36 mem_gb:52.90
+Training:  16%|█▌        | 9400/60000 [7:58:38<43:57:55,  3.13s/step]INFO 2026-07-19 13:18:30 ot_train.py:649 step:9K smpl:1M ep:27K epch:5.51 loss:0.051 grdn:0.307 lr:9.7e-05 updt_s:1.642 data_s:1.454 smp/s:41 mem_gb:52.98
+Training:  16%|█▌        | 9600/60000 [8:10:18<50:39:31,  3.62s/step]INFO 2026-07-19 13:30:09 ot_train.py:649 step:10K smpl:1M ep:28K epch:5.63 loss:0.048 grdn:0.283 lr:9.7e-05 updt_s:1.761 data_s:1.730 smp/s:37 mem_gb:52.98
+Training:  16%|█▋        | 9800/60000 [8:21:25<34:31:50,  2.48s/step]INFO 2026-07-19 13:41:17 ot_train.py:649 step:10K smpl:1M ep:28K epch:5.74 loss:0.046 grdn:0.271 lr:9.7e-05 updt_s:1.702 data_s:1.628 smp/s:38 mem_gb:52.98
+Training:  17%|█▋        | 10000/60000 [8:31:41<49:56:09,  3.60s/step]INFO 2026-07-19 13:51:33 ot_train.py:649 step:10K smpl:1M ep:29K epch:5.86 loss:0.046 grdn:0.266 lr:9.6e-05 updt_s:1.633 data_s:1.441 smp/s:42 mem_gb:52.98
+Training:  17%|█▋        | 10200/60000 [8:43:36<45:05:31,  3.26s/step]INFO 2026-07-19 14:03:28 ot_train.py:649 step:10K smpl:1M ep:29K epch:5.98 loss:0.045 grdn:0.267 lr:9.6e-05 updt_s:1.809 data_s:1.759 smp/s:36 mem_gb:52.90
+Training:  17%|█▋        | 10400/60000 [8:53:10<46:24:36,  3.37s/step]INFO 2026-07-19 14:13:01 ot_train.py:649 step:10K smpl:1M ep:30K epch:6.10 loss:0.044 grdn:0.265 lr:9.6e-05 updt_s:1.549 data_s:1.313 smp/s:45 mem_gb:52.98
+Training:  18%|█▊        | 10600/60000 [9:04:59<49:55:11,  3.64s/step]INFO 2026-07-19 14:24:51 ot_train.py:649 step:11K smpl:1M ep:31K epch:6.21 loss:0.043 grdn:0.260 lr:9.6e-05 updt_s:1.780 data_s:1.760 smp/s:36 mem_gb:52.98
+Training:  18%|█▊        | 10800/60000 [9:14:54<34:47:29,  2.55s/step]INFO 2026-07-19 14:34:45 ot_train.py:649 step:11K smpl:1M ep:31K epch:6.33 loss:0.044 grdn:0.267 lr:9.6e-05 updt_s:1.566 data_s:1.402 smp/s:43 mem_gb:52.98
+Training:  18%|█▊        | 11000/60000 [9:25:56<48:05:14,  3.53s/step]INFO 2026-07-19 14:45:47 ot_train.py:649 step:11K smpl:1M ep:32K epch:6.45 loss:0.045 grdn:0.265 lr:9.5e-05 updt_s:1.686 data_s:1.616 smp/s:39 mem_gb:52.98
+Training:  19%|█▊        | 11200/60000 [9:37:22<37:00:09,  2.73s/step]INFO 2026-07-19 14:57:14 ot_train.py:649 step:11K smpl:1M ep:32K epch:6.57 loss:0.045 grdn:0.269 lr:9.5e-05 updt_s:1.732 data_s:1.692 smp/s:37 mem_gb:52.98
+Training:  19%|█▉        | 11400/60000 [9:47:36<47:30:49,  3.52s/step]INFO 2026-07-19 15:07:27 ot_train.py:649 step:11K smpl:1M ep:33K epch:6.68 loss:0.042 grdn:0.263 lr:9.5e-05 updt_s:1.630 data_s:1.432 smp/s:42 mem_gb:52.90
+Training:  19%|█▉        | 11600/60000 [9:59:27<42:30:45,  3.16s/step]INFO 2026-07-19 15:19:19 ot_train.py:649 step:12K smpl:1M ep:34K epch:6.80 loss:0.042 grdn:0.268 lr:9.5e-05 updt_s:1.768 data_s:1.783 smp/s:36 mem_gb:52.98
+Training:  20%|█▉        | 11800/60000 [10:08:45<40:05:20,  2.99s/step]INFO 2026-07-19 15:28:36 ot_train.py:649 step:12K smpl:2M ep:34K epch:6.92 loss:0.040 grdn:0.250 lr:9.4e-05 updt_s:1.502 data_s:1.280 smp/s:46 mem_gb:52.98
+Training:  20%|█▉        | 11823/60000 [10:10:00<44:44:21,  3.34s/step]

Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K/wandb/run-20260719_051936-2mdgc559/files/requirements.txt

@@ -0,0 +1,117 @@
+pyarrow==25.0.0
+importlib_metadata==9.0.0
+wheel==0.47.0
+cloudpickle==3.1.2
+nvidia-cusolver-cu12==11.7.3.90
+multiprocess==0.70.19
+torch==2.11.0+cu128
+nvidia-nvjitlink-cu12==12.8.93
+fsspec==2026.2.0
+pytz==2026.2
+hf-xet==1.5.1
+tokenizers==0.22.2
+opencv-python-headless==4.13.0.92
+cuda-pathfinder==1.2.2
+frozenlist==1.8.0
+smmap==5.0.3
+httpcore==1.0.9
+nvidia-nvshmem-cu12==3.4.5
+nvidia-cuda-nvrtc-cu12==12.8.93
+annotated-types==0.7.0
+platformdirs==4.10.0
+click==8.4.2
+pyyaml-include==1.4.1
+MarkupSafe==3.0.3
+Jinja2==3.1.6
+urllib3==2.7.0
+nvidia-curand-cu12==10.3.9.90
+certifi==2026.6.17
+nvidia-cuda-runtime-cu12==12.8.90
+torchcodec==0.11.1
+toml==0.10.2
+av==15.1.0
+nvidia-cufft-cu12==11.3.3.83
+attrs==26.1.0
+lerobot==0.6.0
+PyYAML==6.0.3
+safetensors==0.8.0
+diffusers==0.35.2
+requests==2.34.2
+nvidia-nvtx-cu12==12.8.90
+charset-normalizer==3.4.9
+networkx==3.6.1
+cmake==4.1.3
+pydantic_core==2.46.4
+timm==1.0.28
+dm-tree==0.1.10
+nvidia-cufile-cu12==1.13.1.3
+termcolor==3.3.0
+multidict==6.7.1
+filelock==3.29.0
+tzdata==2026.3
+aiohappyeyeballs==2.7.1
+httpx==0.28.1
+nvidia-nccl-cu12==2.28.9
+Pygments==2.20.0
+einops==0.8.2
+nvidia-cusparse-cu12==12.5.8.93
+torchvision==0.26.0+cu128
+dill==0.4.1
+GitPython==3.1.51
+nvidia-cuda-cupti-cu12==12.8.90
+decord==0.6.0
+sympy==1.14.0
+cuda-toolkit==12.8.1
+python-dateutil==2.9.0.post0
+shellingham==1.5.4
+propcache==0.5.2
+Farama-Notifications==0.0.6
+pillow==12.2.0
+cuda-bindings==12.9.4
+huggingface_hub==1.23.0
+pandas==2.3.3
+typing-inspect==0.9.0
+packaging==25.0
+rich==15.0.0
+nvidia-cusparselt-cu12==0.7.1
+setuptools==78.1.0
+wrapt==2.2.2
+mypy_extensions==1.1.0
+tqdm==4.68.4
+draccus==0.10.0
+aiosignal==1.4.0
+mpmath==1.3.0
+numpy==2.2.6
+typing_extensions==4.15.0
+psutil==7.2.2
+mdurl==0.1.2
+nvidia-cublas-cu12==12.8.4.1
+absl-py==2.5.0
+gitdb==4.0.12
+transformers==5.5.4
+accelerate==1.14.0
+pydantic==2.13.4
+pip==26.1.2
+markdown-it-py==4.2.0
+typer==0.26.8
+triton==3.6.0
+wandb==0.27.2
+regex==2026.7.10
+yarl==1.24.2
+jsonlines==4.0.0
+mergedeep==1.3.4
+aiohttp==3.14.1
+datasets==4.8.5
+gymnasium==1.3.0
+peft==0.19.1
+zipp==4.1.0
+idna==3.18
+nvidia-cudnn-cu12==9.19.0.56
+h11==0.16.0
+sentry-sdk==2.65.0
+annotated-doc==0.0.4
+protobuf==7.35.1
+typing-inspection==0.4.2
+anyio==4.14.2
+xxhash==3.8.1
+six==1.17.0

Abs_6D + DeltaEE/RKD_TimewarpVAE/RSCLstyle_cosine_60K/wandb/run-20260719_051936-2mdgc559/files/wandb-summary.json

@@ -0,0 +1 @@
+{"train/training_phase_index":0,"train/weighted_rkd_loss":0.008915836922824383,"_step":11800,"_runtime":36616,"train/gpu_mem_gb":52.9777626991272,"train/flow_matching_loss":0.04272415488958359,"train/epochs":6.916796036031085,"train/episodes":34099.804457633254,"train/weighted_flow_matching_loss":0.04272415488958359,"train/rkd_fm_loss_weight":1,"train/rkd_distance_loss":0.002336467383429408,"train/rkd_angle_loss":0.003743645502254367,"train/grad_norm":0.25020759910345075,"train/rkd_student_pooling_id":1,"train/rkd_teacher_latent_norm":1,"train/update_s":1.5015871298756975,"train/steps":11800,"train/training_phase_until_step":60000,"train/rkd_loss":0.009823758155107498,"train/lr":9.436028069070851e-05,"train/rkd_runtime_enabled":1,"train/dataloading_s":1.2798837350898975,"train/rkd_loss_weight":0.9075790628512187,"train/training_phase_start_step":0,"train/samples_per_s":46.01881745814486,"_timestamp":1.7844425167144637e+09,"train/samples":1510400,"_wandb":{"runtime":36616},"train/loss":0.051639992743730545,"train/training_phase_trainable_params":1.622090881e+09,"train/rkd_student_feature_norm":5.604428768157959}

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/Baseline/wandb/debug-internal.log

@@ -0,0 +1,89 @@
+{"time":"2026-07-19T23:04:31.813242533+09:00","level":"INFO","msg":"wandb-core"}
+{"time":"2026-07-19T23:04:31.81382052+09:00","level":"INFO","msg":"stream: starting","core version":"0.27.2"}
+{"time":"2026-07-19T23:04:32.344730077+09:00","level":"INFO","msg":"stream: created new stream","id":"ajlv9qc5"}
+{"time":"2026-07-19T23:04:32.344968538+09:00","level":"INFO","msg":"stream: started"}
+{"time":"2026-07-19T23:04:32.345008264+09:00","level":"INFO","msg":"writer: started","stream_id":"ajlv9qc5"}
+{"time":"2026-07-19T23:04:32.34508556+09:00","level":"INFO","msg":"sender: started"}
+{"time":"2026-07-19T23:04:32.345059474+09:00","level":"INFO","msg":"handler: started"}
+{"time":"2026-07-19T23:04:32.860950364+09:00","level":"INFO","msg":"filestream: sending request","total_files":1,"console_offset":0,"console_lines":1}
+{"time":"2026-07-19T23:04:33.243629437+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:04:47.86138057+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":0,"events_lines":1,"console_offset":1,"console_lines":15,"uploaded_len":2}
+{"time":"2026-07-19T23:04:48.18030455+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:02.861828226+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":1,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:03.262381224+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:17.861165647+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":3,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:18.148957301+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:32.861865868+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":5,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:33.23129356+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:47.861003074+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":7,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:48.246172948+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:02.86159547+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":9,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:03.269857393+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:17.861688399+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":11,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:18.289365504+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:32.861609778+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":13,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:33.315776295+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:47.861626977+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":15,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:48.362124826+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:02.861392553+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":17,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:03.328352457+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:17.860930238+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":19,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:18.180524239+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:32.861289033+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":21,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:33.243317934+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:47.861114611+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":23,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:48.165733874+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:02.861065838+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":25,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:03.266581507+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:17.861924025+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":27,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:18.263160805+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:32.861265948+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":29,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:33.221054916+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:47.861669879+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":31,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:48.256566549+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:02.861346575+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":33,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:03.287396886+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:17.861498281+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":35,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:18.295465227+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:32.861265597+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":37,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:33.31683242+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:47.861651377+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":39,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:48.27119445+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:02.861398324+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":41,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:03.248283028+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:17.861807031+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":43,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:18.299030334+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:32.861482457+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":45,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:33.270874405+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:47.860914999+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":47,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:48.152543457+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:02.860838398+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":49,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:03.231975975+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:17.861078604+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":51,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:18.196012531+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:32.861061466+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":53,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:33.283575595+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:47.86093486+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":55,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:48.266688556+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:02.861659187+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":57,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:03.237623334+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:17.861674818+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":59,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:18.287218148+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:32.861204007+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":61,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:33.335261363+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:47.861786253+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":63,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:48.129544067+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:02.861912949+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":65,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:03.229115242+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:17.861389905+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":67,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:18.15830388+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:32.861324836+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":69,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:33.296722029+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:47.861380357+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":71,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:48.223402479+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:02.861782715+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":73,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:03.208286934+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:17.860889195+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":75,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:18.181581208+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:32.861876269+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":77,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:33.282595209+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/Baseline/wandb/debug.log

@@ -0,0 +1,19 @@
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_setup.py:_flush():81] Configure stats pid to 2036848
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/Baseline/wandb/run-20260719_230431-ajlv9qc5/logs/debug.log
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/Baseline/wandb/run-20260719_230431-ajlv9qc5/logs/debug-internal.log
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_init.py:init():766] calling init triggers
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/Baseline', 'job_name': 'INSIGHT_RD_baseline', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 500, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_init.py:init():814] starting backend
+2026-07-19 23:04:31,802 INFO    MainThread:2036848 [wandb_init.py:init():829] sending inform_init request
+2026-07-19 23:04:32,346 INFO    MainThread:2036848 [wandb_init.py:init():834] backend started and connected
+2026-07-19 23:04:32,348 INFO    MainThread:2036848 [wandb_init.py:init():904] updated telemetry
+2026-07-19 23:04:32,356 INFO    MainThread:2036848 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-19 23:04:32,748 INFO    MainThread:2036848 [wandb_init.py:init():972] starting run threads in backend
+2026-07-19 23:04:32,856 INFO    MainThread:2036848 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-19 23:04:32,857 INFO    MainThread:2036848 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-19 23:04:32,857 INFO    MainThread:2036848 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-19 23:04:32,857 INFO    MainThread:2036848 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-19 23:04:32,859 INFO    MainThread:2036848 [wandb_init.py:init():1010] run started, returning control to user process

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/Baseline/wandb/run-20260719_230431-ajlv9qc5/files/output.log

@@ -0,0 +1,17 @@
+INFO 2026-07-19 23:04:32 db_utils.py:121 Logs will be synced with wandb.
+INFO 2026-07-19 23:04:32 db_utils.py:122 Track this run --> https://wandb.ai/minje227_hyu-hanyang-university/lerobot/runs/ajlv9qc5
+INFO 2026-07-19 23:04:32 ot_train.py:298 Creating dataset
+INFO 2026-07-19 23:04:36 ot_train.py:332 Creating policy
+Fetching 27 files: 100%|██████████| 27/27 [00:00<00:00, 1586.64it/s]
+`torch_dtype` is deprecated! Use `dtype` instead!
+Loading weights: 100%|██████████| 1031/1031 [00:00<00:00, 1493.53it/s]
+INFO 2026-07-19 23:04:43 ot_train.py:405 Creating optimizer and scheduler
+INFO 2026-07-19 23:04:43 ot_train.py:437 Output dir: /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/Baseline
+INFO 2026-07-19 23:04:43 ot_train.py:444 cfg.steps=60000 (60K)
+INFO 2026-07-19 23:04:43 ot_train.py:445 dataset.num_frames=218367 (218K)
+INFO 2026-07-19 23:04:43 ot_train.py:446 dataset.num_episodes=4930
+INFO 2026-07-19 23:04:43 ot_train.py:449 Effective batch size: 128 x 1 = 128
+INFO 2026-07-19 23:04:43 ot_train.py:450 num_learnable_params=1620515968 (2B)
+INFO 2026-07-19 23:04:43 ot_train.py:451 num_total_params=3144016000 (3B)
+Training:   0%|          | 0/60000 [00:00<?, ?step/s]INFO 2026-07-19 23:04:43 ot_train.py:602 Start offline training on a fixed dataset, with effective batch size: 128
+Training:   0%|          | 184/60000 [09:50<51:27:10,  3.10s/step]

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/Baseline/wandb/run-20260719_230431-ajlv9qc5/logs/debug-internal.log

@@ -0,0 +1,89 @@
+{"time":"2026-07-19T23:04:31.813242533+09:00","level":"INFO","msg":"wandb-core"}
+{"time":"2026-07-19T23:04:31.81382052+09:00","level":"INFO","msg":"stream: starting","core version":"0.27.2"}
+{"time":"2026-07-19T23:04:32.344730077+09:00","level":"INFO","msg":"stream: created new stream","id":"ajlv9qc5"}
+{"time":"2026-07-19T23:04:32.344968538+09:00","level":"INFO","msg":"stream: started"}
+{"time":"2026-07-19T23:04:32.345008264+09:00","level":"INFO","msg":"writer: started","stream_id":"ajlv9qc5"}
+{"time":"2026-07-19T23:04:32.34508556+09:00","level":"INFO","msg":"sender: started"}
+{"time":"2026-07-19T23:04:32.345059474+09:00","level":"INFO","msg":"handler: started"}
+{"time":"2026-07-19T23:04:32.860950364+09:00","level":"INFO","msg":"filestream: sending request","total_files":1,"console_offset":0,"console_lines":1}
+{"time":"2026-07-19T23:04:33.243629437+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:04:47.86138057+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":0,"events_lines":1,"console_offset":1,"console_lines":15,"uploaded_len":2}
+{"time":"2026-07-19T23:04:48.18030455+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:02.861828226+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":1,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:03.262381224+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:17.861165647+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":3,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:18.148957301+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:32.861865868+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":5,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:33.23129356+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:47.861003074+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":7,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:48.246172948+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:02.86159547+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":9,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:03.269857393+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:17.861688399+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":11,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:18.289365504+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:32.861609778+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":13,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:33.315776295+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:47.861626977+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":15,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:48.362124826+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:02.861392553+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":17,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:03.328352457+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:17.860930238+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":19,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:18.180524239+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:32.861289033+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":21,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:33.243317934+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:47.861114611+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":23,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:48.165733874+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:02.861065838+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":25,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:03.266581507+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:17.861924025+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":27,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:18.263160805+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:32.861265948+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":29,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:33.221054916+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:47.861669879+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":31,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:48.256566549+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:02.861346575+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":33,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:03.287396886+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:17.861498281+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":35,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:18.295465227+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:32.861265597+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":37,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:33.31683242+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:47.861651377+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":39,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:48.27119445+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:02.861398324+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":41,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:03.248283028+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:17.861807031+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":43,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:18.299030334+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:32.861482457+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":45,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:33.270874405+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:47.860914999+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":47,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:48.152543457+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:02.860838398+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":49,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:03.231975975+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:17.861078604+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":51,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:18.196012531+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:32.861061466+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":53,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:33.283575595+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:47.86093486+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":55,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:48.266688556+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:02.861659187+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":57,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:03.237623334+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:17.861674818+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":59,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:18.287218148+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:32.861204007+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":61,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:33.335261363+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:47.861786253+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":63,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:48.129544067+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:02.861912949+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":65,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:03.229115242+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:17.861389905+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":67,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:18.15830388+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:32.861324836+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":69,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:33.296722029+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:47.861380357+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":71,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:48.223402479+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:02.861782715+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":73,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:03.208286934+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:17.860889195+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":75,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:18.181581208+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:32.861876269+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":77,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:33.282595209+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/Baseline/wandb/run-20260719_230431-ajlv9qc5/logs/debug.log

@@ -0,0 +1,19 @@
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_setup.py:_flush():81] Configure stats pid to 2036848
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/Baseline/wandb/run-20260719_230431-ajlv9qc5/logs/debug.log
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/Baseline/wandb/run-20260719_230431-ajlv9qc5/logs/debug-internal.log
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_init.py:init():766] calling init triggers
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/Baseline', 'job_name': 'INSIGHT_RD_baseline', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 500, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-19 23:04:31,586 INFO    MainThread:2036848 [wandb_init.py:init():814] starting backend
+2026-07-19 23:04:31,802 INFO    MainThread:2036848 [wandb_init.py:init():829] sending inform_init request
+2026-07-19 23:04:32,346 INFO    MainThread:2036848 [wandb_init.py:init():834] backend started and connected
+2026-07-19 23:04:32,348 INFO    MainThread:2036848 [wandb_init.py:init():904] updated telemetry
+2026-07-19 23:04:32,356 INFO    MainThread:2036848 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-19 23:04:32,748 INFO    MainThread:2036848 [wandb_init.py:init():972] starting run threads in backend
+2026-07-19 23:04:32,856 INFO    MainThread:2036848 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-19 23:04:32,857 INFO    MainThread:2036848 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-19 23:04:32,857 INFO    MainThread:2036848 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-19 23:04:32,857 INFO    MainThread:2036848 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-19 23:04:32,859 INFO    MainThread:2036848 [wandb_init.py:init():1010] run started, returning control to user process

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_10K/wandb/debug-internal.log

@@ -0,0 +1,89 @@
+{"time":"2026-07-19T23:04:35.960247185+09:00","level":"INFO","msg":"wandb-core"}
+{"time":"2026-07-19T23:04:35.960754745+09:00","level":"INFO","msg":"stream: starting","core version":"0.27.2"}
+{"time":"2026-07-19T23:04:36.639007187+09:00","level":"INFO","msg":"stream: created new stream","id":"7ob3zmbd"}
+{"time":"2026-07-19T23:04:36.639131622+09:00","level":"INFO","msg":"handler: started"}
+{"time":"2026-07-19T23:04:36.639327255+09:00","level":"INFO","msg":"stream: started"}
+{"time":"2026-07-19T23:04:36.639379585+09:00","level":"INFO","msg":"writer: started","stream_id":"7ob3zmbd"}
+{"time":"2026-07-19T23:04:36.63950064+09:00","level":"INFO","msg":"sender: started"}
+{"time":"2026-07-19T23:04:37.267802748+09:00","level":"INFO","msg":"filestream: sending request","total_files":1,"console_offset":0,"console_lines":1}
+{"time":"2026-07-19T23:04:37.751082168+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:04:52.268769544+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":0,"events_lines":2,"console_offset":1,"console_lines":15,"uploaded_len":2}
+{"time":"2026-07-19T23:04:52.573885823+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:07.268389868+09:00","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":2,"events_lines":2}
+{"time":"2026-07-19T23:05:07.606875177+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:22.268050665+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":4,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:22.715586991+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:37.268124734+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":5,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:37.645658129+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:52.268366171+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":8,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:52.557736918+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:07.268106669+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":10,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:07.621886852+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:22.268682413+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":11,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:22.726555788+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:37.268359275+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":14,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:37.990790739+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:52.268295041+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":16,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:52.682552499+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:07.268549568+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":18,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:07.673104633+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:22.268615978+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":20,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:22.683091689+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:37.268682698+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":22,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:37.681312757+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:52.268339864+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":23,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:52.640850521+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:07.268194607+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":26,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:07.702499163+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:22.268328372+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":28,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:22.715021671+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:37.268193555+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":30,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:37.626022328+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:52.267803239+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":32,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:52.672064821+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:07.268317912+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":34,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:07.74037807+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:22.267592835+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":36,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:22.710706328+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:37.268099216+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":37,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:37.700480492+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:52.268238695+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":40,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:52.69520669+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:07.268072235+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":42,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:07.662257161+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:22.268470825+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":44,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:22.686271986+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:37.268174148+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":46,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:37.575139887+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:52.268481192+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":48,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:52.595653047+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:07.268211489+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":50,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:07.628512238+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:22.267761494+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":51,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:22.673196918+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:37.267983859+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":54,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:37.658592212+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:52.26826316+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":56,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:52.735918521+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:07.267639278+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":58,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:07.654674405+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:22.26833269+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":59,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:22.711237184+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:37.268231295+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":62,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:37.740143018+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:52.268083109+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":63,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:52.728951786+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:07.267809308+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":66,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:07.573546038+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:22.267745129+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":68,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:22.577801422+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:37.26817292+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":70,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:37.709547258+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:52.268335918+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":72,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:52.716685388+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:07.268130681+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":74,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:07.706210103+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:22.268283925+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":76,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:22.570987679+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:37.268090837+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":78,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:37.638228428+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_10K/wandb/run-20260719_230435-7ob3zmbd/files/wandb-metadata.json

@@ -0,0 +1,91 @@
+{
+  "os": "Linux-6.8.0-107-generic-x86_64-with-glibc2.39",
+  "python": "CPython 3.12.13",
+  "startedAt": "2026-07-19T14:04:35.721770Z",
+  "args": [
+    "--config_path=/tmp/insight_lerobot_cfg_49uglsrc/LAPstyle_RKD_linear_decay_10K.yaml",
+    "--batch_size=128",
+    "--output_dir=/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_10K"
+  ],
+  "program": "/home/ext_minje/clvla/benchmarks/INSIGHT/my_scripts/lerobot_train_yaml.py",
+  "git": {
+    "remote": "git@github.com:paragon7060/clvla.git",
+    "commit": "a9df08f7da2d8eb1fa7f28708b4eb6f6dc81b1f9"
+  },
+  "root": "/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_10K",
+  "host": "DGX-H200-01",
+  "executable": "/home/ext_minje/miniforge3/envs/lerobot060_groot/bin/python",
+  "cpu_count": 112,
+  "cpu_count_logical": 224,
+  "gpu": "NVIDIA H200",
+  "gpu_count": 8,
+  "disk": {
+    "/": {
+      "total": "1888425144320",
+      "used": "813929852928"
+    }
+  },
+  "memory": {
+    "total": "2164177309696"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-1e531a45-480e-3c1a-e9a0-2ba2cc27b0a3"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-5d3d5ae2-ffde-89d9-8818-dc19e11ad2fb"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-305c0dae-dcf5-d7bb-7dd3-5477d2194275"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-ed8b7920-8881-4e8f-3e88-be1bbecafa21"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-a91176db-55f2-b284-bbbf-56626596947d"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-922ff6d5-a948-205a-0426-dbee495fc3dd"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-1fde7c75-18fe-e3db-a4c1-85e67522f55f"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-a9b1c49b-d527-97d8-ce91-7e50bd02154c"
+    }
+  ],
+  "cudaVersion": "12.9",
+  "writerId": "f3wbl1tz4dd8p8pxgdll61uby3zd2y4t"
+}

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_10K/wandb/run-20260719_230435-7ob3zmbd/logs/debug-internal.log

@@ -0,0 +1,89 @@
+{"time":"2026-07-19T23:04:35.960247185+09:00","level":"INFO","msg":"wandb-core"}
+{"time":"2026-07-19T23:04:35.960754745+09:00","level":"INFO","msg":"stream: starting","core version":"0.27.2"}
+{"time":"2026-07-19T23:04:36.639007187+09:00","level":"INFO","msg":"stream: created new stream","id":"7ob3zmbd"}
+{"time":"2026-07-19T23:04:36.639131622+09:00","level":"INFO","msg":"handler: started"}
+{"time":"2026-07-19T23:04:36.639327255+09:00","level":"INFO","msg":"stream: started"}
+{"time":"2026-07-19T23:04:36.639379585+09:00","level":"INFO","msg":"writer: started","stream_id":"7ob3zmbd"}
+{"time":"2026-07-19T23:04:36.63950064+09:00","level":"INFO","msg":"sender: started"}
+{"time":"2026-07-19T23:04:37.267802748+09:00","level":"INFO","msg":"filestream: sending request","total_files":1,"console_offset":0,"console_lines":1}
+{"time":"2026-07-19T23:04:37.751082168+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:04:52.268769544+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":0,"events_lines":2,"console_offset":1,"console_lines":15,"uploaded_len":2}
+{"time":"2026-07-19T23:04:52.573885823+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:07.268389868+09:00","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":2,"events_lines":2}
+{"time":"2026-07-19T23:05:07.606875177+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:22.268050665+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":4,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:22.715586991+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:37.268124734+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":5,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:37.645658129+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:52.268366171+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":8,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:52.557736918+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:07.268106669+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":10,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:07.621886852+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:22.268682413+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":11,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:22.726555788+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:37.268359275+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":14,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:37.990790739+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:52.268295041+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":16,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:52.682552499+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:07.268549568+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":18,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:07.673104633+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:22.268615978+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":20,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:22.683091689+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:37.268682698+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":22,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:37.681312757+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:52.268339864+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":23,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:52.640850521+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:07.268194607+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":26,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:07.702499163+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:22.268328372+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":28,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:22.715021671+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:37.268193555+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":30,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:37.626022328+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:52.267803239+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":32,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:52.672064821+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:07.268317912+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":34,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:07.74037807+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:22.267592835+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":36,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:22.710706328+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:37.268099216+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":37,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:37.700480492+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:52.268238695+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":40,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:52.69520669+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:07.268072235+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":42,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:07.662257161+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:22.268470825+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":44,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:22.686271986+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:37.268174148+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":46,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:37.575139887+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:52.268481192+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":48,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:52.595653047+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:07.268211489+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":50,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:07.628512238+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:22.267761494+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":51,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:22.673196918+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:37.267983859+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":54,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:37.658592212+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:52.26826316+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":56,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:52.735918521+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:07.267639278+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":58,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:07.654674405+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:22.26833269+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":59,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:22.711237184+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:37.268231295+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":62,"events_lines":1,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:37.740143018+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:52.268083109+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":63,"events_lines":3,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:52.728951786+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:07.267809308+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":66,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:07.573546038+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:22.267745129+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":68,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:22.577801422+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:37.26817292+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":70,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:37.709547258+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:52.268335918+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":72,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:52.716685388+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:07.268130681+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":74,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:07.706210103+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:22.268283925+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":76,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:22.570987679+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:37.268090837+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":78,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:37.638228428+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_10K/wandb/run-20260719_230435-7ob3zmbd/logs/debug.log

@@ -0,0 +1,19 @@
+2026-07-19 23:04:35,723 INFO    MainThread:2036850 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-19 23:04:35,723 INFO    MainThread:2036850 [wandb_setup.py:_flush():81] Configure stats pid to 2036850
+2026-07-19 23:04:35,723 INFO    MainThread:2036850 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-19 23:04:35,723 INFO    MainThread:2036850 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_10K/wandb/run-20260719_230435-7ob3zmbd/logs/debug.log
+2026-07-19 23:04:35,723 INFO    MainThread:2036850 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_10K/wandb/run-20260719_230435-7ob3zmbd/logs/debug-internal.log
+2026-07-19 23:04:35,723 INFO    MainThread:2036850 [wandb_init.py:init():766] calling init triggers
+2026-07-19 23:04:35,723 INFO    MainThread:2036850 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot_rkd', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_steps': 6000, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False, 'rkd_enabled': True, 'rkd_vae_checkpoint': '/home/ext_minje/groot_insight/TimewarpVAE_ckpt/multispace_gripper_dedup_batch_ablation_v1/b512/RD/K08_z16_g4_h256/checkpoint-step-20000-epoch-46.84', 'rkd_student_pooling': 'attention', 'rkd_student_attention_hidden_dim': 512, 'rkd_student_attention_output_dim': 512, 'rkd_fm_loss_weight': 1.0, 'rkd_loss_weight': 1.0, 'rkd_distance_weight': 1.0, 'rkd_angle_weight': 2.0}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_10K', 'job_name': 'INSIGHT_RD_RKD_TimewarpVAE_LAPstyle_10K', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 6000, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-19 23:04:35,723 INFO    MainThread:2036850 [wandb_init.py:init():814] starting backend
+2026-07-19 23:04:35,947 INFO    MainThread:2036850 [wandb_init.py:init():829] sending inform_init request
+2026-07-19 23:04:36,641 INFO    MainThread:2036850 [wandb_init.py:init():834] backend started and connected
+2026-07-19 23:04:36,645 INFO    MainThread:2036850 [wandb_init.py:init():904] updated telemetry
+2026-07-19 23:04:36,657 INFO    MainThread:2036850 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-19 23:04:37,058 INFO    MainThread:2036850 [wandb_init.py:init():972] starting run threads in backend
+2026-07-19 23:04:37,260 INFO    MainThread:2036850 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-19 23:04:37,260 INFO    MainThread:2036850 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-19 23:04:37,260 INFO    MainThread:2036850 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-19 23:04:37,260 INFO    MainThread:2036850 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-19 23:04:37,264 INFO    MainThread:2036850 [wandb_init.py:init():1010] run started, returning control to user process

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_6K/wandb/debug.log

@@ -0,0 +1,19 @@
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_setup.py:_flush():81] Configure stats pid to 2036851
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_6K/wandb/run-20260719_230435-bvj1u4hd/logs/debug.log
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_6K/wandb/run-20260719_230435-bvj1u4hd/logs/debug-internal.log
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_init.py:init():766] calling init triggers
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot_rkd', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_steps': 6000, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False, 'rkd_enabled': True, 'rkd_vae_checkpoint': '/home/ext_minje/groot_insight/TimewarpVAE_ckpt/multispace_gripper_dedup_batch_ablation_v1/b512/RD/K08_z16_g4_h256/checkpoint-step-20000-epoch-46.84', 'rkd_student_pooling': 'attention', 'rkd_student_attention_hidden_dim': 512, 'rkd_student_attention_output_dim': 512, 'rkd_fm_loss_weight': 1.0, 'rkd_loss_weight': 1.0, 'rkd_distance_weight': 1.0, 'rkd_angle_weight': 2.0}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_6K', 'job_name': 'INSIGHT_RD_RKD_TimewarpVAE_LAPstyle_6K', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 6000, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_init.py:init():814] starting backend
+2026-07-19 23:04:35,830 INFO    MainThread:2036851 [wandb_init.py:init():829] sending inform_init request
+2026-07-19 23:04:36,541 INFO    MainThread:2036851 [wandb_init.py:init():834] backend started and connected
+2026-07-19 23:04:36,546 INFO    MainThread:2036851 [wandb_init.py:init():904] updated telemetry
+2026-07-19 23:04:36,557 INFO    MainThread:2036851 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-19 23:04:37,200 INFO    MainThread:2036851 [wandb_init.py:init():972] starting run threads in backend
+2026-07-19 23:04:37,351 INFO    MainThread:2036851 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-19 23:04:37,353 INFO    MainThread:2036851 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-19 23:04:37,353 INFO    MainThread:2036851 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-19 23:04:37,353 INFO    MainThread:2036851 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-19 23:04:37,357 INFO    MainThread:2036851 [wandb_init.py:init():1010] run started, returning control to user process

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_6K/wandb/run-20260719_230435-bvj1u4hd/logs/debug-internal.log

@@ -0,0 +1,89 @@
+{"time":"2026-07-19T23:04:35.840125776+09:00","level":"INFO","msg":"wandb-core"}
+{"time":"2026-07-19T23:04:35.840638556+09:00","level":"INFO","msg":"stream: starting","core version":"0.27.2"}
+{"time":"2026-07-19T23:04:36.54060383+09:00","level":"INFO","msg":"stream: created new stream","id":"bvj1u4hd"}
+{"time":"2026-07-19T23:04:36.54078243+09:00","level":"INFO","msg":"handler: started"}
+{"time":"2026-07-19T23:04:36.540866846+09:00","level":"INFO","msg":"stream: started"}
+{"time":"2026-07-19T23:04:36.540882054+09:00","level":"INFO","msg":"writer: started","stream_id":"bvj1u4hd"}
+{"time":"2026-07-19T23:04:36.540911877+09:00","level":"INFO","msg":"sender: started"}
+{"time":"2026-07-19T23:04:37.360831701+09:00","level":"INFO","msg":"filestream: sending request","total_files":1,"console_offset":0,"console_lines":1}
+{"time":"2026-07-19T23:04:37.768316877+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:04:52.361524384+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":0,"events_lines":1,"console_offset":1,"console_lines":15,"uploaded_len":2}
+{"time":"2026-07-19T23:04:52.611456834+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:07.361087657+09:00","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":1,"events_lines":2}
+{"time":"2026-07-19T23:05:07.675579166+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:22.36129589+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":3,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:22.690724217+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:37.361504492+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":5,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:37.748248702+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:52.361539549+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":7,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:52.746890982+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:07.361761595+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":9,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:07.758189689+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:22.360814312+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":11,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:22.732027144+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:37.360944034+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":13,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:37.957533437+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:52.360824557+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":15,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:52.709354336+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:07.361770298+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":17,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:07.77667497+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:22.361532494+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":19,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:22.750366731+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:37.361366156+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":21,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:37.7047906+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:52.360964029+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":23,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:52.741271853+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:07.361007665+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":25,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:07.759270115+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:22.360784311+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":27,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:22.739878533+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:37.362067792+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":29,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:37.697247556+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:52.361783353+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":31,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:52.663210353+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:07.361805829+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":33,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:07.825541965+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:22.360992854+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":35,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:22.78458095+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:37.361584281+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":37,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:37.794040736+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:52.361131877+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":39,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:52.834839905+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:07.361007717+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":41,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:07.749488213+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:22.361854828+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":43,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:22.781090405+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:37.360963306+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":45,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:37.642609437+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:52.36137222+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":47,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:52.842313674+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:07.361553301+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":49,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:07.726615867+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:22.361762701+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":51,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:22.786651142+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:37.360846503+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":53,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:37.816027731+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:52.361149588+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":55,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:52.760242274+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:07.361205542+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":57,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:07.669311643+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:22.361560676+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":59,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:22.795167228+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:37.361547436+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":61,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:37.812676351+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:52.360836304+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":63,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:52.658061715+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:07.361209022+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":65,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:07.659851244+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:22.361327822+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":67,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:22.86967031+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:37.361324551+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":69,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:37.79306621+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:52.361469725+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":71,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:52.84583428+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:07.361146485+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":73,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:07.751834249+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:22.361067976+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":75,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:22.727458065+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:37.36095564+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":77,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:37.755069398+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_6K/wandb/run-20260719_230435-bvj1u4hd/logs/debug.log

@@ -0,0 +1,19 @@
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_setup.py:_flush():81] Configure stats pid to 2036851
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_6K/wandb/run-20260719_230435-bvj1u4hd/logs/debug.log
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_6K/wandb/run-20260719_230435-bvj1u4hd/logs/debug-internal.log
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_init.py:init():766] calling init triggers
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot_rkd', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_steps': 6000, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False, 'rkd_enabled': True, 'rkd_vae_checkpoint': '/home/ext_minje/groot_insight/TimewarpVAE_ckpt/multispace_gripper_dedup_batch_ablation_v1/b512/RD/K08_z16_g4_h256/checkpoint-step-20000-epoch-46.84', 'rkd_student_pooling': 'attention', 'rkd_student_attention_hidden_dim': 512, 'rkd_student_attention_output_dim': 512, 'rkd_fm_loss_weight': 1.0, 'rkd_loss_weight': 1.0, 'rkd_distance_weight': 1.0, 'rkd_angle_weight': 2.0}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/LAPstyle_linear_6K', 'job_name': 'INSIGHT_RD_RKD_TimewarpVAE_LAPstyle_6K', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 6000, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-19 23:04:35,606 INFO    MainThread:2036851 [wandb_init.py:init():814] starting backend
+2026-07-19 23:04:35,830 INFO    MainThread:2036851 [wandb_init.py:init():829] sending inform_init request
+2026-07-19 23:04:36,541 INFO    MainThread:2036851 [wandb_init.py:init():834] backend started and connected
+2026-07-19 23:04:36,546 INFO    MainThread:2036851 [wandb_init.py:init():904] updated telemetry
+2026-07-19 23:04:36,557 INFO    MainThread:2036851 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-19 23:04:37,200 INFO    MainThread:2036851 [wandb_init.py:init():972] starting run threads in backend
+2026-07-19 23:04:37,351 INFO    MainThread:2036851 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-19 23:04:37,353 INFO    MainThread:2036851 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-19 23:04:37,353 INFO    MainThread:2036851 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-19 23:04:37,353 INFO    MainThread:2036851 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-19 23:04:37,357 INFO    MainThread:2036851 [wandb_init.py:init():1010] run started, returning control to user process

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/RSCLstyle_cosine_60K/wandb/run-20260719_230435-5e3ytmnu/files/wandb-metadata.json

@@ -0,0 +1,91 @@
+{
+  "os": "Linux-6.8.0-107-generic-x86_64-with-glibc2.39",
+  "python": "CPython 3.12.13",
+  "startedAt": "2026-07-19T14:04:35.704983Z",
+  "args": [
+    "--config_path=/tmp/insight_lerobot_cfg_7rsw3qpw/RSCLstyle_RKD_cosine_decay_60K.yaml",
+    "--batch_size=128",
+    "--output_dir=/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/RSCLstyle_cosine_60K"
+  ],
+  "program": "/home/ext_minje/clvla/benchmarks/INSIGHT/my_scripts/lerobot_train_yaml.py",
+  "git": {
+    "remote": "git@github.com:paragon7060/clvla.git",
+    "commit": "a9df08f7da2d8eb1fa7f28708b4eb6f6dc81b1f9"
+  },
+  "root": "/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/RSCLstyle_cosine_60K",
+  "host": "DGX-H200-01",
+  "executable": "/home/ext_minje/miniforge3/envs/lerobot060_groot/bin/python",
+  "cpu_count": 112,
+  "cpu_count_logical": 224,
+  "gpu": "NVIDIA H200",
+  "gpu_count": 8,
+  "disk": {
+    "/": {
+      "total": "1888425144320",
+      "used": "813929852928"
+    }
+  },
+  "memory": {
+    "total": "2164177309696"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-1e531a45-480e-3c1a-e9a0-2ba2cc27b0a3"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-5d3d5ae2-ffde-89d9-8818-dc19e11ad2fb"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-305c0dae-dcf5-d7bb-7dd3-5477d2194275"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-ed8b7920-8881-4e8f-3e88-be1bbecafa21"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-a91176db-55f2-b284-bbbf-56626596947d"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-922ff6d5-a948-205a-0426-dbee495fc3dd"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-1fde7c75-18fe-e3db-a4c1-85e67522f55f"
+    },
+    {
+      "name": "NVIDIA H200",
+      "memoryTotal": "150754820096",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-a9b1c49b-d527-97d8-ce91-7e50bd02154c"
+    }
+  ],
+  "cudaVersion": "12.9",
+  "writerId": "gv4s93wd7dns6rviy8dm02jp22skkffq"
+}

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/RSCLstyle_cosine_60K/wandb/run-20260719_230435-5e3ytmnu/logs/debug-internal.log

@@ -0,0 +1,89 @@
+{"time":"2026-07-19T23:04:35.943151157+09:00","level":"INFO","msg":"wandb-core"}
+{"time":"2026-07-19T23:04:35.943433868+09:00","level":"INFO","msg":"stream: starting","core version":"0.27.2"}
+{"time":"2026-07-19T23:04:36.611467609+09:00","level":"INFO","msg":"stream: created new stream","id":"5e3ytmnu"}
+{"time":"2026-07-19T23:04:36.611572242+09:00","level":"INFO","msg":"handler: started"}
+{"time":"2026-07-19T23:04:36.611807563+09:00","level":"INFO","msg":"stream: started"}
+{"time":"2026-07-19T23:04:36.612175087+09:00","level":"INFO","msg":"sender: started"}
+{"time":"2026-07-19T23:04:36.612167211+09:00","level":"INFO","msg":"writer: started","stream_id":"5e3ytmnu"}
+{"time":"2026-07-19T23:04:37.257178205+09:00","level":"INFO","msg":"filestream: sending request","total_files":1,"console_offset":0,"console_lines":1}
+{"time":"2026-07-19T23:04:37.743087617+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:04:52.257392987+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":0,"events_lines":1,"console_offset":1,"console_lines":15,"uploaded_len":2}
+{"time":"2026-07-19T23:04:52.542319433+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:07.258163461+09:00","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":1,"events_lines":2}
+{"time":"2026-07-19T23:05:07.619337894+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:22.257738789+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":3,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:22.7112944+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:37.257907034+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":5,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:37.672011801+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:05:52.258117309+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":7,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:05:52.582021599+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:07.257961398+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":9,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:07.734131298+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:22.257932664+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":11,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:22.547638132+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:37.257258428+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":13,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:37.648933562+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:06:52.259011675+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":15,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:06:52.598570588+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:07.257905651+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":17,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:07.578635523+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:22.257818058+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":19,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:22.696487664+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:37.257806194+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":21,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:37.595511834+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:07:52.257643582+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":23,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:07:52.60856563+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:07.257778282+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":25,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:07.701328257+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:22.257912985+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":27,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:23.185841344+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:37.258052154+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":29,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:37.808582839+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:08:52.257994401+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":31,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:08:52.665256328+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:07.258605414+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":33,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:07.723621911+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:22.257272261+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":35,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:22.635904793+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:37.257219472+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":37,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:37.624689221+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:09:52.257438339+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":39,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:09:52.665381427+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:07.258034994+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":41,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:07.669561015+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:22.257845366+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":43,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:22.634963442+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:37.258233116+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":45,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:37.684343702+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:10:52.258036518+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":47,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:10:52.547934491+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:07.257729046+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":49,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:07.667207125+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:22.257769827+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":51,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:22.626627456+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:37.257737929+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":53,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:37.603320965+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:11:52.257670755+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":55,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:11:52.580805954+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:07.257668422+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":57,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:07.681613981+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:22.257992557+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":59,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:22.625503565+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:37.257937592+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":61,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:37.629617001+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:12:52.258152559+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":63,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:12:52.578713432+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:07.25791458+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":65,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:07.780341076+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:22.258040527+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":67,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:22.552576795+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:37.25756868+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":69,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:37.548953068+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:13:52.257907671+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":71,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:13:52.624452804+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:07.258147285+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":73,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:07.738971805+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:22.257660414+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":75,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:22.743119329+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-07-19T23:14:37.258115457+09:00","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":77,"events_lines":2,"console_offset":16,"console_lines":1}
+{"time":"2026-07-19T23:14:37.561528774+09:00","level":"INFO","msg":"filestream: request sent","status":"200 OK"}

Abs_6D + DeltaEE/restarts/20260719_230419_rerun/RSCLstyle_cosine_60K/wandb/run-20260719_230435-5e3ytmnu/logs/debug.log

@@ -0,0 +1,19 @@
+2026-07-19 23:04:35,706 INFO    MainThread:2036849 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-07-19 23:04:35,706 INFO    MainThread:2036849 [wandb_setup.py:_flush():81] Configure stats pid to 2036849
+2026-07-19 23:04:35,706 INFO    MainThread:2036849 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-07-19 23:04:35,706 INFO    MainThread:2036849 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/RSCLstyle_cosine_60K/wandb/run-20260719_230435-5e3ytmnu/logs/debug.log
+2026-07-19 23:04:35,706 INFO    MainThread:2036849 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/RSCLstyle_cosine_60K/wandb/run-20260719_230435-5e3ytmnu/logs/debug-internal.log
+2026-07-19 23:04:35,706 INFO    MainThread:2036849 [wandb_init.py:init():766] calling init triggers
+2026-07-19 23:04:35,706 INFO    MainThread:2036849 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot_rkd', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_steps': 3000, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False, 'rkd_enabled': True, 'rkd_vae_checkpoint': '/home/ext_minje/groot_insight/TimewarpVAE_ckpt/multispace_gripper_dedup_batch_ablation_v1/b512/RD/K08_z16_g4_h256/checkpoint-step-20000-epoch-46.84', 'rkd_student_pooling': 'attention', 'rkd_student_attention_hidden_dim': 512, 'rkd_student_attention_output_dim': 512, 'rkd_fm_loss_weight': 1.0, 'rkd_loss_weight': 1.0, 'rkd_distance_weight': 1.0, 'rkd_angle_weight': 2.0}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_230419_rerun/RSCLstyle_cosine_60K', 'job_name': 'INSIGHT_RD_RKD_TimewarpVAE_RSCLstyle_C60', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 128, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 3000, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
+2026-07-19 23:04:35,707 INFO    MainThread:2036849 [wandb_init.py:init():814] starting backend
+2026-07-19 23:04:35,932 INFO    MainThread:2036849 [wandb_init.py:init():829] sending inform_init request
+2026-07-19 23:04:36,613 INFO    MainThread:2036849 [wandb_init.py:init():834] backend started and connected
+2026-07-19 23:04:36,615 INFO    MainThread:2036849 [wandb_init.py:init():904] updated telemetry
+2026-07-19 23:04:36,623 INFO    MainThread:2036849 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-07-19 23:04:37,083 INFO    MainThread:2036849 [wandb_init.py:init():972] starting run threads in backend
+2026-07-19 23:04:37,250 INFO    MainThread:2036849 [wandb_run.py:_console_start():2523] atexit reg
+2026-07-19 23:04:37,250 INFO    MainThread:2036849 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-07-19 23:04:37,250 INFO    MainThread:2036849 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-07-19 23:04:37,251 INFO    MainThread:2036849 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-07-19 23:04:37,254 INFO    MainThread:2036849 [wandb_init.py:init():1010] run started, returning control to user process

Abs_6D + DeltaEE/restarts/20260719_231441_rerun_save20k40k60k/Baseline/wandb/run-20260719_231456-eexlcl2l/files/output.log

@@ -0,0 +1,17 @@
+INFO 2026-07-19 23:14:58 db_utils.py:121 Logs will be synced with wandb.
+INFO 2026-07-19 23:14:58 db_utils.py:122 Track this run --> https://wandb.ai/minje227_hyu-hanyang-university/lerobot/runs/eexlcl2l
+INFO 2026-07-19 23:14:58 ot_train.py:298 Creating dataset
+INFO 2026-07-19 23:15:01 ot_train.py:332 Creating policy
+Fetching 27 files: 100%|██████████| 27/27 [00:00<00:00, 1905.25it/s]
+`torch_dtype` is deprecated! Use `dtype` instead!
+Loading weights: 100%|██████████| 1031/1031 [00:00<00:00, 1052.67it/s]
+INFO 2026-07-19 23:15:12 ot_train.py:405 Creating optimizer and scheduler
+INFO 2026-07-19 23:15:12 ot_train.py:437 Output dir: /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_231441_rerun_save20k40k60k/Baseline
+INFO 2026-07-19 23:15:12 ot_train.py:444 cfg.steps=60000 (60K)
+INFO 2026-07-19 23:15:12 ot_train.py:445 dataset.num_frames=218367 (218K)
+INFO 2026-07-19 23:15:12 ot_train.py:446 dataset.num_episodes=4930
+INFO 2026-07-19 23:15:12 ot_train.py:449 Effective batch size: 128 x 1 = 128
+INFO 2026-07-19 23:15:12 ot_train.py:450 num_learnable_params=1620515968 (2B)
+INFO 2026-07-19 23:15:12 ot_train.py:451 num_total_params=3144016000 (3B)
+Training:   0%|          | 0/60000 [00:00<?, ?step/s]INFO 2026-07-19 23:15:12 ot_train.py:602 Start offline training on a fixed dataset, with effective batch size: 128
+Training:   0%|          | 19/60000 [01:18<52:02:57,  3.12s/step]

Abs_6D + DeltaEE/restarts/20260719_231441_rerun_save20k40k60k/LAPstyle_linear_6K/wandb/run-20260719_231455-je3hccha/files/output.log

@@ -0,0 +1,17 @@
+INFO 2026-07-19 23:14:57 db_utils.py:121 Logs will be synced with wandb.
+INFO 2026-07-19 23:14:57 db_utils.py:122 Track this run --> https://wandb.ai/minje227_hyu-hanyang-university/lerobot/runs/je3hccha
+INFO 2026-07-19 23:14:57 ot_train.py:298 Creating dataset
+INFO 2026-07-19 23:15:00 ot_train.py:332 Creating policy
+Fetching 27 files: 100%|██████████| 27/27 [00:00<00:00, 3766.21it/s]
+`torch_dtype` is deprecated! Use `dtype` instead!
+Loading weights: 100%|██████████| 1031/1031 [00:00<00:00, 1673.91it/s]
+INFO 2026-07-19 23:15:10 ot_train.py:405 Creating optimizer and scheduler
+INFO 2026-07-19 23:15:10 ot_train.py:439 Output dir: /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/restarts/20260719_231441_rerun_save20k40k60k/LAPstyle_linear_6K
+INFO 2026-07-19 23:15:10 ot_train.py:446 cfg.steps=60000 (60K)
+INFO 2026-07-19 23:15:10 ot_train.py:447 dataset.num_frames=218367 (218K)
+INFO 2026-07-19 23:15:10 ot_train.py:448 dataset.num_episodes=4930
+INFO 2026-07-19 23:15:10 ot_train.py:451 Effective batch size: 128 x 1 = 128
+INFO 2026-07-19 23:15:10 ot_train.py:452 num_learnable_params=1622090881 (2B)
+INFO 2026-07-19 23:15:10 ot_train.py:453 num_total_params=3145712033 (3B)
+Training:   0%|          | 0/60000 [00:00<?, ?step/s]INFO 2026-07-19 23:15:10 ot_train.py:604 Start offline training on a fixed dataset, with effective batch size: 128
+Training:   0%|          | 20/60000 [01:20<51:30:40,  3.09s/step]

Abs_6D + DeltaEE/restarts/20260719_231636_rerun_save30k60k/Baseline/resolved_config.yaml

@@ -0,0 +1,24 @@
+dataset:
+  repo_id: Whalswp/INSIGHTfixposV4_filtered_multispace_v2
+  root: /home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2
+policy:
+  type: groot
+  device: cuda
+  chunk_size: 16
+  n_action_steps: 16
+  push_to_hub: false
+  tune_llm: false
+  tune_visual: false
+  tune_top_llm_layers: 0
+  tune_projector: true
+  tune_diffusion_model: true
+  tune_vlln: true
+seed: 42
+batch_size: 128
+steps: 60000
+log_freq: 200
+output_dir: /home/ext_minje/groot_insight/train_ckpt/Abs_6D + DeltaEE/Baseline
+job_name: INSIGHT_RD_baseline
+wandb:
+  enable: true
+  disable_artifact: true