wandb/debug.log

2026-03-23 00:53:11,859 INFO    MainThread:190649 [wandb_setup.py:_flush():81] Current SDK version is 0.25.1
2026-03-23 00:53:11,860 INFO    MainThread:190649 [wandb_setup.py:_flush():81] Configure stats pid to 190649
2026-03-23 00:53:11,860 INFO    MainThread:190649 [wandb_setup.py:_flush():81] Loading settings from environment variables
2026-03-23 00:53:11,860 INFO    MainThread:190649 [wandb_init.py:setup_run_log_directory():717] Logging user logs to /global/scratch/users/ghr/Projects/szang/Psi0/.runs/finetune/robocasa-pnpbottletocabinetclose.real.flow1000.cosine.lr5.0e-04.b128.gpus8.2603230052/wandb/run-20260323_005311-u1ed6upc/logs/debug.log
2026-03-23 00:53:11,860 INFO    MainThread:190649 [wandb_init.py:setup_run_log_directory():718] Logging internal logs to /global/scratch/users/ghr/Projects/szang/Psi0/.runs/finetune/robocasa-pnpbottletocabinetclose.real.flow1000.cosine.lr5.0e-04.b128.gpus8.2603230052/wandb/run-20260323_005311-u1ed6upc/logs/debug-internal.log
2026-03-23 00:53:11,860 INFO    MainThread:190649 [wandb_init.py:init():844] calling init triggers
2026-03-23 00:53:11,860 INFO    MainThread:190649 [wandb_init.py:init():849] wandb.init called with sweep_config: {}
config: {'_wandb': {}}
2026-03-23 00:53:11,860 INFO    MainThread:190649 [wandb_init.py:init():892] starting backend
2026-03-23 00:53:12,842 INFO    MainThread:190649 [wandb_init.py:init():895] sending inform_init request
2026-03-23 00:53:13,022 INFO    MainThread:190649 [wandb_init.py:init():903] backend started and connected
2026-03-23 00:53:13,023 INFO    MainThread:190649 [wandb_init.py:init():973] updated telemetry
2026-03-23 00:53:13,038 INFO    MainThread:190649 [wandb_init.py:init():997] communicating run to backend with 90.0 second timeout
2026-03-23 00:53:13,747 INFO    MainThread:190649 [wandb_init.py:init():1042] starting run threads in backend
2026-03-23 00:53:14,170 INFO    MainThread:190649 [wandb_run.py:_console_start():2524] atexit reg
2026-03-23 00:53:14,170 INFO    MainThread:190649 [wandb_run.py:_redirect():2373] redirect: wrap_raw
2026-03-23 00:53:14,170 INFO    MainThread:190649 [wandb_run.py:_redirect():2442] Wrapping output streams.
2026-03-23 00:53:14,170 INFO    MainThread:190649 [wandb_run.py:_redirect():2465] Redirects installed.
2026-03-23 00:53:14,181 INFO    MainThread:190649 [wandb_init.py:init():1082] run started, returning control to user process
2026-03-23 00:53:14,182 INFO    MainThread:190649 [wandb_run.py:_config_callback():1403] config_cb None None {'exp': 'robocasa-pnpbottletocabinetclose', 'seed': 292285, 'auto_tag_run': False, 'eval': False, 'debug': False, 'timestamp': '2603230052', 'log': {'logging_dir': 'logs', 'report_to': 'wandb', 'log_freq': 100}, 'wandb': {'project': 'psi', 'entity': 'szangrl', 'group': None, 'id': None, 'name': None, 'resume': 'allow'}, 'train': {'num_workers': 8, 'overfit_single_batch': False, 'name': 'finetune', 'resume_from_checkpoint': None, 'skip_resumed_steps': False, 'hf_token': '.hf_token', 'lora': False, 'output_dir': '.runs', 'gradient_accumulation_steps': 1, 'mixed_precision': 'bf16', 'max_grad_norm': 1.0, 'optimizer_foreach': None, 'train_batch_size': 16, 'val_batch_size': 16, 'val_num_batches': 20, 'checkpointing_steps': 5000, 'max_checkpoints_to_keep': 10, 'validation_steps': 800, 'learning_rate': 0.0005, 'lr_scheduler_type': 'cosine', 'lr_scheduler_kwargs': {'betas': [0.95, 0.999], 'weight_decay': 1e-06, 'eps': 1e-08}, 'scheduler_specific_kwargs': {}, 'data_parallel': 'ddp', 'sharding_strategy': 'full-shard', 'deepspeed_config': '/global/scratch/users/ghr/Projects/szang/Psi0/src/InternVLA/config/deepseeds/zero3.json', 'enable_gradient_checkpointing': True, 'enable_mixed_precision_training': True, 'reduce_in_full_precision': True, 'max_training_steps': 40000, 'num_train_epochs': None, 'warmup_steps': 1000, 'warmup_ratio': None}, 'data': {'transform': {'repack': {'dataset_name': 'real', 'conditions': [], 'image_keys': ['observation.images.ego_view'], 'state_key': 'observation.state', 'action_key': 'action', 'instruction_key': 'task', 'num_past_frames': 0, 'action_chunk_size': 16, 'pad_action_dim': 44, 'pad_state_dim': 44}, 'model': {'resize': {'size': [256, 256]}, 'center_crop': {'size': [256, 256]}, 'color_jitter': {'brightness': 0.2, 'contrast': [0.8, 1.2], 'saturation': [0.8, 1.2], 'hue': 0.05}, 'gaussian_noise': {'mean': 0.0, 'std': 3.0, 'prob_skip': 0.1}, 'img_aug': True, 'adaptive_resize': False, 'img_sizes': {'egodex': [270, 480], 'he': [240, 320]}}, 'field': {'stat_path': 'meta/stats.json', 'action_norm_type': 'bounds', 'stat_action_key': 'action', 'stat_state_key': 'observation.state', 'use_norm_mask': False, 'action_norm_masks': [True, True, True, True, True, True, False], 'action_min': [-1.5839895009994507, -0.0008472761255688965, -1.8800768852233887, -2.8324925899505615, -1.9416913986206055, -1.1918531656265259, -1.4997141361236572, -1.5, -1.5, -1.5, -1.5, -3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -2.514977216720581, -3.0000972747802734, -3.0000174045562744, -2.815241575241089, -2.121727228164673, -1.3774508237838745, -1.4853956699371338, -1.5, -1.5, -1.5, -1.5, -3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -0.3876988887786865, -0.30079999566078186, -0.15515021979808807], 'action_max': [1.8006138801574707, 1.7875969409942627, 1.3964422941207886, 0.0012261775555089116, 2.9997270107269287, 1.500012755393982, 1.3617151975631714, 1.5, 1.5, 1.5, 1.5, 3.0, 3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.2466375827789307, 0.0002109118941007182, 1.7149735689163208, 2.7035512175643817e-05, 2.4049243927001953, 1.49424147605896, 1.3308935165405273, 1.5, 1.5, 1.5, 1.5, 3.0, 3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.019441843032837, 0.3486739993095398, 0.07686164975166321], 'state_min': [-1.6229853630065918, -0.04118623957037926, -1.8720369338989258, -2.5177226066589355, -1.9119305610656738, -1.0593844652175903, -1.4942827224731445, -0.8076969385147095, -1.7868800163269043, -1.0153224468231201, -0.8283111453056335, -0.7472333908081055, -0.03325841948390007, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -2.4912281036376953, -3.0019447803497314, -3.001241683959961, -2.5321874618530273, -2.0514044761657715, -1.362187385559082, -1.2079144716262817, -1.8965649604797363, -2.766759157180786, -1.6092761754989624, -1.4268920421600342, -0.7526456713676453, -0.015466930344700813, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -0.34590843319892883, -0.29124873876571655, -0.14527782797813416], 'state_max': [1.7968251705169678, 1.7782310247421265, 1.3914952278137207, 0.03404686599969864, 2.994903326034546, 1.501129150390625, 1.1614720821380615, 3.1853458881378174, 1.8330978155136108, 1.91548490524292, 2.152785301208496, 1.5519596338272095, 2.1858060359954834, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.1541643142700195, 0.004218237940222025, 1.6931140422821045, 0.0011083680437877774, 2.390566349029541, 1.3855408430099487, 1.1490544080734253, 2.570661783218384, 2.145352602005005, 2.930386781692505, 3.145815372467041, 1.5530203580856323, 2.10929012298584, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.9515821933746338, 0.2942240536212921, 0.1514909416437149], 'normalize_state': True, 'pad_action_dim': 44, 'pad_state_dim': 44}}, 'root_dir': '../Isaac-GR00T/demo_data/teleop_sim/LeRobot', 'train_repo_ids': ['gr1_unified.PnPBottleToCabinetClose'], 'val_repo_ids': ['gr1_unified.PnPBottleToCabinetClose']}, 'model': {'resnet_store_path': None, 'pretrained_action_header_path': None, 'rtc': False, 'max_delay': 8, 'action_dim': 44, 'action_chunk_size': 16, 'action_exec_horizon': 16, 'observation_horizon': 1, 'img_chunk': 1, 'n_cams': 1, 'use_obs': 'add_token', 'dropout': 0.1, 'noise_scheduler': 'flow', 'train_diffusion_steps': 1000, 'eval_diffusion_steps': 10, 'share_cam_features': False, 'early_fusion': False, 'odim': 44, 'n_conditions': 0, 'token_fusion': 'concat', 'loss_w': [0.1, 0.2, 0.1], 'time_dim': 256, 'hidden_dim': 1536, 'num_blocks': 6, 'dim_feedforward': 2048, 'nhead': 24, 'activation': 'gelu', 'view_feature_dim': 2048, 'use_film': False, 'combined_temb': False, 'use_dit': False, 'weight_decay': 0.01, 'model_name_or_path': '../video-pretraining/psi0_checkpoints/psi0/pre.fast.egodex.2512241941.ckpt200k', 'vlm_ckpt_step': None, 'tune_vlm': False, 'tune_mm_llm': False, 'tune_mm_vision': False, 'tune_mm_mlp': False, 'gradient_checkpointing': True, 'lang_backbone_lr': 1e-05, 'mm_projector_lr': 1e-05, 'vision_tower_lr': 1e-06, 'optim': 'adamw_torch', 'model_max_length': 4096, 'data_flatten': True, 'data_packing': True, 'max_pixels': 451584, 'min_pixels': 12544}, 'environment_variables': {'OMP_NUM_THREADS': '32', 'HF_HOME': '/var/tmp/szang', 'TORCH_HOME': '/var/tmp/szang', 'HF_TOKEN': 'hf_...DZIw', 'HF_LEROBOT_HOME': '/var/tmp/szang/hfm/data/lerobot', 'WE_HOME': 'Not Set', 'DATA_HOME': '/var/tmp/szang/hfm/data', 'UV_CACHE_DIR': '/var/tmp/szang/hfm/cache', 'WANDB_API_KEY': 'f02...c7e9', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'python', 'CUDA_VISIBLE_DEVICES': '0,1,2,3,4,5,6,7', 'WORLD_SIZE': '8', 'LOCAL_WORLD_SIZE': '8', 'RANK': '0', 'LOCAL_RANK': '0', 'MASTER_ADDR': '127.0.0.1', 'MASTER_PORT': '29500'}}
2026-03-23 11:26:52,215 INFO    MainThread:190649 [wandb_run.py:_finish():2291] finishing run szangrl/psi/u1ed6upc
2026-03-23 11:26:52,219 INFO    MainThread:190649 [wandb_run.py:_atexit_cleanup():2490] got exitcode: 0
2026-03-23 11:26:52,219 INFO    MainThread:190649 [wandb_run.py:_restore():2472] restore
2026-03-23 11:26:52,220 INFO    MainThread:190649 [wandb_run.py:_restore():2478] restore done
2026-03-23 11:27:42,901 INFO    MainThread:190649 [wandb_run.py:_footer_sync_info():3868] logging synced files