debug.log

2026-01-25 04:30:35,121 INFO    MainThread:1165115 [wandb_setup.py:_flush():81] Current SDK version is 0.21.4
2026-01-25 04:30:35,121 INFO    MainThread:1165115 [wandb_setup.py:_flush():81] Configure stats pid to 1165115
2026-01-25 04:30:35,121 INFO    MainThread:1165115 [wandb_setup.py:_flush():81] Loading settings from /pfss/mlde/workspaces/mlde_wsp_PI_Hauschild/.config/wandb/settings
2026-01-25 04:30:35,121 INFO    MainThread:1165115 [wandb_setup.py:_flush():81] Loading settings from /pfss/mlde/workspaces/mlde_wsp_PI_Hauschild/VLA/duc/VLA_custom/wandb/settings
2026-01-25 04:30:35,121 INFO    MainThread:1165115 [wandb_setup.py:_flush():81] Loading settings from environment variables
2026-01-25 04:30:35,121 INFO    MainThread:1165115 [wandb_init.py:setup_run_log_directory():686] Logging user logs to /pfss/mlde/workspaces/mlde_wsp_PI_Hauschild/VLA/duc/VLA_custom/outputs/train/2026-01-25/04-30-23_aloha_100%_merged_synthetic_high_view_PRETRAIN/wandb/run-20260125_043035-g0u01bsh/logs/debug.log
2026-01-25 04:30:35,121 INFO    MainThread:1165115 [wandb_init.py:setup_run_log_directory():687] Logging internal logs to /pfss/mlde/workspaces/mlde_wsp_PI_Hauschild/VLA/duc/VLA_custom/outputs/train/2026-01-25/04-30-23_aloha_100%_merged_synthetic_high_view_PRETRAIN/wandb/run-20260125_043035-g0u01bsh/logs/debug-internal.log
2026-01-25 04:30:35,121 INFO    MainThread:1165115 [wandb_init.py:init():813] calling init triggers
2026-01-25 04:30:35,121 INFO    MainThread:1165115 [wandb_init.py:init():818] wandb.init called with sweep_config: {}
config: {'dataset': {'repo_id': 'ducido/calvin_task_D_D_scale_100_lerobo_format', 'root': '/pfss/mlde/workspaces/mlde_wsp_PI_Hauschild/VLA/ALOHA/aloha_3_tasks_large_100_original_synthetic_high_view', 'episodes': None, 'image_transforms': {'enable': True, 'max_num_transforms': 3, 'random_order': False, 'image_tfs': {'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'crop_resize': {'weight': 1.0, 'type': 'RandomResizedCrop', 'kwargs': {'size': [256, 256], 'ratio': [1, 1], 'scale': [0.9, 0.95]}}, 'rotate': {'weight': 1.0, 'type': 'RandomRotate', 'kwargs': {'degrees': [-5, 5]}}}, 'wrist_tfs': {'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'crop_resize': {'weight': 1.0, 'type': 'RandomResizedCrop', 'kwargs': {'size': [256, 256], 'ratio': [1, 1], 'scale': [0.9, 0.95]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'vqa_data_path': None}, 'env': None, 'policy': {'type': 'pi0', 'n_obs_steps': 1, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.MEAN_STD: 'MEAN_STD'>, 'ACTION': <NormalizationMode.MEAN_STD: 'MEAN_STD'>}, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'gradient_accumulation_steps': 2, 'chunk_size': 50, 'n_action_steps': 50, 'visual_prompt': False, 'use_both_origin': False, 'zero_init': False, 'coordinate_text_prod': False, 'coordinate_posEmbed_prod': False, 'posEmbed_to_preTrained': False, 'num_obs_layers': 30, 'last_obs_layers': False, 'add_posEmbed': False, 'not_used_coord': False, 'use_depth': False, 'depth_model_type': 'frozen_siglip', 'use_track': False, 'use_detect_object': False, 'track_mask': False, 'use_slot_att': 'none', 'future_obj': 'none', 'original_pi0': False, 'max_state_dim': 32, 'max_action_dim': 32, 'resize_imgs_with_padding': [224, 224], 'empty_cameras': 0, 'adapt_to_pi_aloha': False, 'use_delta_joint_actions_aloha': False, 'tokenizer_max_length': 48, 'proj_width': 1024, 'num_steps': 10, 'use_cache': True, 'attention_implementation': 'eager', 'freeze_vision_encoder': True, 'train_expert_only': False, 'train_state_proj': True, 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.95], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-10, 'scheduler_warmup_steps': 1000, 'scheduler_decay_steps': 240000, 'scheduler_decay_lr': 2.5e-06}, 'output_dir': '/pfss/mlde/workspaces/mlde_wsp_PI_Hauschild/VLA/duc/VLA_custom/outputs/train/2026-01-25/04-30-23_aloha_100%_merged_synthetic_high_view_PRETRAIN', 'job_name': 'aloha_100%_merged_synthetic_high_view_PRETRAIN', 'visual_prompt': False, 'use_both_origin': False, 'zero_init': False, 'coordinate_text_prod': False, 'coordinate_posEmbed_prod': False, 'posEmbed_to_preTrained': False, 'num_obs_layers': 30, 'last_obs_layers': False, 'add_posEmbed': False, 'not_used_coord': False, 'train_expert_only': False, 'use_depth': False, 'use_track': True, 'use_detect_object': False, 'track_mask': False, 'use_slot_att': 'none', 'future_obj': 'all', 'depth_model_type': 'frozen_siglip', 'original_pi0': False, 'resume': False, 'seed': 42, 'num_workers': 8, 'batch_size': 13, 'steps': 100000, 'eval_freq': 20000, 'log_freq': 200, 'save_checkpoint': True, 'save_freq': 10000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-10, 'grad_clip_norm': 10.0, 'betas': [0.9, 0.95], 'eps': 1e-08}, 'scheduler': {'type': 'cosine_decay_with_warmup', 'num_warmup_steps': 1000, 'num_decay_steps': 240000, 'peak_lr': 0.0001, 'decay_lr': 2.5e-06}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'pi0_lerobot', 'entity': 'Robotics_VLA', 'notes': None, 'run_id': None, 'mode': 'online'}, '_wandb': {}}
2026-01-25 04:30:35,122 INFO    MainThread:1165115 [wandb_init.py:init():854] starting backend
2026-01-25 04:30:35,327 INFO    MainThread:1165115 [wandb_init.py:init():857] sending inform_init request
2026-01-25 04:30:35,332 INFO    MainThread:1165115 [wandb_init.py:init():865] backend started and connected
2026-01-25 04:30:35,334 INFO    MainThread:1165115 [wandb_init.py:init():936] updated telemetry
2026-01-25 04:30:35,344 INFO    MainThread:1165115 [wandb_init.py:init():960] communicating run to backend with 90.0 second timeout
2026-01-25 04:30:35,923 INFO    MainThread:1165115 [wandb_init.py:init():1011] starting run threads in backend
2026-01-25 04:30:36,126 INFO    MainThread:1165115 [wandb_run.py:_console_start():2506] atexit reg
2026-01-25 04:30:36,127 INFO    MainThread:1165115 [wandb_run.py:_redirect():2354] redirect: wrap_raw
2026-01-25 04:30:36,127 INFO    MainThread:1165115 [wandb_run.py:_redirect():2423] Wrapping output streams.
2026-01-25 04:30:36,127 INFO    MainThread:1165115 [wandb_run.py:_redirect():2446] Redirects installed.
2026-01-25 04:30:36,129 INFO    MainThread:1165115 [wandb_init.py:init():1049] run started, returning control to user process
2026-01-26 03:42:47,248 INFO    wandb-AsyncioManager-main:1165115 [service_client.py:_forward_responses():84] Reached EOF.
2026-01-26 03:42:47,248 INFO    wandb-AsyncioManager-main:1165115 [mailbox.py:close():137] Closing mailbox, abandoning 1 handles.