wandb/debug.log

2025-08-26 02:32:25,919 INFO    MainThread:2423864 [wandb_setup.py:_flush():80] Current SDK version is 0.21.0
2025-08-26 02:32:25,919 INFO    MainThread:2423864 [wandb_setup.py:_flush():80] Configure stats pid to 2423864
2025-08-26 02:32:25,919 INFO    MainThread:2423864 [wandb_setup.py:_flush():80] Loading settings from /home/ubuntu/.config/wandb/settings
2025-08-26 02:32:25,919 INFO    MainThread:2423864 [wandb_setup.py:_flush():80] Loading settings from /home/ubuntu/projects/policy/wandb/settings
2025-08-26 02:32:25,919 INFO    MainThread:2423864 [wandb_setup.py:_flush():80] Loading settings from environment variables
2025-08-26 02:32:25,919 INFO    MainThread:2423864 [wandb_init.py:setup_run_log_directory():703] Logging user logs to outputs/train/black_table_insert_rope_downsampled_v7/2025-08-26_02-32-20_act_Te50_abs_ds_1000/wandb/run-20250826_023225-66tdnuo5/logs/debug.log
2025-08-26 02:32:25,919 INFO    MainThread:2423864 [wandb_init.py:setup_run_log_directory():704] Logging internal logs to outputs/train/black_table_insert_rope_downsampled_v7/2025-08-26_02-32-20_act_Te50_abs_ds_1000/wandb/run-20250826_023225-66tdnuo5/logs/debug-internal.log
2025-08-26 02:32:25,919 INFO    MainThread:2423864 [wandb_init.py:init():830] calling init triggers
2025-08-26 02:32:25,919 INFO    MainThread:2423864 [wandb_init.py:init():835] wandb.init called with sweep_config: {}
config: {'dataset': {'repo_id': 'black_table_insert_rope_downsampled_v7', 'root': None, 'episodes': None, 'image_transforms': {'enable': True, 'max_num_transforms': 3, 'random_order': True, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'perspective': {'weight': 1.0, 'type': 'RandomPerspective', 'kwargs': {'distortion_scale': 0.025, 'p': 1.0}}, 'rotation': {'weight': 1.0, 'type': 'RandomRotation', 'kwargs': {'degrees': [-2, 2]}}, 'crop': {'weight': 1.0, 'type': 'RandomCrop', 'kwargs': {'size': [110, 202]}}}, 'downsample': {'resize': {'weight': 1.0, 'type': 'Resize', 'kwargs': {'size': [120, 212]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec'}, 'env': None, 'policy': {'type': 'act', 'n_obs_steps': 1, 'normalization_mapping': {'VISUAL': <NormalizationMode.MEAN_STD: 'MEAN_STD'>, 'STATE': <NormalizationMode.MEAN_STD: 'MEAN_STD'>, 'ACTION': <NormalizationMode.MEAN_STD: 'MEAN_STD'>}, 'relative_actions': False, 'input_features': {'observation.image.side': {'type': <FeatureType.VISUAL: 'VISUAL'>, 'shape': [3, 120, 212]}, 'observation.image.wrist': {'type': <FeatureType.VISUAL: 'VISUAL'>, 'shape': [3, 120, 212]}, 'observation.state': {'type': <FeatureType.STATE: 'STATE'>, 'shape': [8]}}, 'output_features': {'action': {'type': <FeatureType.ACTION: 'ACTION'>, 'shape': [8]}}, 'device': 'cuda', 'use_amp': False, 'chunk_size': 50, 'n_action_steps': 50, 'vision_backbone': 'resnet18', 'pretrained_backbone_weights': 'ResNet18_Weights.IMAGENET1K_V1', 'replace_final_stride_with_dilation': 0, 'pre_norm': False, 'dim_model': 512, 'n_heads': 8, 'dim_feedforward': 3200, 'feedforward_activation': 'relu', 'n_encoder_layers': 4, 'n_decoder_layers': 1, 'use_vae': True, 'latent_dim': 32, 'n_vae_encoder_layers': 4, 'temporal_ensemble_coeff': None, 'dropout': 0.1, 'kl_weight': 10.0, 'optimizer_lr': 1e-05, 'optimizer_weight_decay': 0.0001, 'optimizer_lr_backbone': 1e-05}, 'output_dir': 'outputs/train/black_table_insert_rope_downsampled_v7/2025-08-26_02-32-20_act_Te50_abs_ds_1000', 'job_name': '2025-08-26_02-32-20_act_Te50_abs_ds_1000', 'resume': False, 'seed': 100000, 'num_workers': 4, 'batch_size': 1024, 'steps': 5000, 'eval_freq': 2000, 'log_freq': 100, 'save_checkpoint': True, 'save_freq': 100, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 1e-05, 'weight_decay': 0.0001, 'grad_clip_norm': 10.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': None, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None}, '_wandb': {}}
2025-08-26 02:32:25,919 INFO    MainThread:2423864 [wandb_init.py:init():871] starting backend
2025-08-26 02:32:26,124 INFO    MainThread:2423864 [wandb_init.py:init():874] sending inform_init request
2025-08-26 02:32:26,127 INFO    MainThread:2423864 [wandb_init.py:init():882] backend started and connected
2025-08-26 02:32:26,129 INFO    MainThread:2423864 [wandb_init.py:init():953] updated telemetry
2025-08-26 02:32:26,132 INFO    MainThread:2423864 [wandb_init.py:init():977] communicating run to backend with 90.0 second timeout
2025-08-26 02:32:26,503 INFO    MainThread:2423864 [wandb_init.py:init():1029] starting run threads in backend
2025-08-26 02:32:26,585 INFO    MainThread:2423864 [wandb_run.py:_console_start():2458] atexit reg
2025-08-26 02:32:26,585 INFO    MainThread:2423864 [wandb_run.py:_redirect():2306] redirect: wrap_raw
2025-08-26 02:32:26,585 INFO    MainThread:2423864 [wandb_run.py:_redirect():2375] Wrapping output streams.
2025-08-26 02:32:26,585 INFO    MainThread:2423864 [wandb_run.py:_redirect():2398] Redirects installed.
2025-08-26 02:32:26,587 INFO    MainThread:2423864 [wandb_init.py:init():1075] run started, returning control to user process
2025-08-26 06:18:21,206 INFO    MsgRouterThr:2423864 [mailbox.py:close():129] [no run ID] Closing mailbox, abandoning 1 handles.