| 2026-07-15 14:32:50,279 INFO MainThread:1450496 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2 | |
| 2026-07-15 14:32:50,279 INFO MainThread:1450496 [wandb_setup.py:_flush():81] Configure stats pid to 1450496 | |
| 2026-07-15 14:32:50,279 INFO MainThread:1450496 [wandb_setup.py:_flush():81] Loading settings from environment variables | |
| 2026-07-15 14:32:50,279 INFO MainThread:1450496 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/ext_minje/groot_insight/Abs_6D/Baseline/wandb/run-20260715_143250-1jmjb68j/logs/debug.log | |
| 2026-07-15 14:32:50,279 INFO MainThread:1450496 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/ext_minje/groot_insight/Abs_6D/Baseline/wandb/run-20260715_143250-1jmjb68j/logs/debug-internal.log | |
| 2026-07-15 14:32:50,279 INFO MainThread:1450496 [wandb_init.py:init():766] calling init triggers | |
| 2026-07-15 14:32:50,280 INFO MainThread:1450496 [wandb_init.py:init():771] wandb.init called with sweep_config: {} | |
| config: {'dataset': {'repo_id': 'Whalswp/INSIGHTfixposV4_filtered_multispace_v2', 'root': '/home/ext_minje/INSIGHTfixposV4_filtered_multispace_v2', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'groot', 'n_obs_steps': 1, 'input_features': {'observation.state': {'type': <FeatureType.STATE: 'STATE'>, 'shape': [16]}, 'observation.images.wrist': {'type': <FeatureType.VISUAL: 'VISUAL'>, 'shape': [3, 224, 224]}, 'observation.images.right_shoulder': {'type': <FeatureType.VISUAL: 'VISUAL'>, 'shape': [3, 224, 224]}, 'observation.images.guide': {'type': <FeatureType.VISUAL: 'VISUAL'>, 'shape': [3, 224, 224]}}, 'output_features': {'action': {'type': <FeatureType.ACTION: 'ACTION'>, 'shape': [10]}}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': '/home/ext_minje/groot_insight/Abs_6D/Baseline/checkpoints/050000/pretrained_model', 'pretrained_revision': None, 'chunk_size': 16, 'n_action_steps': 16, 'max_state_dim': 132, 'max_action_dim': 132, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'ACTION': <NormalizationMode.IDENTITY: 'IDENTITY'>}, 'base_model_path': 'nvidia/GR00T-N1.7-3B', 'action_decode_transform': None, 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': True, 'tune_vlln': True, 'tune_top_llm_layers': 0, 'num_inference_timesteps': None, 'rtc_ramp_rate': None, 'use_flash_attention': False, 'use_relative_actions': False, 'relative_exclude_joints': [], 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_ratio': 0.05, 'use_bf16': True, 'model_params_fp32': True, 'image_size': [256, 256], 'tokenizer_assets_repo': None, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False}, 'reward_model': None, 'output_dir': '/home/ext_minje/groot_insight/Abs_6D/Baseline', 'job_name': 'INSIGHT_6D_baseline', 'resume': True, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 4, 'batch_size': 64, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 60000, 'env_eval_freq': 20000, 'log_freq': 200, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 1.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 500, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': '1jmjb68j', 'mode': None, 'add_tags': True}, 'peft': None, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': '/home/ext_minje/groot_insight/Abs_6D/Baseline/checkpoints/050000', '_wandb': {}} | |
| 2026-07-15 14:32:50,280 INFO MainThread:1450496 [wandb_init.py:init():814] starting backend | |
| 2026-07-15 14:32:50,495 INFO MainThread:1450496 [wandb_init.py:init():829] sending inform_init request | |
| 2026-07-15 14:32:50,989 INFO MainThread:1450496 [wandb_init.py:init():834] backend started and connected | |
| 2026-07-15 14:32:50,999 INFO MainThread:1450496 [wandb_init.py:init():904] updated telemetry | |
| 2026-07-15 14:32:51,006 INFO MainThread:1450496 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout | |
| 2026-07-15 14:32:51,687 INFO MainThread:1450496 [wandb_init.py:init():967] run resumed | |
| 2026-07-15 14:32:51,692 INFO MainThread:1450496 [wandb_init.py:init():972] starting run threads in backend | |
| 2026-07-15 14:32:51,783 INFO MainThread:1450496 [wandb_run.py:_console_start():2523] atexit reg | |
| 2026-07-15 14:32:51,783 INFO MainThread:1450496 [wandb_run.py:_redirect():2373] redirect: wrap_raw | |
| 2026-07-15 14:32:51,783 INFO MainThread:1450496 [wandb_run.py:_redirect():2442] Wrapping output streams. | |
| 2026-07-15 14:32:51,783 INFO MainThread:1450496 [wandb_run.py:_redirect():2465] Redirects installed. | |
| 2026-07-15 14:32:51,785 INFO MainThread:1450496 [wandb_init.py:init():1010] run started, returning control to user process | |
| 2026-07-15 19:10:44,422 INFO wandb-AsyncioManager-main:1450496 [service_client.py:_forward_responses():122] Reached EOF. | |
| 2026-07-15 19:10:44,422 INFO wandb-AsyncioManager-main:1450496 [mailbox.py:close():154] Closing mailbox, abandoning 2 handles. | |
| 2026-07-15 19:10:44,423 ERROR wandb-AsyncioManager-main:1450496 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. | |
| Traceback (most recent call last): | |
| File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions | |
| await fn() | |
| File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop | |
| await asyncio_compat.race( | |
| File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 278, in race | |
| async with open_task_group(race=True) as tg: | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/contextlib.py", line 217, in __aexit__ | |
| await anext(self.gen) | |
| File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 240, in open_task_group | |
| await task_group._wait_all(race=race, timeout=exit_timeout) | |
| File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 180, in _wait_all | |
| raise exc | |
| File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all | |
| result = await handle.wait_async(timeout=None) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async | |
| response = await self._handle.wait_async(timeout=timeout) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/home/ext_minje/miniforge3/envs/lerobot060_groot/lib/python3.12/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async | |
| raise HandleAbandonedError() | |
| wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError | |