2025-12-25 07:01:22,156 INFO MainThread:10525 [wandb_setup.py:_flush():81] Current SDK version is 0.21.4 2025-12-25 07:01:22,156 INFO MainThread:10525 [wandb_setup.py:_flush():81] Configure stats pid to 10525 2025-12-25 07:01:22,156 INFO MainThread:10525 [wandb_setup.py:_flush():81] Loading settings from /home/zkj/.config/wandb/settings 2025-12-25 07:01:22,156 INFO MainThread:10525 [wandb_setup.py:_flush():81] Loading settings from /mars_data/zuo/unitree_IL_lerobot/unitree_lerobot/lerobot/wandb/settings 2025-12-25 07:01:22,156 INFO MainThread:10525 [wandb_setup.py:_flush():81] Loading settings from environment variables 2025-12-25 07:01:22,156 INFO MainThread:10525 [wandb_init.py:setup_run_log_directory():686] Logging user logs to outputs/train/groot_handover/wandb/run-20251225_070122-icrg7bac/logs/debug.log 2025-12-25 07:01:22,156 INFO MainThread:10525 [wandb_init.py:setup_run_log_directory():687] Logging internal logs to outputs/train/groot_handover/wandb/run-20251225_070122-icrg7bac/logs/debug-internal.log 2025-12-25 07:01:22,156 INFO MainThread:10525 [wandb_init.py:init():813] calling init triggers 2025-12-25 07:01:22,156 INFO MainThread:10525 [wandb_init.py:init():818] wandb.init called with sweep_config: {} config: {'dataset': {'repo_id': 'ZUO66/handover_drinks', 'root': None, 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'streaming': False}, 'env': None, 'policy': {'type': 'groot', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'push_to_hub': False, 'repo_id': 'ZUO66/groot_handover', 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'chunk_size': 50, 'n_action_steps': 50, 'max_state_dim': 64, 'max_action_dim': 32, 'normalization_mapping': {'VISUAL': , 'STATE': , 'ACTION': }, 'image_size': [224, 224], 'base_model_path': 'nvidia/GR00T-N1.5-3B', 'tokenizer_assets_repo': 'lerobot/eagle2hg-processor-groot-n1p5', 'embodiment_tag': 'new_embodiment', 'tune_llm': False, 'tune_visual': False, 'tune_projector': True, 'tune_diffusion_model': False, 'lora_rank': 0, 'lora_alpha': 16, 'lora_dropout': 0.1, 'lora_full_model': False, 'optimizer_lr': 0.0001, 'optimizer_betas': [0.95, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-05, 'warmup_ratio': 0.05, 'use_bf16': True, 'video_backend': 'decord', 'balance_dataset_weights': True, 'balance_trajectory_weights': True, 'dataset_paths': None, 'output_dir': './tmp/gr00t', 'save_steps': 1000, 'max_steps': 10000, 'batch_size': 32, 'dataloader_num_workers': 8, 'report_to': 'wandb', 'resume': False}, 'output_dir': 'outputs/train/groot_handover', 'job_name': 'groot_training', 'resume': False, 'seed': 1000, 'num_workers': 4, 'batch_size': 256, 'steps': 200000, 'eval_freq': 20000, 'log_freq': 200, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-05, 'grad_clip_norm': 10.0, 'betas': [0.95, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'cosine_decay_with_warmup', 'num_warmup_steps': 500, 'num_decay_steps': 10000, 'peak_lr': 0.0001, 'decay_lr': 1e-05}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': False}, 'wandb': {'enable': True, 'disable_artifact': False, 'project': 'zuo_handover', 'entity': None, 'notes': None, 'run_id': None, 'mode': None}, 'checkpoint_path': None, 'rename_map': {}, '_wandb': {}} 2025-12-25 07:01:22,157 INFO MainThread:10525 [wandb_init.py:init():854] starting backend 2025-12-25 07:01:22,362 INFO MainThread:10525 [wandb_init.py:init():857] sending inform_init request 2025-12-25 07:01:22,366 INFO MainThread:10525 [wandb_init.py:init():865] backend started and connected 2025-12-25 07:01:22,368 INFO MainThread:10525 [wandb_init.py:init():936] updated telemetry 2025-12-25 07:01:22,374 INFO MainThread:10525 [wandb_init.py:init():960] communicating run to backend with 90.0 second timeout 2025-12-25 07:01:23,010 INFO MainThread:10525 [wandb_init.py:init():1011] starting run threads in backend 2025-12-25 07:01:23,113 INFO MainThread:10525 [wandb_run.py:_console_start():2506] atexit reg 2025-12-25 07:01:23,114 INFO MainThread:10525 [wandb_run.py:_redirect():2354] redirect: wrap_raw 2025-12-25 07:01:23,114 INFO MainThread:10525 [wandb_run.py:_redirect():2423] Wrapping output streams. 2025-12-25 07:01:23,114 INFO MainThread:10525 [wandb_run.py:_redirect():2446] Redirects installed. 2025-12-25 07:01:23,118 INFO MainThread:10525 [wandb_init.py:init():1049] run started, returning control to user process 2025-12-26 13:05:05,336 INFO wandb-AsyncioManager-main:10525 [service_client.py:_forward_responses():84] Reached EOF. 2025-12-26 13:05:05,336 INFO wandb-AsyncioManager-main:10525 [mailbox.py:close():137] Closing mailbox, abandoning 2 handles. 2025-12-26 13:05:05,884 ERROR wandb-AsyncioManager-main:10525 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions await fn() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 65, in _send_server_request await self._writer.drain() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/asyncio/streams.py", line 371, in drain await self._protocol._drain_helper() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2025-12-26 13:05:05,886 ERROR wandb-AsyncioManager-main:10525 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions await fn() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 65, in _send_server_request await self._writer.drain() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/asyncio/streams.py", line 371, in drain await self._protocol._drain_helper() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2025-12-26 13:05:05,886 ERROR wandb-AsyncioManager-main:10525 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions await fn() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 65, in _send_server_request await self._writer.drain() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/asyncio/streams.py", line 371, in drain await self._protocol._drain_helper() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2025-12-26 13:05:05,887 ERROR wandb-AsyncioManager-main:10525 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions await fn() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 65, in _send_server_request await self._writer.drain() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/asyncio/streams.py", line 371, in drain await self._protocol._drain_helper() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2025-12-26 13:05:05,887 ERROR wandb-AsyncioManager-main:10525 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions await fn() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 65, in _send_server_request await self._writer.drain() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/asyncio/streams.py", line 371, in drain await self._protocol._drain_helper() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost 2025-12-26 13:05:05,889 ERROR wandb-AsyncioManager-main:10525 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions await fn() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 65, in _send_server_request await self._writer.drain() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/asyncio/streams.py", line 371, in drain await self._protocol._drain_helper() File "/home/zkj/miniconda3/envs/unitree_groot/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost