| 2026-01-30 20:35:17,575 INFO MainThread:1266645 [wandb_setup.py:_flush():81] Current SDK version is 0.21.4 |
| 2026-01-30 20:35:17,575 INFO MainThread:1266645 [wandb_setup.py:_flush():81] Configure stats pid to 1266645 |
| 2026-01-30 20:35:17,575 INFO MainThread:1266645 [wandb_setup.py:_flush():81] Loading settings from /home/zeno/.config/wandb/settings |
| 2026-01-30 20:35:17,575 INFO MainThread:1266645 [wandb_setup.py:_flush():81] Loading settings from /home/zeno/wandb/settings |
| 2026-01-30 20:35:17,575 INFO MainThread:1266645 [wandb_setup.py:_flush():81] Loading settings from environment variables |
| 2026-01-30 20:35:17,575 INFO MainThread:1266645 [wandb_init.py:setup_run_log_directory():686] Logging user logs to /home/zeno/ACT-wholebody-torque/models/ACT-final-promax/wandb/run-20260130_203517-hejdf200/logs/debug.log |
| 2026-01-30 20:35:17,575 INFO MainThread:1266645 [wandb_init.py:setup_run_log_directory():687] Logging internal logs to /home/zeno/ACT-wholebody-torque/models/ACT-final-promax/wandb/run-20260130_203517-hejdf200/logs/debug-internal.log |
| 2026-01-30 20:35:17,575 INFO MainThread:1266645 [wandb_init.py:init():813] calling init triggers |
| 2026-01-30 20:35:17,575 INFO MainThread:1266645 [wandb_init.py:init():818] wandb.init called with sweep_config: {} |
| config: {'dataset': {'repo_id': 'ACT-final-V30', 'root': '/home/zeno/ACT-wholebody-torque/data/ACT-final-V30', 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'pyav', 'streaming': False}, 'env': None, 'policy': {'type': 'act', 'n_obs_steps': 1, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'push_to_hub': False, 'repo_id': 'local/ACT-final-promax', 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'chunk_size': 100, 'n_action_steps': 100, 'normalization_mapping': {'VISUAL': <NormalizationMode.MEAN_STD: 'MEAN_STD'>, 'STATE': <NormalizationMode.MEAN_STD: 'MEAN_STD'>, 'ACTION': <NormalizationMode.MEAN_STD: 'MEAN_STD'>}, 'vision_backbone': 'resnet18', 'pretrained_backbone_weights': 'ResNet18_Weights.IMAGENET1K_V1', 'replace_final_stride_with_dilation': False, 'pre_norm': False, 'dim_model': 512, 'n_heads': 8, 'dim_feedforward': 3200, 'feedforward_activation': 'relu', 'n_encoder_layers': 4, 'n_decoder_layers': 1, 'use_vae': True, 'latent_dim': 32, 'n_vae_encoder_layers': 4, 'temporal_ensemble_coeff': None, 'dropout': 0.1, 'kl_weight': 10.0, 'use_torque': True, 'action_dim_offset': 0, 'use_base': True, 'optimizer_lr': 1e-05, 'optimizer_weight_decay': 0.0001, 'optimizer_lr_backbone': 1e-05}, 'output_dir': '/home/zeno/ACT-wholebody-torque/models/ACT-final-promax', 'job_name': 'ACT-final-promax', 'resume': False, 'seed': 1000, 'num_workers': 4, 'batch_size': 32, 'steps': 40000, 'eval_freq': 20000, 'log_freq': 100, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 1e-05, 'weight_decay': 0.0001, 'grad_clip_norm': 10.0, 'betas': [0.9, 0.999], 'eps': 1e-08}, 'scheduler': None, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': False}, 'wandb': {'enable': True, 'disable_artifact': False, 'project': 'ACT-final-promax', 'entity': None, 'notes': None, 'run_id': None, 'mode': None}, 'checkpoint_path': None, 'rename_map': {}, '_wandb': {}} |
| 2026-01-30 20:35:17,575 INFO MainThread:1266645 [wandb_init.py:init():854] starting backend |
| 2026-01-30 20:35:17,779 INFO MainThread:1266645 [wandb_init.py:init():857] sending inform_init request |
| 2026-01-30 20:35:17,784 INFO MainThread:1266645 [wandb_init.py:init():865] backend started and connected |
| 2026-01-30 20:35:17,787 INFO MainThread:1266645 [wandb_init.py:init():936] updated telemetry |
| 2026-01-30 20:35:17,788 INFO MainThread:1266645 [wandb_init.py:init():960] communicating run to backend with 90.0 second timeout |
| 2026-01-30 20:35:20,625 INFO MainThread:1266645 [wandb_init.py:init():1011] starting run threads in backend |
| 2026-01-30 20:35:20,697 INFO MainThread:1266645 [wandb_run.py:_console_start():2506] atexit reg |
| 2026-01-30 20:35:20,697 INFO MainThread:1266645 [wandb_run.py:_redirect():2354] redirect: wrap_raw |
| 2026-01-30 20:35:20,697 INFO MainThread:1266645 [wandb_run.py:_redirect():2423] Wrapping output streams. |
| 2026-01-30 20:35:20,697 INFO MainThread:1266645 [wandb_run.py:_redirect():2446] Redirects installed. |
| 2026-01-30 20:35:20,698 INFO MainThread:1266645 [wandb_init.py:init():1049] run started, returning control to user process |
| 2026-01-30 23:30:14,198 INFO wandb-AsyncioManager-main:1266645 [service_client.py:_forward_responses():84] Reached EOF. |
| 2026-01-30 23:30:14,198 INFO wandb-AsyncioManager-main:1266645 [mailbox.py:close():137] Closing mailbox, abandoning 1 handles. |
| 2026-01-30 23:30:14,593 ERROR wandb-AsyncioManager-main:1266645 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 65, in _send_server_request |
| await self._writer.drain() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2026-01-30 23:30:14,594 ERROR wandb-AsyncioManager-main:1266645 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 65, in _send_server_request |
| await self._writer.drain() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2026-01-30 23:30:14,594 ERROR wandb-AsyncioManager-main:1266645 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 65, in _send_server_request |
| await self._writer.drain() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2026-01-30 23:30:14,595 ERROR wandb-AsyncioManager-main:1266645 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 65, in _send_server_request |
| await self._writer.drain() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2026-01-30 23:30:14,595 ERROR wandb-AsyncioManager-main:1266645 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 65, in _send_server_request |
| await self._writer.drain() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2026-01-30 23:30:14,595 ERROR wandb-AsyncioManager-main:1266645 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 65, in _send_server_request |
| await self._writer.drain() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/zeno/miniconda3/envs/lerobot/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
|
|
