2026-05-16 20:45:26,914 INFO MainThread:2610908 [wandb_setup.py:_flush():81] Current SDK version is 0.27.0 2026-05-16 20:45:26,914 INFO MainThread:2610908 [wandb_setup.py:_flush():81] Configure stats pid to 2610908 2026-05-16 20:45:26,914 INFO MainThread:2610908 [wandb_setup.py:_flush():81] Loading settings from environment variables 2026-05-16 20:45:26,914 INFO MainThread:2610908 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /home/romain/Desktop/robot_learning/outputs/train/diffusion_laptop_20260516_204518/wandb/run-20260516_204526-1c59314z/logs/debug.log 2026-05-16 20:45:26,914 INFO MainThread:2610908 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /home/romain/Desktop/robot_learning/outputs/train/diffusion_laptop_20260516_204518/wandb/run-20260516_204526-1c59314z/logs/debug-internal.log 2026-05-16 20:45:26,914 INFO MainThread:2610908 [wandb_init.py:init():851] calling init triggers 2026-05-16 20:45:26,914 INFO MainThread:2610908 [wandb_init.py:init():856] wandb.init called with sweep_config: {} config: {'dataset': {'repo_id': 'local/act_collected_episodes', 'root': '/home/romain/Desktop/robot_learning/dataset/act_collected_episodes', 'episodes': None, 'image_transforms': {'enable': True, 'max_num_transforms': 3, 'random_order': False, 'same_cloth_color': True, 'tfs': {'color_jitter': {'weight': 1.5, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.7, 1.3], 'contrast': [0.7, 1.3], 'saturation': [1.0, 1.0], 'hue': [0.0, 0.0]}}, 'grayscale': {'weight': 0.0, 'type': 'RandomGrayscale', 'kwargs': {'p': 1.0}}, 'shadow': {'weight': 0.8, 'type': 'RandomShadow', 'kwargs': {'num_shadows': [1, 2], 'intensity': [0.4, 0.7], 'blur_sigma': [4.0, 20.0]}}, 'highlight': {'weight': 0.5, 'type': 'RandomHighlight', 'kwargs': {'num_highlights': [0, 1], 'intensity': [0.2, 0.5], 'sigma': [20.0, 60.0]}}, 'gamma': {'weight': 0.7, 'type': 'RandomGamma', 'kwargs': {'gamma': [0.6, 1.4]}}, 'blur': {'weight': 0.0, 'type': 'GaussianBlur', 'kwargs': {'kernel_size': [3, 3], 'sigma': [0.1, 1.5]}}, 'jpeg': {'weight': 0.0, 'type': 'RandomJPEG', 'kwargs': {'quality': [60, 95]}}, 'crop': {'weight': 0.5, 'type': 'RandomCropPreserveSize', 'kwargs': {'scale': [0.97, 1.0], 'ratio': [0.99, 1.01]}}, 'affine': {'weight': 0.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'pyav', 'return_uint8': False, 'streaming': False}, 'env': None, 'policy': {'type': 'diffusion', 'n_obs_steps': 2, 'input_features': {}, 'output_features': {}, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': None, 'private': None, 'tags': None, 'license': None, 'pretrained_path': None, 'horizon': 16, 'n_action_steps': 8, 'normalization_mapping': {'VISUAL': , 'STATE': , 'ACTION': }, 'drop_n_last_frames': 7, 'vision_backbone_type': 'dinov2', 'vision_encoder_name': 'facebook/dinov2-small', 'freeze_vision_encoder': True, 'vision_backbone': 'resnet18', 'resize_shape': [224, 224], 'crop_ratio': 1.0, 'crop_shape': None, 'crop_is_random': True, 'pretrained_backbone_weights': 'ResNet18_Weights.IMAGENET1K_V1', 'use_group_norm': False, 'spatial_softmax_num_keypoints': 32, 'use_separate_rgb_encoder_per_camera': True, 'down_dims': [512, 1024, 2048], 'kernel_size': 5, 'n_groups': 8, 'diffusion_step_embed_dim': 128, 'use_film_scale_modulation': True, 'noise_scheduler_type': 'DDPM', 'num_train_timesteps': 100, 'beta_schedule': 'squaredcos_cap_v2', 'beta_start': 0.0001, 'beta_end': 0.02, 'prediction_type': 'epsilon', 'clip_sample': True, 'clip_sample_range': 1.0, 'num_inference_steps': 100, 'compile_model': False, 'compile_mode': 'reduce-overhead', 'do_mask_loss_for_padding': False, 'optimizer_lr': 0.0001, 'optimizer_betas': [0.95, 0.999], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-06, 'scheduler_name': 'cosine', 'scheduler_warmup_steps': 500}, 'reward_model': None, 'output_dir': '/home/romain/Desktop/robot_learning/outputs/train/diffusion_laptop_20260516_204518', 'job_name': 'diffusion_laptop_20260516_204518', 'resume': False, 'seed': 1000, 'cudnn_deterministic': False, 'num_workers': 16, 'batch_size': 64, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 200000, 'eval_freq': 0, 'log_freq': 20, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 20000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adam', 'lr': 0.0001, 'weight_decay': 1e-06, 'grad_clip_norm': 10.0, 'betas': [0.95, 0.999], 'eps': 1e-08}, 'scheduler': {'type': 'diffuser', 'num_warmup_steps': 500, 'name': 'cosine'}, 'eval': {'n_episodes': 50, 'batch_size': 44, 'use_async_envs': True}, 'wandb': {'enable': True, 'disable_artifact': False, 'project': 'lerobot', 'entity': None, 'notes': None, 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': None, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}} 2026-05-16 20:45:26,914 INFO MainThread:2610908 [wandb_init.py:init():899] starting backend 2026-05-16 20:45:27,126 INFO MainThread:2610908 [wandb_init.py:init():914] sending inform_init request 2026-05-16 20:45:27,383 INFO MainThread:2610908 [wandb_init.py:init():919] backend started and connected 2026-05-16 20:45:27,384 INFO MainThread:2610908 [wandb_init.py:init():989] updated telemetry 2026-05-16 20:45:27,387 INFO MainThread:2610908 [wandb_init.py:init():1012] communicating run to backend with 90.0 second timeout 2026-05-16 20:45:27,718 INFO MainThread:2610908 [wandb_init.py:init():1057] starting run threads in backend 2026-05-16 20:45:27,756 INFO MainThread:2610908 [wandb_run.py:_console_start():2509] atexit reg 2026-05-16 20:45:27,756 INFO MainThread:2610908 [wandb_run.py:_redirect():2359] redirect: wrap_raw 2026-05-16 20:45:27,756 INFO MainThread:2610908 [wandb_run.py:_redirect():2428] Wrapping output streams. 2026-05-16 20:45:27,757 INFO MainThread:2610908 [wandb_run.py:_redirect():2451] Redirects installed. 2026-05-16 20:45:27,758 INFO MainThread:2610908 [wandb_init.py:init():1095] run started, returning control to user process 2026-05-17 11:43:44,157 INFO wandb-AsyncioManager-main:2610908 [service_client.py:_forward_responses():122] Reached EOF. 2026-05-17 11:43:44,159 INFO wandb-AsyncioManager-main:2610908 [mailbox.py:close():154] Closing mailbox, abandoning 2 handles. 2026-05-17 11:43:51,873 ERROR wandb-AsyncioManager-main:2610908 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/romain/Desktop/robot_learning/lerobot/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/romain/Desktop/robot_learning/lerobot/.venv/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 105, in loop await self._poll_and_print(timeout=None) File "/home/romain/Desktop/robot_learning/lerobot/.venv/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 133, in _poll_and_print handle = await self._interface.deliver_async(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/romain/Desktop/robot_learning/lerobot/.venv/lib/python3.12/site-packages/wandb/sdk/interface/interface_sock.py", line 58, in deliver_async handle = await self._client.deliver(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/romain/Desktop/robot_learning/lerobot/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_client.py", line 57, in deliver handle = self._mailbox.require_response(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/romain/Desktop/robot_learning/lerobot/.venv/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox.py", line 89, in require_response raise MailboxClosedError() wandb.sdk.mailbox.mailbox.MailboxClosedError 2026-05-17 11:43:54,015 ERROR wandb-AsyncioManager-main:2610908 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. Traceback (most recent call last): File "/home/romain/Desktop/robot_learning/lerobot/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions await fn() File "/home/romain/Desktop/robot_learning/lerobot/.venv/lib/python3.12/site-packages/wandb/sdk/lib/run_stopping.py", line 87, in loop should_stop = await self._check_should_stop() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/romain/Desktop/robot_learning/lerobot/.venv/lib/python3.12/site-packages/wandb/sdk/lib/run_stopping.py", line 115, in _check_should_stop handle = await self._interface.deliver_async(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/romain/Desktop/robot_learning/lerobot/.venv/lib/python3.12/site-packages/wandb/sdk/interface/interface_sock.py", line 58, in deliver_async handle = await self._client.deliver(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/romain/Desktop/robot_learning/lerobot/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_client.py", line 57, in deliver handle = self._mailbox.require_response(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/romain/Desktop/robot_learning/lerobot/.venv/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox.py", line 89, in require_response raise MailboxClosedError() wandb.sdk.mailbox.mailbox.MailboxClosedError