2026-08-09 07:53:23,043 INFO    MainThread:288 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
2026-08-09 07:53:23,043 INFO    MainThread:288 [wandb_setup.py:_flush():81] Configure stats pid to 288
2026-08-09 07:53:23,043 INFO    MainThread:288 [wandb_setup.py:_flush():81] Loading settings from environment variables
2026-08-09 07:53:23,043 INFO    MainThread:288 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /tmp/official_smolvla_output_1786261973/wandb/run-20260809_075323-t1hbhznt/logs/debug.log
2026-08-09 07:53:23,043 INFO    MainThread:288 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /tmp/official_smolvla_output_1786261973/wandb/run-20260809_075323-t1hbhznt/logs/debug-internal.log
2026-08-09 07:53:23,044 INFO    MainThread:288 [wandb_init.py:init():766] calling init triggers
2026-08-09 07:53:23,044 INFO    MainThread:288 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
config: {'dataset': {'repo_id': 'lerobot/libero_plus', 'root': None, 'episodes': None, 'image_transforms': {'enable': False, 'max_num_transforms': 3, 'random_order': False, 'tfs': {'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'affine': {'weight': 1.0, 'type': 'RandomAffine', 'kwargs': {'degrees': [-5.0, 5.0], 'translate': [0.05, 0.05]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'return_uint8': False, 'depth_output_unit': 'mm', 'streaming': False, 'eval_split': 0.0}, 'env': None, 'policy': {'type': 'smolvla', 'n_obs_steps': 1, 'input_features': None, 'output_features': None, 'device': 'cuda', 'use_amp': False, 'use_peft': False, 'push_to_hub': False, 'repo_id': 'pepijn223/smolvla_libero_plus', 'private': None, 'tags': None, 'license': None, 'pretrained_path': '/root/.cache/huggingface/hub/models--lerobot--smolvla_libero_plus/snapshots/7bb70aa5bc92b82c9239142775d3a173103567ff', 'pretrained_revision': None, 'chunk_size': 50, 'n_action_steps': 50, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.MEAN_STD: 'MEAN_STD'>, 'ACTION': <NormalizationMode.MEAN_STD: 'MEAN_STD'>}, 'max_state_dim': 32, 'max_action_dim': 32, 'resize_imgs_with_padding': [512, 512], 'empty_cameras': 0, 'adapt_to_pi_aloha': False, 'use_delta_joint_actions_aloha': False, 'tokenizer_max_length': 48, 'num_steps': 10, 'use_cache': True, 'freeze_vision_encoder': True, 'train_expert_only': True, 'train_state_proj': True, 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.95], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-10, 'optimizer_grad_clip_norm': 10.0, 'scheduler_warmup_steps': 300, 'scheduler_decay_steps': 15000, 'scheduler_decay_lr': 3e-05, 'vlm_model_name': 'HuggingFaceTB/SmolVLM2-500M-Video-Instruct', 'load_vlm_weights': True, 'add_image_special_tokens': False, 'attention_mode': 'cross_attn', 'prefix_length': 0, 'pad_language_to': 'max_length', 'num_expert_layers': 0, 'num_vlm_layers': 16, 'self_attn_every_n_layers': 2, 'expert_width_multiplier': 0.75, 'min_period': 0.004, 'max_period': 4.0, 'rtc_config': None, 'compile_model': False, 'compile_mode': 'max-autotune'}, 'reward_model': None, 'output_dir': '/tmp/official_smolvla_output_1786261973', 'job_name': 'parc2026_smolvla_lora', 'resume': False, 'seed': 42, 'cudnn_deterministic': False, 'num_workers': 0, 'batch_size': 1, 'prefetch_factor': 4, 'persistent_workers': True, 'steps': 15000, 'env_eval_freq': 20000, 'log_freq': 100, 'eval_steps': 0, 'max_eval_samples': 0, 'tolerance_s': 0.0001, 'save_checkpoint': True, 'save_freq': 1000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-10, 'grad_clip_norm': 10.0, 'betas': [0.9, 0.95], 'eps': 1e-08}, 'scheduler': {'type': 'cosine_decay_with_warmup', 'num_warmup_steps': 300, 'num_decay_steps': 15000, 'peak_lr': 0.0001, 'decay_lr': 3e-05}, 'eval': {'n_episodes': 50, 'batch_size': 11, 'use_async_envs': True, 'recording': False, 'recording_repo_id': None, 'recording_private': False}, 'wandb': {'enable': True, 'disable_artifact': False, 'project': 'PARC2026', 'entity': None, 'notes': 'EXP_003 SmolVLA LoRA r=8 alpha=16 steps=15000', 'run_id': None, 'mode': None, 'add_tags': True}, 'peft': {'target_modules': None, 'full_training_modules': None, 'method_type': 'LORA', 'init_type': None, 'r': 8, 'lora_alpha': 16}, 'job': {'target': None, 'image': 'huggingface/lerobot-gpu:latest', 'timeout': '2d', 'detach': False, 'tags': []}, 'save_checkpoint_to_hub': False, 'sample_weighting': None, 'rename_map': {}, 'checkpoint_path': None, '_wandb': {}}
2026-08-09 07:53:23,044 INFO    MainThread:288 [wandb_init.py:init():814] starting backend
2026-08-09 07:53:23,325 INFO    MainThread:288 [wandb_init.py:init():829] sending inform_init request
2026-08-09 07:53:23,569 INFO    MainThread:288 [wandb_init.py:init():834] backend started and connected
2026-08-09 07:53:23,574 INFO    MainThread:288 [wandb_init.py:init():904] updated telemetry
2026-08-09 07:53:23,575 INFO    MainThread:288 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
2026-08-09 07:53:23,814 INFO    MainThread:288 [wandb_init.py:init():972] starting run threads in backend
2026-08-09 07:53:24,080 INFO    MainThread:288 [wandb_run.py:_console_start():2523] atexit reg
2026-08-09 07:53:24,080 INFO    MainThread:288 [wandb_run.py:_redirect():2373] redirect: wrap_raw
2026-08-09 07:53:24,080 INFO    MainThread:288 [wandb_run.py:_redirect():2442] Wrapping output streams.
2026-08-09 07:53:24,080 INFO    MainThread:288 [wandb_run.py:_redirect():2465] Redirects installed.
2026-08-09 07:53:24,086 INFO    MainThread:288 [wandb_init.py:init():1010] run started, returning control to user process
2026-08-09 08:49:39,762 INFO    wandb-AsyncioManager-main:288 [service_client.py:_forward_responses():122] Reached EOF.
2026-08-09 08:49:39,763 INFO    wandb-AsyncioManager-main:288 [mailbox.py:close():154] Closing mailbox, abandoning 3 handles.
2026-08-09 08:49:39,763 ERROR   wandb-AsyncioManager-main:288 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions
    await fn()
  File "/usr/local/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop
    await asyncio_compat.race(
  File "/usr/local/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 278, in race
    async with open_task_group(race=True) as tg:
  File "/usr/local/lib/python3.12/contextlib.py", line 217, in __aexit__
    await anext(self.gen)
  File "/usr/local/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 240, in open_task_group
    await task_group._wait_all(race=race, timeout=exit_timeout)
  File "/usr/local/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 180, in _wait_all
    raise exc
  File "/usr/local/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all
    result = await handle.wait_async(timeout=None)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async
    response = await self._handle.wait_async(timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async
    raise HandleAbandonedError()
wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError