2025-11-27 08:22:05,477 INFO    MainThread:1740673 [wandb_setup.py:_flush():81] Current SDK version is 0.22.3
2025-11-27 08:22:05,477 INFO    MainThread:1740673 [wandb_setup.py:_flush():81] Configure stats pid to 1740673
2025-11-27 08:22:05,477 INFO    MainThread:1740673 [wandb_setup.py:_flush():81] Loading settings from /user/hominhduy.nguyen/u15271/.config/wandb/settings
2025-11-27 08:22:05,477 INFO    MainThread:1740673 [wandb_setup.py:_flush():81] Loading settings from /mnt/vast-kisski/projects/kisski-umg-fairpact-2/VLA/binh/Smolvla_capstone_project/wandb/settings
2025-11-27 08:22:05,478 INFO    MainThread:1740673 [wandb_setup.py:_flush():81] Loading settings from environment variables
2025-11-27 08:22:05,478 INFO    MainThread:1740673 [wandb_init.py:setup_run_log_directory():706] Logging user logs to /projects/extern/kisski/kisski-umg-fairpact-2/dir.project/VLA/binh/Smolvla_capstone_project/outputs/train/2025-11-25/16-45-46_libero_100%_baseline_correct_ourdata_16layers/wandb/offline-run-20251127_082205-9pfedoc2/logs/debug.log
2025-11-27 08:22:05,478 INFO    MainThread:1740673 [wandb_init.py:setup_run_log_directory():707] Logging internal logs to /projects/extern/kisski/kisski-umg-fairpact-2/dir.project/VLA/binh/Smolvla_capstone_project/outputs/train/2025-11-25/16-45-46_libero_100%_baseline_correct_ourdata_16layers/wandb/offline-run-20251127_082205-9pfedoc2/logs/debug-internal.log
2025-11-27 08:22:05,478 INFO    MainThread:1740673 [wandb_init.py:init():833] calling init triggers
2025-11-27 08:22:05,478 INFO    MainThread:1740673 [wandb_init.py:init():838] wandb.init called with sweep_config: {}
config: {'dataset': {'repo_id': '.', 'root': '/mnt/vast-kisski/projects/kisski-umg-fairpact-2/VLA/LIBERO/merged_libero_scale_100_mask_depth_noops_lerobot', 'episodes': None, 'image_transforms': {'enable': True, 'max_num_transforms': 3, 'random_order': False, 'image_tfs': {'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}, 'crop_resize': {'weight': 1.0, 'type': 'RandomResizedCrop', 'kwargs': {'size': [256, 256], 'ratio': [1, 1], 'scale': [0.9, 0.95]}}, 'rotate': {'weight': 1.0, 'type': 'RandomRotate', 'kwargs': {'degrees': [-5, 5]}}}, 'wrist_tfs': {'hue': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'hue': [-0.05, 0.05]}}, 'contrast': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'contrast': [0.8, 1.2]}}, 'sharpness': {'weight': 1.0, 'type': 'SharpnessJitter', 'kwargs': {'sharpness': [0.5, 1.5]}}, 'brightness': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'brightness': [0.8, 1.2]}}, 'saturation': {'weight': 1.0, 'type': 'ColorJitter', 'kwargs': {'saturation': [0.5, 1.5]}}}}, 'revision': None, 'use_imagenet_stats': True, 'video_backend': 'torchcodec', 'vqa_data_path': None}, 'env': None, 'policy': {'type': 'smolvla', 'n_obs_steps': 1, 'normalization_mapping': {'VISUAL': <NormalizationMode.IDENTITY: 'IDENTITY'>, 'STATE': <NormalizationMode.MEAN_STD: 'MEAN_STD'>, 'ACTION': <NormalizationMode.MEAN_STD: 'MEAN_STD'>}, 'input_features': {'observation.images.image': {'type': <FeatureType.VISUAL: 'VISUAL'>, 'shape': [3, 256, 256]}, 'observation.images.wrist_image': {'type': <FeatureType.VISUAL: 'VISUAL'>, 'shape': [3, 256, 256]}, 'observation.images.image_mask': {'type': <FeatureType.VISUAL: 'VISUAL'>, 'shape': [3, 256, 256]}, 'observation.images.wrist_mask': {'type': <FeatureType.VISUAL: 'VISUAL'>, 'shape': [3, 256, 256]}, 'observation.images.object_of_interest_mask': {'type': <FeatureType.VISUAL: 'VISUAL'>, 'shape': [3, 256, 256]}, 'observation.images.object_of_interest_wrist_mask': {'type': <FeatureType.VISUAL: 'VISUAL'>, 'shape': [3, 256, 256]}, 'observation.state': {'type': <FeatureType.STATE: 'STATE'>, 'shape': [8]}, 'observation.states.ee_state': {'type': <FeatureType.STATE: 'STATE'>, 'shape': [6]}, 'observation.states.joint_state': {'type': <FeatureType.STATE: 'STATE'>, 'shape': [7]}, 'observation.states.gripper_state': {'type': <FeatureType.STATE: 'STATE'>, 'shape': [2]}}, 'output_features': {'action': {'type': <FeatureType.ACTION: 'ACTION'>, 'shape': [7]}}, 'device': 'cuda', 'use_amp': False, 'gradient_accumulation_steps': 1, 'chunk_size': 50, 'n_action_steps': 50, 'max_state_dim': 32, 'max_action_dim': 32, 'resize_imgs_with_padding': [512, 512], 'empty_cameras': 0, 'adapt_to_pi_aloha': False, 'use_delta_joint_actions_aloha': False, 'tokenizer_max_length': 48, 'num_steps': 10, 'use_cache': True, 'freeze_vision_encoder': True, 'train_expert_only': True, 'train_state_proj': True, 'optimizer_lr': 0.0001, 'optimizer_betas': [0.9, 0.95], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 1e-10, 'optimizer_grad_clip_norm': 10.0, 'scheduler_warmup_steps': 1000, 'scheduler_decay_steps': 30000, 'scheduler_decay_lr': 2.5e-06, 'vlm_model_name': 'HuggingFaceTB/SmolVLM2-500M-Video-Instruct', 'load_vlm_weights': True, 'add_image_special_tokens': False, 'attention_mode': 'cross_attn', 'prefix_length': 0, 'pad_language_to': 'max_length', 'num_expert_layers': 0, 'num_vlm_layers': 16, 'self_attn_every_n_layers': 2, 'expert_width_multiplier': 0.75, 'min_period': 0.004, 'max_period': 4.0}, 'output_dir': '/projects/extern/kisski/kisski-umg-fairpact-2/dir.project/VLA/binh/Smolvla_capstone_project/outputs/train/2025-11-25/16-45-46_libero_100%_baseline_correct_ourdata_16layers', 'job_name': 'libero_100%_baseline_correct_ourdata_16layers', 'resume': True, 'seed': 42, 'num_workers': 8, 'batch_size': 64, 'steps': 100000, 'eval_freq': 20000, 'log_freq': 10, 'save_checkpoint': True, 'save_freq': 10000, 'use_policy_training_preset': True, 'optimizer': {'type': 'adamw', 'lr': 0.0001, 'weight_decay': 1e-10, 'grad_clip_norm': 10.0, 'betas': [0.9, 0.95], 'eps': 1e-08}, 'scheduler': {'type': 'cosine_decay_with_warmup', 'num_warmup_steps': 1000, 'num_decay_steps': 30000, 'peak_lr': 0.0001, 'decay_lr': 2.5e-06}, 'eval': {'n_episodes': 50, 'batch_size': 50, 'use_async_envs': False}, 'wandb': {'enable': True, 'disable_artifact': True, 'project': 'smolvla', 'entity': 'Robotics_VLA', 'notes': None, 'run_id': None, 'mode': 'offline'}, '_wandb': {}}
2025-11-27 08:22:05,478 INFO    MainThread:1740673 [wandb_init.py:init():881] starting backend
2025-11-27 08:22:07,375 INFO    MainThread:1740673 [wandb_init.py:init():884] sending inform_init request
2025-11-27 08:22:07,391 INFO    MainThread:1740673 [wandb_init.py:init():892] backend started and connected
2025-11-27 08:22:07,393 INFO    MainThread:1740673 [wandb_init.py:init():962] updated telemetry
2025-11-27 08:22:07,425 INFO    MainThread:1740673 [wandb_init.py:init():986] communicating run to backend with 90.0 second timeout
2025-11-27 08:22:07,623 INFO    MainThread:1740673 [wandb_init.py:init():1033] starting run threads in backend
2025-11-27 08:22:08,898 INFO    MainThread:1740673 [wandb_run.py:_console_start():2506] atexit reg
2025-11-27 08:22:08,898 INFO    MainThread:1740673 [wandb_run.py:_redirect():2354] redirect: wrap_raw
2025-11-27 08:22:08,898 INFO    MainThread:1740673 [wandb_run.py:_redirect():2423] Wrapping output streams.
2025-11-27 08:22:08,898 INFO    MainThread:1740673 [wandb_run.py:_redirect():2446] Redirects installed.
2025-11-27 08:22:08,921 INFO    MainThread:1740673 [wandb_init.py:init():1073] run started, returning control to user process
2025-11-27 14:52:02,433 INFO    wandb-AsyncioManager-main:1740673 [service_client.py:_forward_responses():80] Reached EOF.
2025-11-27 14:52:02,442 INFO    wandb-AsyncioManager-main:1740673 [mailbox.py:close():137] Closing mailbox, abandoning 0 handles.
2025-11-27 14:52:02,547 ERROR   wandb-AsyncioManager-main:1740673 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/projects/extern/kisski/kisski-umg-fairpact-2/dir.project/miniconda3/envs/smol/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/projects/extern/kisski/kisski-umg-fairpact-2/dir.project/miniconda3/envs/smol/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/projects/extern/kisski/kisski-umg-fairpact-2/dir.project/miniconda3/envs/smol/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/projects/extern/kisski/kisski-umg-fairpact-2/dir.project/miniconda3/envs/smol/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/projects/extern/kisski/kisski-umg-fairpact-2/dir.project/miniconda3/envs/smol/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost