phase1/wandb/debug.log

2026-08-06 13:49:58,672 INFO    MainThread:4075244 [wandb_setup.py:_flush():81] Current SDK version is 0.28.1
2026-08-06 13:49:58,672 INFO    MainThread:4075244 [wandb_setup.py:_flush():81] Configure stats pid to 4075244
2026-08-06 13:49:58,672 INFO    MainThread:4075244 [wandb_setup.py:_flush():81] Loading settings from environment variables
2026-08-06 13:49:58,672 INFO    MainThread:4075244 [wandb_init.py:setup_run_log_directory():729] Logging user logs to /mnt/data/experiments/snapflow-multi-obj/phase1/wandb/run-20260806_134958-uwqkxwz8/logs/debug.log
2026-08-06 13:49:58,672 INFO    MainThread:4075244 [wandb_init.py:setup_run_log_directory():730] Logging internal logs to /mnt/data/experiments/snapflow-multi-obj/phase1/wandb/run-20260806_134958-uwqkxwz8/logs/debug-internal.log
2026-08-06 13:49:58,672 INFO    MainThread:4075244 [wandb_init.py:init():772] calling init triggers
2026-08-06 13:49:58,672 INFO    MainThread:4075244 [wandb_init.py:init():777] wandb.init called with sweep_config: {}
config: {'_wandb': {}}
2026-08-06 13:49:58,672 INFO    MainThread:4075244 [wandb_init.py:init():820] starting backend
2026-08-06 13:49:58,672 INFO    MainThread:4075244 [wandb_init.py:init():835] sending inform_init request
2026-08-06 13:49:58,796 INFO    MainThread:4075244 [wandb_init.py:init():840] backend started and connected
2026-08-06 13:49:58,799 INFO    MainThread:4075244 [wandb_init.py:init():910] updated telemetry
2026-08-06 13:49:58,809 INFO    MainThread:4075244 [wandb_init.py:init():933] communicating run to backend with 90.0 second timeout
2026-08-06 13:49:59,540 INFO    MainThread:4075244 [wandb_init.py:init():978] starting run threads in backend
2026-08-06 13:49:59,644 INFO    MainThread:4075244 [wandb_run.py:_console_start():2621] atexit reg
2026-08-06 13:49:59,644 INFO    MainThread:4075244 [wandb_run.py:_redirect():2471] redirect: wrap_raw
2026-08-06 13:49:59,644 INFO    MainThread:4075244 [wandb_run.py:_redirect():2540] Wrapping output streams.
2026-08-06 13:49:59,644 INFO    MainThread:4075244 [wandb_run.py:_redirect():2563] Redirects installed.
2026-08-06 13:49:59,647 INFO    MainThread:4075244 [wandb_init.py:init():1016] run started, returning control to user process
2026-08-06 13:49:59,648 INFO    MainThread:4075244 [wandb_run.py:_config_callback():1346] config_cb None None {'phase': 'phase1', 'dataset': 'Daankrol/pick-and-place-multi-obj', 'pretrained': 'lerobot/pi05_base', 'seed': 42, 'val_split': 0.1, 'val_split_seed': 42, 'augment': True, 'augment_n_subset': 3, 'batch_size': 16, 'devices': 1, 'accumulate_grad_batches': 1, 'epochs': 7, 'train_frames': 86093, 'global_batch': 16, 'steps_per_epoch': 5380, 'total_steps': 37660, 'warmup_steps': 1883, 'warmup_frac': 0.05, 'snapflow_alpha': 0.5, 'snapflow_lambda': 0.1, 'snapflow_steps': 1}
2026-08-06 13:50:02,686 INFO    MainThread:4075244 [wandb_run.py:_config_callback():1346] config_cb None None {'paligemma_variant': 'gemma_2b', 'action_expert_variant': 'gemma_300m', 'dtype': 'bfloat16', 'n_obs_steps': 1, 'chunk_size': 50, 'n_action_steps': 50, 'max_state_dim': 32, 'max_action_dim': 32, 'num_inference_steps': 10, 'time_sampling_beta_alpha': 1.5, 'time_sampling_beta_beta': 1.0, 'time_sampling_scale': 0.999, 'time_sampling_offset': 0.001, 'min_period': 0.004, 'max_period': 4.0, 'use_random_input_noise': True, 'snapflow_enabled': False, 'snapflow_alpha': 0.5, 'snapflow_lambda': 1.0, 'snapflow_num_inference_steps': 1, 'image_resolution': [224, 224], 'empty_cameras': 0, 'tokenizer_max_length': 200, 'gradient_checkpointing': True, 'compile_mode': 'max-autotune', 'freeze_vision_encoder': False, 'train_expert_only': False, 'normalization_mode': 'QUANTILES', 'optimizer_lr': 2.5e-05, 'optimizer_betas': [0.9, 0.95], 'optimizer_eps': 1e-08, 'optimizer_weight_decay': 0.01, 'optimizer_grad_clip_norm': 1.0, 'scheduler_warmup_steps': 1883, 'scheduler_decay_steps': None, 'scheduler_decay_lr': 2.5e-06, 'dataset_stats': {'observation.images.base_0_rgb': {'name': 'observation.images.base_0_rgb', 'shape': [3, 224, 224], 'type': 'VISUAL', 'mean': [0.0, 0.0, 0.0], 'std': [1.0, 1.0, 1.0], 'q01': [-1.0, -1.0, -1.0], 'q99': [1.0, 1.0, 1.0]}, 'observation.images.left_wrist_0_rgb': {'name': 'observation.images.left_wrist_0_rgb', 'shape': [3, 224, 224], 'type': 'VISUAL', 'mean': [0.0, 0.0, 0.0], 'std': [1.0, 1.0, 1.0], 'q01': [-1.0, -1.0, -1.0], 'q99': [1.0, 1.0, 1.0]}, 'observation.images.right_wrist_0_rgb': {'name': 'observation.images.right_wrist_0_rgb', 'shape': [3, 224, 224], 'type': 'VISUAL', 'mean': [0.0, 0.0, 0.0], 'std': [1.0, 1.0, 1.0], 'q01': [-1.0, -1.0, -1.0], 'q99': [1.0, 1.0, 1.0]}, 'observation.state': {'name': 'observation.state', 'shape': [32], 'type': 'STATE', 'mean': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'std': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], 'q01': [-1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0], 'q99': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]}, 'action': {'name': 'action', 'shape': [32], 'type': 'ACTION', 'mean': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'std': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], 'q01': [-1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0], 'q99': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]}}}
2026-08-06 23:54:26,913 INFO    MainThread:4075244 [wandb_run.py:_finish():2383] finishing run daankrol/physicalai_snapflow/uwqkxwz8
2026-08-06 23:54:26,914 INFO    MainThread:4075244 [wandb_run.py:_atexit_cleanup():2588] got exitcode: 0
2026-08-06 23:54:26,914 INFO    MainThread:4075244 [wandb_run.py:_restore():2570] restore
2026-08-06 23:54:26,914 INFO    MainThread:4075244 [wandb_run.py:_restore():2576] restore done
2026-08-06 23:54:28,757 INFO    MainThread:4075244 [wandb_run.py:_footer_sync_info():3993] logging synced files