Upload policy_checkpoints/CoffeePreparation_D1/2025-02-01-23-45-24_E32.0-CoffeePreparation_D1-500/wandb/run-20250202_122743-wd94ve42/logs/debug.log with huggingface_hub
Browse files
policy_checkpoints/CoffeePreparation_D1/2025-02-01-23-45-24_E32.0-CoffeePreparation_D1-500/wandb/run-20250202_122743-wd94ve42/logs/debug.log
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2025-02-02 12:27:43,584 INFO MainThread:3658287 [wandb_setup.py:_flush():68] Current SDK version is 0.19.5
|
| 2 |
+
2025-02-02 12:27:43,584 INFO MainThread:3658287 [wandb_setup.py:_flush():68] Configure stats pid to 3658287
|
| 3 |
+
2025-02-02 12:27:43,584 INFO MainThread:3658287 [wandb_setup.py:_flush():68] Loading settings from /mnt/data1/kevin/.config/wandb/settings
|
| 4 |
+
2025-02-02 12:27:43,584 INFO MainThread:3658287 [wandb_setup.py:_flush():68] Loading settings from /mnt/data1/kevin/autom/adaflow/wandb/settings
|
| 5 |
+
2025-02-02 12:27:43,584 INFO MainThread:3658287 [wandb_setup.py:_flush():68] Loading settings from environment variables
|
| 6 |
+
2025-02-02 12:27:43,584 INFO MainThread:3658287 [wandb_init.py:setup_run_log_directory():637] Logging user logs to /mnt/data1/kevin/autom/adaflow/policy_checkpoints/CoffeePreparation_D1/2025-02-01-23-45-24_E32.0-CoffeePreparation_D1-500/wandb/run-20250202_122743-wd94ve42/logs/debug.log
|
| 7 |
+
2025-02-02 12:27:43,584 INFO MainThread:3658287 [wandb_init.py:setup_run_log_directory():638] Logging internal logs to /mnt/data1/kevin/autom/adaflow/policy_checkpoints/CoffeePreparation_D1/2025-02-01-23-45-24_E32.0-CoffeePreparation_D1-500/wandb/run-20250202_122743-wd94ve42/logs/debug-internal.log
|
| 8 |
+
2025-02-02 12:27:43,585 INFO MainThread:3658287 [wandb_init.py:init():756] calling init triggers
|
| 9 |
+
2025-02-02 12:27:43,585 INFO MainThread:3658287 [wandb_init.py:init():761] wandb.init called with sweep_config: {}
|
| 10 |
+
config: {'name': 'train_diffusion_unet_image', '_target_': 'adaflow.workspace.train_diffusion_unet_image_workspace.TrainDiffusionUnetImageWorkspace', 'task_name': 'coffee_image', 'dataset_type': 'ph', 'shape_meta': {'obs': {'agentview_image': {'shape': [3, 84, 84], 'type': 'rgb'}, 'robot0_eye_in_hand_image': {'shape': [3, 84, 84], 'type': 'rgb'}, 'robot0_eef_pos': {'shape': [3]}, 'robot0_eef_quat': {'shape': [4]}, 'robot0_gripper_qpos': {'shape': [2]}}, 'action': {'shape': [10]}}, 'exp_name': 'default', 'horizon': 16, 'n_obs_steps': 2, 'n_action_steps': 8, 'n_latency_steps': 0, 'dataset_obs_steps': 2, 'past_action_visible': False, 'keypoint_visible_rate': 1.0, 'obs_as_global_cond': True, 'normalize_pixel': False, 'policy': {'_target_': 'adaflow.policy.diffusion_unet_image_policy.DiffusionUnetImagePolicy', 'shape_meta': {'obs': {'agentview_image': {'shape': [3, 84, 84], 'type': 'rgb'}, 'robot0_eye_in_hand_image': {'shape': [3, 84, 84], 'type': 'rgb'}, 'robot0_eef_pos': {'shape': [3]}, 'robot0_eef_quat': {'shape': [4]}, 'robot0_gripper_qpos': {'shape': [2]}}, 'action': {'shape': [10]}}, 'noise_scheduler': {'_target_': 'diffusers.schedulers.scheduling_ddpm.DDPMScheduler', 'num_train_timesteps': 100, 'beta_start': 0.0001, 'beta_end': 0.02, 'beta_schedule': 'squaredcos_cap_v2', 'variance_type': 'fixed_small', 'clip_sample': True, 'prediction_type': 'epsilon'}, 'obs_encoder': {'_target_': 'adaflow.model.vision.multi_image_obs_encoder.MultiImageObsEncoder', 'shape_meta': {'obs': {'agentview_image': {'shape': [3, 84, 84], 'type': 'rgb'}, 'robot0_eye_in_hand_image': {'shape': [3, 84, 84], 'type': 'rgb'}, 'robot0_eef_pos': {'shape': [3]}, 'robot0_eef_quat': {'shape': [4]}, 'robot0_gripper_qpos': {'shape': [2]}}, 'action': {'shape': [10]}}, 'rgb_model': {'_target_': 'adaflow.model.vision.model_getter.get_resnet', 'name': 'resnet18', 'weights': None}, 'resize_shape': None, 'crop_shape': [76, 76], 'random_crop': True, 'use_group_norm': True, 'share_rgb_model': False, 'imagenet_norm': True, 'depth_norm': True, 'norm_args': {'robot0_eye_in_hand_depth': {'clamp_max': 1.0, 'mean': 0.177, 'std': 0.1699}, 'agentview_depth': {'clamp_max': 1.75, 'mean': 0.778, 'std': 0.349}}}, 'horizon': 16, 'n_action_steps': 8, 'n_obs_steps': 2, 'num_inference_steps': 100, 'obs_as_global_cond': True, 'diffusion_step_embed_dim': 128, 'down_dims': [512, 1024, 2048], 'kernel_size': 5, 'n_groups': 8, 'cond_predict_scale': True}, 'ema': {'_target_': 'adaflow.model.diffusion.ema_model.EMAModel', 'update_after_step': 0, 'inv_gamma': 1.0, 'power': 0.75, 'min_value': 0.0, 'max_value': 0.9999}, 'dataloader': {'batch_size': 64, 'num_workers': 4, 'shuffle': True, 'pin_memory': True, 'persistent_workers': False}, 'val_dataloader': {'batch_size': 64, 'num_workers': 8, 'shuffle': False, 'pin_memory': True, 'persistent_workers': False}, 'optimizer': {'_target_': 'torch.optim.AdamW', 'lr': 0.0001, 'betas': [0.95, 0.999], 'eps': 1e-08, 'weight_decay': 1e-06}, 'training': {'device': 'cuda:0', 'seed': 42, 'debug': False, 'resume': True, 'resume_dir': None, 'lr_scheduler': 'cosine', 'lr_warmup_steps': 500, 'num_epochs': 500, 'gradient_accumulate_every': 1, 'use_ema': True, 'freeze_encoder': False, 'rollout_every': 20, 'checkpoint_every': 20, 'val_every': 5, 'sample_every': 5, 'max_train_steps': None, 'max_val_steps': None, 'tqdm_interval_sec': 1.0, 'pretrained_ckpt': None}, 'logging': {'project': 'adaflow', 'resume': True, 'mode': 'online', 'name': 'E32.0-CoffeePreparation_D1-500_CoffeePreparation_D1_2025-02-01-23-45-24', 'tags': ['train_diffusion_unet_image', 'coffee_image', 'default'], 'id': None, 'group': None}, 'checkpoint': {'topk': {'monitor_key': 'test_mean_score', 'mode': 'max', 'k': 1, 'format_str': 'epoch={epoch:04d}-test_mean_score={test_mean_score:.3f}.ckpt'}, 'save_last_ckpt': True, 'save_last_snapshot': False}, 'multi_run': {'run_dir': '/mnt/data1/kevin/autom/adaflow//policy_checkpoints/CoffeePreparation_D1/2025-02-01-23-45-24_E32.0-CoffeePreparation_D1-500', 'wandb_name_base': '2025.02.02-12.25.49_train_diffusion_unet_image_coffee_image_ph'}, 'task': {'name': 'coffee_image', 'shape_meta': {'obs': {'agentview_image': {'shape': [3, 84, 84], 'type': 'rgb'}, 'robot0_eye_in_hand_image': {'shape': [3, 84, 84], 'type': 'rgb'}, 'robot0_eef_pos': {'shape': [3]}, 'robot0_eef_quat': {'shape': [4]}, 'robot0_gripper_qpos': {'shape': [2]}}, 'action': {'shape': [10]}}, 'task_name': 'coffee', 'dataset_type': 'ph', 'dataset_path': '/mnt/data1/kevin/autom/demo-aug//datasets/generated/CoffeePreparation_D1/2025-02-01-23-45-24/merged_E32.0-CoffeePreparation_D1-500_all.hdf5', 'abs_action': True, 'env_runner': {'_target_': 'adaflow.env_runner.robomimic_image_runner.RobomimicImageRunner', 'dataset_path': '/mnt/data1/kevin/autom/demo-aug//datasets/generated/CoffeePreparation_D1/2025-02-01-23-45-24/merged_E32.0-CoffeePreparation_D1-500_all.hdf5', 'shape_meta': {'obs': {'agentview_image': {'shape': [3, 84, 84], 'type': 'rgb'}, 'robot0_eye_in_hand_image': {'shape': [3, 84, 84], 'type': 'rgb'}, 'robot0_eef_pos': {'shape': [3]}, 'robot0_eef_quat': {'shape': [4]}, 'robot0_gripper_qpos': {'shape': [2]}}, 'action': {'shape': [10]}}, 'n_train': 1, 'n_train_vis': 1, 'train_start_idx': 0, 'n_test': 50, 'n_test_vis': 50, 'test_start_seed': 100000, 'max_steps': 1100, 'n_obs_steps': 2, 'n_action_steps': 8, 'render_obs_key': 'agentview_image', 'fps': 10, 'crf': 22, 'past_action': False, 'abs_action': True, 'tqdm_interval_sec': 1.0, 'n_envs': 25, 'camera_depths': False}, 'dataset': {'_target_': 'adaflow.dataset.robomimic_replay_image_dataset.RobomimicReplayImageDataset', 'shape_meta': {'obs': {'agentview_image': {'shape': [3, 84, 84], 'type': 'rgb'}, 'robot0_eye_in_hand_image': {'shape': [3, 84, 84], 'type': 'rgb'}, 'robot0_eef_pos': {'shape': [3]}, 'robot0_eef_quat': {'shape': [4]}, 'robot0_gripper_qpos': {'shape': [2]}}, 'action': {'shape': [10]}}, 'dataset_path': '/mnt/data1/kevin/autom/demo-aug//datasets/generated/CoffeePreparation_D1/2025-02-01-23-45-24/merged_E32.0-CoffeePreparation_D1-500_all.hdf5', 'horizon': 16, 'pad_before': 1, 'pad_after': 7, 'n_obs_steps': 2, 'abs_action': True, 'rotation_rep': 'rotation_6d', 'use_legacy_normalizer': False, 'use_cache': True, 'seed': 42, 'val_ratio': 0.02}}, '_wandb': {'code_path': 'code/train.py'}}
|
| 11 |
+
2025-02-02 12:27:43,585 INFO MainThread:3658287 [wandb_init.py:init():789] starting backend
|
| 12 |
+
2025-02-02 12:27:43,801 INFO MainThread:3658287 [wandb_init.py:init():793] sending inform_init request
|
| 13 |
+
2025-02-02 12:27:43,806 INFO MainThread:3658287 [backend.py:_multiprocessing_setup():97] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
|
| 14 |
+
2025-02-02 12:27:43,807 INFO MainThread:3658287 [wandb_init.py:init():808] backend started and connected
|
| 15 |
+
2025-02-02 12:27:43,820 INFO MainThread:3658287 [wandb_init.py:init():901] updated telemetry
|
| 16 |
+
2025-02-02 12:27:43,827 INFO MainThread:3658287 [wandb_init.py:init():926] communicating run to backend with 90.0 second timeout
|
| 17 |
+
2025-02-02 12:27:44,297 INFO MainThread:3658287 [wandb_init.py:init():984] starting run threads in backend
|
| 18 |
+
2025-02-02 12:27:44,499 INFO MainThread:3658287 [wandb_run.py:_console_start():2385] atexit reg
|
| 19 |
+
2025-02-02 12:27:44,499 INFO MainThread:3658287 [wandb_run.py:_redirect():2235] redirect: wrap_raw
|
| 20 |
+
2025-02-02 12:27:44,500 INFO MainThread:3658287 [wandb_run.py:_redirect():2300] Wrapping output streams.
|
| 21 |
+
2025-02-02 12:27:44,500 INFO MainThread:3658287 [wandb_run.py:_redirect():2325] Redirects installed.
|
| 22 |
+
2025-02-02 12:27:44,501 INFO MainThread:3658287 [wandb_init.py:init():1026] run started, returning control to user process
|
| 23 |
+
2025-02-02 12:27:44,502 INFO MainThread:3658287 [wandb_run.py:_config_callback():1253] config_cb None None {'output_dir': '/mnt/data1/kevin/autom/adaflow/policy_checkpoints/CoffeePreparation_D1/2025-02-01-23-45-24_E32.0-CoffeePreparation_D1-500'}
|
| 24 |
+
2025-02-02 12:27:44,620 INFO MainThread:3658287 [wandb_run.py:_config_callback():1253] config_cb None None {'wandb_run_url_0': 'https://wandb.ai/kevin-lin/cpgen/runs/05vn1ybm'}
|
| 25 |
+
2025-02-02 12:27:44,620 INFO MainThread:3658287 [wandb_run.py:_config_callback():1253] config_cb None None {'wandb_run_url_1': 'https://wandb.ai/kevin-lin/cpgen/runs/e4t4jc38'}
|
| 26 |
+
2025-02-02 12:27:44,621 INFO MainThread:3658287 [wandb_run.py:_config_callback():1253] config_cb None None {'wandb_run_url_2': 'https://wandb.ai/kevin-lin/cpgen/runs/497mxoma'}
|
| 27 |
+
2025-02-02 12:27:44,621 INFO MainThread:3658287 [wandb_run.py:_config_callback():1253] config_cb None None {'wandb_run_url_3': 'https://wandb.ai/kevin-lin/cpgen/runs/77xsnsxo'}
|