2026-05-19 08:18:41,909 INFO    MainThread:879065 [wandb_setup.py:_flush():81] Current SDK version is 0.26.0
2026-05-19 08:18:41,909 INFO    MainThread:879065 [wandb_setup.py:_flush():81] Configure stats pid to 879065
2026-05-19 08:18:41,909 INFO    MainThread:879065 [wandb_setup.py:_flush():81] Loading settings from environment variables
2026-05-19 08:18:41,909 INFO    MainThread:879065 [wandb_init.py:setup_run_log_directory():721] Logging user logs to outputs/local_training/adapt_finetune/wandb/run-20260519_081841-0s2iob1d/logs/debug.log
2026-05-19 08:18:41,909 INFO    MainThread:879065 [wandb_init.py:setup_run_log_directory():722] Logging internal logs to outputs/local_training/adapt_finetune/wandb/run-20260519_081841-0s2iob1d/logs/debug-internal.log
2026-05-19 08:18:41,909 INFO    MainThread:879065 [wandb_init.py:init():848] calling init triggers
2026-05-19 08:18:41,910 INFO    MainThread:879065 [wandb_init.py:init():853] wandb.init called with sweep_config: {}
config: {'target_dataset': <TargetDataset.CARLA_LEADERBOARD2_3CAMERAS: 1>, 'num_available_cameras': 3, 'used_cameras': [True, True, True], 'num_used_cameras': 3, 'pixels_per_meter': 4.0, 'min_x_meter': -32, 'max_x_meter': 64, 'min_y_meter': -40, 'max_y_meter': 40, 'lidar_width_pixel': 384, 'lidar_height_pixel': 320, 'lidar_width_meter': 96, 'lidar_height_meter': 80, 'model_type': 'adapt', 'visualize_dataset': False, 'visualize_failed_scenarios': False, 'load_bev_3rd_person_images': False, 'seed': 0, 'wandb_id': None, 'wandb_resume': 'never', 'wandb_project_name': 'lead_posttrain', 'description': 'An example experiment description.', 'visualize_training': True, 'id': 'Experiment 1', 'load_file': '/home/divyanshu/carla/lead-adapt/outputs/local_training/posttrain/model_0019.pth', 'continue_failed_training': False, 'epoch_checkpoints_keep': [], 'use_training_session_cache': True, 'use_persistent_cache': True, 'force_rebuild_data_cache': False, 'carla_cache_path': ('1152', '384', '-32', '64', '-40', '40', 'True', 'True', 'True', 'True', 'False', '10'), 'training_session_cache_path': '/tmp/divyanshu', 'carla_root': 'data/carla_leaderboard2', 'carla_data': 'data/carla_leaderboard2/data', 'logdir': 'outputs/local_training/adapt_finetune', 'training_png_compression_level': 6, 'vehicle_min_num_lidar_points': 1, 'vehicle_min_num_visible_pixels': 1, 'pedestrian_min_num_lidar_points': 5, 'pedestrian_min_num_visible_pixels': 15, 'parking_vehicle_min_num_lidar_points': 3, 'parking_vehicle_min_num_visible_pixels': 5, 'grad_scaler_init_scale': 1024, 'grad_scaler_growth_factor': 2, 'grad_scaler_backoff_factor': 0.5, 'grad_scaler_growth_interval': 256, 'grad_scaler_max_grad_scale': 65536, 'sync_batchnorm': False, 'epochs': 5, 'batch_size': 128, 'torch_float_type': torch.bfloat16, 'use_mixed_precision_training': True, 'need_grad_scaler': False, 'use_zero_redundancy': False, 'save_model_checkpoint': True, 'is_pretraining': False, 'prefetch_factor': 16, 'compile': True, 'channel_last': True, 'lr': 1e-05, 'skip_first': 8, 'skip_last': 8, 'radar_detection': False, 'use_radar_detection': False, 'num_radar_points_per_sensor': 75, 'num_radar_queries': 20, 'radar_hidden_dim_tokenizer': 1024, 'radar_token_dim': 256, 'radar_tf_dim_ff': 1024, 'radar_dropout': 0.1, 'radar_num_heads': 8, 'radar_num_layers': 4, 'radar_hidden_dim_decoder': 1024, 'radar_classification_loss_weight': 1.0, 'radar_regression_loss_weight': 5.0, 'num_radar_sensors': 4, 'force_rebuild_bucket': False, 'randomize_route_order': False, 'hold_out_town13_routes': False, 'carla_bucket_collection': <class 'lead.data_buckets.full_posttrain_bucket_collection.FullPosttrainBucketCollection'>, 'bucket_collection_path': 'data/carla_leaderboard2/buckets', 'use_sensor_perburtation': True, 'use_sensor_perburtation_prob': 0.5, 'use_color_aug': True, 'use_color_aug_prob': 0.2, 'weight_decay': 0.01, 'use_cosine_annealing_with_restarts': True, 'use_rgb': True, 'use_depth': True, 'use_lidar': True, 'training_used_lidar_steps': 10, 'min_z': -4, 'max_z': 4, 'hist_max_per_pixel': 5, 'lidar_vert_anchors': 10, 'lidar_horz_anchors': 12, 'detect_boxes': True, 'data_bb_static_types_white_list': ['static.prop.constructioncone', 'static.prop.trafficwarning'], 'bb_confidence_threshold': 0.3, 'max_num_bbs': 90, 'num_dir_bins': 12, 'top_k_center_keypoints': 100, 'center_net_max_pooling_kernel': 3, 'bb_input_channel': 64, 'car_open_door_extra_width': 1.2, 'num_bb_classes': 8, 'use_discrete_command': True, 'discrete_command_dim': 6, 'use_noisy_tp': False, 'use_kalman_filter_for_gps': True, 'use_velocity': True, 'max_speed': 25.0, 'use_acceleration': False, 'max_acceleration': 10.0, 'use_previous_tp': True, 'use_next_tp': True, 'use_tp': True, 'target_points_normalization_constants': [[200.0, 50.0]], 'tp_pop_distance': 3.25, 'use_past_positions': False, 'use_past_speeds': False, 'num_past_samples_used': 0, 'transfuser_num_bev_cross_attention_layers': 6, 'transfuser_num_bev_cross_attention_heads': 8, 'transfuser_token_dim': 256, 'predict_target_speed': True, 'predict_spatial_path': True, 'predict_temporal_spatial_waypoints': True, 'use_planning_decoder': False, 'use_adapt_decoder': True, 'kdisks_vocab_path': 'lead/adapt/codebooks/kdisks_carla.pkl', 'use_kdisks': True, 'kdisks_heading_weight': 1.0, 'normalize_kinematics': False, 'use_soft_ce': False, 'soft_ce_sigma': 0.1, 'soft_ce_min_prob': 1e-06, 'kinematic_vocab_size': 4096, 'kinematic_embed_dim': 256, 'decoder_ffn_dim': 1024, 'decoder_num_heads': 8, 'decoder_dropout': 0.1, 'decoder_num_layers': 6, 'max_sequence_length': 64, 'frechet_gamma': 0.1, 'num_history_poses': 5, 'use_history_poses': True, 'target_speed_classes': [0.0, 4.0, 8.0, 10.0, 13.88888888, 16.0, 17.77777777, 20.0], 'target_speeds': [0.0, 4.0, 8.0, 10.0, 13.88888888, 16.0, 17.77777777, 20.0], 'smooth_route': True, 'num_route_points_smoothing': 20, 'num_route_points_prediction': 10, 'max_distance_future_waypoint': 10.0, 'num_way_points_prediction': 8, 'waypoints_spacing': 5, 'horizontal_fov_reduction': 0, 'crop_height': 0, 'carla_crop_height_type': <CarlaImageCroppingType.NONE: 3>, 'image_width_before_camera_subselection': 1152, 'final_image_width': 1152, 'final_image_height': 384, 'img_vert_anchors': 12, 'img_horz_anchors': 36, 'freeze_backbone': False, 'image_architecture': 'resnet34', 'lidar_architecture': 'resnet34', 'LTF': False, 'block_exp': 4, 'n_layer': 2, 'n_head': 4, 'embd_pdrop': 0.1, 'resid_pdrop': 0.1, 'attn_pdrop': 0.1, 'gpt_linear_layer_init_mean': 0.0, 'gpt_linear_layer_init_std': 0.02, 'gpt_layer_norm_init_weight': 1.0, 'use_semantic': True, 'num_semantic_classes': 10, 'perspective_downsample_factor': 1, 'deconv_channel_num_0': 128, 'deconv_channel_num_1': 64, 'deconv_channel_num_2': 32, 'deconv_scale_factor_0': 4, 'deconv_scale_factor_1': 8, 'use_bev_semantic': True, 'num_bev_semantic_classes': 13, 'scale_pedestrian_bev_semantic_size': 2.5, 'pedestrian_bev_min_extent': 0.4, 'bev_features_chanels': 64, 'bev_down_sample_factor': 4, 'bev_upsample_factor': 2, 'use_carla_data': True, 'carla_num_samples': -1, 'use_navsim_data': False, 'navsim_data_root': 'data/navsim_training_cache/navtrain', 'navsim_num_samples': -1, 'schedule_carla_num_samples': False, 'use_waymo_e2e_data': False, 'waymo_e2e_num_training_samples': -1, 'waymo_e2e_training_data_root': 'data/waymo_open_dataset_end_to_end_camera_v_1_0_0_training', 'waymo_e2e_val_data_root': 'data/waymo_open_dataset_end_to_end_camera_v_1_0_0_val_rfm', 'waymo_e2e_test_data_root': 'data/waymo_open_dataset_end_to_end_camera_v_1_0_0_test_submission', 'waymo_e2e_subsample_factor': 5, 'navsim_num_bev_semantic_classes': 7, 'navsim_num_bb_classes': 1, 'mixed_data_training': False, 'carla_leaderboard_mode': True, 'log_scalars_frequency': 1, 'log_images_frequency': 100, 'log_wandb': True, 'rank': 0, 'world_size': 2, 'local_rank': 0, 'device': device(type='cuda', index=0), 'assigned_cpu_cores': 8, 'workers_per_cpu_cores': 1, 'debug_mode': True, 'use_radars': False, '_loaded_config': {'target_dataset': 1, 'num_available_cameras': 3, 'used_cameras': [True, True, True], 'num_used_cameras': 3, 'pixels_per_meter': 4.0, 'min_x_meter': -32, 'max_x_meter': 64, 'min_y_meter': -40, 'max_y_meter': 40, 'lidar_width_pixel': 384, 'lidar_height_pixel': 320, 'lidar_width_meter': 96, 'lidar_height_meter': 80, 'model_type': 'adapt', 'visualize_dataset': False, 'visualize_failed_scenarios': False, 'load_bev_3rd_person_images': False, 'seed': 0, 'wandb_id': None, 'wandb_resume': 'never', 'wandb_project_name': 'lead_posttrain', 'description': 'An example experiment description.', 'visualize_training': True, 'id': 'Experiment 1', 'load_file': '/home/divyanshu/carla/lead-adapt/outputs/local_training/posttrain/model_0019.pth', 'continue_failed_training': False, 'epoch_checkpoints_keep': [], 'use_training_session_cache': True, 'use_persistent_cache': True, 'force_rebuild_data_cache': False, 'carla_cache_path': ['1152', '384', '-32', '64', '-40', '40', 'True', 'True', 'True', 'True', 'False', '10'], 'training_session_cache_path': '/tmp/divyanshu', 'carla_root': 'data/carla_leaderboard2', 'carla_data': 'data/carla_leaderboard2/data', 'logdir': 'outputs/local_training/adapt_finetune', 'training_png_compression_level': 6, 'vehicle_min_num_lidar_points': 1, 'vehicle_min_num_visible_pixels': 1, 'pedestrian_min_num_lidar_points': 5, 'pedestrian_min_num_visible_pixels': 15, 'parking_vehicle_min_num_lidar_points': 3, 'parking_vehicle_min_num_visible_pixels': 5, 'grad_scaler_init_scale': 1024, 'grad_scaler_growth_factor': 2, 'grad_scaler_backoff_factor': 0.5, 'grad_scaler_growth_interval': 256, 'grad_scaler_max_grad_scale': 65536, 'sync_batchnorm': False, 'epochs': 5, 'batch_size': 128, 'use_mixed_precision_training': True, 'need_grad_scaler': False, 'use_zero_redundancy': False, 'save_model_checkpoint': True, 'is_pretraining': False, 'prefetch_factor': 16, 'compile': True, 'channel_last': True, 'lr': 1e-05, 'skip_first': 8, 'skip_last': 8, 'radar_detection': False, 'use_radar_detection': False, 'num_radar_points_per_sensor': 75, 'num_radar_queries': 20, 'radar_hidden_dim_tokenizer': 1024, 'radar_token_dim': 256, 'radar_tf_dim_ff': 1024, 'radar_dropout': 0.1, 'radar_num_heads': 8, 'radar_num_layers': 4, 'radar_hidden_dim_decoder': 1024, 'radar_classification_loss_weight': 1.0, 'radar_regression_loss_weight': 5.0, 'num_radar_sensors': 4, 'force_rebuild_bucket': False, 'randomize_route_order': False, 'hold_out_town13_routes': False, 'bucket_collection_path': 'data/carla_leaderboard2/buckets', 'use_sensor_perburtation': True, 'use_sensor_perburtation_prob': 0.5, 'use_color_aug': True, 'use_color_aug_prob': 0.2, 'weight_decay': 0.01, 'use_cosine_annealing_with_restarts': True, 'use_rgb': True, 'use_depth': True, 'use_lidar': True, 'training_used_lidar_steps': 10, 'min_z': -4, 'max_z': 4, 'hist_max_per_pixel': 5, 'lidar_vert_anchors': 10, 'lidar_horz_anchors': 12, 'detect_boxes': True, 'data_bb_static_types_white_list': ['static.prop.constructioncone', 'static.prop.trafficwarning'], 'bb_confidence_threshold': 0.3, 'max_num_bbs': 90, 'num_dir_bins': 12, 'top_k_center_keypoints': 100, 'center_net_max_pooling_kernel': 3, 'bb_input_channel': 64, 'car_open_door_extra_width': 1.2, 'num_bb_classes': 8, 'use_discrete_command': True, 'discrete_command_dim': 6, 'use_noisy_tp': False, 'use_kalman_filter_for_gps': True, 'use_velocity': True, 'max_speed': 25.0, 'use_acceleration': False, 'max_acceleration': 10.0, 'use_previous_tp': True, 'use_next_tp': True, 'use_tp': True, 'target_points_normalization_constants': [[200.0, 50.0]], 'tp_pop_distance': 3.25, 'use_past_positions': False, 'use_past_speeds': False, 'num_past_samples_used': 0, 'transfuser_num_bev_cross_attention_layers': 6, 'transfuser_num_bev_cross_attention_heads': 8, 'transfuser_token_dim': 256, 'predict_target_speed': True, 'predict_spatial_path': True, 'predict_temporal_spatial_waypoints': True, 'use_planning_decoder': False, 'use_adapt_decoder': True, 'kdisks_vocab_path': 'lead/adapt/codebooks/kdisks_carla.pkl', 'use_kdisks': True, 'kdisks_heading_weight': 1.0, 'normalize_kinematics': False, 'use_soft_ce': False, 'soft_ce_sigma': 0.1, 'soft_ce_min_prob': 1e-06, 'kinematic_vocab_size': 4096, 'kinematic_embed_dim': 256, 'decoder_ffn_dim': 1024, 'decoder_num_heads': 8, 'decoder_dropout': 0.1, 'decoder_num_layers': 6, 'max_sequence_length': 64, 'frechet_gamma': 0.1, 'num_history_poses': 5, 'use_history_poses': True, 'target_speed_classes': [0.0, 4.0, 8.0, 10.0, 13.88888888, 16.0, 17.77777777, 20.0], 'target_speeds': [0.0, 4.0, 8.0, 10.0, 13.88888888, 16.0, 17.77777777, 20.0], 'smooth_route': True, 'num_route_points_smoothing': 20, 'num_route_points_prediction': 10, 'max_distance_future_waypoint': 10.0, 'num_way_points_prediction': 8, 'waypoints_spacing': 5, 'horizontal_fov_reduction': 0, 'crop_height': 0, 'carla_crop_height_type': 3, 'image_width_before_camera_subselection': 1152, 'final_image_width': 1152, 'final_image_height': 384, 'img_vert_anchors': 12, 'img_horz_anchors': 36, 'freeze_backbone': False, 'image_architecture': 'resnet34', 'lidar_architecture': 'resnet34', 'LTF': False, 'block_exp': 4, 'n_layer': 2, 'n_head': 4, 'embd_pdrop': 0.1, 'resid_pdrop': 0.1, 'attn_pdrop': 0.1, 'gpt_linear_layer_init_mean': 0.0, 'gpt_linear_layer_init_std': 0.02, 'gpt_layer_norm_init_weight': 1.0, 'use_semantic': True, 'num_semantic_classes': 10, 'perspective_downsample_factor': 1, 'deconv_channel_num_0': 128, 'deconv_channel_num_1': 64, 'deconv_channel_num_2': 32, 'deconv_scale_factor_0': 4, 'deconv_scale_factor_1': 8, 'use_bev_semantic': True, 'num_bev_semantic_classes': 13, 'scale_pedestrian_bev_semantic_size': 2.5, 'pedestrian_bev_min_extent': 0.4, 'bev_features_chanels': 64, 'bev_down_sample_factor': 4, 'bev_upsample_factor': 2, 'use_carla_data': True, 'carla_num_samples': -1, 'use_navsim_data': False, 'navsim_data_root': 'data/navsim_training_cache/navtrain', 'navsim_num_samples': -1, 'schedule_carla_num_samples': False, 'use_waymo_e2e_data': False, 'waymo_e2e_num_training_samples': -1, 'waymo_e2e_training_data_root': 'data/waymo_open_dataset_end_to_end_camera_v_1_0_0_training', 'waymo_e2e_val_data_root': 'data/waymo_open_dataset_end_to_end_camera_v_1_0_0_val_rfm', 'waymo_e2e_test_data_root': 'data/waymo_open_dataset_end_to_end_camera_v_1_0_0_test_submission', 'waymo_e2e_subsample_factor': 5, 'navsim_num_bev_semantic_classes': 7, 'navsim_num_bb_classes': 1, 'mixed_data_training': False, 'carla_leaderboard_mode': True, 'log_scalars_frequency': 1, 'log_images_frequency': 100, 'log_wandb': True, 'rank': 0, 'world_size': 2, 'local_rank': 0, 'assigned_cpu_cores': 8, 'workers_per_cpu_cores': 1, 'debug_mode': True, 'use_radars': False}, '_wandb': {}}
2026-05-19 08:18:41,912 INFO    MainThread:879065 [wandb_init.py:init():896] starting backend
2026-05-19 08:18:42,234 INFO    MainThread:879065 [wandb_init.py:init():911] sending inform_init request
2026-05-19 08:18:42,251 INFO    MainThread:879065 [wandb_init.py:init():919] backend started and connected
2026-05-19 08:18:42,277 INFO    MainThread:879065 [wandb_init.py:init():989] updated telemetry
2026-05-19 08:18:42,398 INFO    MainThread:879065 [wandb_init.py:init():1013] communicating run to backend with 90.0 second timeout
2026-05-19 08:18:55,246 INFO    MainThread:879065 [wandb_init.py:init():1058] starting run threads in backend
2026-05-19 08:18:55,782 INFO    MainThread:879065 [wandb_run.py:_console_start():2542] atexit reg
2026-05-19 08:18:55,783 INFO    MainThread:879065 [wandb_run.py:_redirect():2391] redirect: wrap_raw
2026-05-19 08:18:55,783 INFO    MainThread:879065 [wandb_run.py:_redirect():2460] Wrapping output streams.
2026-05-19 08:18:55,783 INFO    MainThread:879065 [wandb_run.py:_redirect():2483] Redirects installed.
2026-05-19 08:18:55,788 INFO    MainThread:879065 [wandb_init.py:init():1098] run started, returning control to user process