2024-12-10 21:57:28,181 INFO StreamThr :843156 [internal.py:wandb_internal():85] W&B internal server running at pid: 843156, started at: 2024-12-10 21:57:28.180540 2024-12-10 21:57:28,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status 2024-12-10 21:57:28,187 INFO WriterThread:843156 [datastore.py:open_for_write():87] open: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/run-bold5k_nsdm1.wandb 2024-12-10 21:57:28,189 DEBUG SenderThread:843156 [sender.py:send():379] send: header 2024-12-10 21:57:28,199 DEBUG SenderThread:843156 [sender.py:send():379] send: run 2024-12-10 21:57:28,204 INFO SenderThread:843156 [sender.py:_setup_resume():749] checking resume status for None/mindeye/bold5k_nsdm1 2024-12-10 21:57:28,554 INFO SenderThread:843156 [sender.py:_setup_resume():829] configured resuming with: ResumeState(resumed=True,step=0,history=0,events=0,output=3,runtime=0,wandb_runtime=None,summary={},config={'seed': {'desc': None, 'value': 42}, '_wandb': {'desc': None, 'value': {'t': {'1': [1, 5, 9, 11, 41, 49, 53, 55, 63, 71, 79, 83, 103], '2': [1, 5, 9, 11, 41, 49, 53, 55, 63, 71, 79, 83, 103], '3': [13, 14, 16, 23], '4': '3.11.10', '5': '0.17.1', '6': '4.37.2', '8': [5], '13': 'linux-x86_64'}, 'code_path': 'code/src/TrainB5k.py', 'framework': 'huggingface', 'start_time': 1733867727, 'cli_version': '0.17.1', 'is_jupyter_run': False, 'python_version': '3.11.10', 'is_kaggle_kernel': False, 'huggingface_version': '4.37.2'}}, 'max_lr': {'desc': None, 'value': 0.0003}, 'num_test': {'desc': None, 'value': 480}, 'mixup_pct': {'desc': None, 'value': 0.33}, 'batch_size': {'desc': None, 'value': 21}, 'blur_scale': {'desc': None, 'value': 0.5}, 'clip_scale': {'desc': None, 'value': 1}, 'model_name': {'desc': None, 'value': 'bold5k_nsdm1'}, 'num_epochs': {'desc': None, 'value': 80}, 'num_params': {'desc': None, 'value': 2239051016}, 'world_size': {'desc': None, 'value': 1}, 'ckpt_saving': {'desc': None, 'value': False}, 'distributed': {'desc': None, 'value': False}, 'num_devices': {'desc': None, 'value': 1}, 'prior_scale': {'desc': None, 'value': 30}, 'num_sessions': {'desc': None, 'value': 15}, 'ckpt_interval': {'desc': None, 'value': 999}, 'use_image_aug': {'desc': None, 'value': True}, 'global_batch_size': {'desc': None, 'value': '21'}, 'num_samples_per_epoch': {'desc': None, 'value': 4323}},tags=[]) 2024-12-10 21:57:28,697 INFO SenderThread:843156 [dir_watcher.py:__init__():211] watching files in: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files 2024-12-10 21:57:28,697 INFO SenderThread:843156 [sender.py:_start_run_threads():1188] run started: bold5k_nsdm1 with start time 1733867848.182576 2024-12-10 21:57:28,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: check_version 2024-12-10 21:57:28,708 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: check_version 2024-12-10 21:57:28,779 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: run_start 2024-12-10 21:57:28,880 DEBUG HandlerThread:843156 [system_info.py:__init__():26] System info init 2024-12-10 21:57:28,880 DEBUG HandlerThread:843156 [system_info.py:__init__():41] System info init done 2024-12-10 21:57:28,880 INFO HandlerThread:843156 [system_monitor.py:start():194] Starting system monitor 2024-12-10 21:57:28,880 INFO SystemMonitor:843156 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-12-10 21:57:28,881 INFO SystemMonitor:843156 [interfaces.py:start():188] Started cpu monitoring 2024-12-10 21:57:28,881 INFO SystemMonitor:843156 [interfaces.py:start():188] Started disk monitoring 2024-12-10 21:57:28,882 INFO SystemMonitor:843156 [interfaces.py:start():188] Started gpu monitoring 2024-12-10 21:57:28,882 INFO SystemMonitor:843156 [interfaces.py:start():188] Started memory monitoring 2024-12-10 21:57:28,883 INFO SystemMonitor:843156 [interfaces.py:start():188] Started network monitoring 2024-12-10 21:57:29,318 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: python_packages 2024-12-10 21:57:29,318 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: python_packages 2024-12-10 21:57:29,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 21:57:29,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:29,320 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 21:57:29,487 DEBUG SenderThread:843156 [sender.py:send():379] send: telemetry 2024-12-10 21:57:29,699 INFO Thread-12 :843156 [dir_watcher.py:_on_file_created():271] file/dir created: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/requirements.txt 2024-12-10 21:57:30,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:30,700 INFO Thread-12 :843156 [dir_watcher.py:_on_file_created():271] file/dir created: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 21:57:31,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:32,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:32,703 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 21:57:33,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:33,705 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 21:57:33,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:57:34,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:35,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:36,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:37,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:38,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:39,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:57:39,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:40,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:41,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:42,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:43,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:44,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 21:57:44,319 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 21:57:44,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:44,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:57:45,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:46,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:47,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:48,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:49,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:50,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:57:50,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:51,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:52,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:53,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:54,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:55,321 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:57:55,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:56,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:57,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:58,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:57:59,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 21:57:59,319 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 21:57:59,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:00,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:00,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:58:00,761 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/config.yaml 2024-12-10 21:58:01,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:02,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:03,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:04,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:05,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:06,321 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:58:06,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:07,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:08,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:09,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:10,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:11,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:12,321 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:58:12,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:13,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:14,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 21:58:14,319 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 21:58:14,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:15,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:16,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:17,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:17,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:58:18,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:19,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:20,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:21,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:22,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:23,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:58:23,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:24,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:25,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:26,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:27,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:28,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:28,883 DEBUG SystemMonitor:843156 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-12-10 21:58:28,885 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 21:58:28,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:58:29,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 21:58:29,319 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 21:58:29,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:30,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:31,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:32,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:33,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:34,321 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:58:34,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:35,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:36,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:37,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:38,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:39,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:58:39,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:40,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:41,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:42,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:43,280 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 21:58:43,283 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 21:58:43,283 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 21:58:43,285 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 21:58:43,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:43,842 INFO Thread-12 :843156 [dir_watcher.py:_on_file_created():271] file/dir created: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 21:58:44,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 21:58:44,320 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 21:58:44,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:44,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:58:44,843 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 21:58:45,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:46,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:47,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:48,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:49,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:50,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:58:50,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:51,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:52,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:53,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:54,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:55,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:58:55,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:56,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:57,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:58,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:58:58,887 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 21:58:59,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 21:58:59,320 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 21:58:59,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:00,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:00,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:59:01,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:02,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:03,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:04,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:05,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:06,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:59:06,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:07,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:08,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:09,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:10,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:11,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:59:11,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:12,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:13,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:14,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 21:59:14,320 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 21:59:14,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:15,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:16,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:16,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:59:17,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:18,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:19,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:20,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:21,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:22,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:59:22,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:23,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:24,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:25,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:26,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:27,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:59:27,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:28,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:28,889 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 21:59:29,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 21:59:29,320 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 21:59:29,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:30,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:31,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:32,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:32,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:59:33,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:34,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:35,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:36,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:37,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:38,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:59:38,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:39,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:40,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:41,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:42,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:43,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:59:43,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:44,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 21:59:44,320 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 21:59:44,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:45,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:46,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:47,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:48,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:48,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:59:49,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:50,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:51,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:52,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:53,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:54,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 21:59:54,456 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 21:59:54,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:54,458 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 21:59:54,459 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 21:59:54,463 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 21:59:54,983 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 21:59:54,984 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 21:59:55,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:56,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:57,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:58,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:58,891 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 21:59:59,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 21:59:59,320 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 21:59:59,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 21:59:59,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:00:00,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:01,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:02,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:03,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:04,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:05,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:00:05,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:06,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:07,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:08,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:09,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:10,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:11,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:00:11,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:12,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:13,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:14,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:00:14,321 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:00:14,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:15,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:16,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:16,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:00:17,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:18,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:19,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:20,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:21,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:22,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:00:22,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:23,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:24,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:25,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:26,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:27,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:00:27,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:28,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:28,893 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:00:29,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:00:29,320 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:00:29,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:30,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:31,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:32,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:32,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:00:33,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:34,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:35,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:36,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:37,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:38,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:00:38,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:39,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:40,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:41,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:42,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:43,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:44,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:00:44,320 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:00:44,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:44,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:00:45,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:46,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:47,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:48,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:49,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:50,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:00:50,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:51,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:52,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:53,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:54,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:55,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:56,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:00:56,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:57,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:58,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:00:58,895 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:00:59,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:00:59,320 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:00:59,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:00,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:01,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:01,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:01:02,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:03,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:04,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:05,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:05,692 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:01:05,693 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:01:05,693 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:01:05,695 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:01:06,146 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:01:06,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:06,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:01:07,149 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:01:07,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:08,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:09,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:10,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:11,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:12,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:01:12,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:13,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:14,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:01:14,321 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:01:14,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:15,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:16,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:17,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:17,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:01:18,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:19,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:20,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:21,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:22,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:23,326 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:01:23,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:24,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:25,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:26,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:27,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:28,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:28,897 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:01:28,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:01:29,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:01:29,321 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:01:29,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:30,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:31,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:32,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:33,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:34,326 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:01:34,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:35,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:36,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:37,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:38,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:39,326 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:01:39,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:40,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:41,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:42,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:43,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:44,321 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:01:44,321 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:01:44,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:44,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:01:45,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:46,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:47,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:48,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:49,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:50,326 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:01:50,456 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:51,456 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:52,456 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:53,456 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:54,456 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:55,327 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:01:55,456 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:56,456 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:57,456 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:58,456 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:01:58,898 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:01:59,321 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:01:59,321 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:01:59,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:00,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:00,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:02:01,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:02,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:03,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:04,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:05,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:06,327 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:02:06,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:07,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:08,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:09,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:10,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:11,327 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:02:11,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:12,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:13,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:14,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:02:14,423 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:02:14,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:15,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:16,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:16,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:02:16,928 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:02:16,929 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:02:16,929 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:02:16,932 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:02:17,289 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:02:17,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:18,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:19,293 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:02:19,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:20,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:21,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:22,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:02:22,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:23,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:24,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:25,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:26,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:27,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:28,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:02:28,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:28,899 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:02:29,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:02:29,422 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:02:29,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:30,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:31,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:32,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:33,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:33,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:02:34,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:35,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:36,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:37,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:38,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:39,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:02:39,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:40,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:41,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:42,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:43,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:44,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:02:44,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:02:44,422 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:02:44,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:45,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:46,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:47,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:48,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:49,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:50,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:02:50,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:51,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:52,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:53,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:54,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:55,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:02:55,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:56,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:57,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:58,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:02:58,903 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:02:59,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:02:59,422 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:02:59,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:00,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:00,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:03:01,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:02,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:03,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:04,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:05,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:06,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:03:06,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:07,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:08,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:09,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:10,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:11,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:03:11,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:12,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:13,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:14,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:03:14,422 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:03:14,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:15,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:16,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:16,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:03:17,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:18,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:19,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:20,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:21,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:22,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:03:22,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:23,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:24,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:25,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:26,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:27,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:03:27,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:27,580 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:03:27,581 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:03:27,582 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:03:27,584 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:03:28,445 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:03:28,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:28,903 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:03:29,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:03:29,422 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:03:29,447 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:03:29,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:30,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:31,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:32,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:32,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:03:33,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:34,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:35,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:36,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:37,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:38,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:03:38,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:39,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:40,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:41,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:42,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:43,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:44,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:03:44,422 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:03:44,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:44,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:03:45,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:46,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:47,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:48,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:49,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:50,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:03:50,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:51,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:52,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:53,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:54,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:55,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:56,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:03:56,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:57,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:58,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:03:58,906 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:03:59,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:03:59,422 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:03:59,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:00,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:01,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:01,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:04:02,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:03,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:04,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:05,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:06,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:07,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:04:07,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:08,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:09,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:10,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:11,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:12,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:13,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:04:13,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:14,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:04:14,423 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:04:14,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:15,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:16,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:17,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:18,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:18,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:04:19,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:20,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:21,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:22,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:23,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:24,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:04:24,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:25,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:26,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:27,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:28,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:28,908 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:04:29,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:04:29,423 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:04:29,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:29,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:04:30,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:31,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:32,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:33,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:34,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:35,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:04:35,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:36,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:37,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:38,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:39,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:40,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:04:40,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:41,084 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:04:41,085 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:04:41,085 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:04:41,088 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:04:41,604 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:04:41,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:42,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:43,608 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:04:43,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:44,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:04:44,423 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:04:44,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:45,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:04:45,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:46,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:47,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:48,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:49,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:50,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:51,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:04:51,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:52,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:53,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:54,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:55,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:56,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:04:56,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:57,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:58,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:04:58,910 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:04:59,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:04:59,423 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:04:59,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:00,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:01,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:05:01,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:02,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:03,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:04,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:05,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:06,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:07,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:05:07,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:08,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:09,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:10,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:11,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:12,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:05:12,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:13,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:14,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:05:14,423 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:05:14,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:15,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:16,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:17,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:05:17,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:18,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:19,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:20,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:21,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:22,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:23,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:05:23,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:24,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:25,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:26,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:27,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:28,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:05:28,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:28,911 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:05:29,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:05:29,423 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:05:29,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:30,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:31,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:32,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:33,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:05:33,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:34,732 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:35,732 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:36,732 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:37,732 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:38,732 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:39,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:05:39,732 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:40,732 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:41,732 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:42,732 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:43,732 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:44,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:05:44,424 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:05:44,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:05:44,733 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:45,733 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:46,733 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:47,733 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:48,733 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:49,733 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:50,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:05:50,734 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:51,734 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:52,734 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:53,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:05:53,017 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:05:53,018 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:05:53,020 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:05:53,736 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:53,741 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:05:54,735 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:55,735 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:55,744 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:05:56,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:05:56,735 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:57,735 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:58,735 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:05:58,913 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:05:59,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:05:59,424 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:05:59,735 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:00,735 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:01,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:06:01,735 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:02,735 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:03,735 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:04,736 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:05,736 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:06,736 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:07,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:06:07,736 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:08,736 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:09,736 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:10,736 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:11,736 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:12,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:06:12,737 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:13,737 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:14,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:06:14,424 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:06:14,737 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:15,737 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:16,737 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:17,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:06:17,737 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:18,737 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:19,737 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:20,738 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:21,738 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:22,738 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:23,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:06:23,738 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:24,738 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:25,738 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:26,738 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:27,738 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:28,738 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:28,915 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:06:28,917 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:06:29,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:06:29,424 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:06:29,738 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:30,739 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:31,739 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:32,739 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:33,739 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:34,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:06:34,739 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:35,739 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:36,739 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:37,739 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:38,740 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:39,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:06:39,739 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:40,740 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:41,740 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:42,740 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:43,740 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:44,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:06:44,424 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:06:44,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:06:44,740 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:45,740 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:46,740 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:47,740 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:48,741 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:49,741 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:50,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:06:50,741 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:51,741 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:52,741 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:53,741 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:54,741 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:55,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:06:55,741 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:56,741 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:57,741 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:58,742 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:06:58,917 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:06:59,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:06:59,425 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:06:59,742 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:00,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:07:00,742 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:01,742 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:02,742 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:03,743 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:04,742 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:05,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:07:05,605 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:07:05,605 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:07:05,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:07:05,607 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:07:05,742 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:05,881 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:07:06,744 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:07,743 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:07,884 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:07:08,743 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:09,744 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:10,743 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:11,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:07:11,743 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:12,744 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:13,744 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:14,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:07:14,425 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:07:14,744 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:15,744 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:16,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:07:16,744 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:17,744 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:18,744 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:19,744 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:20,744 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:21,744 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:22,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:07:22,745 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:23,745 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:24,745 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:25,745 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:26,745 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:27,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:07:27,745 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:28,745 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:28,918 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:07:29,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:07:29,425 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:07:29,745 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:30,745 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:31,745 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:32,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:07:32,745 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:33,746 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:34,746 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:35,746 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:36,746 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:37,746 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:38,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:07:38,746 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:39,746 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:40,747 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:41,747 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:42,747 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:43,747 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:44,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:07:44,425 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:07:44,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:07:44,747 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:45,747 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:46,747 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:47,747 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:48,747 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:49,748 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:50,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:07:50,748 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:51,748 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:52,748 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:53,748 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:54,748 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:55,748 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:56,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:07:56,748 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:57,749 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:58,749 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:07:58,921 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:07:59,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:07:59,425 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:07:59,749 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:00,749 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:01,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:08:01,749 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:02,749 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:03,749 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:04,750 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:05,750 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:06,750 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:07,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:08:07,750 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:08,750 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:09,750 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:10,750 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:11,750 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:12,750 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:13,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:08:13,751 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:14,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:08:14,425 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:08:14,751 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:15,751 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:16,751 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:17,751 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:18,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:08:18,061 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:08:18,061 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:08:18,063 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:08:18,799 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:19,018 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:08:19,300 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:08:19,797 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:20,020 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:08:20,798 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:21,798 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:22,798 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:23,798 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:24,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:08:24,798 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:25,798 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:26,798 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:27,798 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:28,799 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:28,923 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:08:29,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:08:29,425 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:08:29,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:08:29,799 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:30,799 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:31,799 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:32,799 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:33,799 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:34,799 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:35,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:08:35,799 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:36,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:37,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:38,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:39,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:40,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:41,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:08:41,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:42,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:43,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:44,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:08:44,425 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:08:44,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:45,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:46,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:08:46,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:47,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:48,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:49,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:50,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:51,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:52,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:08:52,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:53,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:54,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:55,802 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:56,802 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:57,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:08:57,802 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:58,802 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:08:58,924 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:08:59,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:08:59,425 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:08:59,802 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:00,802 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:01,802 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:02,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:09:02,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:03,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:04,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:05,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:06,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:07,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:08,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:09:08,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:09,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:10,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:11,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:12,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:13,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:14,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:09:14,426 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:09:14,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:09:14,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:15,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:16,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:17,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:18,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:19,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:20,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:09:20,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:21,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:22,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:23,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:24,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:25,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:26,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:09:26,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:27,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:28,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:28,926 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:09:29,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:09:29,426 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:09:29,818 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:30,819 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:31,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:09:31,234 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:09:31,234 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:09:31,236 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:09:31,819 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:32,152 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:09:32,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:09:32,819 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:33,819 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:34,157 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:09:34,819 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:35,819 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:36,819 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:37,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:09:37,819 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:38,819 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:39,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:40,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:41,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:42,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:43,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:09:43,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:44,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:09:44,426 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:09:44,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:45,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:46,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:47,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:48,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:09:48,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:49,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:50,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:51,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:52,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:53,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:54,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:09:54,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:55,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:56,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:57,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:58,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:09:58,927 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:09:59,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:09:59,426 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:09:59,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:09:59,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:00,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:01,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:02,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:03,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:04,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:05,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:10:05,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:06,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:07,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:08,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:09,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:10,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:10:10,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:11,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:12,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:13,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:14,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:10:14,426 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:10:14,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:15,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:10:15,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:16,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:17,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:18,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:19,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:20,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:21,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:10:21,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:22,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:23,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:24,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:25,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:26,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:10:26,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:27,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:28,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:28,929 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:10:29,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:10:29,426 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:10:29,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:30,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:31,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:10:31,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:32,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:33,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:34,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:35,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:36,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:37,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:10:37,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:38,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:39,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:40,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:41,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:42,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:10:42,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:43,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:10:43,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:43,830 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:10:43,830 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:10:43,832 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:10:44,296 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:10:44,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:10:44,427 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:10:44,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:45,298 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:10:45,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:46,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:47,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:10:47,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:48,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:49,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:50,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:51,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:52,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:53,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:10:53,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:54,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:55,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:56,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:57,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:58,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:10:58,931 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:10:58,934 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:10:59,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:10:59,426 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:10:59,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:00,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:01,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:02,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:03,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:04,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:11:04,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:05,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:06,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:07,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:08,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:09,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:11:09,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:10,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:11,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:12,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:13,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:14,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:11:14,427 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:11:14,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:11:14,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:15,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:16,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:17,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:18,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:19,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:20,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:11:20,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:21,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:22,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:23,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:24,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:25,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:26,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:11:26,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:27,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:28,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:28,934 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:11:29,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:11:29,427 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:11:29,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:30,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:31,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:11:31,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:32,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:33,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:34,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:35,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:36,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:37,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:11:37,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:38,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:39,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:40,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:41,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:42,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:43,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:11:43,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:44,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:11:44,427 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:11:44,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:45,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:46,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:47,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:48,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:11:48,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:49,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:50,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:51,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:52,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:53,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:54,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:11:54,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:55,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:11:55,802 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:11:55,802 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:11:55,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:55,835 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:11:56,341 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:11:56,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:57,345 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:11:57,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:58,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:11:58,937 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:11:59,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:11:59,427 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:11:59,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:11:59,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:00,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:01,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:02,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:03,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:04,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:05,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:12:05,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:06,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:07,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:08,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:09,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:10,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:12:10,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:11,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:12,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:13,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:14,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:12:14,427 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:12:14,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:15,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:12:15,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:16,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:17,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:18,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:19,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:20,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:21,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:12:21,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:22,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:23,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:24,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:25,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:26,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:12:26,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:27,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:28,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:28,939 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:12:29,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:12:29,427 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:12:29,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:30,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:31,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:12:31,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:32,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:33,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:34,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:35,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:36,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:37,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:12:37,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:38,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:39,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:40,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:41,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:42,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:12:42,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:43,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:44,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:12:44,428 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:12:44,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:45,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:46,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:47,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:12:47,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:48,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:49,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:50,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:51,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:52,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:53,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:12:53,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:54,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:55,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:56,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:57,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:58,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:12:58,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:12:58,940 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:12:59,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:12:59,428 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:12:59,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:00,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:01,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:02,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:03,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:13:03,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:04,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:05,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:06,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:07,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:08,516 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:13:08,517 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:13:08,517 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:13:08,519 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:13:08,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:09,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:13:09,504 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:13:09,504 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:13:09,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:10,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:11,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:12,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:13,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:14,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:13:14,428 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:13:14,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:13:14,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:15,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:16,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:17,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:18,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:19,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:20,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:13:20,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:21,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:22,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:23,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:24,947 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:25,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:13:25,947 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:26,947 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:27,947 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:28,941 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:13:28,947 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:29,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:13:29,428 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:13:29,947 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:30,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:13:30,947 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:31,948 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:32,947 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:33,948 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:34,948 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:35,948 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:36,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:13:36,948 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:37,948 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:38,948 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:39,948 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:40,948 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:41,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:13:41,949 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:42,949 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:43,949 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:44,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:13:44,428 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:13:44,949 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:45,949 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:46,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:13:46,949 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:47,949 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:48,949 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:49,949 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:50,950 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:51,950 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:52,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:13:52,950 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:53,950 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:54,950 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:55,950 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:56,950 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:57,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:13:57,950 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:58,943 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:13:58,951 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:13:59,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:13:59,428 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:13:59,951 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:00,951 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:01,951 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:02,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:14:02,951 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:03,951 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:04,951 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:05,951 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:06,951 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:07,952 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:08,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:14:08,952 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:09,952 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:10,952 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:11,952 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:12,952 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:13,952 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:14,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:14:14,428 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:14:14,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:14:14,952 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:15,953 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:16,953 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:17,953 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:18,953 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:19,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:14:19,912 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:14:19,914 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:14:19,914 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:14:19,916 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:14:19,953 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:20,656 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:14:20,954 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:21,658 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:14:21,954 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:22,954 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:23,954 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:24,954 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:25,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:14:25,954 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:26,954 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:27,954 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:28,944 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:14:28,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:29,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:14:29,428 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:14:29,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:30,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:14:30,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:31,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:32,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:33,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:34,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:35,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:36,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:14:36,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:37,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:38,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:39,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:40,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:41,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:14:41,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:42,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:43,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:44,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:14:44,429 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:14:44,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:45,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:46,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:14:46,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:47,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:48,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:49,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:50,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:51,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:52,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:14:52,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:53,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:54,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:55,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:56,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:57,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:58,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:14:58,946 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:14:58,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:14:59,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:14:59,428 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:14:59,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:00,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:01,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:02,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:03,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:15:03,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:04,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:05,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:06,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:07,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:08,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:09,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:15:09,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:10,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:11,962 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:12,962 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:13,962 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:14,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:15:14,429 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:15:14,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:15:14,962 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:15,963 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:16,962 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:17,962 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:18,962 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:19,962 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:20,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:15:20,963 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:21,963 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:22,963 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:23,963 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:24,963 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:25,963 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:26,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:15:26,963 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:27,964 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:28,948 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:15:28,964 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:29,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:15:29,429 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:15:29,964 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:31,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:31,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:15:32,056 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:33,056 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:33,529 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:15:33,530 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:15:33,530 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:15:33,533 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:15:33,837 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:15:34,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:35,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:35,840 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:15:36,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:37,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:37,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:15:38,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:39,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:40,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:41,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:42,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:43,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:43,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:15:44,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:44,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:15:44,429 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:15:45,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:46,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:47,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:48,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:48,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:15:49,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:50,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:51,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:52,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:53,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:54,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:54,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:15:55,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:56,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:57,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:58,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:58,950 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:15:59,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:15:59,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:15:59,429 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:15:59,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:16:00,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:01,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:02,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:03,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:04,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:05,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:05,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:16:06,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:07,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:08,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:09,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:10,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:10,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:16:11,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:12,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:13,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:14,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:14,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:16:14,429 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:16:15,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:15,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:16:16,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:17,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:18,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:19,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:20,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:21,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:21,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:16:22,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:23,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:24,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:25,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:26,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:27,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:27,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:16:28,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:28,951 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:16:29,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:29,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:16:29,429 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:16:30,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:31,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:32,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:32,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:16:33,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:34,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:35,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:36,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:37,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:38,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:38,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:16:39,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:40,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:41,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:42,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:43,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:44,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:44,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:16:44,430 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:16:44,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:16:45,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:46,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:47,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:48,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:16:48,017 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:16:48,017 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:16:48,019 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:16:48,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:48,988 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:16:49,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:49,991 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:16:50,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:50,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:16:51,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:52,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:53,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:54,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:55,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:55,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:16:56,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:57,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:58,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:58,953 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:16:59,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:16:59,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:16:59,430 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:17:00,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:00,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:17:01,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:02,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:03,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:04,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:05,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:06,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:06,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:17:07,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:08,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:09,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:10,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:11,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:11,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:17:12,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:13,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:14,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:14,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:17:14,430 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:17:15,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:16,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:16,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:17:17,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:18,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:19,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:20,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:21,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:22,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:22,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:17:23,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:24,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:25,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:26,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:27,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:27,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:17:28,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:28,955 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:17:29,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:29,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:17:29,430 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:17:30,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:31,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:32,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:32,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:17:33,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:34,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:35,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:36,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:37,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:38,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:38,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:17:39,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:40,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:41,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:42,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:43,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:44,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:44,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:17:44,431 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:17:44,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:17:45,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:46,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:47,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:48,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:49,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:50,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:50,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:17:51,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:52,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:53,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:54,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:55,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:56,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:56,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:17:57,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:58,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:58,956 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:17:59,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:17:59,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:17:59,431 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:18:00,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:01,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:01,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:18:02,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:03,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:04,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:04,560 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:18:04,561 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:18:04,561 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:18:04,562 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:18:05,150 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:18:05,531 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:06,154 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:18:06,529 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:06,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:18:07,529 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:08,529 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:09,529 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:10,529 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:11,530 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:12,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:18:12,529 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:13,530 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:14,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:18:14,431 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:18:14,529 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:15,530 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:16,530 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:17,530 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:17,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:18:18,530 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:19,530 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:20,531 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:21,530 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:22,531 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:23,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:18:23,531 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:24,531 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:25,531 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:26,531 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:27,531 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:28,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:18:28,531 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:28,959 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:18:29,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:18:29,431 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:18:29,531 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:30,532 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:31,532 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:32,532 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:33,532 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:33,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:18:34,532 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:35,532 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:36,532 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:37,533 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:38,532 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:39,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:18:39,532 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:40,532 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:41,533 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:42,533 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:43,533 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:44,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:18:44,431 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:18:44,533 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:44,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:18:45,533 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:46,533 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:47,534 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:48,534 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:49,534 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:50,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:18:50,534 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:51,534 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:52,534 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:53,534 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:54,534 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:55,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:18:55,534 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:56,534 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:57,535 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:58,535 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:18:58,961 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:18:59,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:18:59,431 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:18:59,535 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:00,535 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:00,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:19:01,535 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:02,535 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:03,535 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:04,535 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:05,536 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:06,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:19:06,536 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:07,536 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:08,536 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:09,536 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:10,536 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:11,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:19:11,536 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:12,536 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:13,537 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:14,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:19:14,432 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:19:14,536 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:15,537 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:16,537 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:16,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:19:17,537 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:18,537 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:19,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:19:19,101 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:19:19,101 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:19:19,102 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:19:19,309 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:19:19,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:20,310 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:19:20,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:21,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:21,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:19:22,314 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:19:22,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:23,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:24,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:25,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:26,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:27,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:27,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:19:28,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:28,963 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:19:29,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:19:29,432 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:19:29,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:30,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:31,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:32,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:19:32,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:33,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:34,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:35,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:36,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:37,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:19:37,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:38,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:39,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:40,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:41,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:42,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:19:42,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:43,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:44,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:19:44,432 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:19:44,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:45,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:46,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:47,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:48,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:19:48,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:49,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:50,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:51,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:52,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:53,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:19:53,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:54,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:55,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:56,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:57,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:58,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:19:58,965 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:19:58,966 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:19:59,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:19:59,432 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:19:59,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:00,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:01,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:02,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:03,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:04,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:20:04,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:05,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:06,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:07,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:08,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:09,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:20:09,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:10,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:11,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:12,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:13,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:14,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:20:14,433 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:20:14,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:15,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:20:15,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:16,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:17,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:18,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:19,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:20,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:20:20,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:21,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:22,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:23,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:24,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:25,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:20:25,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:26,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:27,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:28,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:28,967 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:20:29,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:20:29,634 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:20:29,676 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:30,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:30,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:20:31,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:32,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:20:32,222 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:20:32,222 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:20:32,224 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:20:32,461 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:20:32,697 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:33,696 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:34,464 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:20:34,696 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:35,696 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:36,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:20:36,696 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:37,697 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:38,697 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:39,697 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:40,697 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:41,697 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:42,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:20:42,697 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:43,697 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:44,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:20:44,632 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:20:44,697 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:45,697 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:46,697 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:47,698 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:47,798 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:20:48,698 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:49,698 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:50,698 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:51,698 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:52,698 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:53,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:20:53,698 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:54,698 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:55,698 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:56,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:57,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:58,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:20:58,969 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:20:58,972 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:20:59,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:20:59,632 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:20:59,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:00,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:01,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:02,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:03,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:04,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:21:04,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:05,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:06,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:07,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:08,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:09,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:21:09,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:10,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:11,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:12,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:13,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:14,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:21:14,632 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:21:14,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:14,799 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:21:15,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:16,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:17,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:18,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:19,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:20,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:21:20,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:21,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:22,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:23,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:24,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:25,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:26,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:21:26,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:27,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:28,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:28,971 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:21:29,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:21:29,632 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:21:29,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:30,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:31,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:31,799 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:21:32,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:33,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:34,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:35,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:36,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:37,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:21:37,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:38,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:39,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:40,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:41,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:42,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:43,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:21:43,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:43,780 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:21:43,781 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:21:43,781 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:21:43,783 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:21:44,620 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:21:44,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:21:44,633 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:21:44,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:45,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:46,624 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:21:46,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:47,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:48,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:48,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:21:49,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:50,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:51,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:52,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:53,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:54,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:21:54,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:55,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:56,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:57,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:58,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:58,972 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:21:59,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:21:59,633 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:21:59,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:21:59,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:22:00,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:01,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:02,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:03,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:04,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:05,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:22:05,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:06,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:07,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:08,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:09,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:10,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:11,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:22:11,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:12,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:13,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:14,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:22:14,633 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:22:14,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:15,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:16,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:16,799 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:22:17,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:18,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:19,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:20,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:21,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:22,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:22:22,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:23,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:24,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:25,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:26,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:27,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:22:27,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:28,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:28,974 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:22:29,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:22:29,633 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:22:29,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:30,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:31,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:32,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:32,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:22:33,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:34,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:35,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:36,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:37,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:38,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:22:38,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:39,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:40,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:41,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:42,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:43,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:22:43,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:44,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:22:44,633 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:22:44,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:45,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:46,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:47,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:48,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:48,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:22:49,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:50,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:51,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:52,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:53,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:54,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:22:54,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:55,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:22:55,209 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:22:55,209 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:22:55,211 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:22:55,777 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:22:55,777 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:22:55,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:56,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:57,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:58,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:22:58,976 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:22:59,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:22:59,633 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:22:59,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:22:59,822 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:00,823 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:01,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:02,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:03,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:04,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:05,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:23:05,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:06,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:07,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:08,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:09,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:10,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:11,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:23:11,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:12,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:13,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:14,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:23:14,634 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:23:14,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:15,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:16,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:23:16,825 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:17,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:18,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:19,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:20,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:21,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:22,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:23:22,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:23,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:24,826 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:25,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:26,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:27,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:28,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:23:28,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:28,977 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:23:29,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:23:29,634 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:23:29,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:30,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:31,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:32,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:33,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:23:33,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:34,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:35,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:36,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:37,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:38,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:39,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:23:39,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:40,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:41,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:42,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:43,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:44,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:23:44,634 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:23:44,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:23:44,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:45,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:46,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:47,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:48,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:49,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:50,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:23:50,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:51,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:52,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:53,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:54,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:55,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:23:55,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:56,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:57,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:58,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:23:58,980 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:23:59,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:23:59,634 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:23:59,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:00,802 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:24:00,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:01,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:02,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:03,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:04,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:05,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:06,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:24:06,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:07,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:24:07,062 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:24:07,062 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:24:07,064 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:24:07,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:07,924 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:24:07,924 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:24:08,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:09,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:10,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:11,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:12,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:24:12,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:13,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:14,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:24:14,634 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:24:14,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:15,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:16,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:17,802 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:24:17,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:18,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:19,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:20,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:21,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:22,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:23,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:24:23,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:24,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:25,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:26,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:27,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:28,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:28,981 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:24:28,983 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:24:29,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:24:29,634 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:24:29,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:30,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:31,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:32,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:33,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:34,771 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:24:34,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:35,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:36,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:37,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:38,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:39,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:24:39,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:40,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:41,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:42,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:43,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:44,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:24:44,635 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:24:44,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:24:44,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:45,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:46,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:47,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:48,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:49,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:50,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:24:50,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:51,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:52,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:53,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:54,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:55,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:56,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:24:56,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:57,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:58,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:24:58,982 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:24:59,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:24:59,635 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:24:59,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:00,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:01,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:02,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:25:02,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:03,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:04,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:05,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:06,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:07,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:25:07,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:08,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:09,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:10,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:11,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:12,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:25:12,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:13,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:14,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:25:14,635 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:25:14,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:15,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:16,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:17,801 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:25:17,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:18,556 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:25:18,557 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:25:18,557 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:25:18,559 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:25:18,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:19,056 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:25:19,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:20,057 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:25:20,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:21,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:22,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:23,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:25:23,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:24,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:25,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:26,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:27,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:28,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:28,983 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:25:28,984 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:25:29,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:25:29,635 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:25:29,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:30,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:31,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:32,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:33,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:34,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:25:34,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:35,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:36,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:37,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:38,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:39,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:25:39,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:40,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:41,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:42,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:43,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:44,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:25:44,635 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:25:44,800 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:25:44,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:45,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:46,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:47,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:48,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:49,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:50,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:25:50,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:51,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:52,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:53,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:54,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:55,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:56,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:25:56,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:57,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:58,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:25:58,986 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:25:59,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:25:59,635 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:25:59,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:00,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:01,802 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:26:01,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:02,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:03,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:04,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:05,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:06,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:07,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:26:07,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:08,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:09,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:10,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:11,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:12,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:26:12,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:13,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:14,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:26:14,636 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:26:14,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:15,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:16,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:17,802 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:26:17,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:18,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:19,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:20,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:21,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:22,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:23,775 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:26:23,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:24,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:25,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:26,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:27,849 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:28,775 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:26:28,849 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:28,986 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:26:29,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:26:29,636 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:26:29,849 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:30,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:26:30,161 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:26:30,161 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:26:30,165 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:26:30,194 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:26:30,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:31,849 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:32,198 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:26:32,849 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:33,849 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:34,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:26:34,849 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:35,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:36,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:37,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:38,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:39,775 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:26:39,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:40,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:41,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:42,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:43,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:44,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:26:44,636 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:26:44,802 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:26:44,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:45,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:46,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:47,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:48,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:49,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:50,775 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:26:50,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:51,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:52,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:53,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:54,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:55,775 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:26:55,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:56,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:57,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:58,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:26:58,988 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:26:59,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:26:59,636 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:26:59,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:00,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:27:00,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:01,853 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:02,853 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:03,853 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:04,853 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:05,853 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:06,776 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:27:06,853 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:07,853 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:08,853 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:09,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:10,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:11,776 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:27:11,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:12,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:13,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:14,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:27:14,636 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:27:14,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:15,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:16,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:27:16,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:17,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:18,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:19,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:20,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:21,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:22,776 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:27:22,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:23,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:24,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:25,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:26,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:27,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:28,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:27:28,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:28,989 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:27:29,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:27:29,637 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:27:29,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:30,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:31,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:32,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:33,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:27:33,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:34,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:35,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:36,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:37,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:38,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:39,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:27:39,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:40,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:41,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:27:41,774 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:27:41,774 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:27:41,776 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:27:41,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:42,335 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:27:42,335 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:27:42,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:43,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:44,339 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:27:44,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:27:44,637 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:27:44,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:27:44,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:45,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:46,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:47,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:48,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:49,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:50,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:27:50,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:51,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:52,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:53,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:54,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:55,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:27:55,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:56,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:57,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:58,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:27:58,991 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:27:59,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:27:59,637 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:27:59,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:00,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:28:00,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:01,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:02,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:03,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:04,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:05,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:06,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:28:06,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:07,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:08,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:09,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:10,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:11,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:28:11,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:12,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:13,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:14,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:28:14,637 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:28:14,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:15,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:16,803 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:28:16,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:17,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:18,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:19,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:20,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:21,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:22,778 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:28:22,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:23,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:24,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:25,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:26,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:27,778 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:28:27,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:28,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:28,993 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:28:29,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:28:29,637 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:28:29,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:30,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:31,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:32,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:28:32,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:33,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:34,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:35,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:36,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:37,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:38,778 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:28:38,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:39,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:40,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:41,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:42,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:43,779 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:28:43,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:44,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:28:44,638 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:28:44,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:45,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:46,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:47,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:48,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:28:48,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:49,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:50,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:51,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:52,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:53,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:53,983 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:28:54,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:28:54,025 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:28:54,025 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:28:54,026 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:28:54,471 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:28:54,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:55,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:56,474 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:28:56,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:57,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:58,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:28:58,995 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:28:58,997 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:28:59,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:28:59,637 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:28:59,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:00,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:01,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:02,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:03,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:04,779 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:29:04,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:05,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:06,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:07,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:08,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:09,780 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:29:09,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:10,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:11,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:12,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:13,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:14,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:29:14,638 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:29:14,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:29:14,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:15,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:16,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:17,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:18,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:19,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:20,780 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:29:20,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:21,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:22,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:23,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:24,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:25,780 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:29:25,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:26,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:27,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:28,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:28,997 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:29:29,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:29:29,638 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:29:29,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:30,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:29:30,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:31,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:32,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:33,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:34,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:35,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:36,780 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:29:36,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:37,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:38,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:39,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:40,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:41,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:42,781 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:29:42,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:43,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:44,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:29:44,638 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:29:44,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:45,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:46,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:47,804 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:29:47,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:48,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:49,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:50,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:51,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:52,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:53,781 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:29:53,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:54,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:55,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:56,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:57,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:58,781 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:29:58,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:29:58,998 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:29:59,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:29:59,638 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:29:59,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:00,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:01,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:02,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:03,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:30:03,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:04,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:05,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:06,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:30:06,269 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:30:06,269 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:30:06,271 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:30:06,634 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:30:06,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:07,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:08,639 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:30:08,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:09,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:30:09,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:10,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:11,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:12,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:13,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:14,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:30:14,638 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:30:14,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:30:14,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:15,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:16,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:17,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:18,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:19,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:20,781 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:30:20,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:21,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:22,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:23,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:24,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:25,782 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:30:25,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:26,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:27,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:28,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:29,000 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:30:29,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:30:29,638 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:30:29,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:30,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:30:30,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:31,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:32,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:33,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:34,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:35,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:36,782 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:30:36,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:37,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:38,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:39,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:40,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:41,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:42,782 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:30:42,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:43,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:44,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:30:44,638 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:30:44,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:45,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:46,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:47,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:30:47,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:48,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:49,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:50,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:51,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:52,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:53,782 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:30:53,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:54,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:55,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:56,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:57,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:58,783 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:30:58,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:30:59,001 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:30:59,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:30:59,639 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:30:59,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:00,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:01,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:02,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:03,806 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:31:03,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:04,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:05,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:06,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:07,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:08,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:09,782 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:31:09,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:10,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:11,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:12,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:13,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:14,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:31:14,639 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:31:14,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:31:14,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:15,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:16,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:17,644 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:31:17,645 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:31:17,645 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:31:17,647 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:31:17,776 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:31:18,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:18,778 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:31:19,078 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:20,078 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:20,783 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:31:21,078 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:22,078 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:23,078 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:24,078 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:25,078 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:25,783 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:31:26,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:27,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:28,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:29,003 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:31:29,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:29,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:31:29,639 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:31:30,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:30,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:31:31,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:32,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:33,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:34,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:35,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:36,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:36,783 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:31:37,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:38,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:39,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:40,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:41,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:42,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:42,783 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:31:43,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:44,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:44,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:31:44,639 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:31:45,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:46,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:47,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:47,806 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:31:48,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:49,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:50,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:51,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:52,082 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:53,082 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:53,784 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:31:54,082 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:55,082 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:56,082 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:57,082 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:58,082 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:58,784 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:31:59,005 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:31:59,082 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:31:59,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:31:59,639 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:32:00,082 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:01,082 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:02,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:03,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:03,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:32:04,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:05,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:06,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:07,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:08,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:09,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:09,784 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:32:10,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:11,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:12,084 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:13,084 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:14,084 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:14,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:32:14,640 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:32:14,805 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:32:15,084 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:16,084 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:17,084 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:18,084 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:19,084 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:20,084 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:20,784 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:32:21,084 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:22,085 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:23,085 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:24,085 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:25,085 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:26,085 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:26,786 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:32:27,085 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:28,086 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:29,007 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:32:29,086 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:29,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:32:29,640 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:32:30,086 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:30,348 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:32:30,350 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:32:30,350 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:32:30,351 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:32:30,916 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:32:31,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:32,086 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:32,352 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:32:32,920 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:32:33,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:34,086 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:35,086 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:36,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:37,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:37,785 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:32:38,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:39,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:40,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:41,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:42,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:42,785 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:32:43,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:44,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:44,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:32:44,640 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:32:45,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:46,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:47,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:47,806 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:32:48,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:49,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:50,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:51,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:52,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:53,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:53,785 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:32:54,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:55,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:56,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:57,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:58,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:59,009 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:32:59,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:32:59,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:32:59,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:32:59,640 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:33:00,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:01,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:02,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:03,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:04,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:04,785 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:33:05,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:06,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:07,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:08,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:09,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:09,785 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:33:10,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:11,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:12,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:13,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:14,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:14,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:33:14,640 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:33:14,806 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:33:15,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:16,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:17,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:18,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:19,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:20,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:20,785 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:33:21,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:22,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:23,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:24,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:25,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:25,786 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:33:26,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:27,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:28,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:29,010 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:33:29,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:29,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:33:29,640 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:33:30,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:30,808 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:33:31,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:32,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:33,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:34,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:35,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:36,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:36,787 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:33:37,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:38,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:39,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:40,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:41,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:41,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:33:42,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:42,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:33:42,233 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:33:42,234 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:33:42,236 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:33:43,080 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:33:43,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:44,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:44,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:33:44,641 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:33:45,085 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:33:45,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:46,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:47,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:47,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:33:48,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:49,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:50,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:51,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:52,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:53,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:53,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:33:54,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:55,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:56,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:57,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:58,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:58,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:33:59,011 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:33:59,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:33:59,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:33:59,641 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:34:00,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:01,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:02,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:03,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:04,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:04,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:34:05,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:06,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:07,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:08,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:09,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:10,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:10,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:34:11,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:12,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:13,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:14,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:14,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:34:14,641 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:34:15,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:16,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:16,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:34:17,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:18,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:19,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:20,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:21,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:21,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:34:22,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:23,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:24,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:25,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:26,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:27,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:27,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:34:28,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:29,013 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:34:29,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:29,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:34:29,641 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:34:30,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:31,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:32,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:32,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:34:33,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:34,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:35,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:36,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:37,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:38,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:38,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:34:39,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:40,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:41,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:42,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:43,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:43,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:34:44,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:44,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:34:44,641 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:34:45,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:46,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:47,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:48,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:48,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:34:49,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:50,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:51,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:52,556 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:53,552 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:53,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:34:54,552 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:55,352 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:34:55,353 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:34:55,353 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:34:55,355 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:34:55,552 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:56,223 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:34:56,553 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:57,225 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:34:57,553 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:58,553 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:59,014 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:34:59,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:34:59,553 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:34:59,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:34:59,641 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:35:00,553 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:01,553 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:02,553 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:03,553 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:04,553 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:04,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:35:05,553 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:06,554 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:07,554 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:08,554 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:09,554 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:09,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:35:10,554 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:11,554 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:12,555 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:13,554 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:14,555 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:14,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:35:14,642 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:35:15,555 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:15,846 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:35:16,555 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:17,555 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:18,556 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:19,556 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:20,556 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:20,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:35:21,556 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:22,557 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:23,556 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:24,557 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:25,557 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:26,557 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:26,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:35:27,557 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:28,557 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:29,016 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:35:29,558 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:29,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:35:29,642 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:35:30,558 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:31,557 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:32,558 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:32,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:35:33,558 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:34,558 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:35,558 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:36,559 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:37,558 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:37,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:35:38,559 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:39,559 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:40,559 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:41,560 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:42,560 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:42,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:35:43,560 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:44,560 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:44,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:35:44,642 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:35:45,560 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:46,560 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:47,560 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:48,560 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:48,847 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:35:49,561 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:50,561 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:51,561 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:52,561 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:53,561 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:53,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:35:54,562 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:55,562 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:56,562 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:57,562 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:58,562 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:58,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:35:59,018 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:35:59,563 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:35:59,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:35:59,642 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:36:00,563 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:01,563 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:02,563 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:03,563 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:04,563 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:04,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:36:05,563 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:06,564 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:07,564 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:08,564 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:08,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:36:08,821 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:36:08,821 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:36:08,823 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:36:09,358 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:36:09,565 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:09,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:36:10,360 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:36:10,565 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:11,565 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:12,565 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:13,565 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:14,565 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:14,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:36:14,642 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:36:15,565 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:15,848 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:36:16,565 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:17,566 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:18,566 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:19,566 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:20,566 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:20,849 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:36:21,566 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:22,566 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:23,567 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:24,567 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:25,567 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:25,849 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:36:26,567 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:27,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:28,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:29,020 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:36:29,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:29,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:36:29,643 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:36:30,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:31,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:31,849 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:36:32,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:33,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:34,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:35,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:36,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:37,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:37,849 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:36:38,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:39,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:40,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:41,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:42,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:43,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:43,849 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:36:44,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:44,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:36:44,642 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:36:45,571 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:46,571 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:47,571 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:48,571 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:48,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:36:49,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:50,571 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:51,571 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:52,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:53,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:53,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:36:54,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:55,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:56,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:57,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:58,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:59,021 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:36:59,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:36:59,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:36:59,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:36:59,643 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:37:00,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:01,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:02,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:03,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:04,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:04,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:37:05,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:06,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:07,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:08,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:09,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:10,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:10,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:37:11,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:12,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:13,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:14,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:14,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:37:14,643 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:37:15,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:16,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:16,850 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:37:17,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:18,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:19,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:20,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:21,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:21,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:37:22,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:37:22,385 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:37:22,385 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:37:22,387 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:37:22,515 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:37:22,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:23,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:24,518 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:37:24,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:25,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:26,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:27,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:27,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:37:28,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:29,023 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:37:29,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:37:29,643 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:37:29,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:30,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:31,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:32,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:33,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:33,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:37:34,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:35,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:36,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:37,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:38,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:38,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:37:39,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:40,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:41,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:42,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:43,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:43,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:37:44,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:37:44,644 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:37:44,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:45,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:46,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:47,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:48,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:48,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:37:49,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:50,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:51,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:52,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:53,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:54,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:54,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:37:55,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:56,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:57,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:58,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:59,024 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:37:59,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:37:59,644 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:37:59,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:37:59,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:38:00,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:01,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:02,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:03,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:04,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:04,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:38:05,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:06,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:07,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:08,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:09,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:09,851 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:38:10,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:11,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:12,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:13,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:14,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:38:14,644 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:38:14,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:14,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:38:15,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:16,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:17,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:18,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:19,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:20,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:20,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:38:21,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:22,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:23,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:24,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:25,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:25,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:38:26,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:27,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:28,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:29,027 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:38:29,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:38:29,644 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:38:29,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:30,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:31,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:31,852 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:38:32,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:33,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:34,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:35,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:36,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:38:36,605 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:38:36,605 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:38:36,607 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:38:36,670 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:38:36,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:37,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:38:37,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:38,674 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:38:38,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:39,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:40,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:41,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:42,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:42,853 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:38:43,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:44,644 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:38:44,644 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:38:44,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:45,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:46,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:47,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:47,853 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:38:48,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:49,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:50,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:51,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:52,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:53,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:53,853 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:38:54,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:55,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:56,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:57,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:58,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:38:58,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:38:59,029 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:38:59,644 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:38:59,645 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:38:59,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:00,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:01,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:02,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:03,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:04,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:04,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:39:05,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:06,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:07,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:08,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:09,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:10,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:10,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:39:11,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:12,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:13,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:14,644 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:39:14,644 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:39:14,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:15,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:15,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:39:16,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:17,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:18,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:19,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:20,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:20,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:39:21,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:22,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:23,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:24,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:25,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:26,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:26,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:39:27,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:28,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:29,030 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:39:29,644 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:39:29,644 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:39:29,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:30,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:31,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:32,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:32,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:39:33,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:34,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:35,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:36,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:37,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:37,854 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:39:38,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:39,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:40,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:41,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:42,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:42,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:39:43,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:44,645 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:39:44,645 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:39:44,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:45,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:46,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:47,755 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:48,754 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:48,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:39:49,754 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:50,284 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:39:50,285 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:39:50,286 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:39:50,287 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:39:50,837 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:39:51,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:52,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:52,841 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:39:53,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:54,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:54,288 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:39:55,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:56,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:57,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:58,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:59,033 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:39:59,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:39:59,644 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:39:59,645 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:39:59,810 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:40:00,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:01,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:02,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:03,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:04,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:04,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:40:05,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:06,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:07,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:08,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:09,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:09,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:40:10,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:11,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:12,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:13,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:14,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:14,645 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:40:14,645 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:40:14,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:40:15,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:16,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:17,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:18,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:19,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:19,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:40:20,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:21,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:22,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:23,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:24,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:24,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:40:25,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:26,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:27,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:28,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:29,034 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:40:29,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:29,645 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:40:29,645 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:40:29,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:40:30,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:31,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:32,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:33,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:34,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:34,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:40:35,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:36,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:37,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:38,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:39,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:39,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:40:40,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:41,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:42,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:43,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:44,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:44,645 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:40:44,645 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:40:44,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:40:45,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:46,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:47,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:48,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:49,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:49,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:40:50,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:51,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:52,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:53,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:54,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:54,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:40:55,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:56,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:57,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:58,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:59,036 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:40:59,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:40:59,645 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:40:59,645 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:40:59,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:41:00,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:01,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:02,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:03,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:41:03,097 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:41:03,097 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:41:03,099 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:41:03,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:03,978 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:41:04,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:04,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:41:04,980 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:41:05,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:06,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:07,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:08,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:09,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:10,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:10,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:41:11,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:12,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:13,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:14,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:14,646 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:41:14,647 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:41:15,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:16,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:16,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:41:17,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:18,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:19,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:20,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:21,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:21,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:41:22,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:23,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:24,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:25,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:26,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:26,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:41:27,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:28,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:29,037 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:41:29,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:29,646 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:41:29,646 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:41:30,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:31,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:32,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:32,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:41:33,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:34,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:35,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:36,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:37,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:37,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:41:38,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:39,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:40,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:41,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:42,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:43,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:43,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:41:44,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:44,646 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:41:44,646 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:41:45,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:46,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:47,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:48,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:49,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:49,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:41:50,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:51,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:52,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:53,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:54,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:54,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:41:55,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:56,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:57,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:58,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:59,040 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:41:59,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:41:59,646 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:41:59,646 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:42:00,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:00,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:42:01,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:02,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:03,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:04,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:05,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:05,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:42:06,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:07,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:08,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:09,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:10,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:10,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:42:11,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:12,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:13,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:14,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:14,646 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:42:14,647 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:42:15,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:16,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:16,310 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:42:16,352 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:42:16,353 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:42:16,353 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:42:16,355 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:42:17,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:17,123 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:42:17,123 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:42:18,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:19,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:20,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:21,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:21,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:42:22,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:23,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:24,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:25,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:26,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:26,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:42:27,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:28,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:29,041 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:42:29,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:29,646 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:42:29,647 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:42:30,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:31,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:32,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:32,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:42:33,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:34,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:35,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:36,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:37,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:38,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:38,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:42:39,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:40,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:41,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:42,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:43,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:43,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:42:44,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:44,646 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:42:44,647 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:42:45,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:46,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:47,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:48,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:48,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:42:49,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:50,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:51,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:52,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:53,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:54,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:54,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:42:55,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:56,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:57,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:58,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:59,042 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:42:59,118 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:42:59,647 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:42:59,647 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:42:59,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:43:00,118 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:01,118 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:02,118 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:03,118 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:04,118 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:04,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:43:05,118 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:06,118 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:07,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:08,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:09,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:09,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:43:10,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:11,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:12,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:13,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:14,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:14,647 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:43:14,647 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:43:14,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:43:15,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:16,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:17,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:18,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:19,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:20,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:20,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:43:21,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:22,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:23,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:24,121 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:25,121 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:26,121 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:26,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:43:27,121 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:28,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:43:28,041 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:43:28,041 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:43:28,043 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:43:28,121 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:28,261 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:43:29,053 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:43:29,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:29,263 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:43:29,647 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:43:29,648 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:43:30,122 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:31,122 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:31,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:43:32,122 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:33,122 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:34,122 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:35,122 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:36,122 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:36,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:43:37,122 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:38,122 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:39,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:40,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:41,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:41,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:43:42,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:43,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:44,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:44,647 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:43:44,647 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:43:45,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:46,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:46,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:43:47,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:48,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:49,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:50,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:51,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:51,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:43:52,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:53,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:54,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:55,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:56,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:57,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:57,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:43:58,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:59,046 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:43:59,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:43:59,647 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:43:59,648 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:44:00,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:01,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:02,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:02,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:44:03,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:04,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:05,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:06,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:07,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:08,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:08,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:44:09,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:10,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:11,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:12,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:13,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:13,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:44:14,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:14,647 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:44:14,648 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:44:15,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:16,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:17,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:18,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:19,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:19,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:44:20,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:21,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:22,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:23,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:24,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:24,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:44:25,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:26,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:27,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:28,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:29,049 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:44:29,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:29,647 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:44:29,648 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:44:30,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:30,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:44:31,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:32,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:33,128 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:34,128 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:35,128 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:35,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:44:36,128 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:37,128 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:38,129 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:39,128 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:40,128 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:40,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:44:40,189 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:44:40,189 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:44:40,191 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:44:40,425 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:44:41,129 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:41,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:44:41,427 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:44:42,129 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:43,129 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:44,129 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:44,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:44:44,648 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:44:45,129 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:46,129 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:46,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:44:47,129 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:48,129 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:49,129 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:50,130 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:51,130 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:51,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:44:52,130 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:53,130 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:54,130 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:55,130 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:56,130 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:56,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:44:57,130 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:58,131 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:59,051 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:44:59,131 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:44:59,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:44:59,648 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:45:00,131 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:01,131 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:02,131 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:02,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:45:03,131 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:04,131 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:05,131 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:06,131 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:07,132 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:07,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:45:08,132 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:09,132 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:10,132 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:11,132 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:12,132 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:12,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:45:13,132 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:14,132 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:14,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:45:14,648 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:45:15,132 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:16,132 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:17,132 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:18,133 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:18,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:45:19,133 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:20,133 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:21,133 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:22,133 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:23,133 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:23,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:45:24,133 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:25,133 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:26,133 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:27,133 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:28,134 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:29,051 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:45:29,052 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:45:29,134 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:29,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:45:29,649 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:45:30,134 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:31,134 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:32,134 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:33,134 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:34,134 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:34,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:45:35,134 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:36,134 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:37,135 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:38,135 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:39,135 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:39,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:45:40,135 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:41,135 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:42,135 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:43,135 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:44,135 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:44,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:45:44,648 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:45:45,135 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:45,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:45:46,136 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:47,136 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:48,136 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:49,136 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:50,136 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:50,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:45:51,136 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:52,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:45:52,037 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:45:52,037 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:45:52,040 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:45:52,136 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:52,567 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:45:53,138 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:53,568 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:45:54,137 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:55,137 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:55,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:45:56,137 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:57,137 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:58,137 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:59,053 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:45:59,137 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:45:59,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:45:59,649 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:46:00,137 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:01,138 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:01,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:46:02,137 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:03,138 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:04,138 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:05,138 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:06,138 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:06,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:46:07,138 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:08,138 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:09,138 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:10,138 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:11,139 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:11,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:46:12,139 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:13,139 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:14,139 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:14,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:46:14,649 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:46:15,139 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:16,139 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:17,139 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:17,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:46:18,139 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:19,140 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:20,139 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:21,140 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:22,140 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:22,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:46:23,140 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:24,140 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:25,140 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:26,140 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:27,140 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:28,141 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:28,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:46:29,057 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:46:29,140 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:29,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:46:29,649 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:46:30,141 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:31,141 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:32,141 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:33,141 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:34,141 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:34,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:46:35,141 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:36,141 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:37,141 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:38,142 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:39,142 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:39,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:46:40,142 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:41,142 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:42,142 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:43,142 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:44,142 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:44,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:46:44,649 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:46:45,142 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:45,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:46:46,142 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:47,142 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:48,143 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:49,143 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:50,143 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:50,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:46:51,143 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:52,143 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:53,143 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:54,143 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:55,143 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:56,143 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:56,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:46:57,144 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:58,144 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:59,058 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:46:59,144 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:46:59,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:46:59,649 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:47:00,144 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:01,300 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:02,299 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:02,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:47:03,299 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:03,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:47:03,669 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:47:03,669 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:47:03,671 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:47:03,716 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:47:04,299 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:05,299 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:05,722 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:47:06,300 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:07,300 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:07,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:47:08,301 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:09,300 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:10,300 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:11,300 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:12,301 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:13,301 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:13,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:47:14,301 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:14,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:47:14,650 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:47:15,301 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:16,301 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:17,301 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:18,301 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:19,301 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:19,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:47:20,302 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:21,302 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:22,302 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:23,302 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:24,302 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:24,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:47:25,302 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:26,302 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:27,302 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:28,303 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:29,060 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:47:29,303 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:29,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:47:29,649 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:47:30,303 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:30,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:47:31,303 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:32,303 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:33,303 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:34,303 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:35,303 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:35,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:47:36,304 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:37,304 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:38,304 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:39,304 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:40,304 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:40,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:47:41,304 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:42,304 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:43,304 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:44,304 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:44,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:47:44,650 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:47:45,305 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:46,305 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:46,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:47:47,305 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:48,305 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:49,305 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:50,305 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:51,305 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:51,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:47:52,305 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:53,305 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:54,306 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:55,306 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:56,306 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:57,306 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:57,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:47:58,306 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:59,061 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:47:59,306 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:47:59,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:47:59,650 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:48:00,306 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:01,306 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:02,307 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:03,307 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:03,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:48:04,307 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:05,307 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:06,307 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:07,307 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:08,307 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:08,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:48:09,307 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:10,307 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:11,307 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:12,307 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:13,308 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:14,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:14,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:48:14,650 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:48:14,816 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:48:15,308 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:16,308 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:17,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:48:17,121 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:48:17,121 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:48:17,123 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:48:17,308 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:17,853 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:48:18,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:18,855 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:48:19,308 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:19,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:48:20,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:21,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:22,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:23,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:24,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:25,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:25,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:48:26,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:27,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:28,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:29,063 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:48:29,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:29,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:48:29,650 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:48:30,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:31,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:31,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:48:32,310 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:33,310 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:34,310 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:35,310 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:36,310 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:36,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:48:37,310 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:38,311 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:39,311 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:40,311 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:41,311 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:42,311 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:42,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:48:43,311 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:44,311 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:44,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:48:44,650 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:48:45,311 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:46,311 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:47,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:48,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:48,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:48:49,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:50,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:51,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:52,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:53,313 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:53,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:48:54,313 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:55,313 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:56,313 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:57,313 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:58,313 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:58,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:48:59,064 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:48:59,313 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:48:59,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:48:59,650 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:49:00,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:01,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:02,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:03,326 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:04,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:04,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:49:05,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:06,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:07,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:08,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:09,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:10,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:10,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:49:11,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:12,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:13,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:14,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:14,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:49:14,651 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:49:15,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:16,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:16,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:49:17,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:18,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:19,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:20,316 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:21,316 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:21,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:49:22,316 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:23,316 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:24,316 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:25,316 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:26,316 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:27,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:27,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:49:28,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:29,066 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:49:29,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:29,651 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:49:29,651 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:49:29,779 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:49:29,816 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:49:29,817 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:49:29,819 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:49:30,010 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:49:30,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:31,012 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:49:31,318 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:32,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:33,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:49:33,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:34,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:35,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:36,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:37,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:38,318 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:38,999 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:49:39,318 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:40,318 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:41,318 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:42,318 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:43,318 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:44,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:49:44,318 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:44,651 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:49:44,651 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:49:45,318 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:46,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:47,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:48,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:49,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:49,999 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:49:50,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:51,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:52,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:53,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:54,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:55,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:55,999 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:49:56,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:57,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:58,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:59,068 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:49:59,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:49:59,651 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:49:59,651 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:50:00,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:01,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:50:01,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:02,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:03,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:04,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:05,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:06,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:07,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:50:07,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:08,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:09,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:10,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:11,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:12,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:50:12,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:13,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:14,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:14,651 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:50:14,652 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:50:15,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:16,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:17,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:18,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:50:18,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:19,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:20,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:21,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:22,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:23,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:50:23,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:24,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:25,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:26,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:27,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:28,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:29,001 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:50:29,070 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:50:29,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:29,651 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:50:29,652 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:50:30,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:31,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:32,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:33,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:34,001 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:50:34,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:35,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:36,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:37,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:38,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:39,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:40,001 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:50:40,326 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:41,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:42,326 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:42,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:50:42,845 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:50:42,845 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:50:42,847 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:50:43,148 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:50:43,379 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:44,377 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:44,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:50:44,652 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:50:45,001 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:50:45,151 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:50:45,377 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:46,377 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:47,378 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:48,377 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:49,378 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:50,378 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:51,001 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:50:51,378 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:52,378 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:53,378 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:54,378 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:55,378 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:56,378 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:57,001 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:50:57,378 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:58,379 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:59,071 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:50:59,379 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:50:59,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:50:59,652 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:51:00,379 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:01,379 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:02,379 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:03,001 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:51:03,379 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:04,379 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:05,379 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:06,380 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:07,380 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:08,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:51:08,380 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:09,380 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:10,380 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:11,380 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:12,380 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:13,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:51:13,380 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:14,381 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:14,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:51:14,653 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:51:15,381 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:16,381 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:17,381 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:18,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:51:18,381 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:19,381 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:20,381 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:21,381 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:22,381 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:23,382 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:24,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:51:24,382 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:25,382 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:26,382 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:27,382 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:28,382 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:29,073 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:51:29,075 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:51:29,382 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:29,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:51:29,652 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:51:30,382 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:31,382 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:32,383 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:33,383 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:34,383 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:35,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:51:35,383 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:36,383 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:37,383 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:38,383 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:39,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:40,383 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:41,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:51:41,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:42,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:43,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:44,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:44,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:51:44,652 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:51:45,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:46,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:47,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:51:47,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:48,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:49,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:50,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:51,385 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:52,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:51:52,385 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:53,385 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:54,385 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:55,385 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:56,136 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:51:56,137 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:51:56,138 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:51:56,140 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:51:56,304 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:51:56,385 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:57,140 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:51:57,305 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:51:57,386 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:58,385 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:59,075 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:51:59,386 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:51:59,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:51:59,653 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:52:00,386 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:01,386 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:02,386 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:03,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:52:03,386 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:04,386 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:05,386 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:06,387 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:07,387 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:08,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:52:08,387 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:09,387 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:10,387 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:11,387 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:12,388 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:13,388 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:14,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:52:14,388 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:14,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:52:14,653 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:52:15,388 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:16,388 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:17,388 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:18,389 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:19,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:52:19,389 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:20,389 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:21,389 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:22,389 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:23,389 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:24,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:52:24,390 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:25,390 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:26,390 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:27,390 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:28,390 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:29,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:52:29,076 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:52:29,390 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:29,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:52:29,653 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:52:30,390 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:31,390 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:32,390 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:33,390 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:34,390 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:35,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:52:35,391 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:36,391 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:37,391 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:38,391 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:39,391 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:40,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:52:40,391 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:41,391 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:42,391 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:43,391 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:44,392 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:44,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:52:44,653 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:52:45,392 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:46,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:52:46,392 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:47,392 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:48,392 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:49,392 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:50,392 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:51,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:52:51,392 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:52,393 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:53,393 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:54,393 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:55,393 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:56,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:52:56,393 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:57,393 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:58,394 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:59,078 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:52:59,394 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:52:59,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:52:59,653 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:53:00,394 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:01,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:53:01,394 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:02,394 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:03,394 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:04,394 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:05,394 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:06,395 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:07,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:53:07,394 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:08,395 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:08,684 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:53:08,685 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:53:08,685 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:53:08,687 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:53:09,395 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:09,448 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:53:09,449 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:53:10,395 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:11,395 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:11,452 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:53:12,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:53:12,395 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:13,395 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:14,395 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:14,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:53:14,653 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:53:15,396 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:16,396 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:17,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:53:17,396 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:18,396 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:19,396 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:20,396 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:21,396 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:22,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:53:22,396 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:23,397 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:24,397 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:25,397 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:26,397 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:27,397 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:28,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:53:28,397 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:29,079 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:53:29,397 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:29,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:53:29,653 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:53:30,397 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:31,397 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:32,398 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:33,398 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:34,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:53:34,398 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:35,398 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:36,398 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:37,398 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:38,399 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:39,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:53:39,398 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:40,398 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:41,399 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:42,399 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:43,399 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:44,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:53:44,399 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:44,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:53:44,653 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:53:45,399 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:46,399 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:47,399 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:48,399 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:49,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:53:49,399 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:50,400 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:51,400 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:52,400 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:53,400 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:54,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:53:54,400 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:55,400 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:56,400 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:57,400 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:58,401 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:59,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:53:59,082 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:53:59,401 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:53:59,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:53:59,654 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:54:00,401 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:01,401 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:02,402 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:03,401 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:04,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:54:04,401 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:05,401 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:06,402 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:07,402 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:08,402 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:09,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:54:09,402 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:10,402 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:11,402 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:12,402 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:13,402 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:14,402 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:14,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:54:14,654 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:54:14,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:54:15,402 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:16,403 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:17,403 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:18,403 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:19,403 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:20,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:54:20,403 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:21,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:54:21,161 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:54:21,161 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:54:21,163 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:54:21,403 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:21,605 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:54:22,405 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:23,404 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:23,609 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:54:24,404 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:25,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:54:25,404 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:26,404 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:27,404 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:28,404 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:29,084 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:54:29,404 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:29,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:54:29,654 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:54:30,405 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:31,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:54:31,405 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:32,405 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:33,405 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:34,405 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:35,405 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:36,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:54:36,405 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:37,405 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:38,405 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:39,406 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:40,406 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:41,406 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:42,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:54:42,406 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:43,406 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:44,406 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:44,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:54:44,654 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:54:45,406 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:46,406 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:47,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:54:47,407 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:48,407 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:49,407 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:50,407 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:51,407 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:52,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:54:52,407 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:53,407 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:54,407 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:55,408 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:56,408 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:57,408 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:58,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:54:58,408 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:59,085 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:54:59,408 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:54:59,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:54:59,654 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:55:00,408 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:01,408 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:02,408 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:03,409 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:04,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:55:04,409 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:05,409 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:06,409 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:07,409 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:08,409 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:09,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:55:09,409 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:10,409 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:11,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:12,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:13,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:14,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:14,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:55:14,655 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:55:14,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:55:15,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:16,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:17,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:18,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:19,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:20,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:55:20,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:21,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:22,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:23,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:24,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:25,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:55:25,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:26,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:27,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:28,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:29,088 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:55:29,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:29,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:55:29,654 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:55:30,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:31,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:55:31,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:32,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:33,292 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:55:33,293 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:55:33,293 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:55:33,295 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:55:33,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:33,745 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:55:34,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:35,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:35,749 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:55:36,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:55:36,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:37,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:38,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:39,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:40,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:41,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:42,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:55:42,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:43,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:44,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:44,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:55:44,655 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:55:45,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:46,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:47,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:55:47,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:48,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:49,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:50,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:51,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:52,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:53,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:55:53,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:54,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:55,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:56,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:57,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:58,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:55:58,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:59,089 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:55:59,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:55:59,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:55:59,655 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:56:00,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:01,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:02,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:03,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:04,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:56:04,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:05,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:06,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:07,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:08,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:09,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:56:09,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:10,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:11,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:12,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:13,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:14,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:14,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:56:14,655 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:56:14,821 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:56:15,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:16,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:17,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:18,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:19,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:20,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:56:20,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:21,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:22,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:23,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:24,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:25,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:26,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:56:26,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:27,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:28,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:29,090 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:56:29,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:29,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:56:29,655 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:56:30,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:31,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:32,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:56:32,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:33,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:34,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:35,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:36,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:37,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:38,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:56:38,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:39,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:40,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:41,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:42,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:43,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:56:43,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:44,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:44,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:56:44,655 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:56:45,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:45,808 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:56:45,809 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:56:45,809 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:56:45,812 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:56:45,886 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:56:46,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:47,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:47,890 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:56:48,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:48,812 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:56:49,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:50,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:51,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:52,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:53,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:54,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:56:54,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:55,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:56,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:57,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:58,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:59,093 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:56:59,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:56:59,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:56:59,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:56:59,655 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:57:00,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:01,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:02,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:03,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:04,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:05,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:57:05,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:06,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:07,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:08,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:09,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:10,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:11,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:57:11,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:12,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:13,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:14,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:14,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:57:14,655 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:57:15,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:16,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:57:16,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:17,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:18,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:19,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:20,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:21,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:57:21,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:22,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:23,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:24,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:25,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:26,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:27,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:57:27,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:28,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:29,093 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:57:29,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:29,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:57:29,655 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:57:30,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:31,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:32,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:33,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:57:33,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:34,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:35,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:36,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:37,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:38,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:57:38,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:39,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:40,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:41,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:42,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:43,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:57:43,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:44,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:44,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:57:44,656 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:57:45,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:46,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:47,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:48,067 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:57:48,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:49,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:50,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:51,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:52,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:53,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:54,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:57:54,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:55,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:56,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:57,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:58,128 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:57:58,129 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:57:58,130 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:57:58,131 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:57:58,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:59,028 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:57:59,095 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:57:59,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:57:59,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:57:59,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:57:59,656 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:58:00,030 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:58:00,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:01,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:02,427 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:03,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:04,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:05,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:58:05,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:06,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:07,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:08,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:09,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:10,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:11,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:58:11,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:12,428 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:13,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:14,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:14,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:58:14,656 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:58:15,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:16,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:58:16,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:17,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:18,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:19,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:20,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:21,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:22,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:58:22,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:23,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:24,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:25,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:26,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:27,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:58:27,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:28,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:29,096 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:58:29,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:29,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:58:29,656 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:58:30,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:31,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:32,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:58:32,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:33,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:34,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:35,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:36,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:37,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:58:37,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:38,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:39,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:40,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:41,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:42,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:43,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:58:43,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:44,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:44,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:58:44,656 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:58:45,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:46,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:47,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:48,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:49,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:58:49,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:50,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:51,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:52,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:53,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:54,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:58:54,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:55,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:56,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:57,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:58,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:59,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:58:59,099 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:58:59,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:58:59,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:58:59,656 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:59:00,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:01,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:02,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:03,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:04,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:59:04,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:05,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:06,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:07,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:08,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:09,014 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:59:09,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:10,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 22:59:10,109 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 22:59:10,109 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 22:59:10,111 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 22:59:10,148 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 22:59:10,624 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:11,624 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:12,151 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 22:59:12,624 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:13,624 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:14,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:59:14,624 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:14,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:59:14,656 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:59:15,624 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:16,624 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:17,625 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:18,625 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:19,625 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:20,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:59:20,625 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:21,625 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:22,625 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:23,625 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:24,626 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:25,014 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:59:25,626 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:26,626 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:27,626 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:28,626 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:29,099 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:59:29,627 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:29,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:59:29,657 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:59:30,014 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:59:30,626 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:31,627 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:32,627 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:33,627 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:34,627 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:35,627 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:36,014 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:59:36,627 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:37,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:38,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:39,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:40,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:41,015 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:59:41,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:42,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:43,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:44,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:44,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:59:44,656 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 22:59:45,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:46,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:47,015 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:59:47,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:48,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:49,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:50,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:51,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:52,015 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:59:52,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:53,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:54,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:55,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:56,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:57,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:58,015 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 22:59:58,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:59,102 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 22:59:59,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 22:59:59,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 22:59:59,657 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:00:00,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:01,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:02,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:03,015 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:00:03,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:04,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:05,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:06,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:07,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:08,015 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:00:08,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:09,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:10,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:11,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:12,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:13,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:14,015 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:00:14,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:14,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:00:14,657 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:00:15,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:16,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:17,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:18,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:19,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:00:19,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:20,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:21,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:21,988 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:00:21,989 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:00:21,989 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:00:21,996 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:00:22,286 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:00:22,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:23,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:24,290 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:00:24,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:24,996 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:00:25,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:26,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:27,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:28,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:29,104 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:00:29,648 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:29,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:00:29,657 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:00:30,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:00:30,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:31,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:32,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:33,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:34,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:35,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:00:35,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:36,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:37,649 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:38,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:39,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:40,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:41,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:00:41,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:42,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:43,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:44,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:44,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:00:44,657 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:00:45,650 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:46,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:00:46,651 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:47,651 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:48,651 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:49,651 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:50,651 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:51,651 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:52,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:00:52,651 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:53,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:54,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:55,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:56,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:57,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:00:57,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:58,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:59,105 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:00:59,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:00:59,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:00:59,657 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:01:00,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:01,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:02,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:01:02,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:03,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:04,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:05,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:06,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:07,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:01:07,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:08,653 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:09,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:10,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:11,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:12,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:13,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:01:13,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:14,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:14,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:01:14,657 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:01:15,654 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:16,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:17,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:18,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:01:18,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:19,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:20,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:21,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:22,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:23,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:24,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:01:24,655 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:25,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:26,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:27,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:28,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:29,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:01:29,106 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:01:29,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:29,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:01:29,657 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:01:30,656 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:31,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:32,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:33,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:34,030 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:01:34,072 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:01:34,073 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:01:34,074 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:01:34,075 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:01:34,436 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:01:34,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:35,438 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:01:35,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:36,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:37,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:38,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:39,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:40,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:01:40,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:41,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:42,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:43,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:44,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:01:44,658 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:01:44,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:45,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:01:45,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:46,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:47,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:48,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:49,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:50,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:01:50,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:51,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:52,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:53,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:54,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:55,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:56,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:01:56,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:57,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:58,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:01:59,108 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:01:59,657 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:01:59,658 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:01:59,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:00,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:01,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:02:01,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:02,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:03,661 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:04,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:05,661 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:06,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:02:06,661 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:07,661 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:08,661 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:09,661 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:10,661 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:11,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:02:11,661 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:12,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:13,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:14,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:02:14,658 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:02:14,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:15,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:16,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:17,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:02:17,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:18,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:19,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:20,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:21,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:22,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:23,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:02:23,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:24,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:25,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:26,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:27,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:28,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:02:28,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:29,111 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:02:29,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:02:29,658 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:02:29,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:30,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:31,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:32,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:33,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:34,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:02:34,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:35,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:36,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:37,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:38,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:39,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:02:39,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:40,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:41,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:42,666 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:43,666 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:44,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:02:44,658 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:02:44,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:44,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:02:45,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:02:45,461 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:02:45,461 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:02:45,463 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:02:45,578 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:02:45,666 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:46,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:47,590 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:02:47,666 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:48,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:49,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:50,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:02:50,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:51,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:52,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:53,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:54,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:55,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:56,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:02:56,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:57,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:58,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:02:59,111 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:02:59,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:02:59,658 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:02:59,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:00,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:01,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:03:01,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:02,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:03,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:04,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:05,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:06,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:07,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:03:07,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:08,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:09,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:10,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:11,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:12,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:03:12,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:13,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:14,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:03:14,659 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:03:14,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:15,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:16,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:17,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:03:17,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:18,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:19,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:20,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:21,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:22,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:03:22,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:23,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:24,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:25,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:26,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:27,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:03:27,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:28,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:29,113 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:03:29,658 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:03:29,659 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:03:29,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:30,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:31,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:32,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:03:32,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:33,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:34,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:35,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:36,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:37,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:38,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:03:38,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:39,673 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:40,673 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:41,673 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:42,673 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:43,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:03:43,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:44,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:03:44,659 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:03:44,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:45,673 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:46,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:47,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:48,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:49,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:03:49,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:50,675 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:51,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:52,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:53,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:54,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:03:54,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:55,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:56,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:57,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:58,148 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:03:58,150 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:03:58,150 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:03:58,152 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:03:58,733 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:03:58,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:03:59,114 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:03:59,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:03:59,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:03:59,660 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:03:59,735 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:03:59,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:00,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:01,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:02,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:03,855 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:04,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:05,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:04:05,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:06,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:07,856 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:08,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:09,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:10,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:04:10,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:11,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:12,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:13,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:14,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:04:14,659 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:04:14,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:15,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:16,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:04:16,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:17,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:18,857 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:19,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:20,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:21,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:04:21,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:22,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:23,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:24,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:25,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:26,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:27,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:04:27,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:28,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:29,115 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:04:29,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:04:29,659 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:04:29,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:30,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:31,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:32,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:04:32,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:33,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:34,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:35,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:36,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:37,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:38,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:04:38,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:39,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:40,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:41,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:42,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:43,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:44,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:04:44,659 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:04:44,659 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:04:44,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:45,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:46,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:47,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:48,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:49,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:04:49,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:50,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:51,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:52,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:53,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:54,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:04:54,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:55,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:56,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:57,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:58,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:04:59,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:04:59,118 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:04:59,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:04:59,660 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:04:59,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:00,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:01,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:02,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:03,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:04,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:05:04,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:05,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:06,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:07,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:08,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:09,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:10,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:05:10,540 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:05:10,541 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:05:10,542 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:05:10,543 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:05:10,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:10,876 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:05:11,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:11,879 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:05:12,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:13,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:14,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:05:14,660 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:05:14,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:15,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:16,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:05:16,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:17,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:18,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:19,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:20,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:21,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:22,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:05:22,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:23,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:24,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:25,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:26,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:27,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:05:27,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:28,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:29,119 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:05:29,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:05:29,660 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:05:29,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:30,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:31,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:32,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:33,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:05:33,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:34,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:35,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:36,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:37,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:38,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:05:38,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:39,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:40,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:41,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:42,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:43,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:05:43,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:44,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:05:44,660 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:05:44,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:45,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:46,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:47,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:48,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:49,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:05:49,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:50,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:51,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:52,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:53,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:54,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:05:54,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:55,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:56,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:57,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:58,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:05:59,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:05:59,120 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:05:59,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:05:59,661 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:05:59,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:00,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:01,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:02,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:03,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:04,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:06:04,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:05,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:06,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:07,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:08,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:09,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:10,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:06:10,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:11,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:12,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:13,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:14,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:06:14,661 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:06:14,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:15,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:16,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:06:16,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:17,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:18,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:19,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:20,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:21,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:06:21,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:22,768 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:06:22,769 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:06:22,770 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:06:22,771 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:06:22,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:23,027 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:06:23,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:24,029 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:06:24,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:25,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:26,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:06:26,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:27,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:28,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:29,122 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:06:29,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:06:29,661 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:06:29,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:30,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:31,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:06:31,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:32,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:33,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:34,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:35,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:36,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:37,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:06:37,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:38,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:39,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:40,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:41,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:42,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:43,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:06:43,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:44,660 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:06:44,661 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:06:44,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:45,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:46,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:47,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:48,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:06:48,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:49,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:50,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:51,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:52,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:53,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:06:53,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:54,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:55,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:56,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:57,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:58,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:06:58,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:06:59,124 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:06:59,661 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:06:59,661 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:06:59,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:00,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:01,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:02,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:03,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:04,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:07:04,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:05,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:06,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:07,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:08,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:09,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:07:09,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:10,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:11,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:12,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:13,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:14,661 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:07:14,661 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:07:14,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:07:14,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:15,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:16,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:17,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:18,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:19,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:20,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:07:20,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:21,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:22,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:23,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:24,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:25,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:07:25,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:26,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:27,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:28,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:29,128 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:07:29,661 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:07:29,661 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:07:29,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:30,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:07:30,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:31,884 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:32,884 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:33,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:34,884 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:35,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:07:35,916 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:07:35,917 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:35,918 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:07:35,918 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:07:35,920 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:07:36,171 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:07:36,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:37,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:38,174 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:07:38,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:39,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:40,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:41,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:07:41,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:42,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:43,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:44,661 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:07:44,662 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:07:44,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:45,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:46,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:07:46,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:47,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:48,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:49,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:50,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:51,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:52,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:07:52,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:53,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:54,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:55,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:56,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:57,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:07:57,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:58,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:07:59,129 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:07:59,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:07:59,662 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:07:59,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:00,899 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:01,899 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:02,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:08:02,899 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:03,899 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:04,899 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:05,899 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:06,899 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:07,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:08:07,899 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:08,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:09,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:10,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:11,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:12,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:13,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:08:13,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:14,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:08:14,662 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:08:14,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:15,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:16,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:17,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:18,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:19,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:08:19,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:20,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:21,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:22,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:23,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:24,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:08:24,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:25,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:26,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:27,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:28,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:29,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:08:29,130 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:08:29,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:08:29,662 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:08:29,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:30,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:31,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:32,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:33,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:34,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:08:34,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:35,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:36,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:37,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:38,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:39,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:40,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:08:40,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:41,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:42,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:43,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:44,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:08:44,662 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:08:44,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:45,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:46,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:08:46,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:47,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:48,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:08:48,257 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:08:48,257 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:08:48,259 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:08:48,314 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:08:48,994 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:49,993 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:50,318 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:08:50,993 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:51,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:08:51,993 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:52,992 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:53,993 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:54,993 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:55,993 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:56,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:08:56,993 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:57,993 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:58,994 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:08:59,132 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:08:59,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:08:59,662 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:08:59,993 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:00,993 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:01,994 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:02,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:09:02,994 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:03,994 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:04,994 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:05,994 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:06,995 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:07,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:09:07,994 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:08,995 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:09,995 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:10,995 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:11,995 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:12,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:09:12,995 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:13,995 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:14,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:09:14,662 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:09:14,995 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:15,995 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:16,995 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:17,996 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:18,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:09:18,996 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:19,996 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:20,996 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:21,996 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:22,996 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:23,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:09:23,996 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:24,996 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:25,997 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:26,997 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:27,997 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:28,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:09:28,997 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:29,133 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:09:29,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:09:29,663 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:09:29,997 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:30,997 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:31,997 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:32,997 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:33,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:09:33,997 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:34,998 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:35,998 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:36,998 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:37,998 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:38,998 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:39,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:09:39,998 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:40,999 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:41,999 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:42,999 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:43,999 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:44,662 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:09:44,663 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:09:44,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:09:44,999 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:45,999 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:46,999 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:47,999 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:49,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:50,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:50,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:09:51,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:52,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:53,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:54,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:55,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:55,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:09:56,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:57,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:58,000 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:59,001 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:09:59,136 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:09:59,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:09:59,663 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:10:00,001 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:00,034 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:10:01,001 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:01,364 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:10:01,365 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:10:01,365 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:10:01,367 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:10:01,458 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:10:02,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:02,459 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:10:03,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:04,001 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:05,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:05,034 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:10:06,001 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:07,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:08,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:09,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:10,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:11,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:11,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:10:12,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:13,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:14,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:14,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:10:14,663 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:10:15,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:16,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:17,002 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:17,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:10:18,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:19,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:20,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:21,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:22,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:23,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:23,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:10:24,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:25,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:26,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:27,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:28,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:28,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:10:29,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:29,139 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:10:29,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:10:29,664 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:10:30,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:31,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:32,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:33,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:33,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:10:34,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:35,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:36,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:37,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:38,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:39,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:39,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:10:40,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:41,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:42,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:43,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:44,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:44,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:10:44,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:10:44,664 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:10:45,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:46,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:47,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:48,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:49,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:50,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:50,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:10:51,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:52,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:53,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:54,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:55,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:56,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:56,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:10:57,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:58,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:59,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:10:59,141 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:10:59,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:10:59,664 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:11:00,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:01,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:02,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:02,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:11:03,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:04,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:05,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:06,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:07,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:07,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:11:08,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:09,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:10,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:11,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:12,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:13,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:13,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:11:13,544 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:11:13,545 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:11:13,545 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:11:13,546 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:11:13,593 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:11:14,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:14,595 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:11:14,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:11:14,664 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:11:15,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:16,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:16,599 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:11:17,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:18,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:11:18,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:19,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:20,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:21,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:22,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:23,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:24,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:11:24,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:25,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:26,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:27,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:28,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:29,142 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:11:29,145 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:11:29,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:29,663 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:11:29,664 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:11:30,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:31,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:32,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:33,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:34,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:35,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:11:35,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:36,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:37,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:38,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:39,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:40,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:41,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:11:41,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:42,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:43,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:44,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:44,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:11:44,664 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:11:45,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:46,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:47,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:11:47,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:48,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:49,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:50,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:51,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:52,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:11:52,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:53,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:54,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:55,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:56,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:57,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:58,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:11:58,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:59,144 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:11:59,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:11:59,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:11:59,664 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:12:00,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:01,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:02,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:03,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:12:03,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:04,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:05,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:06,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:07,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:08,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:12:08,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:09,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:10,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:11,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:12,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:13,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:14,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:12:14,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:14,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:12:14,664 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:12:15,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:16,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:17,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:18,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:19,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:12:19,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:20,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:21,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:22,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:23,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:24,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:25,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:12:25,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:25,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:12:25,466 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:12:25,466 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:12:25,468 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:12:25,795 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:12:26,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:26,796 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:12:27,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:28,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:29,146 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:12:29,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:29,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:12:29,664 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:12:30,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:31,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:12:31,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:32,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:33,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:34,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:35,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:36,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:12:36,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:37,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:38,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:39,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:40,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:41,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:42,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:12:42,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:43,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:44,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:44,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:12:44,664 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:12:45,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:46,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:47,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:48,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:12:48,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:49,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:50,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:51,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:52,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:53,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:12:53,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:54,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:55,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:56,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:57,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:58,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:12:58,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:59,148 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:12:59,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:12:59,664 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:12:59,665 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:13:00,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:01,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:02,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:03,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:04,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:13:04,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:05,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:06,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:07,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:08,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:09,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:13:09,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:10,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:11,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:12,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:13,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:14,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:14,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:13:14,665 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:13:14,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:13:15,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:16,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:17,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:18,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:19,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:20,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:13:20,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:21,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:22,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:23,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:24,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:25,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:13:25,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:26,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:27,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:28,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:29,150 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:13:29,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:29,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:13:29,665 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:13:30,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:13:30,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:31,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:32,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:33,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:34,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:35,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:13:35,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:36,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:37,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:37,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:13:37,605 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:13:37,605 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:13:37,607 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:13:37,929 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:13:37,929 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:13:38,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:39,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:40,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:41,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:13:41,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:42,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:43,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:44,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:44,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:13:44,665 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:13:45,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:46,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:47,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:13:47,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:48,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:49,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:50,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:51,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:52,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:13:52,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:53,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:54,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:55,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:56,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:57,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:13:57,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:58,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:59,152 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:13:59,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:13:59,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:13:59,665 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:14:00,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:01,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:02,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:14:02,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:03,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:04,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:05,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:06,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:07,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:14:07,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:08,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:09,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:10,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:11,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:12,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:13,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:14:13,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:14,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:14,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:14:14,665 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:14:15,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:16,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:17,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:18,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:14:18,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:19,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:20,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:21,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:22,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:23,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:14:23,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:24,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:25,238 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:26,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:27,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:28,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:14:28,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:29,153 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:14:29,238 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:29,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:14:29,665 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:14:30,238 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:31,238 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:32,238 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:33,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:14:33,238 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:34,238 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:35,238 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:36,238 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:37,238 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:38,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:14:38,239 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:39,239 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:40,239 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:41,239 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:42,239 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:43,239 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:44,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:14:44,239 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:44,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:14:44,666 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:14:45,240 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:46,240 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:47,240 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:48,241 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:49,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:14:49,240 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:49,928 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:14:49,929 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:14:49,929 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:14:49,931 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:14:50,080 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:14:50,240 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:51,241 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:52,088 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:14:52,240 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:53,241 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:54,241 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:55,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:14:55,241 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:56,241 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:57,241 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:58,241 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:59,155 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:14:59,242 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:14:59,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:14:59,666 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:15:00,241 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:01,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:15:01,242 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:02,242 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:03,242 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:04,242 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:05,242 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:06,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:15:06,242 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:07,242 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:08,242 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:09,242 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:10,243 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:11,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:15:11,243 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:12,243 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:13,243 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:14,243 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:14,666 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:15:14,666 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:15:15,243 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:16,243 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:17,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:15:17,243 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:18,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:19,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:20,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:21,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:22,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:23,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:15:23,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:24,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:25,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:26,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:27,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:28,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:15:28,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:29,157 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:15:29,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:29,666 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:15:29,666 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:15:30,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:31,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:32,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:33,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:15:33,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:34,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:35,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:36,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:37,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:38,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:39,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:15:39,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:40,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:41,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:42,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:43,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:44,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:44,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:15:44,667 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:15:44,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:15:45,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:46,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:47,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:48,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:49,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:50,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:15:50,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:51,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:52,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:53,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:54,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:55,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:15:55,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:56,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:57,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:58,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:59,159 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:15:59,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:15:59,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:15:59,667 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:16:00,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:16:00,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:01,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:01,308 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:16:01,309 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:16:01,309 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:16:01,311 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:16:02,242 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:16:02,242 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:16:02,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:03,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:04,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:05,044 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:16:05,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:06,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:07,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:08,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:09,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:10,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:11,044 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:16:11,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:12,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:13,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:14,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:14,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:16:14,667 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:16:15,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:16,044 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:16:16,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:17,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:18,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:19,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:20,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:21,044 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:16:21,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:22,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:23,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:24,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:25,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:26,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:27,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:16:27,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:28,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:29,160 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:16:29,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:29,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:16:29,667 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:16:30,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:31,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:32,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:33,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:16:33,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:34,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:35,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:36,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:37,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:38,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:16:38,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:39,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:40,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:41,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:42,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:43,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:44,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:16:44,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:44,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:16:44,668 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:16:45,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:46,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:47,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:48,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:49,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:50,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:16:50,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:51,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:52,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:53,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:54,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:55,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:16:55,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:56,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:57,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:58,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:59,161 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:16:59,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:16:59,667 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:16:59,668 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:17:00,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:17:00,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:01,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:02,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:03,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:04,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:05,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:06,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:17:06,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:07,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:08,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:09,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:10,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:11,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:17:11,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:12,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:13,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:17:13,249 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:17:13,249 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:17:13,251 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:17:13,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:13,401 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:17:14,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:14,403 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:17:14,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:17:14,669 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:17:15,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:16,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:17,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:17:17,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:18,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:19,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:20,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:21,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:22,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:23,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:17:23,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:24,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:25,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:26,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:27,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:28,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:17:28,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:29,165 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:17:29,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:29,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:17:29,668 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:17:30,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:31,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:32,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:33,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:34,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:17:34,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:35,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:36,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:37,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:38,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:39,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:17:39,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:40,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:41,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:42,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:43,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:44,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:17:44,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:44,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:17:44,668 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:17:45,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:46,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:47,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:48,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:49,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:17:49,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:50,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:51,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:52,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:53,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:54,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:17:54,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:55,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:56,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:57,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:58,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:59,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:17:59,165 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:17:59,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:17:59,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:17:59,668 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:18:00,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:01,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:02,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:03,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:04,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:18:04,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:05,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:06,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:07,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:08,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:09,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:10,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:18:10,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:11,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:12,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:13,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:14,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:14,668 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:18:14,669 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:18:15,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:18:15,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:16,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:17,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:18,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:19,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:20,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:18:20,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:21,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:22,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:23,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:24,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:25,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:18:25,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:18:25,157 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:18:25,158 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:18:25,160 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:18:25,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:25,570 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:18:26,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:26,572 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:18:27,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:28,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:29,167 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:18:29,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:29,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:18:29,669 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:18:30,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:18:30,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:31,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:32,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:33,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:34,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:35,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:18:35,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:36,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:37,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:38,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:39,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:40,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:41,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:18:41,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:42,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:43,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:44,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:44,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:18:44,669 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:18:45,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:46,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:47,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:18:47,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:48,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:49,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:50,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:51,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:52,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:18:52,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:53,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:54,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:55,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:56,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:57,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:18:57,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:58,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:59,169 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:18:59,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:18:59,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:18:59,669 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:19:00,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:01,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:02,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:03,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:19:03,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:04,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:05,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:06,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:07,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:08,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:19:08,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:09,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:10,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:11,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:12,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:13,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:19:13,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:14,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:14,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:19:14,669 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:19:15,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:16,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:17,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:18,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:19,050 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:19:19,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:20,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:21,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:22,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:23,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:24,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:25,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:19:25,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:26,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:27,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:28,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:29,171 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:19:29,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:29,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:19:29,669 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:19:30,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:19:30,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:31,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:32,275 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:33,275 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:34,275 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:35,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:19:35,275 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:36,275 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:37,275 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:37,920 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:19:37,921 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:19:37,922 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:19:37,924 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:19:38,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:38,717 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:19:39,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:40,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:19:40,409 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:40,726 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:19:41,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:42,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:43,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:44,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:44,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:19:44,669 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:19:45,050 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:19:45,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:46,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:47,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:48,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:49,410 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:50,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:51,050 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:19:51,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:52,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:53,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:54,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:55,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:56,050 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:19:56,411 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:57,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:58,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:59,172 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:19:59,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:19:59,669 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:19:59,670 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:20:00,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:01,050 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:20:01,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:02,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:03,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:04,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:05,412 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:06,050 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:20:06,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:07,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:08,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:09,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:10,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:11,051 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:20:11,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:12,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:13,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:14,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:14,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:20:14,670 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:20:15,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:16,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:17,051 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:20:17,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:18,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:19,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:20,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:21,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:22,052 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:20:22,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:23,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:24,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:25,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:26,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:27,052 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:20:27,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:28,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:29,176 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:20:29,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:29,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:20:29,670 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:20:30,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:31,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:32,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:33,051 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:20:33,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:34,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:35,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:36,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:37,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:38,052 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:20:38,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:39,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:40,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:41,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:42,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:43,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:44,051 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:20:44,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:44,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:20:44,670 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:20:45,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:46,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:47,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:48,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:49,052 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:20:49,284 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:20:49,285 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:20:49,285 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:20:49,287 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:20:49,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:49,861 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:20:50,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:50,863 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:20:51,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:52,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:53,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:54,053 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:20:54,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:55,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:56,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:57,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:58,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:59,177 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:20:59,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:20:59,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:20:59,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:20:59,670 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:21:00,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:01,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:02,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:03,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:04,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:05,052 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:21:05,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:06,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:07,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:08,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:09,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:10,052 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:21:10,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:11,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:12,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:13,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:14,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:14,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:21:14,671 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:21:15,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:16,052 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:21:16,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:17,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:18,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:19,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:20,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:21,052 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:21:21,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:22,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:23,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:24,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:25,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:26,053 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:21:26,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:27,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:28,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:29,178 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:21:29,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:29,670 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:21:29,671 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:21:30,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:31,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:32,053 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:21:32,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:33,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:34,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:35,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:36,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:37,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:38,053 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:21:38,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:39,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:40,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:41,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:42,424 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:43,053 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:21:43,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:44,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:44,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:21:44,671 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:21:45,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:46,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:47,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:48,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:49,053 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:21:49,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:50,425 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:51,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:52,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:53,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:54,053 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:21:54,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:55,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:56,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:57,426 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:58,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:59,054 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:21:59,180 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:21:59,627 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:21:59,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:21:59,671 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:22:00,627 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:00,948 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:22:00,949 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:22:00,949 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:22:00,951 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:22:00,996 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:22:01,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:02,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:03,000 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:22:03,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:04,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:04,952 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:22:05,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:06,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:07,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:08,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:09,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:10,054 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:22:10,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:11,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:12,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:13,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:14,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:14,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:22:14,671 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:22:15,629 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:16,054 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:22:16,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:17,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:18,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:19,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:20,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:21,054 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:22:21,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:22,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:23,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:24,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:25,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:26,055 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:22:26,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:27,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:28,631 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:29,182 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:22:29,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:29,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:22:29,671 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:22:30,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:31,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:22:31,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:32,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:33,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:34,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:35,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:36,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:37,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:22:37,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:38,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:39,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:40,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:41,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:42,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:22:42,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:43,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:44,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:44,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:22:44,672 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:22:45,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:46,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:47,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:48,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:22:48,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:49,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:50,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:51,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:52,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:53,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:54,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:22:54,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:55,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:56,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:57,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:58,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:59,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:22:59,183 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:22:59,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:22:59,671 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:22:59,672 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:23:00,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:01,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:02,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:03,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:04,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:05,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:23:05,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:06,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:07,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:08,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:09,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:10,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:11,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:23:11,939 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:12,937 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:13,938 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:14,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:23:14,445 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:23:14,445 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:23:14,447 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:23:14,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:23:14,672 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:23:15,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:15,162 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:23:16,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:16,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:23:17,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:17,165 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:23:18,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:19,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:20,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:21,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:21,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:23:22,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:23,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:24,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:25,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:26,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:27,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:27,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:23:28,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:29,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:29,185 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:23:29,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:23:29,672 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:23:30,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:31,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:32,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:33,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:33,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:23:34,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:35,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:36,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:37,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:38,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:39,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:39,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:23:40,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:41,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:42,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:43,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:44,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:44,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:23:44,672 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:23:44,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:23:45,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:46,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:47,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:48,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:49,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:50,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:50,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:23:51,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:52,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:53,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:54,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:55,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:55,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:23:56,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:57,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:58,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:59,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:23:59,186 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:23:59,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:23:59,672 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:24:00,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:01,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:01,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:24:02,026 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:03,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:04,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:05,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:06,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:06,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:24:07,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:08,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:09,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:10,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:11,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:12,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:12,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:24:13,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:14,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:14,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:24:14,672 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:24:15,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:16,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:17,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:17,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:24:18,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:19,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:20,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:21,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:22,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:22,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:24:23,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:24,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:25,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:26,029 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:27,030 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:27,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:24:28,030 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:29,030 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:29,189 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:24:29,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:24:29,673 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:24:29,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:24:29,873 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:24:29,873 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:24:29,874 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:24:30,030 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:30,315 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:24:31,031 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:31,316 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:24:32,031 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:33,031 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:33,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:24:33,320 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:24:34,031 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:35,031 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:36,031 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:37,031 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:38,032 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:39,032 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:39,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:24:40,032 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:41,032 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:42,032 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:43,032 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:44,032 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:44,672 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:24:44,673 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:24:44,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:24:45,032 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:46,032 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:47,032 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:48,033 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:49,033 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:50,033 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:50,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:24:51,033 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:52,033 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:53,034 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:54,034 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:55,034 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:55,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:24:56,034 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:57,034 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:58,034 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:59,034 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:24:59,190 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:24:59,673 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:24:59,673 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:25:00,034 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:00,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:25:01,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:02,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:03,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:04,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:05,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:06,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:06,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:25:07,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:08,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:09,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:10,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:11,036 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:11,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:25:12,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:13,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:14,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:14,673 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:25:14,673 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:25:15,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:16,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:16,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:25:17,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:18,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:19,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:20,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:21,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:22,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:22,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:25:23,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:24,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:25,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:26,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:27,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:27,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:25:28,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:29,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:29,191 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:25:29,673 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:25:29,673 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:25:30,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:31,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:32,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:32,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:25:33,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:34,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:35,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:36,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:37,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:37,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:25:38,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:39,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:40,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:41,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:42,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:43,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:43,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:25:44,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:44,388 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:25:44,389 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:25:44,390 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:25:44,391 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:25:44,495 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:25:44,495 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:25:44,673 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:25:44,673 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:25:45,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:46,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:47,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:48,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:48,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:25:49,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:50,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:51,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:52,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:53,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:53,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:25:54,042 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:55,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:56,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:57,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:58,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:59,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:25:59,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:25:59,194 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:25:59,673 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:25:59,673 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:26:00,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:01,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:02,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:03,044 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:04,044 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:05,044 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:05,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:26:06,044 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:07,044 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:08,044 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:09,044 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:10,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:10,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:26:11,044 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:12,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:13,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:14,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:14,673 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:26:14,674 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:26:15,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:15,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:26:16,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:17,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:18,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:19,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:20,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:20,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:26:21,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:22,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:23,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:24,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:25,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:26,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:26,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:26:27,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:28,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:29,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:29,194 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:26:29,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:26:29,674 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:26:30,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:31,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:31,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:26:32,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:33,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:34,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:35,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:36,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:36,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:26:37,047 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:38,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:39,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:40,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:41,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:41,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:26:42,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:43,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:44,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:44,673 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:26:44,674 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:26:45,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:46,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:46,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:26:47,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:48,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:49,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:50,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:51,050 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:51,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:26:52,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:53,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:54,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:55,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:56,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:56,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:26:57,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:57,652 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:26:57,653 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:26:57,653 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:26:57,655 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:26:57,655 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:26:58,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:58,656 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:26:59,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:26:59,196 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:26:59,675 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:26:59,675 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:27:00,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:01,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:02,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:27:02,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:03,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:04,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:05,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:06,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:07,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:08,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:27:08,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:09,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:10,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:11,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:12,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:13,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:14,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:27:14,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:14,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:27:14,675 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:27:15,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:16,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:17,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:18,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:19,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:20,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:27:20,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:21,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:22,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:23,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:24,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:25,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:27:25,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:26,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:27,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:28,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:29,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:29,199 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:27:29,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:27:29,675 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:27:30,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:27:30,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:31,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:32,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:33,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:34,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:35,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:36,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:27:36,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:37,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:38,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:39,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:40,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:41,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:42,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:27:42,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:43,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:44,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:44,674 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:27:44,675 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:27:45,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:46,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:47,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:27:47,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:48,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:49,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:50,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:51,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:52,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:27:52,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:53,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:54,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:55,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:56,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:57,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:58,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:27:58,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:59,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:27:59,200 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:27:59,675 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:27:59,675 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:28:00,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:01,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:02,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:03,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:04,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:28:04,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:05,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:06,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:07,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:08,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:09,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:28:09,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:10,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:10,552 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:28:10,553 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:28:10,554 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:28:10,556 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:28:10,816 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:28:11,145 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:12,145 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:12,820 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:28:13,145 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:14,145 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:14,557 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:28:14,675 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:28:14,676 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:28:15,145 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:16,145 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:17,145 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:18,146 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:19,146 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:20,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:28:20,145 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:21,146 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:22,146 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:23,146 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:24,146 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:25,146 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:26,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:28:26,147 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:27,149 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:28,149 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:29,150 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:29,201 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:28:29,675 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:28:29,676 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:28:30,149 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:31,149 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:32,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:28:32,149 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:33,149 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:34,150 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:35,150 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:36,150 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:37,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:28:37,150 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:38,150 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:39,152 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:40,151 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:41,151 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:42,151 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:43,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:28:43,151 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:44,151 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:44,675 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:28:44,676 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:28:45,152 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:46,152 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:47,152 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:48,152 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:49,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:28:49,152 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:50,152 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:51,152 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:52,153 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:53,153 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:54,153 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:55,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:28:55,153 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:56,153 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:57,153 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:58,153 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:59,153 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:28:59,205 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:28:59,675 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:28:59,676 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:29:00,153 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:01,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:29:01,153 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:02,154 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:03,154 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:04,154 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:05,154 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:06,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:29:06,154 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:07,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:08,154 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:09,154 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:10,154 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:11,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:12,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:29:12,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:13,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:14,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:14,676 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:29:14,676 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:29:15,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:16,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:17,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:18,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:29:18,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:19,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:20,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:21,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:22,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:23,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:29:23,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:24,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:24,328 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:29:24,329 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:29:24,330 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:29:24,331 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:29:24,952 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:29:24,952 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:29:25,157 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:26,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:27,157 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:28,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:29,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:29:29,157 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:29,206 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:29:29,676 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:29:29,676 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:29:30,157 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:31,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:32,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:33,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:34,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:35,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:29:35,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:36,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:37,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:38,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:39,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:40,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:29:40,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:41,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:42,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:43,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:44,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:44,676 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:29:44,677 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:29:45,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:29:45,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:46,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:47,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:48,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:49,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:50,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:51,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:29:51,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:52,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:53,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:54,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:55,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:56,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:57,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:29:57,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:58,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:59,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:29:59,211 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:29:59,676 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:29:59,676 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:30:00,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:01,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:02,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:30:02,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:03,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:04,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:05,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:06,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:07,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:30:07,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:08,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:09,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:10,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:11,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:12,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:13,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:30:13,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:14,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:14,676 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:30:14,677 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:30:15,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:16,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:17,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:18,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:19,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:30:19,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:20,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:21,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:22,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:23,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:24,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:30:24,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:25,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:26,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:27,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:28,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:29,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:29,210 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:30:29,213 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:30:29,677 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:30:29,677 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:30:30,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:31,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:32,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:33,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:34,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:35,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:30:35,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:36,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:36,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:30:36,897 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:30:36,898 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:30:36,900 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:30:37,119 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:30:37,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:38,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:39,122 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:30:39,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:40,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:30:40,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:41,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:42,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:43,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:44,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:44,677 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:30:44,677 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:30:45,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:46,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:30:46,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:47,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:48,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:49,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:50,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:51,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:52,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:30:52,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:53,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:54,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:55,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:56,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:57,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:58,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:30:58,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:59,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:30:59,216 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:30:59,677 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:30:59,677 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:31:00,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:01,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:02,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:03,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:04,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:31:04,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:05,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:06,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:07,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:08,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:09,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:10,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:31:10,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:11,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:12,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:13,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:14,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:14,677 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:31:14,678 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:31:15,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:31:15,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:16,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:17,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:18,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:19,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:20,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:31:20,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:21,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:22,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:23,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:24,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:25,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:26,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:31:26,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:27,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:28,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:29,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:29,216 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:31:29,677 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:31:29,677 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:31:30,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:31,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:32,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:31:32,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:33,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:34,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:35,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:36,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:37,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:31:37,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:38,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:39,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:40,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:41,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:42,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:43,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:31:43,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:44,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:44,677 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:31:44,678 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:31:45,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:46,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:47,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:48,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:49,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:31:49,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:49,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:31:49,269 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:31:49,269 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:31:49,272 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:31:49,273 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:31:50,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:51,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:51,276 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:31:52,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:53,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:54,122 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:31:54,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:55,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:56,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:57,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:58,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:59,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:31:59,218 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:31:59,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:31:59,677 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:31:59,678 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:32:00,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:01,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:02,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:03,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:04,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:05,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:32:05,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:06,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:07,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:08,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:09,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:10,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:32:10,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:11,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:12,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:13,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:14,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:14,678 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:32:14,678 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:32:15,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:16,117 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:32:16,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:17,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:18,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:19,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:20,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:21,118 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:32:21,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:22,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:23,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:24,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:25,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:26,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:27,118 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:32:27,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:28,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:29,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:29,220 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:32:29,678 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:32:29,678 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:32:30,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:31,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:32,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:32:32,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:33,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:34,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:35,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:36,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:37,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:38,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:32:38,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:39,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:40,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:41,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:42,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:43,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:44,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:32:44,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:44,678 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:32:44,678 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:32:45,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:46,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:47,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:48,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:49,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:50,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:32:50,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:51,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:52,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:53,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:54,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:55,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:32:55,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:56,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:57,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:58,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:59,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:32:59,222 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:32:59,679 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:32:59,679 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:33:00,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:01,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:33:01,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:01,456 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:33:01,457 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:33:01,457 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:33:01,459 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:33:02,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:02,433 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:33:03,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:03,435 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:33:04,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:05,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:06,119 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:33:06,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:07,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:08,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:09,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:10,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:11,121 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:33:11,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:12,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:13,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:14,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:14,678 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:33:14,679 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:33:15,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:16,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:17,121 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:33:17,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:18,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:19,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:20,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:21,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:22,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:23,121 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:33:23,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:24,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:25,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:26,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:27,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:28,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:29,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:33:29,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:29,223 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:33:29,679 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:33:29,679 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:33:30,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:31,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:32,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:33,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:34,121 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:33:34,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:35,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:36,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:37,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:38,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:39,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:40,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:33:40,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:41,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:42,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:43,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:44,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:44,679 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:33:44,679 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:33:45,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:33:45,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:46,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:47,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:48,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:49,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:50,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:51,120 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:33:51,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:52,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:53,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:54,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:55,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:56,121 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:33:56,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:57,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:58,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:59,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:33:59,225 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:33:59,679 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:33:59,679 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:34:00,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:01,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:34:01,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:02,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:03,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:04,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:05,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:06,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:07,122 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:34:07,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:08,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:09,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:10,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:11,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:12,122 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:34:12,203 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:13,203 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:14,203 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:14,284 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:34:14,285 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:34:14,285 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:34:14,287 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:34:14,606 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:34:14,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:34:14,860 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:34:15,203 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:15,607 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:34:16,203 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:17,204 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:18,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:34:18,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:19,205 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:20,205 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:21,205 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:22,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:23,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:24,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:34:24,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:25,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:26,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:27,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:28,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:29,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:34:29,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:29,227 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:34:29,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:34:29,858 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:34:30,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:31,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:32,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:33,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:34,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:35,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:34:35,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:36,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:37,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:38,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:39,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:40,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:34:40,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:41,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:42,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:43,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:44,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:44,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:34:44,858 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:34:45,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:46,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:34:46,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:47,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:48,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:49,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:50,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:51,123 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:34:51,210 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:52,210 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:53,210 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:54,210 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:55,210 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:56,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:34:56,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:57,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:58,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:59,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:34:59,231 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:34:59,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:34:59,859 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:35:00,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:01,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:35:01,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:02,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:03,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:04,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:05,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:06,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:35:06,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:07,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:08,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:09,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:10,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:11,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:12,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:35:12,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:13,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:14,213 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:14,858 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:35:14,858 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:35:15,216 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:16,216 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:17,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:18,124 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:35:18,216 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:19,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:20,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:21,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:22,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:23,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:35:23,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:24,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:25,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:26,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:27,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:27,344 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:35:27,345 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:35:27,345 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:35:27,347 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:35:27,751 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:35:28,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:28,348 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:35:29,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:29,232 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:35:29,754 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:35:29,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:35:29,860 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:35:30,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:31,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:32,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:33,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:34,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:35:34,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:35,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:36,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:37,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:38,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:39,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:35:39,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:40,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:41,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:42,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:43,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:44,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:35:44,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:44,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:35:44,860 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:35:45,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:46,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:47,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:48,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:49,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:50,125 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:35:50,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:51,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:52,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:53,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:54,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:55,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:35:55,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:56,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:57,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:58,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:59,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:35:59,235 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:35:59,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:35:59,859 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:36:00,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:01,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:36:01,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:02,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:03,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:04,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:05,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:06,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:36:06,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:07,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:08,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:09,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:10,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:11,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:12,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:36:12,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:13,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:14,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:14,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:36:14,860 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:36:15,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:16,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:17,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:18,126 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:36:18,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:19,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:20,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:21,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:22,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:23,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:36:23,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:24,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:25,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:26,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:27,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:28,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:36:28,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:29,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:29,236 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:36:29,859 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:36:29,860 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:36:30,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:31,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:32,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:33,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:34,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:36:34,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:35,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:36,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:37,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:38,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:39,127 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:36:39,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:39,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:36:39,705 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:36:39,705 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:36:39,706 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:36:39,892 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:36:40,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:41,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:41,896 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:36:42,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:43,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:44,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:44,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:36:44,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:36:44,860 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:36:45,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:46,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:47,244 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:48,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:49,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:50,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:36:50,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:51,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:52,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:53,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:54,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:55,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:56,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:36:56,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:57,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:58,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:59,238 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:36:59,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:36:59,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:36:59,860 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:37:00,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:01,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:37:01,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:02,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:03,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:04,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:05,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:06,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:07,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:37:07,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:08,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:09,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:10,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:11,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:12,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:37:12,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:13,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:14,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:14,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:37:14,860 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:37:15,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:16,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:17,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:18,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:37:18,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:19,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:20,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:21,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:22,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:23,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:37:23,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:24,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:25,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:26,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:27,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:28,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:37:28,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:29,241 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:37:29,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:29,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:37:29,861 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:37:30,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:31,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:32,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:33,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:34,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:37:34,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:35,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:36,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:37,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:38,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:39,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:40,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:37:40,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:41,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:42,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:43,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:44,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:44,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:37:44,861 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:37:45,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:37:45,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:46,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:47,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:48,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:49,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:50,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:37:50,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:51,308 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:52,308 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:52,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:37:52,841 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:37:52,841 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:37:52,843 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:37:53,043 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:37:53,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:54,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:55,047 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:37:55,311 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:37:55,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:56,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:57,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:58,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:59,241 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:37:59,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:37:59,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:37:59,861 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:38:00,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:38:00,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:01,429 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:02,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:03,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:04,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:05,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:06,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:38:06,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:07,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:08,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:09,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:10,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:11,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:12,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:38:12,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:13,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:14,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:14,860 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:38:14,861 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:38:15,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:16,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:17,313 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:38:17,431 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:18,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:19,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:20,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:21,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:22,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:23,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:38:23,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:24,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:25,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:26,432 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:27,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:28,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:29,243 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:38:29,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:38:29,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:29,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:38:29,861 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:38:30,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:31,433 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:32,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:33,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:34,313 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:38:34,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:35,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:36,434 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:37,436 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:38,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:39,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:40,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:38:40,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:41,435 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:42,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:43,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:44,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:44,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:38:44,861 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:38:45,312 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:38:45,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:46,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:47,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:48,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:49,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:50,313 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:38:50,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:51,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:52,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:53,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:54,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:55,313 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:38:55,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:56,439 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:57,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:58,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:59,245 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:38:59,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:38:59,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:38:59,861 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:39:00,313 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:39:00,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:01,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:02,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:03,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:04,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:05,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:39:05,441 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:05,816 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:39:05,817 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:39:05,818 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:39:05,820 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:39:06,209 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:39:06,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:07,211 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:39:07,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:08,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:09,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:10,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:11,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:39:11,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:12,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:13,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:14,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:14,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:39:14,862 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:39:15,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:16,442 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:17,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:39:17,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:18,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:19,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:20,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:21,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:22,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:23,314 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:39:23,443 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:24,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:25,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:26,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:27,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:28,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:39:28,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:29,247 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:39:29,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:29,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:39:29,862 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:39:30,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:31,444 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:32,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:33,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:34,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:39:34,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:35,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:36,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:37,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:38,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:39,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:39:39,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:40,445 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:41,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:42,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:43,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:44,316 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:39:44,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:44,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:39:44,862 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:39:45,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:46,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:47,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:48,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:49,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:50,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:39:50,446 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:51,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:52,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:53,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:54,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:55,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:56,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:39:56,447 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:57,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:58,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:59,251 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:39:59,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:39:59,861 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:39:59,862 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:40:00,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:01,315 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:40:01,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:02,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:03,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:04,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:05,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:06,316 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:40:06,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:07,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:08,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:09,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:10,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:11,316 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:40:11,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:12,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:13,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:14,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:14,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:40:14,864 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:40:15,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:16,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:17,316 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:40:17,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:18,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:19,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:40:19,190 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:40:19,190 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:40:19,192 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:40:19,359 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:40:19,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:20,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:21,363 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:40:21,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:22,318 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:40:22,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:23,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:24,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:25,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:26,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:27,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:28,316 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:40:28,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:29,252 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:40:29,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:29,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:40:29,863 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:40:30,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:31,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:32,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:33,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:40:33,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:34,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:35,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:36,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:37,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:38,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:39,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:40:39,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:40,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:41,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:42,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:43,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:44,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:40:44,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:44,862 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:40:44,863 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:40:45,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:46,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:47,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:48,456 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:49,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:40:49,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:50,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:51,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:52,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:53,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:54,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:55,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:40:55,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:56,457 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:57,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:58,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:59,255 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:40:59,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:40:59,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:40:59,864 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:41:00,317 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:41:00,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:01,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:02,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:03,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:04,458 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:05,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:06,326 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:41:06,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:07,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:08,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:09,459 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:10,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:11,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:12,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:41:12,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:13,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:14,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:14,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:41:14,863 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:41:15,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:16,460 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:17,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:18,318 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:41:18,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:19,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:20,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:21,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:22,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:23,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:41:23,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:24,461 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:25,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:26,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:27,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:28,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:29,257 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:41:29,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:41:29,462 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:29,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:41:29,863 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:41:30,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:31,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:32,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:33,463 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:33,680 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:41:33,681 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:41:33,681 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:41:33,682 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:41:34,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:34,506 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:41:34,787 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:41:35,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:35,507 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:41:36,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:37,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:38,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:39,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:40,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:41:40,464 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:41,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:42,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:43,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:44,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:44,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:41:44,863 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:41:45,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:41:45,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:46,465 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:47,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:48,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:49,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:50,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:41:50,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:51,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:52,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:53,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:54,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:55,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:56,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:41:56,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:57,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:58,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:59,260 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:41:59,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:41:59,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:41:59,864 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:42:00,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:01,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:42:01,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:02,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:03,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:04,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:05,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:06,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:07,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:42:07,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:08,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:09,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:10,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:11,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:12,320 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:42:12,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:13,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:14,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:14,863 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:42:14,864 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:42:15,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:16,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:17,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:42:17,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:18,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:19,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:20,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:21,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:22,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:23,321 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:42:23,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:24,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:25,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:26,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:27,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:28,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:42:28,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:29,262 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:42:29,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:29,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:42:29,864 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:42:30,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:31,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:32,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:33,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:34,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:42:34,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:35,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:36,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:37,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:38,474 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:39,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:42:39,474 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:40,474 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:41,474 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:42,474 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:43,474 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:44,476 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:44,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:42:44,864 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:42:45,033 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:42:45,474 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:46,475 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:46,733 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:42:46,734 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:42:46,735 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:42:46,736 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:42:47,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:47,663 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:42:47,663 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:42:48,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:49,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:50,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:42:50,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:51,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:52,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:53,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:54,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:55,322 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:42:55,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:56,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:57,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:58,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:59,265 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:42:59,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:42:59,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:42:59,864 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:43:00,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:43:00,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:01,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:02,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:03,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:04,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:05,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:06,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:43:06,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:07,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:08,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:09,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:10,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:11,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:12,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:43:12,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:13,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:14,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:14,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:43:14,864 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:43:15,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:16,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:17,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:18,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:43:18,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:19,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:20,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:21,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:22,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:23,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:24,323 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:43:24,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:25,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:26,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:27,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:28,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:29,265 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:43:29,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:29,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:43:29,865 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:43:30,035 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:43:30,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:31,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:32,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:33,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:34,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:35,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:43:35,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:36,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:37,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:38,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:39,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:40,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:41,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:43:41,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:42,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:43,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:44,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:44,864 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:43:44,866 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:43:45,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:46,325 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:43:46,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:47,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:48,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:49,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:50,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:51,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:52,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:43:52,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:53,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:54,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:55,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:56,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:57,324 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:43:57,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:58,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:59,267 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:43:59,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:43:59,552 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:43:59,553 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:43:59,553 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:43:59,555 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:43:59,833 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:43:59,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:43:59,865 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:44:00,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:01,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:01,837 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:44:02,329 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:44:02,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:03,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:04,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:05,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:06,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:07,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:44:07,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:08,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:09,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:10,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:11,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:12,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:13,329 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:44:13,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:14,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:14,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:44:14,865 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:44:15,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:16,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:17,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:18,329 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:44:18,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:19,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:20,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:21,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:22,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:23,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:44:23,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:24,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:25,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:26,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:27,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:28,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:44:28,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:29,269 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:44:29,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:29,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:44:29,866 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:44:30,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:31,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:32,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:33,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:34,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:44:34,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:35,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:36,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:37,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:38,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:39,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:44:39,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:40,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:41,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:42,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:43,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:44,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:44,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:44:44,866 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:44:45,037 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:44:45,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:46,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:47,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:48,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:49,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:50,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:44:50,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:51,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:52,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:53,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:54,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:55,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:56,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:44:56,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:57,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:58,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:59,271 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:44:59,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:44:59,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:44:59,865 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:45:00,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:01,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:45:01,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:02,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:03,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:04,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:05,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:06,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:07,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:45:07,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:08,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:09,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:10,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:11,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:12,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:45:12,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:13,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:13,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:45:13,833 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:45:13,833 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:45:13,835 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:45:13,972 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:45:14,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:14,865 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:45:14,866 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:45:15,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:15,975 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:45:16,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:17,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:45:17,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:18,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:19,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:20,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:21,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:22,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:23,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:45:23,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:24,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:25,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:26,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:27,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:28,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:45:28,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:29,273 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:45:29,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:29,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:45:29,866 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:45:30,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:31,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:32,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:33,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:45:33,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:34,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:35,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:36,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:37,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:38,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:39,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:45:39,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:40,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:41,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:42,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:43,508 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:44,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:45:44,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:44,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:45:44,866 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:45:45,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:46,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:47,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:48,508 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:49,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:50,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:45:50,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:51,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:52,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:53,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:54,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:55,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:45:55,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:56,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:57,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:58,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:59,276 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:45:59,510 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:45:59,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:45:59,866 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:46:00,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:46:00,510 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:01,510 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:02,510 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:03,510 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:04,511 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:05,510 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:06,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:46:06,510 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:07,510 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:08,510 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:09,510 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:10,511 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:11,335 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:46:11,511 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:12,511 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:13,512 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:14,511 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:14,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:46:14,866 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:46:15,512 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:16,512 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:17,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:46:17,512 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:18,513 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:19,512 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:20,512 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:21,513 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:22,335 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:46:22,513 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:23,513 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:24,513 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:25,513 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:26,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:27,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:28,334 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:46:28,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:29,278 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:46:29,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:46:29,498 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:46:29,498 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:46:29,500 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:46:29,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:46:29,867 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:46:29,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:30,126 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:46:30,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:31,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:32,129 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:46:32,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:33,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:46:33,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:34,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:35,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:36,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:37,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:38,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:39,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:46:39,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:40,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:41,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:42,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:43,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:44,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:46:44,866 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:46:44,867 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:46:44,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:45,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:46,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:47,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:48,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:49,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:50,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:46:50,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:51,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:52,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:53,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:54,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:55,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:46:55,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:56,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:57,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:58,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:46:59,279 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:46:59,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:46:59,867 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:46:59,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:00,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:47:00,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:01,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:02,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:03,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:04,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:05,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:47:05,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:06,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:07,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:08,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:09,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:10,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:47:10,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:11,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:12,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:13,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:14,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:47:14,867 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:47:14,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:15,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:16,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:47:16,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:17,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:18,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:19,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:20,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:21,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:47:21,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:22,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:23,884 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:24,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:25,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:26,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:47:26,884 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:27,884 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:28,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:29,283 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:47:29,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:47:29,868 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:47:29,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:30,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:31,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:47:31,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:32,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:33,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:34,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:35,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:36,886 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:37,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:47:37,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:38,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:39,886 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:40,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:41,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:42,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:43,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:47:43,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:44,824 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:47:44,825 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:47:44,826 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:47:44,828 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:47:44,867 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:47:44,868 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:47:44,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:45,290 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:47:45,888 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:46,291 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:47:46,889 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:47,888 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:48,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:47:48,888 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:49,888 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:50,889 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:51,889 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:52,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:53,889 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:54,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:47:54,889 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:55,889 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:56,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:57,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:58,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:47:59,283 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:47:59,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:47:59,868 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:47:59,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:00,041 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:48:00,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:01,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:02,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:03,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:04,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:05,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:48:05,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:06,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:07,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:08,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:09,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:10,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:11,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:48:11,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:12,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:13,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:14,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:48:14,868 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:48:14,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:15,892 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:16,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:48:16,892 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:17,892 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:18,892 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:19,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:20,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:21,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:22,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:48:22,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:23,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:24,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:25,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:26,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:27,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:48:27,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:28,894 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:29,285 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:48:29,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:48:29,868 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:48:29,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:30,894 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:31,894 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:32,894 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:33,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:48:33,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:34,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:35,894 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:36,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:37,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:38,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:48:38,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:39,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:40,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:41,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:42,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:43,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:44,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:48:44,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:48:44,868 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:48:44,912 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:45,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:46,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:47,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:48,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:49,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:50,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:48:50,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:51,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:52,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:53,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:55,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:56,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:56,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:48:57,040 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:57,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:48:57,493 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:48:57,493 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:48:57,495 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:48:58,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:58,441 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:48:59,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:48:59,287 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:48:59,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:48:59,869 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:49:00,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:00,444 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:49:01,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:01,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:49:02,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:03,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:04,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:05,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:06,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:07,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:07,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:49:08,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:09,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:10,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:11,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:12,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:13,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:13,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:49:14,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:14,868 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:49:14,869 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:49:15,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:16,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:17,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:18,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:19,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:19,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:49:20,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:21,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:22,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:23,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:24,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:24,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:49:25,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:26,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:27,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:28,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:29,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:29,289 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:49:29,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:49:29,869 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:49:30,038 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:49:30,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:31,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:32,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:33,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:34,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:35,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:35,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:49:36,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:37,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:38,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:39,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:40,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:41,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:41,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:49:42,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:43,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:44,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:44,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:49:44,869 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:49:45,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:46,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:47,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:47,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:49:48,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:49,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:50,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:51,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:52,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:52,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:49:53,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:54,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:55,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:56,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:57,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:58,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:58,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:49:59,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:49:59,290 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:49:59,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:49:59,869 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:50:00,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:01,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:02,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:03,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:03,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:50:04,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:05,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:06,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:07,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:08,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:08,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:50:09,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:10,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:11,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:11,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:50:11,503 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:50:11,503 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:50:11,505 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:50:11,618 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:50:12,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:13,352 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:13,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:50:13,622 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:50:14,352 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:14,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:50:14,870 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:50:15,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:16,352 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:17,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:18,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:19,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:19,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:50:20,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:21,354 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:22,354 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:23,354 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:24,354 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:25,354 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:25,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:50:26,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:27,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:28,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:29,292 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:50:29,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:29,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:50:29,870 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:50:30,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:31,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:31,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:50:32,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:33,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:34,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:35,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:36,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:37,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:37,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:50:38,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:39,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:40,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:41,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:42,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:42,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:50:43,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:44,357 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:44,869 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:50:44,870 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:50:45,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:46,357 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:47,357 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:48,358 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:48,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:50:49,357 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:50,358 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:51,358 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:52,358 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:53,358 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:53,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:50:54,359 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:55,359 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:56,358 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:57,359 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:58,359 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:59,294 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:50:59,297 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:50:59,359 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:50:59,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:50:59,870 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:51:00,359 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:01,359 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:02,359 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:03,359 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:04,359 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:04,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:51:05,359 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:06,360 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:07,360 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:08,360 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:09,361 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:09,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:51:10,361 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:11,361 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:12,361 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:13,361 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:14,362 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:14,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:51:14,870 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:51:15,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:51:15,361 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:16,361 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:17,362 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:18,362 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:19,362 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:20,362 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:20,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:51:21,362 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:22,362 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:23,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:24,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:25,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:25,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:51:25,976 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:51:25,977 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:51:25,978 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:51:25,979 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:51:26,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:26,760 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:51:27,567 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:27,766 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:51:28,567 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:29,295 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:51:29,567 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:29,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:51:29,871 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:51:30,519 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:51:30,567 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:31,567 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:32,567 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:33,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:34,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:35,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:36,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:51:36,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:37,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:38,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:39,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:40,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:41,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:42,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:51:42,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:43,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:44,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:44,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:51:44,870 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:51:45,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:46,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:47,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:51:47,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:48,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:49,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:50,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:51,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:52,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:51:52,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:53,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:54,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:55,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:56,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:57,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:51:57,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:58,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:59,298 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:51:59,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:51:59,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:51:59,871 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:52:00,571 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:01,571 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:02,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:03,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:52:03,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:04,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:05,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:06,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:07,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:08,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:09,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:52:09,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:10,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:11,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:12,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:13,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:14,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:14,870 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:52:14,870 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:52:15,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:52:15,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:16,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:17,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:18,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:19,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:20,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:52:20,573 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:21,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:22,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:23,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:24,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:25,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:52:25,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:26,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:27,574 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:28,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:29,301 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:52:29,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:29,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:52:29,871 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:52:30,508 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:52:30,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:31,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:32,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:33,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:34,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:35,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:36,508 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:52:36,575 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:37,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:38,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:39,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:40,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:52:40,046 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:52:40,046 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:52:40,048 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:52:40,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:40,899 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:52:41,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:52:41,633 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:41,901 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:52:42,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:43,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:44,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:44,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:52:44,872 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:52:45,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:46,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:52:46,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:47,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:48,634 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:49,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:50,635 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:51,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:52,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:52:52,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:53,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:54,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:55,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:56,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:57,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:52:57,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:58,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:59,303 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:52:59,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:52:59,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:52:59,872 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:53:00,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:01,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:02,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:53:02,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:03,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:04,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:05,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:06,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:07,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:08,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:53:08,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:09,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:10,637 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:11,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:12,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:13,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:14,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:53:14,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:14,871 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:53:14,872 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:53:15,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:16,638 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:17,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:18,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:19,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:53:19,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:20,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:21,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:22,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:23,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:24,639 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:25,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:53:25,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:26,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:27,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:28,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:29,305 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:53:29,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:29,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:53:29,872 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:53:30,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:53:30,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:31,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:32,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:33,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:34,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:35,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:53:35,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:36,641 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:37,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:38,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:39,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:40,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:41,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:53:41,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:42,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:43,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:44,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:44,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:53:44,872 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:53:45,642 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:46,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:47,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:53:47,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:48,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:49,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:50,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:51,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:52,608 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:53:52,643 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:53,644 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:54,644 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:55,644 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:56,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:53:56,113 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:53:56,114 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:53:56,115 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:53:57,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:57,041 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:53:58,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:58,043 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:53:58,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:53:59,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:53:59,307 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:53:59,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:53:59,873 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:54:00,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:01,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:02,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:03,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:03,608 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:54:04,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:05,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:06,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:07,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:08,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:09,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:09,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:54:10,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:11,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:12,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:13,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:14,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:14,872 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:54:14,873 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:54:15,017 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:15,043 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:54:16,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:17,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:18,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:19,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:20,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:20,608 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:54:21,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:22,018 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:23,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:24,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:25,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:26,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:26,608 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:54:27,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:28,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:29,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:29,308 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:54:29,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:54:29,873 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:54:30,019 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:31,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:32,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:32,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:54:33,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:34,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:35,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:36,020 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:37,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:37,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:54:38,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:39,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:40,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:41,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:42,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:42,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:54:43,021 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:44,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:44,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:54:44,873 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:54:45,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:46,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:47,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:48,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:48,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:54:49,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:50,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:51,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:52,022 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:53,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:54,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:54,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:54:55,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:56,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:57,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:58,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:59,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:54:59,310 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:54:59,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:54:59,873 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:55:00,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:00,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:55:01,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:02,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:03,023 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:04,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:05,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:05,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:55:06,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:07,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:08,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:09,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:10,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:10,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:55:11,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:12,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:13,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:13,128 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:55:13,129 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:55:13,129 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:55:13,131 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:55:13,194 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:55:14,074 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:14,195 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:55:14,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:55:14,873 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:55:15,072 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:15,687 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:55:16,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:17,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:18,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:19,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:20,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:20,687 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:55:21,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:22,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:23,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:24,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:25,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:26,074 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:26,687 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:55:27,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:28,074 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:29,074 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:29,311 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:55:29,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:55:29,873 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:55:30,074 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:31,074 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:31,688 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:55:32,074 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:33,074 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:34,074 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:35,074 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:36,075 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:37,075 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:37,687 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:55:38,075 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:39,075 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:40,075 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:41,075 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:42,076 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:42,688 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:55:43,076 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:44,076 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:44,873 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:55:44,873 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:55:45,076 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:46,076 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:47,076 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:47,688 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:55:48,076 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:49,076 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:50,077 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:51,077 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:52,077 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:53,077 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:53,688 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:55:54,077 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:55,077 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:56,077 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:57,077 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:58,078 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:58,688 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:55:59,078 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:55:59,314 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:55:59,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:55:59,874 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:56:00,078 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:01,078 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:02,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:03,078 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:03,689 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:56:04,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:05,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:06,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:07,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:08,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:09,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:09,689 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:56:10,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:11,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:12,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:13,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:14,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:14,690 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:56:14,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:56:14,874 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:56:15,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:16,080 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:17,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:18,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:19,082 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:20,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:20,689 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:56:21,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:22,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:23,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:24,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:25,082 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:26,081 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:26,689 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:56:27,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:28,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:29,083 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:29,315 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:56:29,404 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:56:29,405 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:56:29,405 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:56:29,406 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:56:29,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:56:29,874 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:56:30,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:30,340 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:56:30,340 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:56:31,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:31,689 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:56:32,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:32,343 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:56:33,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:34,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:35,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:36,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:36,689 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:56:37,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:38,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:39,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:40,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:41,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:42,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:42,689 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:56:43,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:44,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:44,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:56:44,874 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:56:45,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:46,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:47,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:48,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:48,689 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:56:49,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:50,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:51,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:52,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:53,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:53,690 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:56:54,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:55,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:56,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:57,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:58,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:59,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:56:59,316 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:56:59,319 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:56:59,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:56:59,874 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:57:00,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:01,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:02,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:03,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:04,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:04,691 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:57:05,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:06,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:07,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:08,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:09,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:10,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:10,690 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:57:11,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:12,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:13,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:14,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:14,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:57:14,874 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:57:15,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:16,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:16,690 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:57:17,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:18,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:19,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:20,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:21,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:21,690 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:57:22,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:23,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:24,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:25,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:26,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:26,690 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:57:27,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:28,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:29,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:29,319 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:57:29,874 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:57:29,875 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:57:30,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:31,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:32,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:32,690 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:57:33,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:34,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:35,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:36,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:37,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:38,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:38,690 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:57:39,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:40,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:41,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:42,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:43,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:43,691 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:57:44,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:57:44,009 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:57:44,009 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:57:44,011 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:57:44,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:44,465 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:57:44,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:57:44,876 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:57:45,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:46,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:46,468 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:57:47,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:48,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:48,691 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:57:49,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:50,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:51,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:52,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:53,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:53,691 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:57:54,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:55,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:56,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:57,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:58,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:58,691 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:57:59,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:57:59,319 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:57:59,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:57:59,875 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:58:00,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:01,194 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:02,194 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:03,194 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:04,194 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:04,691 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:58:05,194 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:06,194 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:07,194 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:08,194 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:09,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:09,692 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:58:10,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:11,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:12,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:13,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:14,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:14,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:58:14,875 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:58:15,045 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:58:15,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:16,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:17,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:18,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:19,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:20,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:20,692 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:58:21,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:22,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:23,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:24,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:25,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:25,692 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:58:26,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:27,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:28,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:29,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:29,322 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:58:29,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:58:29,875 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:58:30,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:31,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:31,692 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:58:32,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:33,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:34,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:35,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:36,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:36,692 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:58:37,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:38,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:39,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:40,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:41,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:41,693 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:58:42,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:43,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:44,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:44,875 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:58:44,875 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:58:45,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:46,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:47,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:47,693 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:58:48,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:49,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:50,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:51,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:52,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:52,693 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:58:53,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:54,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:55,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:56,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:57,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:57,981 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:58:58,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:58,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-10 23:58:58,441 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-10 23:58:58,441 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-10 23:58:58,442 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-10 23:58:58,630 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-10 23:58:59,324 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:58:59,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:58:59,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:58:59,876 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:59:00,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:00,635 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-10 23:59:01,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:02,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:03,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:03,980 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:59:04,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:05,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:06,466 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:07,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:08,467 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:09,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:09,979 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:59:10,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:11,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:12,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:13,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:14,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:14,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:59:14,876 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:59:15,046 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:59:15,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:16,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:17,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:18,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:19,469 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:20,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:20,980 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:59:21,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:22,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:23,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:24,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:25,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:26,470 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:26,980 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:59:27,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:28,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:29,325 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:59:29,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:29,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:59:29,876 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:59:30,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:31,471 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:31,980 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:59:32,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:33,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:34,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:35,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:36,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:36,980 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:59:37,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:38,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:39,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:40,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:41,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:41,981 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:59:42,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:43,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:44,473 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:44,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:59:44,877 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-10 23:59:45,474 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:46,474 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:46,981 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:59:47,474 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:48,475 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:49,475 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:50,475 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:51,475 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:52,475 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:52,981 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:59:53,475 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:54,475 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:55,476 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:56,475 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:57,476 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:57,981 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-10 23:59:58,476 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:59,327 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-10 23:59:59,476 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-10 23:59:59,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-10 23:59:59,876 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:00:00,476 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:01,476 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:02,476 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:03,476 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:03,981 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:00:04,476 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:05,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:06,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:07,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:08,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:08,981 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:00:09,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:10,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:11,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:12,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:13,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:13,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:00:13,597 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:00:13,597 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:00:13,600 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:00:13,769 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:00:14,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:14,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:00:14,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:00:14,877 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:00:15,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:16,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:16,775 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:00:17,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:18,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:19,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:19,982 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:00:20,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:21,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:22,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:23,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:24,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:25,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:25,982 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:00:26,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:27,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:28,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:29,328 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:00:29,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:29,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:00:29,877 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:00:30,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:31,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:31,982 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:00:32,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:33,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:34,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:35,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:36,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:36,982 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:00:37,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:38,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:39,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:40,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:41,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:42,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:42,982 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:00:43,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:44,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:44,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:00:44,877 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:00:45,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:46,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:47,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:47,982 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:00:48,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:49,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:50,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:51,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:52,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:53,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:53,983 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:00:54,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:55,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:56,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:57,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:58,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:58,983 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:00:59,331 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:00:59,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:00:59,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:00:59,877 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:01:00,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:01,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:02,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:03,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:03,983 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:01:04,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:05,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:06,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:07,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:08,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:08,983 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:01:09,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:10,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:11,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:12,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:13,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:13,984 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:01:14,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:14,877 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:01:14,877 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:01:15,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:16,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:17,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:18,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:19,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:19,984 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:01:20,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:21,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:22,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:23,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:24,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:24,984 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:01:25,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:26,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:27,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:28,472 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:01:28,473 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:01:28,473 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:01:28,475 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:01:28,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:28,910 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:01:29,333 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:01:29,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:29,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:01:29,878 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:01:30,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:01:30,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:30,913 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:01:31,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:32,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:33,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:34,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:35,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:01:35,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:36,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:37,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:38,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:39,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:40,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:01:40,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:41,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:42,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:43,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:44,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:44,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:01:44,878 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:01:45,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:46,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:01:46,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:47,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:48,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:49,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:50,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:51,154 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:01:51,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:52,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:53,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:54,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:55,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:56,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:01:56,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:57,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:58,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:59,334 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:01:59,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:01:59,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:01:59,878 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:02:00,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:01,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:02:01,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:02,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:03,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:04,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:05,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:06,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:02:06,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:07,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:08,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:09,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:10,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:11,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:12,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:02:12,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:13,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:14,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:14,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:02:14,878 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:02:15,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:16,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:17,155 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:02:17,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:18,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:19,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:20,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:21,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:22,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:23,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:02:23,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:24,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:25,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:26,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:27,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:28,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:02:28,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:29,338 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:02:29,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:29,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:02:29,878 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:02:30,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:31,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:32,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:33,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:34,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:02:34,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:35,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:36,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:37,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:38,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:39,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:02:39,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:40,630 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:41,627 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:42,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:43,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:02:43,093 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:02:43,093 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:02:43,095 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:02:44,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:44,036 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:02:44,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:02:44,878 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:02:45,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:45,038 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:02:45,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:02:46,003 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:47,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:48,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:49,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:50,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:50,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:02:51,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:52,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:53,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:54,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:55,004 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:56,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:56,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:02:57,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:58,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:59,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:02:59,338 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:02:59,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:02:59,879 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:03:00,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:01,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:01,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:03:02,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:03,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:04,005 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:05,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:06,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:07,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:07,156 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:03:08,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:09,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:10,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:11,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:12,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:13,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:13,157 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:03:14,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:14,878 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:03:14,879 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:03:15,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:16,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:17,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:18,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:18,157 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:03:19,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:20,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:21,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:22,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:23,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:23,157 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:03:24,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:25,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:26,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:27,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:28,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:28,157 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:03:29,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:29,340 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:03:29,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:03:29,879 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:03:30,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:31,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:32,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:33,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:34,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:34,157 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:03:35,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:36,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:37,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:38,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:39,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:40,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:40,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:03:41,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:42,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:43,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:44,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:44,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:03:44,879 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:03:45,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:46,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:46,049 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:03:47,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:48,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:49,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:50,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:51,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:51,157 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:03:52,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:53,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:54,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:55,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:56,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:56,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:03:57,085 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:58,085 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:58,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:03:58,597 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:03:58,597 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:03:58,599 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:03:59,085 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:03:59,199 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:03:59,344 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:03:59,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:03:59,880 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:04:00,085 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:00,200 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:04:01,086 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:01,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:04:02,086 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:03,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:04,086 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:05,086 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:06,086 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:07,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:07,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:04:08,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:09,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:10,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:11,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:12,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:13,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:13,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:04:14,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:14,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:04:14,880 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:04:15,087 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:16,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:17,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:18,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:18,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:04:19,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:20,088 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:21,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:22,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:23,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:24,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:24,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:04:25,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:26,089 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:27,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:28,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:29,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:29,158 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:04:29,343 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:04:29,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:04:29,880 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:04:30,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:31,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:32,090 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:33,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:34,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:34,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:04:35,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:36,091 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:37,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:38,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:39,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:39,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:04:40,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:41,092 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:42,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:43,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:44,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:44,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:04:44,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:04:44,880 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:04:45,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:46,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:47,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:48,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:49,093 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:50,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:50,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:04:51,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:52,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:53,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:54,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:55,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:55,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:04:56,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:57,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:58,094 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:59,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:04:59,346 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:04:59,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:04:59,880 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:05:00,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:01,051 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:05:01,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:02,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:03,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:04,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:05,095 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:06,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:06,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:05:07,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:08,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:09,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:10,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:11,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:11,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:05:12,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:13,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:13,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:05:13,237 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:05:13,238 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:05:13,239 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:05:13,328 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:05:14,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:14,330 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:05:14,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:05:14,881 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:05:15,097 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:16,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:16,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:05:17,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:18,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:19,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:20,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:21,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:22,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:22,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:05:23,098 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:24,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:25,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:26,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:27,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:27,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:05:28,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:29,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:29,347 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:05:29,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:05:29,881 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:05:30,099 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:31,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:32,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:32,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:05:33,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:34,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:35,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:36,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:37,100 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:37,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:05:38,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:39,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:40,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:41,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:42,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:43,101 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:43,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:05:44,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:44,880 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:05:44,881 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:05:45,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:46,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:47,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:48,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:49,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:49,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:05:50,102 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:51,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:52,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:53,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:54,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:54,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:05:55,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:56,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:57,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:58,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:59,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:05:59,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:05:59,349 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:05:59,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:05:59,881 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:06:00,103 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:01,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:02,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:03,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:04,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:04,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:06:05,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:06,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:07,104 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:08,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:09,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:09,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:06:10,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:11,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:12,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:13,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:14,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:14,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:06:14,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:06:14,881 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:06:15,105 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:16,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:17,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:18,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:19,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:20,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:20,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:06:21,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:22,106 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:23,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:24,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:24,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:06:24,877 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:06:24,877 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:06:24,879 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:06:25,107 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:25,463 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:06:25,879 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:06:26,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:26,465 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:06:27,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:28,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:29,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:29,349 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:06:29,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:06:29,881 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:06:30,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:31,051 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:06:31,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:32,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:33,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:34,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:35,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:36,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:36,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:06:37,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:38,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:39,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:40,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:41,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:41,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:06:42,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:43,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:44,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:44,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:06:44,882 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:06:45,109 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:46,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:46,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:06:47,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:48,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:49,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:50,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:51,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:51,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:06:52,110 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:53,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:54,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:55,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:56,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:57,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:57,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:06:58,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:59,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:06:59,351 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:06:59,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:06:59,882 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:07:00,111 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:01,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:02,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:02,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:07:03,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:04,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:05,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:06,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:07,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:07,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:07:08,112 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:09,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:10,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:11,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:12,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:12,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:07:13,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:14,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:14,881 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:07:14,882 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:07:15,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:16,113 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:17,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:17,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:07:18,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:19,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:20,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:21,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:22,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:22,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:07:23,114 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:24,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:25,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:26,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:27,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:28,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:28,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:07:29,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:29,353 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:07:29,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:07:29,882 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:07:30,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:31,115 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:32,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:33,116 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:33,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:07:34,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:35,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:36,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:36,628 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:07:36,629 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:07:36,629 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:07:36,631 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:07:37,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:37,609 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:07:38,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:38,611 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:07:38,632 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:07:39,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:40,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:41,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:42,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:43,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:44,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:07:44,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:44,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:07:44,882 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:07:45,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:46,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:47,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:48,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:49,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:50,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:07:50,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:51,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:52,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:53,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:54,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:55,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:07:55,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:56,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:57,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:58,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:59,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:07:59,355 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:07:59,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:07:59,883 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:08:00,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:01,063 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:08:01,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:02,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:03,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:04,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:05,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:06,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:08:06,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:07,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:08,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:09,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:10,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:11,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:12,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:08:12,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:13,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:14,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:14,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:08:14,883 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:08:15,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:16,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:17,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:08:17,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:18,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:19,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:20,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:21,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:22,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:08:22,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:23,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:24,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:25,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:26,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:27,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:28,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:08:28,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:29,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:29,356 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:08:29,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:08:29,883 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:08:30,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:31,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:32,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:33,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:08:33,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:34,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:35,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:36,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:37,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:38,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:39,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:08:39,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:40,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:41,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:42,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:43,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:44,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:08:44,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:44,882 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:08:44,883 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:08:45,238 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:46,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:47,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:48,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:48,820 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:08:48,821 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:08:48,821 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:08:48,822 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:08:49,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:49,730 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:08:50,025 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:08:50,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:50,732 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:08:51,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:52,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:53,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:54,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:55,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:08:55,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:56,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:57,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:58,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:59,358 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:08:59,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:08:59,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:08:59,883 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:09:00,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:01,054 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:09:01,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:02,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:03,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:04,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:05,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:06,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:09:06,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:07,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:08,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:09,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:10,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:11,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:09:11,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:12,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:13,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:14,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:14,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:09:14,883 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:09:15,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:16,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:17,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:09:17,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:18,585 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:19,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:20,585 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:21,585 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:22,585 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:23,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:09:23,585 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:24,585 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:25,585 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:26,585 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:27,585 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:28,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:09:28,586 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:29,359 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:09:29,586 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:29,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:09:29,883 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:09:30,586 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:31,586 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:32,586 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:33,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:09:33,586 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:34,586 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:35,586 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:36,587 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:37,587 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:38,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:09:38,587 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:39,587 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:40,587 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:41,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:42,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:43,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:09:43,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:44,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:44,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:09:44,883 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:09:45,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:46,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:47,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:48,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:49,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:09:49,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:50,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:51,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:52,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:53,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:54,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:09:54,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:55,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:56,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:57,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:58,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:59,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:09:59,362 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:09:59,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:09:59,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:09:59,884 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:10:00,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:10:00,502 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:10:00,502 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:10:00,504 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:10:00,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:00,858 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:10:01,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:02,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:02,861 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:10:03,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:04,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:10:04,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:05,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:06,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:07,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:08,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:09,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:10,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:10:10,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:11,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:12,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:13,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:14,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:14,883 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:10:14,884 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:10:15,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:16,054 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:10:16,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:17,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:18,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:19,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:20,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:21,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:10:21,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:22,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:23,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:24,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:25,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:26,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:10:26,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:27,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:28,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:29,363 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:10:29,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:29,884 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:10:29,884 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:10:30,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:31,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:10:31,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:32,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:33,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:34,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:35,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:36,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:10:36,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:37,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:38,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:39,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:40,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:41,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:10:41,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:42,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:43,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:44,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:44,884 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:10:44,884 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:10:45,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:46,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:10:46,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:47,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:48,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:49,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:50,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:51,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:52,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:10:52,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:53,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:54,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:55,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:56,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:57,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:10:57,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:58,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:59,378 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:10:59,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:10:59,884 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:10:59,884 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:11:00,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:01,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:02,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:11:02,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:03,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:04,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:05,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:06,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:07,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:11:07,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:08,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:09,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:10,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:11,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:12,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:11:12,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:12,968 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:11:12,969 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:11:12,969 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:11:12,971 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:11:13,069 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:11:13,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:14,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:14,884 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:11:14,885 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:11:15,073 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:11:15,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:16,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:17,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:11:17,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:18,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:19,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:20,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:21,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:22,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:23,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:11:23,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:24,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:25,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:26,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:27,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:28,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:29,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:11:29,367 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:11:29,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:29,884 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:11:29,885 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:11:30,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:31,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:32,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:33,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:34,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:35,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:11:35,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:36,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:37,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:38,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:39,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:40,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:41,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:11:41,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:42,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:43,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:44,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:44,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:11:44,885 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:11:45,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:46,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:47,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:11:47,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:48,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:49,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:50,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:51,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:52,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:11:52,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:53,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:54,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:55,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:56,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:57,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:11:57,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:58,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:59,371 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:11:59,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:11:59,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:11:59,885 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:12:00,608 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:01,608 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:02,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:03,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:12:03,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:04,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:05,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:06,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:07,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:08,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:12:08,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:09,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:10,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:11,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:12,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:13,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:14,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:12:14,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:14,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:12:14,885 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:12:15,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:16,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:17,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:18,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:19,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:12:19,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:20,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:21,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:22,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:23,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:24,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:12:24,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:25,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:26,096 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:12:26,097 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:12:26,098 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:12:26,100 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:12:26,133 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:12:26,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:27,136 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:12:27,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:28,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:29,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:12:29,371 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:12:29,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:29,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:12:29,897 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:12:30,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:31,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:32,699 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:33,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:34,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:35,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:12:35,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:36,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:37,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:38,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:39,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:40,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:12:40,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:41,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:42,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:43,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:44,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:44,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:12:44,886 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:12:45,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:46,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:12:46,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:47,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:48,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:49,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:50,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:51,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:12:51,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:52,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:53,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:54,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:55,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:56,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:57,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:12:57,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:58,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:59,373 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:12:59,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:12:59,885 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:12:59,885 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:13:00,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:01,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:02,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:13:02,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:03,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:04,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:05,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:06,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:07,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:08,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:13:08,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:09,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:10,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:11,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:12,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:13,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:14,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:13:14,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:14,886 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:13:14,886 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:13:15,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:16,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:17,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:18,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:19,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:13:19,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:20,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:21,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:22,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:23,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:24,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:25,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:13:25,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:26,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:27,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:28,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:29,376 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:13:29,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:29,886 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:13:29,886 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:13:30,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:31,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:13:31,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:32,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:33,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:34,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:35,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:36,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:13:36,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:37,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:38,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:39,376 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:13:39,377 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:13:39,377 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:13:39,379 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:13:39,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:40,294 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:13:40,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:41,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:13:41,296 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:13:41,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:42,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:43,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:44,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:44,886 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:13:44,886 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:13:45,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:46,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:47,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:13:47,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:48,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:49,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:50,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:51,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:52,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:13:52,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:53,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:54,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:55,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:56,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:57,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:58,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:13:58,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:59,379 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:13:59,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:13:59,886 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:13:59,886 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:14:00,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:01,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:02,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:03,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:04,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:14:04,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:05,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:06,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:07,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:08,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:09,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:10,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:14:10,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:11,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:12,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:13,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:14,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:14,886 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:14:14,886 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:14:15,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:16,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:14:16,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:17,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:18,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:19,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:20,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:21,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:14:21,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:22,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:23,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:24,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:25,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:26,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:14:26,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:27,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:28,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:29,381 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:14:29,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:29,886 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:14:29,886 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:14:30,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:31,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:32,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:14:32,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:33,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:34,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:35,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:36,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:37,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:14:37,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:38,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:39,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:40,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:41,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:42,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:14:42,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:43,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:44,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:44,886 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:14:44,886 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:14:45,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:46,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:47,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:48,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:14:48,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:49,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:50,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:51,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:51,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:14:51,941 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:14:51,942 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:14:51,944 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:14:52,440 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:14:52,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:53,442 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:14:53,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:53,944 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:14:54,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:55,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:56,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:57,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:58,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:59,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:14:59,382 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:14:59,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:14:59,886 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:14:59,886 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:15:00,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:01,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:02,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:03,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:04,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:15:04,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:05,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:06,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:07,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:08,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:09,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:10,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:15:10,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:11,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:12,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:13,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:14,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:14,886 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:15:14,887 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:15:15,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:16,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:15:16,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:17,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:18,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:19,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:20,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:21,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:15:21,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:22,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:23,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:24,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:25,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:26,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:15:26,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:27,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:28,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:29,384 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:15:29,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:29,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:15:29,887 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:15:30,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:31,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:32,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:15:32,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:33,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:34,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:35,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:36,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:37,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:15:37,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:38,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:39,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:40,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:41,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:42,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:15:42,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:43,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:44,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:44,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:15:44,887 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:15:45,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:46,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:47,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:15:47,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:48,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:49,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:50,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:51,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:52,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:15:52,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:53,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:54,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:55,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:56,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:57,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:58,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:15:58,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:59,386 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:15:59,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:15:59,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:15:59,887 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:16:00,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:01,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:02,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:03,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:04,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:16:04,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:05,024 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:16:05,025 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:16:05,025 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:16:05,026 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:16:05,619 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:16:05,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:06,770 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:07,623 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:16:07,771 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:08,771 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:09,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:16:09,771 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:10,771 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:11,771 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:12,771 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:13,771 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:14,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:16:14,771 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:14,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:16:14,887 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:16:15,771 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:16,771 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:17,771 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:18,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:19,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:16:19,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:20,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:21,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:22,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:23,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:24,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:25,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:16:25,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:26,772 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:27,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:28,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:29,387 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:16:29,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:29,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:16:29,887 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:16:30,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:31,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:16:31,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:32,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:33,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:34,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:35,773 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:36,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:16:36,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:37,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:38,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:39,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:40,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:41,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:42,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:16:42,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:43,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:44,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:44,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:16:44,887 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:16:45,774 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:46,775 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:47,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:16:47,775 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:48,775 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:49,775 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:50,775 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:51,775 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:52,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:16:52,775 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:53,776 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:54,776 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:55,776 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:56,776 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:57,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:16:57,776 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:58,776 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:59,389 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:16:59,776 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:16:59,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:16:59,887 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:17:00,776 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:01,776 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:02,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:03,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:17:03,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:04,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:05,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:06,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:07,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:08,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:17:08,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:09,777 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:10,778 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:11,778 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:12,778 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:13,179 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:17:13,778 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:14,782 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:14,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:17:14,887 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:17:15,780 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:16,780 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:17,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:17:17,713 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:17:17,713 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:17:17,714 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:17:17,757 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:17:17,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:18,759 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:17:18,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:18,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:17:19,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:20,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:21,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:22,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:23,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:24,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:17:24,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:25,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:26,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:27,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:28,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:29,391 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:17:29,393 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:17:29,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:29,887 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:17:29,888 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:17:30,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:31,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:32,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:33,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:34,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:35,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:17:35,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:36,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:37,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:38,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:39,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:40,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:17:40,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:41,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:42,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:43,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:44,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:44,888 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:17:44,888 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:17:45,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:46,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:17:46,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:47,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:48,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:49,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:50,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:51,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:17:51,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:52,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:53,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:54,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:55,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:56,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:17:56,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:57,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:58,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:59,393 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:17:59,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:17:59,888 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:17:59,888 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:18:00,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:01,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:18:01,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:02,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:03,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:04,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:05,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:06,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:18:06,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:07,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:08,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:09,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:10,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:11,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:12,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:18:12,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:13,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:14,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:14,892 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:18:14,893 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:18:15,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:16,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:17,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:18:17,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:18,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:19,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:20,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:21,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:22,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:18:22,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:23,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:24,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:25,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:26,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:27,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:18:27,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:28,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:29,395 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:18:29,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:29,888 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:18:29,888 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:18:30,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:18:30,273 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:18:30,274 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:18:30,275 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:18:30,904 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:18:30,904 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:18:30,932 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:31,931 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:32,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:18:32,931 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:33,931 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:34,931 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:35,931 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:36,931 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:37,931 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:38,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:18:38,931 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:39,931 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:40,932 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:41,932 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:42,932 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:43,932 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:44,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:18:44,888 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:18:44,889 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:18:44,932 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:45,932 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:46,932 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:47,932 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:48,932 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:49,933 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:50,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:18:50,933 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:51,933 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:52,933 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:53,933 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:54,933 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:55,933 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:56,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:18:56,933 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:57,934 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:58,934 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:18:59,397 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:18:59,889 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:18:59,889 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:18:59,934 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:00,934 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:01,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:19:01,934 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:02,934 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:03,934 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:04,934 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:05,934 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:06,935 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:07,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:19:07,935 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:08,935 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:09,935 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:10,935 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:11,935 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:12,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:19:12,935 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:13,935 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:14,889 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:19:14,889 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:19:14,936 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:15,936 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:16,936 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:17,936 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:18,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:19:18,936 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:19,936 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:20,936 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:21,936 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:22,936 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:23,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:19:23,937 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:24,937 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:25,937 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:26,937 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:27,937 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:28,937 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:29,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:19:29,398 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:19:29,889 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:19:29,889 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:19:29,937 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:30,938 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:31,938 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:32,938 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:33,938 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:34,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:19:34,938 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:35,938 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:36,938 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:37,938 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:38,938 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:39,939 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:40,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:19:40,939 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:41,939 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:42,939 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:43,008 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:19:43,009 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:19:43,009 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:19:43,013 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:19:43,035 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:19:43,941 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:44,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:19:44,890 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:19:44,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:45,039 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:19:45,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:46,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:19:46,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:47,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:48,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:49,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:50,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:51,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:19:51,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:52,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:53,941 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:54,941 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:55,941 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:56,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:19:56,941 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:57,941 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:58,941 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:19:59,400 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:19:59,889 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:19:59,890 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:19:59,941 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:00,941 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:01,941 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:02,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:20:02,941 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:03,941 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:04,942 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:05,942 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:06,942 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:07,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:20:07,942 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:08,942 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:09,942 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:10,942 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:11,942 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:12,943 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:13,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:20:13,943 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:14,889 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:20:14,890 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:20:14,943 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:15,943 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:16,943 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:17,943 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:18,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:20:18,943 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:19,943 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:20,943 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:21,943 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:22,944 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:23,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:20:23,944 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:24,944 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:25,944 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:26,944 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:27,944 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:28,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:20:28,944 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:29,402 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:20:29,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:20:29,890 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:20:29,944 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:30,944 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:31,944 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:32,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:33,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:34,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:20:34,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:35,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:36,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:37,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:38,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:39,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:20:39,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:40,945 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:41,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:42,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:43,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:44,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:20:44,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:20:44,890 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:20:44,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:45,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:46,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:47,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:48,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:49,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:20:49,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:50,947 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:51,946 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:52,947 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:53,947 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:54,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:20:54,947 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:55,384 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:20:55,385 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:20:55,386 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:20:55,389 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:20:55,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:56,170 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:20:56,954 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:57,174 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:20:57,954 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:58,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:20:59,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:20:59,402 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:20:59,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:20:59,890 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:20:59,954 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:00,954 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:01,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:02,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:03,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:04,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:05,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:21:05,955 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:06,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:07,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:08,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:09,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:10,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:11,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:21:11,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:12,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:13,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:14,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:21:14,890 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:21:14,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:15,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:16,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:21:16,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:17,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:18,956 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:19,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:20,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:21,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:22,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:21:22,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:23,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:24,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:25,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:26,957 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:27,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:21:27,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:28,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:29,404 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:21:29,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:21:29,891 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:21:29,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:30,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:31,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:32,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:21:32,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:33,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:34,958 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:35,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:36,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:37,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:21:37,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:38,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:39,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:40,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:41,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:42,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:43,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:21:43,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:44,890 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:21:44,892 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:21:44,959 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:45,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:46,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:47,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:48,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:21:48,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:49,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:50,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:51,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:52,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:53,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:21:53,960 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:54,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:55,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:56,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:57,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:58,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:21:59,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:21:59,405 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:21:59,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:21:59,891 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:21:59,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:00,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:01,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:02,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:03,961 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:05,179 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:05,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:22:06,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:07,179 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:07,636 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:22:07,637 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:22:07,637 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:22:07,641 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:22:08,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:08,313 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:22:09,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:09,315 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:22:10,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:10,677 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:22:11,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:12,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:13,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:14,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:14,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:22:14,891 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:22:15,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:16,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:22:16,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:17,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:18,246 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:19,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:20,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:21,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:22:21,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:22,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:23,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:24,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:25,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:26,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:22:26,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:27,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:28,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:29,247 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:29,406 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:22:29,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:22:29,891 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:22:30,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:31,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:22:31,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:32,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:33,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:34,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:35,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:36,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:22:36,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:37,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:38,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:39,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:40,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:41,248 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:42,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:22:42,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:43,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:44,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:44,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:22:44,891 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:22:45,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:46,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:47,187 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:22:47,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:48,249 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:49,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:50,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:51,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:52,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:22:52,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:53,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:54,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:55,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:56,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:57,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:22:57,250 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:58,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:59,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:22:59,409 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:22:59,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:22:59,891 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:23:00,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:01,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:02,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:03,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:23:03,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:04,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:05,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:06,251 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:07,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:08,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:23:08,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:09,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:10,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:11,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:12,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:13,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:23:13,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:14,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:14,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:23:14,891 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:23:15,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:16,252 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:17,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:18,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:19,188 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:23:19,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:19,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:23:19,605 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:23:19,605 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:23:19,608 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:23:20,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:20,455 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:23:21,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:21,456 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:23:22,253 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:23,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:24,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:23:24,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:25,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:26,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:27,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:28,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:29,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:29,411 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:23:29,413 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:23:29,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:23:29,892 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:23:30,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:31,254 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:32,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:33,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:34,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:35,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:23:35,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:36,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:37,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:38,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:39,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:40,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:23:40,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:41,255 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:42,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:43,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:44,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:44,891 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:23:44,892 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:23:45,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:46,063 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:23:46,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:47,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:48,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:49,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:50,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:51,189 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:23:51,256 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:52,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:53,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:54,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:55,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:56,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:23:56,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:57,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:58,257 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:59,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:23:59,413 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:23:59,892 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:23:59,892 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:24:00,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:01,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:02,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:24:02,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:03,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:04,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:05,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:06,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:07,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:24:07,258 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:08,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:09,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:10,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:11,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:12,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:24:12,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:13,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:14,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:14,892 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:24:14,892 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:24:15,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:16,259 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:17,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:18,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:24:18,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:19,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:20,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:21,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:22,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:23,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:24,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:24:24,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:25,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:26,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:27,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:28,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:29,190 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:24:29,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:29,414 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:24:29,892 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:24:29,892 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:24:30,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:31,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:32,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:33,064 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:24:33,065 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:24:33,065 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:24:33,068 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:24:33,261 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:33,624 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:24:34,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:35,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:24:35,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:35,627 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:24:36,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:37,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:38,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:39,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:40,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:24:40,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:41,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:42,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:43,262 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:44,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:44,892 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:24:44,893 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:24:45,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:46,064 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:24:46,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:47,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:48,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:49,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:50,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:51,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:24:51,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:52,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:53,263 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:54,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:55,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:56,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:57,191 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:24:57,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:58,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:59,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:24:59,416 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:24:59,892 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:24:59,893 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:25:00,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:01,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:02,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:25:02,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:03,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:04,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:05,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:06,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:07,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:25:07,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:08,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:09,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:10,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:11,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:12,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:13,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:25:13,265 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:14,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:14,892 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:25:14,893 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:25:15,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:16,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:17,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:18,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:25:18,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:19,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:20,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:21,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:22,266 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:23,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:24,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:25:24,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:25,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:26,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:27,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:28,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:29,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:25:29,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:29,418 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:25:29,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:25:29,893 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:25:30,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:31,267 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:32,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:33,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:34,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:35,192 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:25:35,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:36,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:37,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:38,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:39,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:40,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:25:40,268 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:41,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:42,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:43,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:44,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:44,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:25:44,893 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:25:45,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:25:45,186 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:25:45,186 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:25:45,190 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:25:45,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:25:45,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:45,750 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:25:46,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:47,269 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:47,753 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:25:48,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:49,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:50,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:51,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:25:51,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:52,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:53,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:54,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:55,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:56,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:25:56,270 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:57,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:58,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:59,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:25:59,418 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:25:59,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:25:59,893 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:26:00,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:01,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:26:01,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:02,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:03,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:04,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:05,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:06,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:26:06,271 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:07,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:08,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:09,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:10,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:11,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:12,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:26:12,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:13,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:14,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:14,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:26:14,893 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:26:15,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:16,272 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:17,193 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:26:17,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:18,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:19,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:20,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:21,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:22,194 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:26:22,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:23,273 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:24,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:25,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:26,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:27,194 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:26:27,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:28,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:29,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:29,420 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:26:29,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:26:29,894 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:26:30,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:31,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:32,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:33,194 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:26:33,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:34,274 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:35,275 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:36,275 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:37,275 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:38,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:26:38,275 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:39,275 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:40,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:41,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:42,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:43,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:44,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:26:44,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:44,893 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:26:44,894 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:26:45,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:46,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:47,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:48,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:49,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:50,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:26:50,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:51,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:52,276 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:53,277 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:54,277 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:55,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:26:55,277 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:56,278 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:57,277 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:58,277 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:58,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:26:58,713 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:26:58,713 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:26:58,717 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:26:58,880 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:26:59,423 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:26:59,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:26:59,881 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:26:59,894 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:26:59,894 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:27:00,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:01,068 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:27:01,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:02,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:03,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:04,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:05,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:06,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:27:06,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:07,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:08,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:09,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:10,700 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:11,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:27:11,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:12,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:13,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:14,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:14,894 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:27:14,894 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:27:15,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:16,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:17,195 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:27:17,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:18,701 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:19,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:20,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:21,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:22,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:27:22,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:23,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:24,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:25,702 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:26,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:27,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:27:27,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:28,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:29,423 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:27:29,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:29,894 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:27:29,894 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:27:30,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:31,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:32,703 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:33,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:27:33,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:34,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:35,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:36,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:37,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:38,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:27:38,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:39,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:40,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:41,704 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:42,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:43,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:27:43,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:44,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:44,894 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:27:44,894 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:27:45,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:46,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:47,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:48,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:49,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:27:49,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:50,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:51,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:52,705 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:53,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:54,196 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:27:54,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:55,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:56,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:57,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:58,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:59,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:27:59,424 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:27:59,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:27:59,894 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:27:59,894 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:28:00,706 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:01,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:02,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:03,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:04,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:28:04,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:05,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:06,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:07,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:08,707 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:09,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:10,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:28:10,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:11,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:11,792 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:28:11,793 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:28:11,793 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:28:11,796 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:28:12,022 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:28:12,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:13,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:14,026 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:28:14,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:14,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:28:14,896 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:28:15,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:16,068 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:28:16,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:17,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:18,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:19,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:20,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:21,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:28:21,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:22,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:23,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:24,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:25,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:26,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:28:26,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:27,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:28,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:29,426 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:28:29,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:29,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:28:29,895 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:28:30,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:31,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:32,197 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:28:32,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:33,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:34,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:35,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:36,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:37,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:28:37,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:38,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:39,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:40,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:41,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:42,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:43,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:28:43,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:44,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:44,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:28:44,895 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:28:45,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:46,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:47,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:48,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:28:48,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:49,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:50,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:51,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:52,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:53,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:54,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:28:54,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:55,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:56,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:57,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:58,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:59,198 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:28:59,428 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:28:59,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:28:59,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:28:59,895 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:29:00,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:01,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:02,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:03,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:04,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:29:04,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:05,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:06,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:07,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:08,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:09,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:29:09,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:10,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:11,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:12,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:13,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:14,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:14,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:29:14,896 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:29:15,066 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:29:15,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:16,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:17,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:18,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:19,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:20,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:29:20,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:21,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:22,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:23,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:23,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:29:23,909 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:29:23,910 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:29:23,913 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:29:24,160 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:29:24,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:25,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:25,913 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:29:26,163 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:29:26,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:27,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:28,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:29,429 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:29:29,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:29,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:29:29,895 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:29:30,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:31,066 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:29:31,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:32,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:33,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:34,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:35,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:36,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:29:36,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:37,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:38,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:39,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:40,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:41,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:42,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:29:42,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:43,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:44,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:44,895 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:29:44,896 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:29:45,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:46,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:47,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:29:47,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:48,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:49,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:50,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:51,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:52,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:53,199 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:29:53,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:54,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:55,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:56,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:57,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:58,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:59,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:29:59,431 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:29:59,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:29:59,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:29:59,896 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:30:00,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:01,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:02,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:03,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:04,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:05,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:30:05,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:06,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:07,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:08,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:09,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:10,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:30:10,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:11,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:12,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:13,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:14,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:14,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:30:14,896 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:30:15,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:16,066 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:30:16,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:17,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:18,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:19,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:20,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:21,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:30:21,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:22,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:23,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:24,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:25,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:26,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:30:26,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:27,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:28,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:29,433 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:30:29,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:29,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:30:29,896 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:30:30,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:31,200 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:30:31,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:32,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:33,921 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:34,921 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:35,921 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:36,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:30:36,260 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:30:36,262 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:30:36,262 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:30:36,267 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:30:36,308 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:30:36,921 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:37,921 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:38,311 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:30:38,921 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:39,921 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:40,922 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:41,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:30:41,922 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:42,922 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:43,922 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:44,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:30:44,897 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:30:44,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:45,922 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:46,922 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:47,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:30:47,922 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:48,923 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:49,923 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:50,923 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:51,923 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:52,923 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:53,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:30:53,923 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:54,923 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:55,923 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:56,923 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:57,923 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:58,201 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:30:58,923 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:30:59,435 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:30:59,896 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:30:59,897 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:30:59,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:00,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:01,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:02,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:03,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:31:03,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:04,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:05,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:06,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:07,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:08,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:09,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:31:09,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:10,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:11,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:12,925 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:13,925 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:14,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:31:14,897 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:31:14,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:15,067 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:31:15,925 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:16,925 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:17,925 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:18,925 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:19,925 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:20,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:31:20,926 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:21,926 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:22,926 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:23,927 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:24,926 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:25,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:31:25,926 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:26,927 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:27,926 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:28,926 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:29,436 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:31:29,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:31:29,897 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:31:29,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:30,927 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:31,068 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:31:31,927 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:32,927 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:33,927 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:34,927 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:35,927 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:36,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:31:36,928 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:37,928 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:38,928 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:39,928 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:40,928 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:41,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:31:41,928 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:42,928 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:43,928 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:44,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:31:44,897 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:31:44,940 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:45,928 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:46,202 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:31:47,137 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:48,135 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:49,135 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:49,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:31:49,481 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:31:49,481 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:31:49,497 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:31:50,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:50,459 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:31:51,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:51,203 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:31:51,461 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:31:52,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:53,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:54,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:55,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:56,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:57,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:57,203 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:31:58,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:59,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:31:59,439 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:31:59,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:31:59,898 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:32:00,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:01,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:02,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:03,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:03,203 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:32:04,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:05,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:06,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:07,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:08,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:08,203 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:32:09,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:10,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:11,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:12,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:13,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:14,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:14,203 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:32:14,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:32:14,897 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:32:15,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:16,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:17,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:18,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:19,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:20,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:20,204 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:32:21,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:22,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:23,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:24,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:25,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:25,204 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:32:26,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:27,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:28,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:29,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:29,439 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:32:29,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:32:29,897 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:32:30,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:31,068 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:32:31,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:32,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:33,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:34,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:35,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:36,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:36,204 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:32:37,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:38,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:39,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:40,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:41,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:42,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:42,204 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:32:43,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:44,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:44,897 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:32:44,898 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:32:45,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:46,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:47,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:47,204 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:32:48,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:49,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:50,169 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:51,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:52,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:52,204 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:32:53,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:54,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:55,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:56,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:57,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:58,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:58,204 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:32:59,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:32:59,441 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:32:59,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:32:59,899 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:33:00,170 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:01,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:01,876 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:33:01,877 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:33:01,877 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:33:01,881 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:33:02,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:02,618 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:33:03,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:03,622 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:33:03,886 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:33:04,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:05,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:06,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:07,171 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:08,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:09,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:09,205 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:33:10,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:11,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:12,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:13,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:14,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:14,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:33:14,898 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:33:15,068 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:33:15,172 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:16,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:17,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:18,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:19,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:20,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:20,205 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:33:21,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:22,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:23,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:24,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:25,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:25,205 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:33:26,173 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:27,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:28,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:29,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:29,450 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:33:29,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:33:29,898 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:33:30,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:31,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:33:31,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:32,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:33,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:34,174 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:35,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:36,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:36,205 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:33:37,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:38,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:39,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:40,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:41,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:42,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:42,205 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:33:43,175 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:44,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:44,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:33:44,898 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:33:45,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:46,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:47,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:48,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:48,205 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:33:49,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:50,176 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:51,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:52,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:53,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:53,205 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:33:54,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:55,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:56,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:57,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:58,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:58,205 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:33:59,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:33:59,451 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:33:59,898 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:33:59,899 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:34:00,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:01,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:02,177 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:03,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:04,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:04,205 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:34:05,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:06,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:07,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:08,178 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:09,179 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:09,206 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:34:10,179 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:11,179 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:12,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:13,179 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:14,048 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:34:14,049 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:34:14,049 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:34:14,053 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:34:14,179 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:14,754 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:34:14,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:34:14,900 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:34:15,070 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:34:15,179 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:15,756 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:34:16,179 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:17,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:18,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:19,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:20,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:20,206 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:34:21,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:22,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:23,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:24,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:25,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:25,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:34:26,180 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:27,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:28,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:29,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:29,453 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:34:29,899 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:34:29,900 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:34:30,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:31,070 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:34:31,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:32,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:33,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:34,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:35,181 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:36,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:36,206 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:34:37,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:38,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:39,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:40,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:41,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:41,206 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:34:42,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:43,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:44,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:44,899 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:34:44,900 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:34:45,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:46,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:47,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:47,206 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:34:48,182 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:49,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:50,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:51,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:52,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:53,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:53,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:34:54,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:55,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:56,183 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:57,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:58,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:58,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:34:59,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:34:59,458 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:34:59,899 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:34:59,900 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:35:00,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:01,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:02,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:03,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:04,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:04,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:35:05,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:06,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:07,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:08,184 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:09,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:09,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:35:10,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:11,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:12,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:13,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:14,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:14,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:35:14,899 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:35:14,900 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:35:15,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:16,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:17,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:18,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:19,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:20,185 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:20,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:35:21,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:22,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:23,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:24,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:25,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:25,207 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:35:26,186 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:26,548 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:35:26,549 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:35:26,549 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:35:26,553 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:35:26,907 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:35:27,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:27,909 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:35:28,329 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:29,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:29,459 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:35:29,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:35:29,900 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:35:30,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:31,006 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:35:31,329 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:32,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:33,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:34,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:35,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:36,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:35:36,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:37,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:38,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:39,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:40,330 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:41,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:35:41,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:42,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:43,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:44,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:44,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:35:44,900 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:35:45,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:46,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:47,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:35:47,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:48,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:49,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:50,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:51,331 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:52,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:35:52,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:53,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:54,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:55,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:56,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:57,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:35:57,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:58,332 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:59,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:35:59,461 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:35:59,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:35:59,900 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:36:00,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:01,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:02,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:36:02,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:03,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:04,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:05,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:06,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:07,334 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:08,208 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:36:08,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:09,334 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:10,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:11,333 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:12,334 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:13,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:36:13,334 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:14,334 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:14,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:36:14,900 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:36:15,334 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:16,334 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:17,334 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:18,334 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:19,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:36:19,335 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:20,334 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:21,334 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:22,335 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:23,335 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:24,335 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:25,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:36:25,335 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:26,335 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:27,335 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:28,335 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:29,335 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:29,462 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:36:29,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:36:29,900 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:36:30,335 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:31,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:36:31,336 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:32,336 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:33,336 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:34,336 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:35,336 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:36,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:36:36,336 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:37,337 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:38,337 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:39,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:36:39,169 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:36:39,169 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:36:39,173 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:36:39,337 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:40,049 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:36:40,049 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:36:40,338 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:41,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:36:41,337 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:42,337 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:43,337 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:44,337 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:44,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:36:44,901 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:36:45,338 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:46,338 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:47,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:36:47,338 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:48,338 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:49,338 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:50,338 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:51,338 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:52,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:36:52,338 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:53,339 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:54,339 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:55,339 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:56,339 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:57,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:36:57,339 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:58,339 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:59,339 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:36:59,464 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:36:59,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:36:59,901 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:37:00,339 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:01,339 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:02,339 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:03,209 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:37:03,339 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:04,340 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:05,340 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:06,340 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:07,340 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:08,210 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:37:08,340 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:09,340 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:10,340 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:11,340 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:12,340 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:13,210 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:37:13,341 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:14,341 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:14,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:37:14,901 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:37:15,341 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:16,341 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:17,341 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:18,210 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:37:18,341 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:19,341 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:20,341 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:21,341 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:22,341 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:23,341 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:24,210 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:37:24,341 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:25,343 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:26,342 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:27,343 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:28,343 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:29,343 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:29,466 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:37:29,468 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:37:29,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:37:29,901 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:37:30,343 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:31,343 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:32,343 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:33,343 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:34,344 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:35,210 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:37:35,344 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:36,344 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:37,344 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:38,344 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:39,344 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:40,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:37:40,344 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:41,344 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:42,344 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:43,345 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:44,345 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:44,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:37:44,901 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:37:45,345 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:46,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:37:46,345 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:47,345 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:48,345 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:49,346 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:50,345 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:51,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:37:51,203 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:37:51,205 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:37:51,205 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:37:51,210 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:37:51,346 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:52,179 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:37:52,179 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:37:52,347 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:53,346 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:54,346 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:55,349 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:56,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:37:56,348 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:57,348 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:58,348 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:59,349 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:37:59,468 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:37:59,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:37:59,901 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:38:00,349 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:01,348 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:02,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:38:02,349 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:03,349 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:04,349 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:05,349 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:06,349 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:07,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:38:07,350 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:08,350 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:09,350 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:10,350 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:11,350 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:12,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:38:12,350 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:13,350 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:14,351 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:14,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:38:14,901 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:38:15,351 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:16,351 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:17,351 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:18,211 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:38:18,351 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:19,351 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:20,351 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:21,351 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:22,351 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:23,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:38:23,352 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:24,352 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:25,352 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:26,352 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:27,352 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:28,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:38:28,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:29,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:29,471 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:38:29,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:38:29,902 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:38:30,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:31,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:32,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:33,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:34,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:38:34,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:35,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:36,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:37,353 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:38,354 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:39,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:38:39,354 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:40,354 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:41,354 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:42,354 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:43,354 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:44,212 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:38:44,354 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:44,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:38:44,901 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:38:45,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:46,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:47,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:48,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:49,214 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:38:49,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:50,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:51,355 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:52,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:53,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:54,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:55,213 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:38:55,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:56,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:57,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:58,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:59,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:38:59,471 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:38:59,901 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:38:59,902 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:39:00,356 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:01,007 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:39:01,357 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:02,357 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:03,358 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:03,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:39:03,709 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:39:03,709 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:39:03,712 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:39:04,332 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:39:04,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:05,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:06,213 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:39:06,335 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:39:06,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:07,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:08,414 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:09,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:10,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:11,214 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:39:11,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:12,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:13,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:14,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:14,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:39:14,902 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:39:15,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:16,415 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:17,213 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:39:17,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:18,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:19,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:20,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:21,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:22,213 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:39:22,416 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:23,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:24,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:25,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:26,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:27,213 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:39:27,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:28,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:29,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:29,473 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:39:29,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:39:29,902 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:39:30,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:31,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:32,417 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:33,213 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:39:33,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:34,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:35,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:36,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:37,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:38,213 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:39:38,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:39,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:40,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:41,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:42,418 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:43,214 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:39:43,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:44,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:44,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:39:44,902 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:39:45,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:46,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:47,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:48,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:49,214 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:39:49,419 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:50,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:51,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:52,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:53,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:54,214 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:39:54,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:55,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:56,420 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:57,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:58,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:59,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:39:59,475 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:39:59,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:39:59,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:39:59,902 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:40:00,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:01,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:02,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:03,421 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:04,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:05,215 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:40:05,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:06,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:07,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:08,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:09,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:10,422 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:11,214 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:40:11,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:12,423 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:13,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:14,437 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:14,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:40:14,902 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:40:15,438 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:15,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:40:15,842 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:40:15,842 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:40:15,846 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:40:16,214 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:40:16,469 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:40:16,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:17,476 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:18,473 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:40:18,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:19,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:20,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:21,215 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:40:21,477 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:22,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:23,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:24,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:25,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:26,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:27,215 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:40:27,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:28,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:29,476 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:40:29,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:29,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:40:29,902 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:40:30,478 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:31,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:32,215 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:40:32,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:33,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:34,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:35,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:36,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:37,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:38,215 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:40:38,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:39,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:40,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:41,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:42,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:43,215 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:40:43,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:44,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:44,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:40:44,903 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:40:45,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:46,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:47,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:48,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:49,215 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:40:49,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:50,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:51,480 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:52,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:53,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:54,216 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:40:54,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:55,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:56,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:57,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:58,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:59,477 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:40:59,479 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:40:59,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:40:59,902 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:40:59,903 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:41:00,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:01,481 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:02,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:03,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:04,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:05,216 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:41:05,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:06,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:07,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:08,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:09,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:10,216 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:41:10,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:11,482 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:12,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:13,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:14,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:14,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:41:14,903 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:41:15,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:16,009 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:41:16,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:17,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:18,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:19,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:20,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:21,216 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:41:21,483 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:22,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:23,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:24,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:25,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:26,216 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:41:26,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:27,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:27,696 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:41:27,697 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:41:27,698 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:41:27,701 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:41:28,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:28,627 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:41:29,480 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:41:29,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:29,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:41:29,903 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:41:30,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:30,630 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:41:31,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:41:31,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:32,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:33,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:34,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:35,485 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:36,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:37,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:41:37,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:38,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:39,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:40,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:41,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:42,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:41:42,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:43,486 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:44,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:44,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:41:44,903 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:41:45,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:46,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:47,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:48,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:41:48,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:49,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:50,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:51,487 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:52,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:53,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:54,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:41:54,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:55,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:56,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:57,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:58,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:59,481 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:41:59,484 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:41:59,488 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:41:59,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:41:59,904 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:42:00,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:01,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:02,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:03,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:04,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:05,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:42:05,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:06,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:07,489 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:08,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:09,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:10,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:11,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:42:11,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:12,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:13,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:14,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:14,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:42:14,904 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:42:15,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:16,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:17,217 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:42:17,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:18,490 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:19,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:20,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:21,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:22,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:42:22,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:23,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:24,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:25,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:26,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:27,491 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:28,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:42:28,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:29,483 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:42:29,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:29,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:42:29,904 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:42:30,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:31,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:32,492 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:33,218 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:42:33,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:34,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:35,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:36,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:37,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:38,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:42:38,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:39,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:40,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:42:40,169 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:42:40,169 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:42:40,172 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:42:40,493 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:40,765 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:42:41,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:42,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:42,768 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:42:43,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:44,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:42:44,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:44,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:42:44,904 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:42:45,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:46,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:47,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:48,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:49,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:50,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:42:50,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:51,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:52,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:53,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:54,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:55,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:56,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:42:56,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:57,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:58,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:59,485 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:42:59,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:42:59,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:42:59,904 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:43:00,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:01,219 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:43:01,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:02,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:03,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:04,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:05,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:06,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:43:06,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:07,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:08,496 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:09,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:10,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:11,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:43:11,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:12,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:13,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:14,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:14,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:43:14,904 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:43:15,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:16,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:17,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:43:17,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:18,497 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:19,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:20,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:21,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:22,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:43:22,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:23,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:24,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:25,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:26,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:27,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:28,220 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:43:28,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:29,486 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:43:29,498 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:29,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:43:29,904 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:43:30,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:31,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:32,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:33,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:34,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:43:34,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:35,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:36,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:37,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:38,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:39,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:40,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:43:40,499 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:41,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:42,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:43,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:44,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:44,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:43:44,905 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:43:45,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:46,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:43:46,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:47,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:48,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:49,500 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:50,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:51,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:43:51,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:52,264 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:43:52,265 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:43:52,265 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:43:52,269 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:43:52,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:52,899 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:43:53,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:53,901 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:43:54,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:55,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:56,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:43:56,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:57,501 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:58,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:59,489 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:43:59,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:43:59,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:43:59,905 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:44:00,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:01,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:44:01,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:02,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:03,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:04,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:05,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:06,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:07,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:44:07,502 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:08,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:09,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:10,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:11,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:12,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:13,221 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:44:13,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:14,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:14,905 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:44:14,905 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:44:15,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:16,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:17,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:18,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:44:18,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:19,503 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:20,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:21,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:22,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:23,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:44:23,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:24,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:25,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:26,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:27,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:28,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:44:28,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:29,490 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:44:29,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:29,904 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:44:29,905 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:44:30,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:31,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:32,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:33,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:34,222 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:44:34,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:35,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:36,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:37,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:38,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:39,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:44:39,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:40,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:41,505 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:42,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:43,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:44,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:44,905 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:44:44,905 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:44:45,010 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:44:45,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:46,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:47,506 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:48,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:49,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:50,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:44:50,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:51,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:52,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:53,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:54,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:55,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:56,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:44:56,507 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:57,508 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:58,508 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:59,492 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:44:59,508 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:44:59,905 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:44:59,905 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:45:00,508 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:01,508 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:02,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:45:02,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:03,509 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:04,440 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:45:04,441 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:45:04,442 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:45:04,445 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:45:04,578 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:05,036 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:45:05,579 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:06,038 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:45:06,579 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:07,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:45:07,579 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:08,579 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:09,579 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:10,579 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:11,579 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:12,579 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:13,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:45:13,579 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:14,579 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:14,905 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:45:14,905 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:45:15,579 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:16,580 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:17,580 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:18,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:45:18,580 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:19,580 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:20,580 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:21,580 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:22,580 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:23,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:45:23,580 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:24,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:25,580 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:26,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:27,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:28,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:45:28,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:29,493 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:45:29,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:29,905 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:45:29,905 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:45:30,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:31,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:32,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:33,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:34,223 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:45:34,581 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:35,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:36,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:37,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:38,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:39,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:45:39,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:40,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:41,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:42,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:43,582 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:44,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:44,905 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:45:44,906 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:45:45,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:45:45,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:46,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:47,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:48,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:49,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:50,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:45:50,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:51,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:52,583 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:53,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:54,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:55,224 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:45:55,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:56,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:57,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:58,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:59,495 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:45:59,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:45:59,905 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:45:59,906 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:46:00,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:01,011 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:46:01,584 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:02,585 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:03,585 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:04,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:05,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:06,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:46:06,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:07,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:08,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:09,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:10,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:11,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:12,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:46:12,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:13,588 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:14,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:14,906 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:46:14,906 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:46:15,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:16,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:16,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:46:16,829 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:46:16,829 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:46:16,831 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:46:17,168 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:46:17,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:17,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:46:18,170 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:46:18,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:19,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:20,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:21,589 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:22,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:23,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:46:23,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:24,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:25,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:26,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:27,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:28,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:46:28,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:29,497 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:46:29,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:29,906 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:46:29,906 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:46:30,590 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:31,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:32,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:33,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:34,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:46:34,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:35,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:36,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:37,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:38,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:39,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:40,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:46:40,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:41,591 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:42,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:43,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:44,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:44,906 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:46:44,906 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:46:45,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:46,012 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:46:46,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:47,592 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:48,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:49,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:50,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:51,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:46:51,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:52,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:53,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:54,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:55,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:56,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:46:56,593 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:57,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:58,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:59,498 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:46:59,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:46:59,906 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:46:59,907 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:47:00,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:01,225 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:47:01,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:02,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:03,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:04,594 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:05,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:06,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:47:06,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:07,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:08,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:09,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:10,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:11,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:47:11,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:12,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:13,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:14,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:14,906 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:47:14,906 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:47:15,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:16,595 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:17,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:47:17,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:18,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:19,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:20,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:21,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:22,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:47:22,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:23,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:24,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:25,596 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:26,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:27,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:47:27,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:28,364 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:47:28,365 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:47:28,365 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:47:28,367 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:47:28,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:29,319 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:47:29,500 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:47:29,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:29,906 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:47:29,907 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:47:30,321 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:47:30,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:31,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:32,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:33,226 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:47:33,597 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:34,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:35,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:36,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:37,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:38,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:47:38,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:39,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:40,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:41,598 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:42,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:43,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:44,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:47:44,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:44,907 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:47:44,907 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:47:45,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:46,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:47,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:48,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:49,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:50,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:47:50,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:51,599 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:52,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:53,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:54,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:55,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:56,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:47:56,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:57,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:58,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:59,502 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:47:59,600 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:47:59,907 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:47:59,907 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:48:00,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:01,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:48:01,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:02,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:03,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:04,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:05,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:06,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:48:06,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:07,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:08,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:09,601 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:10,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:11,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:12,227 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:48:12,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:13,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:14,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:14,907 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:48:14,907 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:48:15,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:16,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:17,602 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:18,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:48:18,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:19,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:20,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:21,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:22,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:23,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:24,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:48:24,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:25,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:26,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:27,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:28,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:29,503 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:48:29,504 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:48:29,603 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:29,907 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:48:29,907 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:48:30,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:31,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:32,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:33,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:34,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:35,228 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:48:35,604 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:36,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:37,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:38,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:39,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:39,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:48:39,925 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:48:39,925 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:48:39,927 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:48:40,447 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:48:40,448 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:48:40,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:40,928 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:48:41,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:42,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:43,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:44,605 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:44,907 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:48:44,908 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:48:45,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:46,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:48:46,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:47,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:48,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:49,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:50,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:51,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:48:51,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:52,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:53,606 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:54,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:55,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:56,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:48:56,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:57,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:58,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:59,504 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:48:59,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:48:59,907 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:48:59,907 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:49:00,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:01,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:49:01,607 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:02,608 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:03,608 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:04,608 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:05,608 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:06,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:49:06,608 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:07,608 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:08,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:09,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:10,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:11,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:49:11,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:12,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:13,609 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:14,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:14,907 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:49:14,908 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:49:15,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:16,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:17,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:49:17,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:18,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:19,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:20,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:21,610 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:22,229 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:49:22,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:23,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:24,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:25,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:26,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:27,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:49:27,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:28,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:29,506 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:49:29,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:29,907 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:49:29,908 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:49:30,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:31,611 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:32,612 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:33,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:49:33,612 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:34,612 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:35,612 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:36,612 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:37,612 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:38,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:49:38,612 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:39,612 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:40,613 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:41,613 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:42,613 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:43,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:49:43,613 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:44,613 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:44,907 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:49:44,908 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:49:45,613 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:46,613 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:47,613 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:48,230 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:49:48,613 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:49,693 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:50,687 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:51,687 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:52,068 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:49:52,069 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:49:52,070 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:49:52,071 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:49:52,600 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:49:52,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:53,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:54,072 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:49:54,603 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:49:54,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:55,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:56,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:57,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:58,708 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:59,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:49:59,508 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:49:59,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:49:59,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:49:59,908 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:50:00,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:01,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:02,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:03,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:04,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:50:04,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:05,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:06,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:07,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:08,709 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:09,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:10,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:50:10,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:11,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:12,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:13,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:14,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:14,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:50:14,908 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:50:15,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:16,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:50:16,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:17,710 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:18,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:19,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:20,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:21,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:50:21,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:22,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:23,711 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:24,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:25,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:26,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:50:26,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:27,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:28,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:29,509 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:50:29,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:29,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:50:29,908 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:50:30,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:31,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:50:31,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:32,712 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:33,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:34,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:35,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:36,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:37,231 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:50:37,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:38,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:39,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:40,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:41,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:42,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:50:42,713 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:43,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:44,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:44,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:50:44,908 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:50:45,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:46,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:47,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:48,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:50:48,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:49,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:50,714 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:51,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:52,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:53,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:54,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:50:54,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:55,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:56,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:57,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:58,715 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:59,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:50:59,510 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:50:59,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:50:59,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:50:59,908 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:51:00,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:01,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:02,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:03,716 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:03,900 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:51:03,901 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:51:03,901 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:51:03,902 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:51:04,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:04,734 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:51:04,734 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:51:04,903 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:51:05,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:06,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:07,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:08,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:09,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:10,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:51:10,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:11,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:12,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:13,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:14,717 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:14,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:51:14,908 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:51:15,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:16,013 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:51:16,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:17,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:18,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:19,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:20,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:21,232 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:51:21,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:22,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:23,718 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:24,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:25,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:26,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:51:26,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:27,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:28,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:29,512 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:51:29,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:29,909 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:51:29,909 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:51:30,719 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:31,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:51:31,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:32,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:33,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:34,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:35,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:36,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:51:36,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:37,720 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:38,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:39,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:40,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:41,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:42,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:51:42,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:43,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:44,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:44,908 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:51:44,909 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:51:45,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:46,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:47,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:48,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:51:48,721 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:49,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:50,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:51,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:52,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:53,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:51:53,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:54,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:55,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:56,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:57,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:58,233 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:51:58,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:59,513 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:51:59,722 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:51:59,909 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:51:59,909 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:52:00,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:01,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:02,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:03,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:52:03,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:04,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:05,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:06,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:07,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:08,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:52:08,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:09,723 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:10,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:11,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:12,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:13,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:14,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:52:14,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:14,909 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:52:14,909 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:52:15,430 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:52:15,431 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:52:15,431 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:52:15,432 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:52:15,724 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:15,862 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:52:16,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:16,864 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:52:17,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:18,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:19,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:52:19,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:20,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:21,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:22,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:23,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:24,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:25,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:52:25,725 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:26,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:27,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:28,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:29,515 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:52:29,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:29,909 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:52:29,909 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:52:30,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:31,014 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:52:31,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:32,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:33,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:34,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:35,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:36,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:52:36,726 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:37,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:38,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:39,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:40,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:41,234 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:52:41,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:42,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:43,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:44,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:44,909 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:52:44,909 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:52:45,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:46,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:52:46,727 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:47,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:48,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:49,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:50,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:51,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:52:51,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:52,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:53,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:54,728 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:55,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:56,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:57,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:52:57,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:58,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:59,516 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:52:59,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:52:59,909 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:52:59,909 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:53:00,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:01,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:02,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:03,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:53:03,729 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:04,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:05,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:06,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:07,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:08,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:53:08,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:09,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:10,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:11,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:12,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:13,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:14,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:53:14,730 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:14,909 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:53:14,909 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:53:15,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:16,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:17,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:18,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:19,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:20,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:53:20,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:21,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:22,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:23,731 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:24,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:25,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:53:25,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:26,827 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:27,144 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:53:27,145 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:53:27,145 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:53:27,147 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:53:27,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:27,981 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:53:28,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:28,983 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:53:29,517 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:53:29,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:29,910 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:53:29,910 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:53:30,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:31,015 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:53:31,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:32,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:33,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:34,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:35,828 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:36,235 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:53:36,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:37,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:38,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:39,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:40,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:41,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:53:41,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:42,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:43,829 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:44,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:44,910 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:53:44,910 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:53:45,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:46,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:53:46,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:47,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:48,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:49,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:50,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:51,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:52,236 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:53:52,830 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:53,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:54,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:55,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:56,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:57,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:58,245 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:53:58,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:59,520 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:53:59,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:53:59,910 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:53:59,910 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:54:00,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:01,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:02,831 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:03,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:04,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:54:04,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:05,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:06,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:07,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:08,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:09,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:10,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:54:10,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:11,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:12,832 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:13,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:14,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:14,910 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:54:14,910 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:54:15,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:16,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:54:16,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:17,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:18,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:19,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:20,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:21,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:54:21,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:22,833 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:23,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:24,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:25,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:26,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:27,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:54:27,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:28,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:29,522 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:54:29,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:29,910 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:54:29,910 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:54:30,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:31,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:32,237 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:54:32,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:33,834 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:34,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:35,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:36,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:37,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:54:37,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:38,732 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:54:38,733 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:54:38,733 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:54:38,735 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:54:38,835 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:39,101 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:54:39,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:40,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:41,104 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:54:41,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:42,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:43,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:54:43,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:44,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:44,910 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:54:44,910 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:54:45,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:46,836 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:47,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:48,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:49,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:54:49,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:50,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:51,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:52,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:53,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:54,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:55,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:54:55,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:56,837 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:57,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:58,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:59,523 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:54:59,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:54:59,910 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:54:59,910 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:55:00,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:01,016 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:55:01,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:02,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:03,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:04,838 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:05,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:06,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:55:06,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:07,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:08,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:09,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:10,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:11,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:55:11,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:12,839 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:13,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:14,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:14,910 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:55:14,911 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:55:15,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:16,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:55:16,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:17,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:18,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:19,840 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:20,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:21,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:55:21,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:22,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:23,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:24,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:25,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:26,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:55:26,841 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:27,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:28,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:29,524 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:55:29,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:29,911 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:55:29,911 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:55:30,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:31,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:55:31,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:32,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:33,842 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:34,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:35,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:36,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:37,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:55:37,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:38,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:39,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:40,843 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:41,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:42,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:55:42,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:43,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:44,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:44,911 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:55:44,911 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:55:45,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:46,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:47,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:55:47,844 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:48,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:49,845 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:50,448 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:55:50,449 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:55:50,449 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:55:50,451 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:55:51,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:51,235 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:55:52,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:52,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:55:53,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:53,239 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:55:54,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:55,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:56,057 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:57,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:58,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:58,449 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:55:59,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:55:59,526 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:55:59,911 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:55:59,911 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:56:00,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:01,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:02,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:03,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:04,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:04,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:56:05,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:06,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:07,058 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:08,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:09,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:09,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:56:10,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:11,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:12,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:13,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:14,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:14,911 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:56:14,911 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:56:15,027 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:56:15,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:16,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:17,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:18,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:19,059 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:20,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:20,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:56:21,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:22,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:23,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:24,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:25,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:25,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:56:26,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:27,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:28,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:29,060 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:29,527 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:56:29,911 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:56:29,911 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:56:30,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:31,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:56:31,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:32,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:33,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:34,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:35,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:36,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:36,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:56:37,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:38,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:39,061 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:40,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:41,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:41,450 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:56:42,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:43,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:44,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:44,911 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:56:44,911 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:56:45,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:46,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:46,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:56:47,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:48,062 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:49,063 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:50,063 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:51,063 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:52,063 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:52,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:56:53,063 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:54,063 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:55,063 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:56,063 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:57,063 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:57,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:56:58,063 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:59,064 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:56:59,528 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:56:59,911 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:56:59,912 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:57:00,064 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:01,064 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:02,076 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:57:02,077 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:02,077 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:57:02,077 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:57:02,079 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:57:02,372 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:57:03,065 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:03,079 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:57:03,373 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:57:04,064 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:05,064 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:05,377 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:57:06,064 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:07,064 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:08,065 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:08,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:57:09,065 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:10,065 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:11,065 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:12,065 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:13,065 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:13,451 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:57:14,065 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:14,912 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:57:14,912 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:57:15,066 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:16,066 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:17,066 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:18,066 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:19,066 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:19,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:57:20,066 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:21,066 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:22,066 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:23,067 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:24,067 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:24,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:57:25,067 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:26,067 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:27,067 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:28,067 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:29,068 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:29,531 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:57:29,532 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:57:29,912 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:57:29,912 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:57:30,068 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:31,068 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:32,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:33,068 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:34,068 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:35,068 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:35,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:57:36,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:37,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:38,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:39,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:40,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:40,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:57:41,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:42,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:43,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:44,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:44,912 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:57:44,912 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:57:45,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:46,028 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:57:46,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:47,069 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:48,070 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:49,070 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:50,070 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:51,070 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:51,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:57:52,070 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:53,070 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:54,070 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:55,070 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:56,071 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:56,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:57:57,071 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:58,071 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:59,071 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:57:59,531 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:57:59,912 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:57:59,912 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:58:00,071 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:01,071 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:01,452 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:58:02,071 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:03,071 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:04,071 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:05,071 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:06,071 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:06,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:58:07,072 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:08,072 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:09,072 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:10,072 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:11,072 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:11,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:58:12,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:13,073 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:14,108 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:58:14,109 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:58:14,109 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:58:14,111 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:58:14,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:14,499 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:58:14,499 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:58:14,913 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:58:14,914 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:58:15,159 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:16,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:16,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:58:16,502 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:58:17,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:18,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:19,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:20,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:21,160 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:22,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:22,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:58:23,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:24,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:25,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:26,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:27,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:27,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:58:28,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:29,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:29,533 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:58:29,912 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:58:29,913 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:58:30,161 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:31,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:32,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:33,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:33,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:58:34,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:35,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:36,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:37,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:38,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:39,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:39,453 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:58:40,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:41,162 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:42,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:43,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:44,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:44,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:58:44,912 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:58:44,912 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:58:45,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:46,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:47,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:48,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:49,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:50,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:50,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:58:51,163 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:52,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:53,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:54,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:55,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:55,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:58:56,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:57,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:58,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:59,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:58:59,534 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:58:59,913 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:58:59,913 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:59:00,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:01,039 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:59:01,164 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:02,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:03,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:04,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:05,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:06,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:06,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:59:07,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:08,165 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:09,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:10,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:11,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:12,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:12,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:59:13,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:14,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:14,913 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: stop_status 2024-12-11 00:59:14,913 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: stop_status 2024-12-11 00:59:15,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:16,166 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:17,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:17,454 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:59:18,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:19,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:20,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:21,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:22,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:23,167 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:23,455 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:59:24,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:25,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:26,168 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:26,304 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: partial_history 2024-12-11 00:59:26,305 DEBUG SenderThread:843156 [sender.py:send():379] send: history 2024-12-11 00:59:26,306 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: summary_record 2024-12-11 00:59:26,307 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:59:26,307 DEBUG SenderThread:843156 [sender.py:send():379] send: exit 2024-12-11 00:59:26,307 INFO SenderThread:843156 [sender.py:send_exit():586] handling exit code: 1 2024-12-11 00:59:26,308 INFO SenderThread:843156 [sender.py:send_exit():588] handling runtime: 10917 2024-12-11 00:59:26,309 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:59:26,309 INFO SenderThread:843156 [sender.py:send_exit():594] send defer 2024-12-11 00:59:26,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:26,309 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 0 2024-12-11 00:59:26,310 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:26,310 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 0 2024-12-11 00:59:26,310 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 1 2024-12-11 00:59:26,310 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:26,310 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 1 2024-12-11 00:59:26,310 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:26,310 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 1 2024-12-11 00:59:26,310 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 2 2024-12-11 00:59:26,310 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:26,310 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 2 2024-12-11 00:59:26,310 INFO HandlerThread:843156 [system_monitor.py:finish():203] Stopping system monitor 2024-12-11 00:59:26,310 DEBUG SystemMonitor:843156 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-12-11 00:59:26,311 DEBUG SystemMonitor:843156 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-12-11 00:59:26,312 INFO HandlerThread:843156 [interfaces.py:finish():200] Joined cpu monitor 2024-12-11 00:59:26,313 INFO HandlerThread:843156 [interfaces.py:finish():200] Joined disk monitor 2024-12-11 00:59:26,653 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:59:27,919 INFO HandlerThread:843156 [interfaces.py:finish():200] Joined gpu monitor 2024-12-11 00:59:27,919 INFO HandlerThread:843156 [interfaces.py:finish():200] Joined memory monitor 2024-12-11 00:59:27,920 INFO HandlerThread:843156 [interfaces.py:finish():200] Joined network monitor 2024-12-11 00:59:27,920 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-11 00:59:27,920 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:27,920 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 2 2024-12-11 00:59:27,920 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 3 2024-12-11 00:59:27,920 DEBUG SenderThread:843156 [sender.py:send():379] send: stats 2024-12-11 00:59:27,920 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:27,921 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: poll_exit 2024-12-11 00:59:27,921 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 3 2024-12-11 00:59:27,922 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:27,922 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 3 2024-12-11 00:59:27,922 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 4 2024-12-11 00:59:27,922 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:27,922 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 4 2024-12-11 00:59:27,922 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:27,922 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 4 2024-12-11 00:59:27,922 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 5 2024-12-11 00:59:27,922 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:27,922 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 5 2024-12-11 00:59:27,922 DEBUG SenderThread:843156 [sender.py:send():379] send: summary 2024-12-11 00:59:27,924 INFO SenderThread:843156 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-11 00:59:27,924 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:27,924 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 5 2024-12-11 00:59:27,924 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 6 2024-12-11 00:59:27,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:27,924 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 6 2024-12-11 00:59:27,924 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:27,924 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 6 2024-12-11 00:59:27,924 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 7 2024-12-11 00:59:27,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: status_report 2024-12-11 00:59:27,924 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:27,924 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 7 2024-12-11 00:59:27,925 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:27,925 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 7 2024-12-11 00:59:28,307 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-11 00:59:28,656 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:59:28,656 INFO Thread-12 :843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:59:30,529 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 8 2024-12-11 00:59:30,529 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: poll_exit 2024-12-11 00:59:30,529 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:30,529 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 8 2024-12-11 00:59:30,529 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:30,529 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 8 2024-12-11 00:59:30,529 INFO SenderThread:843156 [job_builder.py:build():440] Attempting to build job artifact 2024-12-11 00:59:30,530 WARNING SenderThread:843156 [job_builder.py:_log_if_verbose():274] Ensure read and write access to run files dir: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files, control this via the WANDB_DIR env var. See https://docs.wandb.ai/guides/track/environment-variables 2024-12-11 00:59:30,530 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 9 2024-12-11 00:59:30,530 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:30,530 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 9 2024-12-11 00:59:30,530 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:30,530 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 9 2024-12-11 00:59:30,530 INFO SenderThread:843156 [dir_watcher.py:finish():358] shutting down directory watcher 2024-12-11 00:59:30,658 INFO SenderThread:843156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:59:30,659 INFO SenderThread:843156 [dir_watcher.py:finish():388] scan: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files 2024-12-11 00:59:30,659 INFO SenderThread:843156 [dir_watcher.py:finish():402] scan save: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log output.log 2024-12-11 00:59:30,659 INFO SenderThread:843156 [dir_watcher.py:finish():402] scan save: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/config.yaml config.yaml 2024-12-11 00:59:30,660 INFO SenderThread:843156 [dir_watcher.py:finish():402] scan save: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/requirements.txt requirements.txt 2024-12-11 00:59:30,661 INFO SenderThread:843156 [dir_watcher.py:finish():402] scan save: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json wandb-summary.json 2024-12-11 00:59:30,663 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 10 2024-12-11 00:59:30,665 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:30,666 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 10 2024-12-11 00:59:30,666 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:30,666 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 10 2024-12-11 00:59:30,666 INFO SenderThread:843156 [file_pusher.py:finish():169] shutting down file pusher 2024-12-11 00:59:31,099 INFO wandb-upload_0:843156 [upload_job.py:push():130] Uploaded file /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/output.log 2024-12-11 00:59:31,258 INFO wandb-upload_2:843156 [upload_job.py:push():130] Uploaded file /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/requirements.txt 2024-12-11 00:59:31,259 INFO wandb-upload_3:843156 [upload_job.py:push():130] Uploaded file /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/wandb-summary.json 2024-12-11 00:59:31,293 INFO wandb-upload_1:843156 [upload_job.py:push():130] Uploaded file /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/files/config.yaml 2024-12-11 00:59:31,309 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-11 00:59:31,309 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: poll_exit 2024-12-11 00:59:31,493 INFO Thread-11 (_thread_body):843156 [sender.py:transition_state():614] send defer: 11 2024-12-11 00:59:31,494 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:31,494 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 11 2024-12-11 00:59:31,494 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:31,494 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 11 2024-12-11 00:59:31,494 INFO SenderThread:843156 [file_pusher.py:join():175] waiting for file pusher 2024-12-11 00:59:31,495 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 12 2024-12-11 00:59:31,495 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:31,495 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 12 2024-12-11 00:59:31,495 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:31,495 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 12 2024-12-11 00:59:31,495 INFO SenderThread:843156 [file_stream.py:finish():601] file stream finish called 2024-12-11 00:59:31,567 INFO SenderThread:843156 [file_stream.py:finish():605] file stream finish is done 2024-12-11 00:59:31,567 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 13 2024-12-11 00:59:31,567 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:31,567 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 13 2024-12-11 00:59:31,567 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:31,567 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 13 2024-12-11 00:59:31,567 INFO SenderThread:843156 [sender.py:transition_state():614] send defer: 14 2024-12-11 00:59:31,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: defer 2024-12-11 00:59:31,568 DEBUG SenderThread:843156 [sender.py:send():379] send: final 2024-12-11 00:59:31,568 INFO HandlerThread:843156 [handler.py:handle_request_defer():184] handle defer: 14 2024-12-11 00:59:31,568 DEBUG SenderThread:843156 [sender.py:send():379] send: footer 2024-12-11 00:59:31,568 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: defer 2024-12-11 00:59:31,568 INFO SenderThread:843156 [sender.py:send_request_defer():610] handle sender defer: 14 2024-12-11 00:59:31,568 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-11 00:59:31,568 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: poll_exit 2024-12-11 00:59:31,569 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: server_info 2024-12-11 00:59:31,569 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: server_info 2024-12-11 00:59:31,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-11 00:59:31,570 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: get_summary 2024-12-11 00:59:31,571 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: sampled_history 2024-12-11 00:59:31,572 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-11 00:59:31,638 DEBUG SenderThread:843156 [sender.py:send_request():406] send_request: poll_exit 2024-12-11 00:59:31,638 INFO MainThread:843156 [wandb_run.py:_footer_history_summary_info():4008] rendering history 2024-12-11 00:59:31,639 INFO MainThread:843156 [wandb_run.py:_footer_history_summary_info():4040] rendering summary 2024-12-11 00:59:31,639 INFO MainThread:843156 [wandb_run.py:_footer_sync_info():3967] logging synced files 2024-12-11 00:59:31,640 DEBUG HandlerThread:843156 [handler.py:handle_request():158] handle_request: shutdown 2024-12-11 00:59:31,640 INFO HandlerThread:843156 [handler.py:finish():882] shutting down handler 2024-12-11 00:59:32,570 INFO WriterThread:843156 [datastore.py:close():296] close: /weka/proj-fmri/ckadirt/MindEyeV2/src/wandb/run-20241210_215728-bold5k_nsdm1/run-bold5k_nsdm1.wandb 2024-12-11 00:59:32,638 INFO SenderThread:843156 [sender.py:finish():1608] shutting down sender 2024-12-11 00:59:32,638 INFO SenderThread:843156 [file_pusher.py:finish():169] shutting down file pusher 2024-12-11 00:59:32,638 INFO SenderThread:843156 [file_pusher.py:join():175] waiting for file pusher