2023-08-31 16:21:12,589 INFO StreamThr :3427956 [internal.py:wandb_internal():86] W&B internal server running at pid: 3427956, started at: 2023-08-31 16:21:12.587723 2023-08-31 16:21:12,590 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status 2023-08-31 16:21:12,598 INFO WriterThread:3427956 [datastore.py:open_for_write():85] open: ./wandb/run-20230831_162112-pftwit53/run-pftwit53.wandb 2023-08-31 16:21:12,600 DEBUG SenderThread:3427956 [sender.py:send():369] send: header 2023-08-31 16:21:12,601 DEBUG SenderThread:3427956 [sender.py:send():369] send: run 2023-08-31 16:21:12,606 INFO SenderThread:3427956 [sender.py:_maybe_setup_resume():737] checking resume status for ckadirt/brain2music/pftwit53 2023-08-31 16:21:13,166 INFO SenderThread:3427956 [dir_watcher.py:__init__():211] watching files in: ./wandb/run-20230831_162112-pftwit53/files 2023-08-31 16:21:13,166 INFO SenderThread:3427956 [sender.py:_start_run_threads():1100] run started: pftwit53 with start time 1693498872.59191 2023-08-31 16:21:13,166 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: summary_record 2023-08-31 16:21:13,168 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-08-31 16:21:13,171 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: check_version 2023-08-31 16:21:13,172 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: check_version 2023-08-31 16:21:13,241 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: run_start 2023-08-31 16:21:13,251 DEBUG HandlerThread:3427956 [system_info.py:__init__():31] System info init 2023-08-31 16:21:13,251 DEBUG HandlerThread:3427956 [system_info.py:__init__():46] System info init done 2023-08-31 16:21:13,251 INFO HandlerThread:3427956 [system_monitor.py:start():181] Starting system monitor 2023-08-31 16:21:13,251 INFO SystemMonitor:3427956 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-08-31 16:21:13,252 INFO HandlerThread:3427956 [system_monitor.py:probe():201] Collecting system info 2023-08-31 16:21:13,252 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started cpu monitoring 2023-08-31 16:21:13,252 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started disk monitoring 2023-08-31 16:21:13,253 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started gpu monitoring 2023-08-31 16:21:13,253 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started memory monitoring 2023-08-31 16:21:13,254 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started network monitoring 2023-08-31 16:21:13,306 DEBUG HandlerThread:3427956 [system_info.py:probe():195] Probing system 2023-08-31 16:21:13,308 DEBUG HandlerThread:3427956 [system_info.py:_probe_git():180] Probing git 2023-08-31 16:21:13,313 ERROR HandlerThread:3427956 [gitlib.py:root():89] git root error: Cmd('git') failed due to: exit code(128) cmdline: git rev-parse --show-toplevel stderr: 'fatal: detected dubious ownership in repository at '/fsx' To add an exception for this directory, call: git config --global --add safe.directory /fsx' 2023-08-31 16:21:13,313 DEBUG HandlerThread:3427956 [system_info.py:_probe_git():188] Probing git done 2023-08-31 16:21:13,313 DEBUG HandlerThread:3427956 [system_info.py:probe():240] Probing system done 2023-08-31 16:21:13,313 DEBUG HandlerThread:3427956 [system_monitor.py:probe():210] {'os': 'Linux-5.15.0-1037-aws-x86_64-with-glibc2.31', 'python': '3.10.8', 'heartbeatAt': '2023-08-31T16:21:13.306106', 'startedAt': '2023-08-31T16:21:12.558810', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': '/fsx/proj-fmri/ckadirt/b2m/src/mlpdummy.py', 'codePath': 'src/mlpdummy.py', 'git': {'remote': None, 'commit': None}, 'email': None, 'root': '/fsx/proj-fmri/ckadirt/b2m', 'host': 'ip-26-0-144-149', 'username': 'ckadirt', 'executable': '/admin/home-ckadirt/miniconda3/envs/mindeye/bin/python', 'cpu_count': 48, 'cpu_count_logical': 96, 'cpu_freq': {'current': 3002.844541666667, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 2773.902, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 2591.475, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 2971.069, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 232.5829315185547, 'used': 89.25303268432617}, 'gpu': 'NVIDIA A100-SXM4-40GB', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}], 'memory': {'total': 1121.815086364746}} 2023-08-31 16:21:13,313 INFO HandlerThread:3427956 [system_monitor.py:probe():211] Finished collecting system info 2023-08-31 16:21:13,313 INFO HandlerThread:3427956 [system_monitor.py:probe():214] Publishing system info 2023-08-31 16:21:13,313 DEBUG HandlerThread:3427956 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment 2023-08-31 16:21:13,316 DEBUG HandlerThread:3427956 [system_info.py:_save_pip():67] Saving pip packages done 2023-08-31 16:21:13,316 DEBUG HandlerThread:3427956 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment 2023-08-31 16:21:14,170 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/conda-environment.yaml 2023-08-31 16:21:14,170 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/requirements.txt 2023-08-31 16:21:14,170 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json 2023-08-31 16:21:19,578 DEBUG HandlerThread:3427956 [system_info.py:_save_conda():86] Saving conda packages done 2023-08-31 16:21:19,578 DEBUG HandlerThread:3427956 [system_info.py:_save_code():89] Saving code 2023-08-31 16:21:19,582 ERROR HandlerThread:3427956 [gitlib.py:root():89] git root error: Cmd('git') failed due to: exit code(128) cmdline: git rev-parse --show-toplevel stderr: 'fatal: detected dubious ownership in repository at '/fsx' To add an exception for this directory, call: git config --global --add safe.directory /fsx' 2023-08-31 16:21:19,588 DEBUG HandlerThread:3427956 [system_info.py:_save_code():110] Saving code done 2023-08-31 16:21:19,588 DEBUG HandlerThread:3427956 [system_info.py:_save_patches():127] Saving git patches 2023-08-31 16:21:19,591 ERROR HandlerThread:3427956 [gitlib.py:root():89] git root error: Cmd('git') failed due to: exit code(128) cmdline: git rev-parse --show-toplevel stderr: 'fatal: detected dubious ownership in repository at '/fsx' To add an exception for this directory, call: git config --global --add safe.directory /fsx' 2023-08-31 16:21:19,600 DEBUG HandlerThread:3427956 [system_info.py:_save_patches():169] Saving git patches done 2023-08-31 16:21:19,604 INFO HandlerThread:3427956 [system_monitor.py:probe():216] Finished publishing system info 2023-08-31 16:21:19,608 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report 2023-08-31 16:21:19,608 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: keepalive 2023-08-31 16:21:19,608 DEBUG SenderThread:3427956 [sender.py:send():369] send: files 2023-08-31 16:21:19,608 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file wandb-metadata.json with policy now 2023-08-31 16:21:19,609 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file code/src/mlpdummy.py with policy now 2023-08-31 16:21:19,616 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: stop_status 2023-08-31 16:21:19,616 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: stop_status 2023-08-31 16:21:19,994 DEBUG SenderThread:3427956 [sender.py:send():369] send: telemetry 2023-08-31 16:21:19,994 DEBUG SenderThread:3427956 [sender.py:send():369] send: metric 2023-08-31 16:21:19,995 DEBUG SenderThread:3427956 [sender.py:send():369] send: telemetry 2023-08-31 16:21:19,995 DEBUG SenderThread:3427956 [sender.py:send():369] send: metric 2023-08-31 16:21:19,995 WARNING SenderThread:3427956 [sender.py:send_metric():1305] Seen metric with glob (shouldn't happen) 2023-08-31 16:21:20,181 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/conda-environment.yaml 2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/code/src/mlpdummy.py 2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/wandb-metadata.json 2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/output.log 2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/code/src 2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/code 2023-08-31 16:21:20,188 INFO wandb-upload_0:3427956 [upload_job.py:push():133] Uploaded file /tmp/tmpcnz000j2wandb/zi0muto5-wandb-metadata.json 2023-08-31 16:21:20,359 INFO wandb-upload_1:3427956 [upload_job.py:push():133] Uploaded file /tmp/tmpcnz000j2wandb/igjvcllv-code/src/mlpdummy.py 2023-08-31 16:21:22,186 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/output.log 2023-08-31 16:21:23,961 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report 2023-08-31 16:21:26,201 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/output.log 2023-08-31 16:21:29,329 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report 2023-08-31 16:21:34,329 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report 2023-08-31 16:21:34,684 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: stop_status 2023-08-31 16:21:34,684 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: stop_status 2023-08-31 16:21:36,096 DEBUG SenderThread:3427956 [sender.py:send():369] send: exit 2023-08-31 16:21:36,096 INFO SenderThread:3427956 [sender.py:send_exit():574] handling exit code: 1 2023-08-31 16:21:36,096 INFO SenderThread:3427956 [sender.py:send_exit():576] handling runtime: 22 2023-08-31 16:21:36,099 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-08-31 16:21:36,100 INFO SenderThread:3427956 [sender.py:send_exit():582] send defer 2023-08-31 16:21:36,100 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:36,100 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 0 2023-08-31 16:21:36,100 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:36,100 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 0 2023-08-31 16:21:36,100 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 1 2023-08-31 16:21:36,101 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:36,101 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 1 2023-08-31 16:21:36,101 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:36,101 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 1 2023-08-31 16:21:36,101 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 2 2023-08-31 16:21:36,101 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:36,101 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 2 2023-08-31 16:21:36,101 INFO HandlerThread:3427956 [system_monitor.py:finish():190] Stopping system monitor 2023-08-31 16:21:36,101 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined cpu monitor 2023-08-31 16:21:36,101 DEBUG SystemMonitor:3427956 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-08-31 16:21:36,102 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined disk monitor 2023-08-31 16:21:36,102 DEBUG SystemMonitor:3427956 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-08-31 16:21:36,102 DEBUG SystemMonitor:3427956 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-08-31 16:21:36,277 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/output.log 2023-08-31 16:21:36,278 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json 2023-08-31 16:21:37,227 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined gpu monitor 2023-08-31 16:21:37,228 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined memory monitor 2023-08-31 16:21:37,228 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined network monitor 2023-08-31 16:21:37,228 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit 2023-08-31 16:21:37,228 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:37,229 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 2 2023-08-31 16:21:37,229 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 3 2023-08-31 16:21:37,229 DEBUG SenderThread:3427956 [sender.py:send():369] send: stats 2023-08-31 16:21:37,229 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:37,229 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit 2023-08-31 16:21:37,229 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 3 2023-08-31 16:21:37,230 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:37,230 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 3 2023-08-31 16:21:37,230 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 4 2023-08-31 16:21:37,230 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:37,230 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 4 2023-08-31 16:21:37,230 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:37,230 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 4 2023-08-31 16:21:37,230 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 5 2023-08-31 16:21:37,230 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:37,230 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 5 2023-08-31 16:21:37,231 DEBUG SenderThread:3427956 [sender.py:send():369] send: summary 2023-08-31 16:21:37,233 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-08-31 16:21:37,233 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:37,233 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 5 2023-08-31 16:21:37,233 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 6 2023-08-31 16:21:37,233 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:37,233 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 6 2023-08-31 16:21:37,234 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:37,234 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 6 2023-08-31 16:21:37,238 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report 2023-08-31 16:21:37,283 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json 2023-08-31 16:21:37,323 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 7 2023-08-31 16:21:37,323 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:37,323 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 7 2023-08-31 16:21:37,324 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:37,324 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 7 2023-08-31 16:21:38,097 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit 2023-08-31 16:21:38,288 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/output.log 2023-08-31 16:21:38,289 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/config.yaml 2023-08-31 16:21:38,923 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 8 2023-08-31 16:21:38,923 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit 2023-08-31 16:21:38,924 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:38,924 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 8 2023-08-31 16:21:38,924 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:38,924 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 8 2023-08-31 16:21:38,924 INFO SenderThread:3427956 [job_builder.py:build():240] Attempting to build job artifact 2023-08-31 16:21:38,926 INFO SenderThread:3427956 [job_builder.py:build():274] no source found 2023-08-31 16:21:38,926 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 9 2023-08-31 16:21:38,927 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:38,927 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 9 2023-08-31 16:21:38,927 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:38,927 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 9 2023-08-31 16:21:38,927 INFO SenderThread:3427956 [dir_watcher.py:finish():359] shutting down directory watcher 2023-08-31 16:21:39,098 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit 2023-08-31 16:21:39,295 INFO SenderThread:3427956 [dir_watcher.py:finish():389] scan: ./wandb/run-20230831_162112-pftwit53/files 2023-08-31 16:21:39,296 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json wandb-summary.json 2023-08-31 16:21:39,296 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/requirements.txt requirements.txt 2023-08-31 16:21:39,297 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/wandb-metadata.json wandb-metadata.json 2023-08-31 16:21:39,299 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/conda-environment.yaml conda-environment.yaml 2023-08-31 16:21:39,301 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/config.yaml config.yaml 2023-08-31 16:21:39,301 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/output.log output.log 2023-08-31 16:21:39,308 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/code/src/mlpdummy.py code/src/mlpdummy.py 2023-08-31 16:21:39,308 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 10 2023-08-31 16:21:39,308 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit 2023-08-31 16:21:39,309 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:39,309 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 10 2023-08-31 16:21:39,309 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:39,309 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 10 2023-08-31 16:21:39,309 INFO SenderThread:3427956 [file_pusher.py:finish():159] shutting down file pusher 2023-08-31 16:21:39,746 INFO wandb-upload_0:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json 2023-08-31 16:21:39,814 INFO wandb-upload_1:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/requirements.txt 2023-08-31 16:21:39,848 INFO wandb-upload_2:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/conda-environment.yaml 2023-08-31 16:21:39,965 INFO wandb-upload_3:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/config.yaml 2023-08-31 16:21:40,100 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit 2023-08-31 16:21:40,100 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit 2023-08-31 16:21:40,182 INFO wandb-upload_4:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/output.log 2023-08-31 16:21:40,383 INFO Thread-11 (_thread_body):3427956 [sender.py:transition_state():602] send defer: 11 2023-08-31 16:21:40,384 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:40,384 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 11 2023-08-31 16:21:40,384 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:40,384 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 11 2023-08-31 16:21:40,384 INFO SenderThread:3427956 [file_pusher.py:join():164] waiting for file pusher 2023-08-31 16:21:40,384 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 12 2023-08-31 16:21:40,384 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:40,384 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 12 2023-08-31 16:21:40,385 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:40,385 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 12 2023-08-31 16:21:40,463 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 13 2023-08-31 16:21:40,464 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:40,464 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 13 2023-08-31 16:21:40,464 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:40,464 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 13 2023-08-31 16:21:40,464 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 14 2023-08-31 16:21:40,464 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer 2023-08-31 16:21:40,465 DEBUG SenderThread:3427956 [sender.py:send():369] send: final 2023-08-31 16:21:40,465 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 14 2023-08-31 16:21:40,465 DEBUG SenderThread:3427956 [sender.py:send():369] send: footer 2023-08-31 16:21:40,465 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer 2023-08-31 16:21:40,465 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 14 2023-08-31 16:21:40,465 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit 2023-08-31 16:21:40,465 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit 2023-08-31 16:21:40,466 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit 2023-08-31 16:21:40,466 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit 2023-08-31 16:21:40,466 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: server_info 2023-08-31 16:21:40,466 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: server_info 2023-08-31 16:21:40,466 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: get_summary 2023-08-31 16:21:40,468 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: sampled_history 2023-08-31 16:21:40,613 INFO MainThread:3427956 [wandb_run.py:_footer_history_summary_info():3464] rendering history 2023-08-31 16:21:40,613 INFO MainThread:3427956 [wandb_run.py:_footer_history_summary_info():3496] rendering summary 2023-08-31 16:21:40,613 INFO MainThread:3427956 [wandb_run.py:_footer_sync_info():3423] logging synced files 2023-08-31 16:21:40,614 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: shutdown 2023-08-31 16:21:40,614 INFO HandlerThread:3427956 [handler.py:finish():854] shutting down handler 2023-08-31 16:21:41,466 INFO WriterThread:3427956 [datastore.py:close():298] close: ./wandb/run-20230831_162112-pftwit53/run-pftwit53.wandb 2023-08-31 16:21:41,613 INFO SenderThread:3427956 [sender.py:finish():1526] shutting down sender 2023-08-31 16:21:41,613 INFO SenderThread:3427956 [file_pusher.py:finish():159] shutting down file pusher 2023-08-31 16:21:41,614 INFO SenderThread:3427956 [file_pusher.py:join():164] waiting for file pusher