2024-07-04 20:03:05,004 INFO StreamThr :136 [internal.py:wandb_internal():85] W&B internal server running at pid: 136, started at: 2024-07-04 20:03:05.004020 2024-07-04 20:03:05,006 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status 2024-07-04 20:03:05,320 INFO WriterThread:136 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240704_200304-v5ofm505/run-v5ofm505.wandb 2024-07-04 20:03:05,321 DEBUG SenderThread:136 [sender.py:send():378] send: header 2024-07-04 20:03:05,324 DEBUG SenderThread:136 [sender.py:send():378] send: run 2024-07-04 20:03:05,534 INFO SenderThread:136 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files 2024-07-04 20:03:05,534 INFO SenderThread:136 [sender.py:_start_run_threads():1123] run started: v5ofm505 with start time 1720123385.005429 2024-07-04 20:03:05,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: check_version 2024-07-04 20:03:05,549 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: check_version 2024-07-04 20:03:05,639 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: run_start 2024-07-04 20:03:05,653 DEBUG HandlerThread:136 [system_info.py:__init__():26] System info init 2024-07-04 20:03:05,653 DEBUG HandlerThread:136 [system_info.py:__init__():41] System info init done 2024-07-04 20:03:05,653 INFO HandlerThread:136 [system_monitor.py:start():194] Starting system monitor 2024-07-04 20:03:05,653 INFO SystemMonitor:136 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-04 20:03:05,654 INFO HandlerThread:136 [system_monitor.py:probe():214] Collecting system info 2024-07-04 20:03:05,654 INFO SystemMonitor:136 [interfaces.py:start():188] Started cpu monitoring 2024-07-04 20:03:05,655 INFO SystemMonitor:136 [interfaces.py:start():188] Started disk monitoring 2024-07-04 20:03:05,655 INFO SystemMonitor:136 [interfaces.py:start():188] Started gpu monitoring 2024-07-04 20:03:05,657 INFO SystemMonitor:136 [interfaces.py:start():188] Started memory monitoring 2024-07-04 20:03:05,658 INFO SystemMonitor:136 [interfaces.py:start():188] Started network monitoring 2024-07-04 20:03:05,663 DEBUG HandlerThread:136 [system_info.py:probe():150] Probing system 2024-07-04 20:03:05,666 DEBUG HandlerThread:136 [gitlib.py:_init_repo():56] git repository is invalid 2024-07-04 20:03:05,666 DEBUG HandlerThread:136 [system_info.py:probe():198] Probing system done 2024-07-04 20:03:05,666 DEBUG HandlerThread:136 [system_monitor.py:probe():223] {'os': 'Linux-5.15.154+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-07-04T20:03:05.663552', 'startedAt': '2024-07-04T20:03:04.997852', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'bcbb291c3718', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.134, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.134, 'min': 0.0, 'max': 0.0}, {'current': 2000.134, 'min': 0.0, 'max': 0.0}, {'current': 2000.134, 'min': 0.0, 'max': 0.0}, {'current': 2000.134, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5689.499000549316}}, 'gpu': 'Tesla P100-PCIE-16GB', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla P100-PCIE-16GB', 'memory_total': 17179869184}], 'memory': {'total': 31.357574462890625}} 2024-07-04 20:03:05,666 INFO HandlerThread:136 [system_monitor.py:probe():224] Finished collecting system info 2024-07-04 20:03:05,666 INFO HandlerThread:136 [system_monitor.py:probe():227] Publishing system info 2024-07-04 20:03:05,667 DEBUG HandlerThread:136 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment 2024-07-04 20:03:06,536 INFO Thread-12 :136 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/conda-environment.yaml 2024-07-04 20:03:20,681 ERROR HandlerThread:136 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda subprocess.call( File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call return p.wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait return self._wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait raise TimeoutExpired(self.args, timeout) subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds 2024-07-04 20:03:20,684 DEBUG HandlerThread:136 [system_info.py:_save_conda():222] Saving conda packages done 2024-07-04 20:03:20,684 INFO HandlerThread:136 [system_monitor.py:probe():229] Finished publishing system info 2024-07-04 20:03:20,693 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:03:20,693 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 20:03:20,694 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:03:20,694 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 20:03:20,694 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:03:20,694 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 20:03:20,694 DEBUG SenderThread:136 [sender.py:send():378] send: files 2024-07-04 20:03:20,695 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-metadata.json with policy now 2024-07-04 20:03:21,011 INFO wandb-upload_0:136 [upload_job.py:push():130] Uploaded file /tmp/tmp8n8odwqhwandb/kxconlu9-wandb-metadata.json 2024-07-04 20:03:21,539 INFO Thread-12 :136 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-metadata.json 2024-07-04 20:03:21,971 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: python_packages 2024-07-04 20:03:21,972 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: python_packages 2024-07-04 20:03:21,974 DEBUG SenderThread:136 [sender.py:send():378] send: telemetry 2024-07-04 20:03:21,975 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:03:21,976 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:03:22,117 DEBUG SenderThread:136 [sender.py:send():378] send: config 2024-07-04 20:03:22,118 DEBUG SenderThread:136 [sender.py:send():378] send: metric 2024-07-04 20:03:22,119 DEBUG SenderThread:136 [sender.py:send():378] send: telemetry 2024-07-04 20:03:22,119 DEBUG SenderThread:136 [sender.py:send():378] send: telemetry 2024-07-04 20:03:22,119 DEBUG SenderThread:136 [sender.py:send():378] send: metric 2024-07-04 20:03:22,119 WARNING SenderThread:136 [sender.py:send_metric():1340] Seen metric with glob (shouldn't happen) 2024-07-04 20:03:22,119 DEBUG SenderThread:136 [sender.py:send():378] send: telemetry 2024-07-04 20:03:22,540 INFO Thread-12 :136 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/requirements.txt 2024-07-04 20:03:26,120 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:03:31,120 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:03:36,126 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:03:36,545 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/config.yaml 2024-07-04 20:03:36,972 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:03:36,973 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:03:42,091 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:03:47,092 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:03:51,972 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:03:51,973 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:03:53,042 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:03:58,043 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:04:03,044 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:04:05,658 DEBUG SystemMonitor:136 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-04 20:04:05,659 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:04:06,972 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:04:06,973 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:04:08,101 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:04:13,102 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:04:18,103 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:04:21,972 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:04:21,973 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:04:24,057 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:04:29,058 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:04:34,059 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:04:35,660 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:04:36,974 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:04:36,975 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:04:39,127 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:04:44,128 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:04:49,129 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:04:51,973 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:04:51,973 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:04:55,076 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:05:00,077 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:05:05,078 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:05:05,661 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:05:07,010 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:05:07,011 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:05:10,100 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:05:15,101 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:05:20,102 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:05:22,046 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:05:22,047 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:05:25,127 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:05:30,128 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:05:35,129 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:05:35,662 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:05:37,025 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:05:37,026 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:05:40,159 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:05:45,161 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:05:50,162 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:05:52,178 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:05:52,178 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:05:55,246 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:06:00,247 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:06:05,249 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:06:05,662 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:06:07,177 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:06:07,178 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:06:11,242 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:06:16,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:06:21,244 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:06:22,177 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:06:22,178 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:06:26,267 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:06:31,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:06:35,663 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:06:36,664 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:06:37,244 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:06:37,245 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:06:42,377 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:06:47,378 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:06:52,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:06:52,244 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:06:53,356 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:06:58,357 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:07:03,359 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:07:05,664 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:07:07,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:07:07,244 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:07:09,315 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:07:14,316 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:07:19,317 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:07:22,288 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:07:22,288 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:07:24,348 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:07:29,349 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:07:34,350 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:07:35,665 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:07:37,275 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:07:37,275 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:07:39,409 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:07:44,410 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:07:49,411 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:07:52,275 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:07:52,275 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:07:55,370 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:08:00,371 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:08:05,372 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:08:05,666 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:08:07,275 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:08:07,276 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:08:10,436 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:08:15,437 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:08:20,438 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:08:22,275 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:08:22,276 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:08:26,367 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:08:31,368 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:08:35,667 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:08:36,668 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:08:37,319 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:08:37,320 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:08:42,438 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:08:47,439 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:08:52,298 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:08:52,298 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:08:53,357 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:08:58,358 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:09:03,359 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:09:05,668 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:09:07,298 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:09:07,298 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:09:08,380 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:09:13,381 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:09:18,383 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:09:22,336 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:09:22,337 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:09:23,495 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:09:28,496 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:09:33,497 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:09:35,669 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:09:37,315 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:09:37,315 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:09:39,440 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:09:44,441 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:09:49,442 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:09:52,339 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:09:52,340 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:09:55,438 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:10:00,439 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:10:05,440 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:10:05,669 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:10:07,518 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:10:07,519 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:10:10,692 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:10:15,693 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:10:20,694 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:10:22,700 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:10:22,700 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:10:25,760 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:10:26,645 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 20:10:26,647 DEBUG SenderThread:136 [sender.py:send():378] send: metric 2024-07-04 20:10:26,648 DEBUG SenderThread:136 [sender.py:send():378] send: metric 2024-07-04 20:10:26,648 DEBUG SenderThread:136 [sender.py:send():378] send: metric 2024-07-04 20:10:26,648 DEBUG SenderThread:136 [sender.py:send():378] send: metric 2024-07-04 20:10:26,648 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 20:10:26,648 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 20:10:26,648 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 20:10:26,700 INFO Thread-12 :136 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 20:10:31,650 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:10:35,670 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:10:36,671 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:10:37,700 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:10:37,700 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:10:41,839 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:10:46,840 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:10:51,846 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:10:52,700 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:10:52,701 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:10:52,710 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/config.yaml 2024-07-04 20:10:57,806 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:11:02,807 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:11:05,671 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:11:07,700 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:11:07,701 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:11:07,845 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:11:12,846 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:11:17,847 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:11:22,701 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:11:22,702 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:11:22,860 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:11:27,861 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:11:32,862 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:11:35,672 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:11:37,701 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:11:37,701 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:11:38,797 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:11:43,797 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:11:48,798 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:11:52,701 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:11:52,701 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:11:53,833 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:11:58,834 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:12:03,834 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:12:05,673 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:12:07,701 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:12:07,701 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:12:08,850 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:12:13,851 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:12:18,852 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:12:22,877 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:12:22,878 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:12:23,991 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:12:28,992 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:12:33,993 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:12:35,674 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:12:37,876 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:12:37,877 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:12:39,035 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:12:44,036 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:12:49,037 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:12:52,876 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:12:52,877 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:12:55,029 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:13:00,030 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:13:05,031 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:13:05,674 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:13:08,085 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:13:08,086 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:13:10,235 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:13:15,236 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:13:20,237 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:13:23,084 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:13:23,085 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:13:26,154 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:13:31,155 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:13:35,675 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:13:36,676 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:13:38,085 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:13:38,085 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:13:42,179 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:13:47,180 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:13:52,182 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:13:53,137 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:13:53,137 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:13:57,242 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:14:02,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:14:05,676 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:14:07,677 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:14:08,194 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:14:08,194 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:14:13,328 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:14:18,329 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:14:23,173 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:14:23,174 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:14:24,309 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:14:29,310 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:14:34,311 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:14:35,677 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:14:38,173 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:14:38,174 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:14:40,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:14:45,244 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:14:50,245 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:14:53,222 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:14:53,223 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:14:55,332 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:15:00,333 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:15:05,334 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:15:05,678 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:15:08,201 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:15:08,201 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:15:11,333 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:15:16,334 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:15:21,335 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:15:23,201 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:15:23,201 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:15:27,318 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:15:32,319 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:15:35,679 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:15:37,680 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:15:38,201 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:15:38,201 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:15:43,317 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:15:48,318 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:15:53,201 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:15:53,202 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:15:54,285 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:15:59,286 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:16:04,287 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:16:05,680 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:16:08,210 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:16:08,210 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:16:09,369 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:16:14,370 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:16:19,371 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:16:23,209 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:16:23,210 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:16:25,282 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:16:30,283 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:16:35,284 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:16:35,681 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:16:38,360 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:16:38,361 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:16:40,448 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:16:45,449 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:16:50,450 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:16:53,361 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:16:53,361 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:16:55,481 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:17:00,482 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:17:05,483 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:17:05,681 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:17:08,361 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:17:08,361 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:17:11,443 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:17:16,444 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:17:21,445 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:17:23,361 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:17:23,362 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:17:27,434 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:17:30,624 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 20:17:30,626 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 20:17:30,626 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 20:17:30,626 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 20:17:30,860 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 20:17:32,627 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:17:35,682 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:17:37,683 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:17:38,398 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:17:38,398 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:17:43,492 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:17:48,493 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:17:53,445 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:17:53,445 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:17:53,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:17:58,550 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:18:03,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:18:05,683 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:18:08,422 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:18:08,423 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:18:09,550 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:18:14,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:18:19,553 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:18:23,423 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:18:23,424 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:18:24,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:18:29,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:18:34,571 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:18:35,684 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:18:38,466 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:18:38,467 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:18:40,540 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:18:45,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:18:50,542 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:18:53,445 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:18:53,445 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:18:56,506 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:19:01,508 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:19:05,685 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:19:06,686 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:19:08,461 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:19:08,462 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:19:12,550 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:19:17,550 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:19:22,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:19:23,460 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:19:23,461 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:19:27,597 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:19:32,598 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:19:35,686 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:19:37,687 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:19:38,460 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:19:38,461 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:19:43,566 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:19:48,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:19:53,468 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:19:53,469 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:19:54,540 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:19:59,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:20:04,542 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:20:05,687 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:20:08,467 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:20:08,467 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:20:10,540 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:20:15,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:20:20,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:20:23,480 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:20:23,481 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:20:25,590 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:20:30,591 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:20:35,592 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:20:35,688 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:20:38,479 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:20:38,480 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:20:40,635 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:20:45,636 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:20:50,637 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:20:53,479 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:20:53,480 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:20:56,631 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:21:01,632 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:21:05,689 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:21:06,690 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:21:08,479 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:21:08,480 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:21:12,620 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:21:17,621 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:21:22,621 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:21:23,480 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:21:23,480 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:21:28,581 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:21:33,582 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:21:35,690 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:21:38,640 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:21:38,641 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:21:38,643 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:21:43,723 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:21:48,724 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:21:53,656 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:21:53,657 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:21:53,756 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:21:58,757 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:22:03,757 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:22:05,691 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:22:08,653 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:22:08,654 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:22:08,760 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:22:13,762 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:22:18,763 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:22:23,695 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:22:23,695 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:22:23,810 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:22:28,811 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:22:33,812 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:22:35,695 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:22:38,731 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:22:38,732 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:22:38,884 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:22:43,885 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:22:48,886 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:22:53,708 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:22:53,708 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:22:54,858 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:22:59,859 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:23:04,860 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:23:05,692 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:23:08,844 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:23:08,844 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:23:09,981 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:23:14,982 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:23:19,983 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:23:23,897 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:23:23,898 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:23:25,043 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:23:30,044 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:23:35,045 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:23:35,695 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:23:39,100 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:23:39,101 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:23:40,165 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:23:45,166 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:23:50,167 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:23:54,139 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:23:54,140 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:23:55,277 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:24:00,279 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:24:05,280 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:24:05,694 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:24:09,119 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:24:09,120 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:24:11,193 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:24:16,194 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:24:21,195 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:24:24,120 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:24:24,120 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:24:26,259 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:24:31,260 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:24:35,695 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:24:35,700 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 20:24:35,702 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 20:24:35,702 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 20:24:35,703 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 20:24:36,020 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 20:24:36,703 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:24:39,180 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:24:39,181 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:24:42,313 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:24:47,314 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:24:52,315 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:24:54,119 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:24:54,120 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:24:58,195 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:25:03,196 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:25:05,695 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:25:08,696 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:25:09,166 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:25:09,166 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:25:14,249 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:25:19,250 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:25:24,163 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:25:24,164 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:25:24,312 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:25:29,314 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:25:34,315 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:25:35,696 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:25:39,306 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:25:39,306 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:25:39,461 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:25:44,461 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:25:49,462 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:25:54,305 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:25:54,306 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:25:55,391 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:26:00,392 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:26:05,393 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:26:05,697 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:26:09,305 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:26:09,306 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:26:10,422 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:26:15,423 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:26:20,424 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:26:24,337 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:26:24,337 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:26:25,499 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:26:30,500 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:26:35,501 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:26:35,698 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:26:39,316 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:26:39,317 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:26:41,398 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:26:46,399 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:26:51,400 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:26:54,316 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:26:54,317 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:26:56,453 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:27:01,454 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:27:05,699 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:27:06,700 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:27:09,518 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:27:09,518 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:27:12,639 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:27:17,640 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:27:22,641 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:27:24,517 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:27:24,518 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:27:28,636 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:27:33,637 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:27:35,699 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:27:38,701 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:27:39,521 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:27:39,521 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:27:44,654 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:27:49,655 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:27:54,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:27:54,573 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:27:54,702 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:27:59,703 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:28:04,704 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:28:05,700 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:28:09,571 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:28:09,571 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:28:09,728 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:28:14,729 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:28:19,730 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:28:24,572 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:28:24,573 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:28:25,687 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:28:30,688 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:28:35,692 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:28:35,701 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:28:39,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:28:39,573 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:28:40,705 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:28:45,706 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:28:50,707 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:28:54,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:28:54,573 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:28:55,712 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:29:00,713 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:29:05,702 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:29:06,703 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:29:09,610 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:29:09,610 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:29:11,727 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:29:16,729 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:29:21,729 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:29:24,597 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:29:24,598 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:29:27,685 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:29:32,686 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:29:35,703 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:29:37,704 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:29:39,597 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:29:39,598 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:29:43,696 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:29:48,697 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:29:53,699 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:29:54,597 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:29:54,598 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:29:58,770 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:30:03,771 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:30:05,704 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:30:09,588 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:30:09,598 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:30:09,598 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:30:14,730 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:30:19,731 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:30:24,810 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:30:24,811 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:30:24,935 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:30:29,937 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:30:34,938 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:30:35,705 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:30:40,019 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:30:40,020 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:30:40,098 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:30:45,100 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:30:50,100 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:30:55,019 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:30:55,020 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:30:55,118 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:31:00,120 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:31:05,121 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:31:05,706 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:31:10,020 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:31:10,020 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:31:10,151 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:31:15,152 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:31:20,153 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:31:25,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:31:25,200 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:31:25,334 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:31:30,335 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:31:35,337 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:31:35,706 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:31:40,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:31:40,200 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:31:41,292 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:31:42,067 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 20:31:42,068 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 20:31:42,068 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 20:31:42,069 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 20:31:42,182 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 20:31:47,070 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:31:52,071 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:31:55,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:31:55,200 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:31:57,269 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:32:02,270 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:32:05,707 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:32:07,708 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:32:10,364 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:32:10,365 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:32:13,467 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:32:18,468 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:32:23,469 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:32:25,364 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:32:25,365 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:32:28,485 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:32:33,486 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:32:35,708 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:32:38,708 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:32:40,383 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:32:40,384 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:32:44,505 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:32:49,506 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:32:54,507 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:32:55,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:32:55,567 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:32:59,717 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:33:04,718 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:33:05,708 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:33:09,769 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:33:10,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:33:10,567 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:33:15,638 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:33:20,639 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:33:25,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:33:25,568 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:33:25,644 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:33:30,645 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:33:35,647 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:33:35,709 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:33:40,709 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:33:40,709 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:33:40,849 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:33:45,850 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:33:50,851 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:33:55,752 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:33:55,753 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:33:56,828 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:34:01,829 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:34:05,710 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:34:07,711 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:34:10,732 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:34:10,733 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:34:12,850 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:34:17,851 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:34:22,852 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:34:25,932 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:34:25,932 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:34:28,032 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:34:33,033 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:34:35,711 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:34:38,712 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:34:40,964 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:34:40,965 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:34:44,073 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:34:49,074 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:34:54,075 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:34:55,944 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:34:55,944 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:34:59,999 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:35:05,000 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:35:05,712 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:35:10,944 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:35:10,944 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:35:11,027 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:35:16,028 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:35:21,029 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:35:25,978 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:35:25,978 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:35:26,122 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:35:31,124 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:35:35,712 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:35:36,713 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:35:40,956 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:35:40,957 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:35:42,094 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:35:47,096 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:35:52,097 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:35:55,979 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:35:55,979 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:35:57,137 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:36:02,138 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:36:05,713 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:36:07,714 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:36:11,089 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:36:11,089 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:36:13,204 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:36:18,205 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:36:23,206 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:36:26,089 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:36:26,089 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:36:28,240 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:36:33,240 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:36:35,714 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:36:38,715 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:36:41,089 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:36:41,089 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:36:44,198 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:36:49,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:36:54,200 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:36:56,089 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:36:56,090 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:36:59,244 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:37:04,245 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:37:05,715 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:37:09,716 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:37:11,089 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:37:11,090 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:37:15,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:37:20,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:37:25,200 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:37:26,253 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:37:26,254 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:37:30,372 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:37:35,373 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:37:35,716 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:37:40,404 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:37:41,253 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:37:41,254 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:37:46,383 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:37:51,384 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:37:56,253 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:37:56,254 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:37:57,324 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:38:02,324 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:38:05,716 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:38:07,717 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:38:11,254 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:38:11,254 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:38:13,355 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:38:18,356 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:38:23,357 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:38:26,254 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:38:26,254 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:38:29,350 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:38:34,351 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:38:35,717 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:38:39,718 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:38:41,302 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:38:41,303 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:38:45,441 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:38:47,016 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 20:38:47,017 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 20:38:47,018 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 20:38:47,018 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 20:38:47,329 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 20:38:51,019 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:38:56,410 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:38:56,410 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:38:56,542 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:39:01,543 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:39:05,718 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:39:06,719 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:39:11,410 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:39:11,410 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:39:12,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:39:17,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:39:22,562 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:39:26,410 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:39:26,410 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:39:27,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:39:32,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:39:35,719 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:39:37,720 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:39:41,484 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:39:41,484 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:39:43,563 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:39:48,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:39:53,565 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:39:56,600 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:39:56,600 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:39:58,676 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:40:03,677 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:40:05,720 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:40:08,721 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:40:11,600 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:40:11,600 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:40:14,701 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:40:19,702 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:40:24,702 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:40:26,600 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:40:26,600 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:40:30,698 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:40:35,698 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:40:35,720 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:40:40,721 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:40:41,713 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:40:41,713 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:40:45,781 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:40:50,782 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:40:55,783 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:40:56,713 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:40:56,714 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:41:00,838 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:41:05,721 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:41:06,722 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:41:11,713 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:41:11,714 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:41:11,802 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:41:16,803 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:41:21,804 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:41:26,728 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:41:26,728 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:41:26,879 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:41:31,880 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:41:35,722 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:41:37,723 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:41:41,756 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:41:41,756 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:41:42,877 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:41:47,878 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:41:52,878 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:41:56,735 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:41:56,736 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:41:58,867 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:42:03,869 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:42:05,723 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:42:09,724 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:42:11,736 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:42:11,736 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:42:14,875 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:42:19,876 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:42:24,876 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:42:26,736 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:42:26,736 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:42:30,858 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:42:35,724 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:42:36,724 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:42:41,784 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:42:41,785 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:42:41,895 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:42:46,897 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:42:51,898 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:42:56,764 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:42:56,765 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:42:57,840 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:43:02,841 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:43:05,725 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:43:08,726 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:43:11,800 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:43:11,800 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:43:13,961 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:43:18,961 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:43:23,962 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:43:26,780 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:43:26,780 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:43:29,887 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:43:34,888 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:43:35,725 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:43:40,726 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:43:41,780 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:43:41,780 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:43:41,896 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 20:43:41,898 DEBUG SenderThread:136 [sender.py:send():378] send: metric 2024-07-04 20:43:41,898 DEBUG SenderThread:136 [sender.py:send():378] send: metric 2024-07-04 20:43:41,898 DEBUG SenderThread:136 [sender.py:send():378] send: metric 2024-07-04 20:43:41,898 DEBUG SenderThread:136 [sender.py:send():378] send: metric 2024-07-04 20:43:41,898 DEBUG SenderThread:136 [sender.py:send():378] send: metric 2024-07-04 20:43:41,899 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 20:43:41,899 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 20:43:41,899 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 20:43:42,432 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 20:43:42,432 INFO Thread-12 :136 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/output.log 2024-07-04 20:43:44,433 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/output.log 2024-07-04 20:43:45,900 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:43:50,902 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:43:56,780 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:43:56,781 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:43:56,950 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:44:01,951 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:44:05,726 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:44:07,727 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:44:11,780 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:44:11,781 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:44:12,905 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:44:13,443 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/config.yaml 2024-07-04 20:44:18,034 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:44:23,035 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:44:26,823 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:44:26,824 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:44:28,961 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:44:33,962 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:44:35,728 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:44:39,729 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:44:41,823 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:44:41,824 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:44:44,916 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:44:49,916 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:44:54,918 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:44:56,824 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:44:56,824 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:45:00,899 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:45:05,727 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:45:06,728 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:45:11,922 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:45:11,922 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:45:12,009 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:45:17,010 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:45:22,011 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:45:27,105 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:45:27,105 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:45:27,253 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:45:32,254 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:45:35,728 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:45:37,729 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:45:42,231 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:45:42,231 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:45:43,322 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:45:48,323 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:45:53,324 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:45:57,232 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:45:57,232 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:45:58,389 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:46:03,390 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:46:05,729 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:46:08,730 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:46:12,231 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:46:12,232 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:46:14,313 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:46:19,315 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:46:24,316 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:46:27,231 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:46:27,232 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:46:29,345 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:46:34,346 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:46:35,735 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:46:39,736 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:46:42,378 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:46:42,379 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:46:45,529 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:46:50,530 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:46:55,531 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:46:57,515 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:46:57,515 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:47:00,647 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:47:05,648 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:47:05,731 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:47:10,732 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:47:12,515 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:47:12,515 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:47:16,625 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:47:21,626 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:47:26,627 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:47:27,515 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:47:27,515 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:47:32,599 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:47:35,733 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:47:37,734 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:47:42,664 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:47:42,664 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:47:42,773 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:47:47,774 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:47:52,775 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:47:57,725 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:47:57,726 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:47:57,883 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:48:02,884 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:48:05,733 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:48:08,734 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:48:12,664 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:48:12,664 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:48:13,793 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:48:18,794 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:48:23,795 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:48:27,664 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:48:27,664 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:48:28,806 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:48:32,303 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 20:48:32,304 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 20:48:32,305 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 20:48:32,305 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 20:48:32,539 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 20:48:34,306 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:48:35,733 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:48:39,734 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:48:42,721 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:48:42,722 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:48:44,785 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:48:49,786 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:48:54,787 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:48:57,721 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:48:57,722 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:48:59,811 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:49:04,812 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:49:05,734 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:49:10,735 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:49:12,722 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:49:12,722 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:49:15,852 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:49:20,853 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:49:25,854 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:49:27,722 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:49:27,722 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:49:31,804 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:49:35,735 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:49:37,736 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:49:42,722 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:49:42,722 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:49:42,935 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:49:47,936 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:49:52,937 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:49:57,777 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:49:57,777 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:49:58,899 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:50:03,900 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:50:05,735 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:50:09,736 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:50:12,777 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:50:12,777 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:50:14,879 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:50:19,880 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:50:24,881 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:50:27,777 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:50:27,777 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:50:30,864 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:50:35,736 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:50:36,737 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:50:41,738 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:50:42,954 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:50:42,955 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:50:47,105 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:50:52,106 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:50:57,107 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:50:58,021 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:50:58,021 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:51:02,126 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:51:05,737 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:51:07,738 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:51:12,995 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:51:12,995 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:51:13,147 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:51:18,148 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:51:23,148 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:51:28,162 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:51:28,163 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:51:28,295 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:51:33,296 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:51:35,738 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:51:38,739 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:51:43,162 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:51:43,163 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:51:44,271 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:51:49,272 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:51:54,273 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:51:58,163 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:51:58,163 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:51:59,286 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:52:04,287 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:52:05,739 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:52:09,739 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:52:13,217 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:52:13,218 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:52:15,370 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:52:20,371 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:52:25,373 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:52:28,196 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:52:28,197 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:52:31,338 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:52:34,984 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: pause 2024-07-04 20:52:34,984 INFO HandlerThread:136 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-04 20:52:34,984 INFO HandlerThread:136 [system_monitor.py:finish():203] Stopping system monitor 2024-07-04 20:52:34,985 INFO HandlerThread:136 [interfaces.py:finish():200] Joined cpu monitor 2024-07-04 20:52:34,985 DEBUG SystemMonitor:136 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-04 20:52:34,985 DEBUG SystemMonitor:136 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-04 20:52:34,986 INFO HandlerThread:136 [interfaces.py:finish():200] Joined disk monitor 2024-07-04 20:52:34,992 INFO HandlerThread:136 [interfaces.py:finish():200] Joined gpu monitor 2024-07-04 20:52:34,992 INFO HandlerThread:136 [interfaces.py:finish():200] Joined memory monitor 2024-07-04 20:52:34,992 INFO HandlerThread:136 [interfaces.py:finish():200] Joined network monitor 2024-07-04 20:52:34,993 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:52:35,214 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: resume 2024-07-04 20:52:35,214 INFO HandlerThread:136 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-04 20:52:35,214 INFO HandlerThread:136 [system_monitor.py:start():194] Starting system monitor 2024-07-04 20:52:35,215 INFO SystemMonitor:136 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-04 20:52:35,215 INFO SystemMonitor:136 [interfaces.py:start():188] Started cpu monitoring 2024-07-04 20:52:35,216 INFO SystemMonitor:136 [interfaces.py:start():188] Started disk monitoring 2024-07-04 20:52:35,216 INFO SystemMonitor:136 [interfaces.py:start():188] Started gpu monitoring 2024-07-04 20:52:35,218 INFO SystemMonitor:136 [interfaces.py:start():188] Started memory monitoring 2024-07-04 20:52:35,220 INFO SystemMonitor:136 [interfaces.py:start():188] Started network monitoring 2024-07-04 20:52:36,554 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:52:36,629 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/output.log 2024-07-04 20:52:41,555 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:52:43,196 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:52:43,197 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:52:45,662 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: pause 2024-07-04 20:52:45,662 INFO HandlerThread:136 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-04 20:52:45,662 INFO HandlerThread:136 [system_monitor.py:finish():203] Stopping system monitor 2024-07-04 20:52:45,662 INFO HandlerThread:136 [interfaces.py:finish():200] Joined cpu monitor 2024-07-04 20:52:45,663 DEBUG SystemMonitor:136 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-04 20:52:45,663 INFO HandlerThread:136 [interfaces.py:finish():200] Joined disk monitor 2024-07-04 20:52:45,663 DEBUG SystemMonitor:136 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-04 20:52:45,664 DEBUG SystemMonitor:136 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-04 20:52:45,670 INFO HandlerThread:136 [interfaces.py:finish():200] Joined gpu monitor 2024-07-04 20:52:45,670 INFO HandlerThread:136 [interfaces.py:finish():200] Joined memory monitor 2024-07-04 20:52:45,670 INFO HandlerThread:136 [interfaces.py:finish():200] Joined network monitor 2024-07-04 20:52:45,671 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:52:46,671 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:52:51,672 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:52:56,674 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:52:58,197 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:52:58,197 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:53:02,357 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:53:07,358 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:53:13,197 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:53:13,197 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:53:13,306 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:53:18,307 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:53:23,308 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:53:28,197 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:53:28,197 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:53:28,362 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:53:33,362 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:53:38,363 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:53:43,197 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:53:43,198 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:53:44,336 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:53:49,337 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:53:54,338 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:53:58,197 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:53:58,198 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:54:00,330 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:54:05,331 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:54:10,333 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:54:13,197 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:54:13,198 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:54:15,345 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:54:20,346 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:54:25,347 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:54:28,198 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:54:28,198 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:54:31,281 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:54:36,282 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:54:41,283 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:54:43,198 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:54:43,198 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:54:47,262 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:54:52,263 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:54:57,264 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:54:58,198 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:54:58,199 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:55:02,351 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:55:07,352 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:55:13,198 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:55:13,199 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:55:13,321 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:55:18,322 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:55:23,323 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:55:28,198 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:55:28,199 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:55:29,260 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:55:34,261 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:55:39,262 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:55:43,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:55:43,199 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:55:44,345 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:55:49,347 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:55:54,348 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:55:58,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:55:58,199 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:56:00,307 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:56:05,308 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:56:10,309 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:56:13,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:56:13,199 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:56:16,296 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:56:16,312 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: resume 2024-07-04 20:56:16,312 INFO HandlerThread:136 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-04 20:56:16,312 INFO HandlerThread:136 [system_monitor.py:start():194] Starting system monitor 2024-07-04 20:56:16,313 INFO SystemMonitor:136 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-04 20:56:16,313 INFO SystemMonitor:136 [interfaces.py:start():188] Started cpu monitoring 2024-07-04 20:56:16,314 INFO SystemMonitor:136 [interfaces.py:start():188] Started disk monitoring 2024-07-04 20:56:16,315 INFO SystemMonitor:136 [interfaces.py:start():188] Started gpu monitoring 2024-07-04 20:56:16,315 INFO SystemMonitor:136 [interfaces.py:start():188] Started memory monitoring 2024-07-04 20:56:16,315 INFO SystemMonitor:136 [interfaces.py:start():188] Started network monitoring 2024-07-04 20:56:21,298 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:56:26,299 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:56:28,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:56:28,200 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:56:32,276 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:56:37,278 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:56:42,278 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:56:43,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:56:43,200 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:56:47,282 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:56:52,284 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:56:57,285 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:56:58,200 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:56:58,200 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:57:03,256 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:57:08,257 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:57:13,200 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:57:13,200 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:57:13,276 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:57:16,316 DEBUG SystemMonitor:136 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-04 20:57:16,317 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:57:18,318 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:57:23,319 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:57:28,200 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:57:28,201 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:57:28,337 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:57:33,338 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:57:38,338 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:57:43,200 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:57:43,200 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:57:44,296 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:57:46,318 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:57:49,319 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:57:54,320 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:57:58,215 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:57:58,215 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:57:59,349 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:58:04,350 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:58:09,351 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:58:13,214 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:58:13,215 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:58:14,364 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:58:16,319 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:58:20,320 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:58:25,321 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:58:28,214 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:58:28,215 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:58:31,311 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:58:36,312 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:58:41,313 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:58:43,215 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:58:43,215 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:58:46,320 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:58:46,320 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:58:51,321 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:58:56,323 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:58:58,246 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:58:58,246 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:59:01,323 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:59:06,325 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:59:11,325 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:59:13,246 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:59:13,246 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:59:16,321 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:59:17,321 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:59:22,322 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:59:27,324 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:59:28,273 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:59:28,274 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:59:32,356 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:59:37,357 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:59:42,357 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:59:43,344 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:59:43,345 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:59:46,321 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 20:59:48,322 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:59:53,323 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 20:59:58,344 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 20:59:58,345 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 20:59:58,491 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:00:03,492 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:00:08,493 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:00:13,344 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:00:13,345 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:00:14,471 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:00:16,322 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:00:20,323 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:00:25,324 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:00:28,345 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:00:28,345 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:00:30,456 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:00:35,457 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:00:40,458 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:00:43,387 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:00:43,388 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:00:45,498 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:00:46,323 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:00:51,324 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:00:56,325 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:00:58,532 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:00:58,532 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:01:01,663 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:01:06,664 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:01:11,665 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:01:12,409 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 21:01:12,410 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 21:01:12,410 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 21:01:12,411 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 21:01:12,825 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 21:01:13,589 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:01:13,590 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:01:16,324 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:01:17,324 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:01:22,325 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:01:27,327 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:01:28,589 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:01:28,590 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:01:32,678 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:01:37,679 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:01:42,680 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:01:43,590 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:01:43,590 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:01:46,324 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:01:48,325 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:01:53,326 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:01:58,590 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:01:58,591 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:01:58,647 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:02:03,648 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:02:08,649 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:02:13,590 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:02:13,590 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:02:14,647 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:02:16,325 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:02:20,326 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:02:25,327 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:02:28,598 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:02:28,599 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:02:30,671 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:02:35,672 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:02:40,673 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:02:43,598 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:02:43,598 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:02:45,723 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:02:46,326 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:02:51,327 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:02:56,328 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:02:58,598 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:02:58,599 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:03:01,717 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:03:06,718 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:03:11,718 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:03:13,598 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:03:13,599 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:03:16,326 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:03:17,327 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:03:22,328 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:03:27,329 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:03:28,685 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:03:28,686 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:03:32,853 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:03:37,853 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:03:43,685 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:03:43,686 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:03:43,775 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:03:46,327 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:03:49,328 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:03:54,329 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:03:58,685 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:03:58,686 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:03:59,844 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:04:04,845 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:04:09,845 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:04:13,685 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:04:13,686 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:04:15,810 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:04:16,328 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:04:21,329 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:04:26,330 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:04:28,686 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:04:28,686 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:04:31,834 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:04:36,835 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:04:41,836 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:04:43,686 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:04:43,686 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:04:46,329 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:04:47,329 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:04:52,330 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:04:57,332 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:04:58,687 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:04:58,687 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:05:02,818 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:05:07,819 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:05:12,820 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:05:13,714 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:05:13,714 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:05:16,330 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:05:18,331 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:05:23,332 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:05:28,714 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:05:28,714 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:05:28,804 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:05:33,805 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:05:38,805 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:05:43,714 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:05:43,714 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:05:43,872 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:05:46,331 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:05:49,331 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:05:54,332 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:05:58,714 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:05:58,714 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:05:59,787 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:06:04,788 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:06:09,788 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:06:13,936 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:06:13,936 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:06:15,096 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:06:16,331 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:06:20,332 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:06:25,333 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:06:28,999 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:06:28,999 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:06:31,108 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:06:36,108 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:06:41,109 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:06:43,999 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:06:43,999 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:06:46,156 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:06:46,332 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:06:51,333 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:06:56,334 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:06:58,999 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:06:58,999 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:07:02,062 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:07:07,063 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:07:12,063 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:07:13,999 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:07:14,000 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:07:16,333 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:07:17,333 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:07:22,335 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:07:27,336 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:07:28,999 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:07:29,000 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:07:33,113 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:07:38,114 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:07:43,115 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:07:43,999 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:07:44,000 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:07:46,334 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:07:48,334 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:07:53,336 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:07:58,999 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:07:59,000 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:07:59,118 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:08:04,119 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:08:09,120 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:08:14,023 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:08:14,023 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:08:15,107 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:08:16,335 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:08:20,336 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:08:25,337 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:08:28,862 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 21:08:28,863 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 21:08:28,864 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 21:08:28,864 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 21:08:28,990 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 21:08:29,175 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:08:29,176 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:08:30,339 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:08:35,340 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:08:40,341 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:08:44,175 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:08:44,176 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:08:46,284 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:08:46,335 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:08:51,336 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:08:56,338 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:08:59,175 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:08:59,176 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:09:02,266 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:09:07,267 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:09:12,269 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:09:14,175 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:09:14,176 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:09:16,336 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:09:17,337 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:09:22,338 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:09:27,339 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:09:29,358 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:09:29,359 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:09:32,519 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:09:37,520 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:09:42,521 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:09:44,358 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:09:44,359 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:09:46,337 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:09:48,338 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:09:53,339 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:09:58,340 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:09:59,377 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:09:59,378 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:10:03,490 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:10:08,491 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:10:13,492 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:10:14,367 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:10:14,368 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:10:16,338 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:10:19,339 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:10:24,340 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:10:29,367 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:10:29,368 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:10:29,527 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:10:34,528 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:10:39,529 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:10:44,367 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:10:44,368 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:10:45,528 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:10:46,339 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:10:51,340 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:10:56,341 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:10:59,368 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:10:59,368 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:11:01,463 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:11:06,464 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:11:11,465 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:11:14,368 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:11:14,368 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:11:16,340 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:11:17,340 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:11:22,341 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:11:27,342 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:11:29,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:11:29,609 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:11:32,670 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:11:37,671 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:11:42,672 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:11:44,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:11:44,609 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:11:46,341 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:11:48,341 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:11:53,343 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:11:58,344 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:11:59,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:11:59,609 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:12:03,705 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:12:08,706 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:12:13,707 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:12:14,609 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:12:14,609 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:12:16,342 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:12:19,342 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:12:24,343 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:12:29,697 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:12:29,697 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:12:29,768 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:12:34,769 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:12:39,770 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:12:44,697 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:12:44,697 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:12:44,833 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:12:46,342 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:12:50,343 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:12:55,344 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:12:59,697 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:12:59,697 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:13:00,776 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:13:05,777 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:13:10,778 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:13:14,904 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:13:14,904 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:13:15,991 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:13:16,343 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:13:21,344 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:13:26,345 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:13:30,037 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:13:30,038 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:13:32,175 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:13:37,176 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:13:42,176 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:13:45,037 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:13:45,038 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:13:46,344 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:13:47,345 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:13:52,346 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:13:57,347 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:14:00,037 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:14:00,038 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:14:03,147 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:14:08,148 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:14:13,149 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:14:15,038 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:14:15,038 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:14:16,349 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:14:18,349 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:14:23,350 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:14:28,352 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:14:30,178 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:14:30,179 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:14:34,338 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:14:39,339 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:14:45,178 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:14:45,179 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:14:45,277 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:14:46,346 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:14:50,347 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:14:55,348 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:15:00,178 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:15:00,179 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:15:01,329 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:15:06,331 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:15:11,332 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:15:15,179 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:15:15,179 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:15:16,348 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:15:16,348 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:15:21,349 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:15:26,350 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:15:30,179 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:15:30,179 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:15:32,312 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:15:37,314 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:15:42,314 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:15:45,179 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:15:45,179 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:15:46,105 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 21:15:46,106 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 21:15:46,107 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 21:15:46,107 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 21:15:46,159 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 21:15:46,347 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:15:47,348 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:15:52,349 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:15:57,350 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:16:00,218 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:16:00,219 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:16:03,292 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:16:08,293 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:16:13,294 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:16:15,198 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:16:15,198 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:16:16,348 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:16:18,349 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:16:23,350 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:16:28,352 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:16:30,303 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:16:30,303 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:16:33,718 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:16:38,719 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:16:43,720 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:16:45,302 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:16:45,303 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:16:46,349 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:16:49,350 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:16:54,351 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:16:59,352 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:17:00,302 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:17:00,303 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:17:04,387 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:17:09,388 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:17:14,389 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:17:15,498 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:17:15,499 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:17:16,349 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:17:20,350 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:17:25,352 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:17:30,498 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:17:30,499 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:17:30,646 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:17:35,647 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:17:40,648 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:17:45,507 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:17:45,508 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:17:46,350 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:17:46,351 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:17:51,352 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:17:56,353 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:18:00,507 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:18:00,508 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:18:01,663 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:18:06,664 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:18:11,664 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:18:15,611 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:18:15,611 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:18:16,351 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:18:17,352 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:18:22,353 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:18:27,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:18:30,611 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:18:30,611 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:18:32,767 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:18:37,767 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:18:42,768 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:18:45,611 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:18:45,611 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:18:46,352 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:18:48,353 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:18:53,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:18:58,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:19:00,611 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:19:00,612 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:19:03,734 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:19:08,735 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:19:13,736 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:19:15,689 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:19:15,690 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:19:16,353 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:19:19,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:19:24,355 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:19:29,355 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:19:30,690 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:19:30,690 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:19:34,845 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:19:39,846 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:19:45,690 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:19:45,690 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:19:45,811 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:19:46,354 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:19:51,355 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:19:56,356 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:20:00,887 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:20:00,888 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:20:01,942 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:20:06,943 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:20:11,944 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:20:15,887 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:20:15,888 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:20:16,354 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:20:17,355 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:20:22,356 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:20:27,357 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:20:31,102 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:20:31,103 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:20:33,170 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:20:38,171 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:20:43,172 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:20:46,102 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:20:46,104 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:20:46,355 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:20:48,357 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:20:53,358 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:20:58,359 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:21:01,102 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:21:01,103 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:21:04,250 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:21:09,251 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:21:14,252 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:21:16,103 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:21:16,103 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:21:16,356 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:21:19,357 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:21:24,358 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:21:29,359 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:21:31,103 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:21:31,103 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:21:35,185 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:21:40,186 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:21:45,187 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:21:46,103 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:21:46,104 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:21:46,357 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:21:50,358 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:21:55,358 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:22:01,103 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:22:01,104 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:22:01,226 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:22:06,227 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:22:11,228 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:22:16,104 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:22:16,104 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:22:16,266 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:22:16,357 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:22:21,358 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:22:26,359 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:22:31,104 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:22:31,104 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:22:32,258 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:22:37,259 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:22:42,260 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:22:46,104 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:22:46,104 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:22:46,358 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:22:47,359 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:22:52,360 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:22:57,361 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:23:01,104 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:23:01,104 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:23:02,173 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 21:23:02,174 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 21:23:02,174 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 21:23:02,175 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 21:23:02,322 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 21:23:03,175 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:23:08,176 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:23:13,177 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:23:16,104 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:23:16,105 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:23:16,359 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:23:18,360 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:23:23,361 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:23:28,361 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:23:31,104 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:23:31,105 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:23:34,253 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:23:39,253 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:23:44,254 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:23:46,135 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:23:46,135 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:23:46,360 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:23:49,361 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:23:54,362 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:23:59,363 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:24:01,115 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:24:01,115 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:24:04,407 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:24:09,409 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:24:14,410 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:24:16,129 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:24:16,129 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:24:16,361 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:24:20,362 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:24:25,362 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:24:30,363 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:24:31,235 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:24:31,235 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:24:36,304 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:24:41,304 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:24:46,235 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:24:46,235 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:24:46,337 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:24:46,361 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:24:51,362 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:24:56,363 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:25:01,235 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:25:01,235 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:25:02,306 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:25:07,307 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:25:12,308 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:25:16,235 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:25:16,235 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:25:16,362 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:25:17,363 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:25:22,364 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:25:27,365 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:25:31,235 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:25:31,235 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:25:33,300 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:25:38,301 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:25:43,302 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:25:46,235 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:25:46,236 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:25:46,363 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:25:48,364 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:25:53,364 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:25:58,365 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:26:01,235 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:26:01,236 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:26:03,388 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:26:08,389 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:26:13,390 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:26:16,238 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:26:16,238 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:26:16,364 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:26:19,364 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:26:24,365 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:26:29,366 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:26:31,437 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:26:31,437 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:26:34,589 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:26:39,590 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:26:44,591 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:26:46,364 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:26:46,437 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:26:46,437 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:26:50,590 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:26:55,591 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:27:00,592 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:27:01,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:27:01,541 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:27:05,657 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:27:10,658 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:27:15,659 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:27:16,365 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:27:16,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:27:16,541 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:27:21,630 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:27:26,631 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:27:31,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:27:31,542 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:27:31,706 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:27:36,707 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:27:41,708 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:27:46,366 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:27:46,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:27:46,542 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:27:47,636 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:27:52,637 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:27:57,638 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:28:01,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:28:01,542 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:28:02,709 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:28:07,710 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:28:12,711 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:28:16,367 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:28:16,607 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:28:16,607 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:28:18,710 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:28:23,711 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:28:28,712 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:28:31,607 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:28:31,607 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:28:34,682 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:28:39,684 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:28:44,685 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:28:46,368 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:28:46,607 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:28:46,607 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:28:49,761 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:28:54,762 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:28:59,762 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:29:01,791 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:29:01,791 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:29:04,913 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:29:09,914 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:29:14,915 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:29:16,368 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:29:16,791 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:29:16,791 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:29:20,914 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:29:25,915 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:29:30,916 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:29:31,791 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:29:31,791 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:29:36,869 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:29:41,870 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:29:46,369 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:29:46,791 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:29:46,791 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:29:46,937 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:29:51,938 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:29:56,939 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:30:01,805 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:30:01,805 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:30:01,945 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:30:06,946 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:30:11,947 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:30:16,370 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:30:16,803 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:30:16,804 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:30:17,875 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:30:18,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 21:30:18,244 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 21:30:18,244 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 21:30:18,244 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 21:30:18,483 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 21:30:23,245 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:30:28,246 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:30:31,803 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:30:31,804 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:30:33,908 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:30:38,909 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:30:43,910 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:30:46,371 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:30:46,803 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:30:46,804 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:30:48,962 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:30:53,963 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:30:58,964 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:31:01,804 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:31:01,804 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:31:04,958 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:31:09,959 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:31:14,960 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:31:16,371 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:31:16,916 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:31:16,917 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:31:20,059 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:31:25,060 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:31:30,060 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:31:31,916 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:31:31,917 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:31:36,023 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:31:41,024 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:31:46,025 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:31:46,372 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:31:46,916 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:31:46,917 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:31:51,052 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:31:56,053 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:32:01,055 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:32:01,932 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:32:01,933 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:32:06,086 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:32:11,087 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:32:16,088 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:32:16,373 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:32:16,922 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:32:16,923 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:32:22,079 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:32:27,079 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:32:31,922 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:32:31,923 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:32:33,037 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:32:38,038 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:32:43,039 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:32:46,374 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:32:46,938 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:32:46,939 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:32:48,049 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:32:53,050 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:32:58,051 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:33:02,092 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:33:02,092 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:33:03,163 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:33:08,164 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:33:13,165 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:33:16,375 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:33:17,092 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:33:17,092 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:33:18,186 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:33:23,187 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:33:28,187 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:33:32,092 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:33:32,092 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:33:33,213 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:33:38,214 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:33:43,215 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:33:46,375 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:33:47,092 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:33:47,093 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:33:48,233 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:33:53,233 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:33:58,235 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:34:02,092 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:34:02,093 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:34:03,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:34:08,244 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:34:13,245 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:34:16,376 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:34:17,092 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:34:17,093 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:34:19,145 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:34:24,146 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:34:29,147 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:34:32,092 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:34:32,093 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:34:34,220 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:34:39,221 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:34:44,222 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:34:46,377 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:34:47,092 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:34:47,093 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:34:49,257 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:34:54,257 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:34:59,258 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:35:02,093 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:35:02,093 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:35:05,156 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:35:10,156 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:35:15,157 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:35:16,378 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:35:17,093 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:35:17,093 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:35:20,227 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:35:25,228 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:35:30,229 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:35:32,093 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:35:32,093 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:35:35,242 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:35:40,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:35:45,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:35:46,378 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:35:47,111 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:35:47,111 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:35:51,194 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:35:56,195 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:36:01,196 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:36:02,101 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:36:02,101 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:36:06,229 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:36:11,230 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:36:16,231 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:36:16,379 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:36:17,135 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:36:17,136 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:36:21,280 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:36:26,281 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:36:31,282 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:36:32,115 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:36:32,116 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:36:37,257 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:36:42,257 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:36:46,380 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:36:47,115 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:36:47,116 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:36:48,191 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:36:53,191 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:36:58,193 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:37:02,116 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:37:02,116 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:37:03,231 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:37:08,232 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:37:13,233 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:37:16,381 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:37:17,116 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:37:17,116 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:37:18,253 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:37:23,253 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:37:28,254 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:37:32,116 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:37:32,116 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:37:33,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:37:35,185 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 21:37:35,186 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 21:37:35,187 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 21:37:35,187 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 21:37:35,643 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/output.log 2024-07-04 21:37:35,643 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 21:37:39,188 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:37:44,189 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:37:46,381 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:37:47,116 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:37:47,117 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:37:50,179 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:37:55,180 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:38:00,181 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:38:02,116 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:38:02,117 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:38:05,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:38:10,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:38:15,245 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:38:17,128 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:38:17,203 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:38:17,204 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:38:20,352 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:38:25,353 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:38:30,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:38:32,128 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:38:32,128 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:38:36,212 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:38:41,213 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:38:46,213 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:38:46,383 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:38:47,128 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:38:47,128 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:38:51,265 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:38:56,266 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:39:01,267 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:39:02,128 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:39:02,128 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:39:06,288 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:39:11,289 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:39:16,290 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:39:16,385 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:39:17,358 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:39:17,359 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:39:21,509 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:39:26,510 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:39:31,511 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:39:32,500 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:39:32,500 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:39:36,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:39:41,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:39:46,384 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:39:46,974 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:39:47,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:39:47,569 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:39:52,737 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:39:57,738 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:40:02,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:40:02,569 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:40:03,689 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:40:08,690 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:40:13,691 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:40:16,385 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:40:16,614 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 21:40:16,615 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 21:40:16,615 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 21:40:16,616 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 21:40:16,702 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 21:40:17,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:40:17,569 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:40:18,757 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:40:23,758 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:40:28,759 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:40:32,782 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:40:32,783 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:40:33,842 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:40:38,843 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:40:43,843 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:40:46,385 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:40:47,782 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:40:47,783 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:40:48,948 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:40:53,950 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:40:58,950 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:41:02,782 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:41:02,783 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:41:04,912 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:41:09,912 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:41:14,913 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:41:16,386 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:41:17,782 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:41:17,783 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:41:20,852 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:41:25,852 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:41:30,853 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:41:32,783 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:41:32,783 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:41:35,871 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:41:40,872 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:41:45,873 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:41:46,387 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:41:47,872 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:41:47,873 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:41:50,974 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:41:55,975 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:42:00,976 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:42:02,872 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:42:02,873 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:42:06,037 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:42:11,038 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:42:16,039 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:42:16,388 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:42:17,872 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:42:17,873 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:42:22,016 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:42:27,017 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:42:32,873 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:42:32,873 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:42:32,985 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:42:37,985 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:42:42,986 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:42:46,388 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:42:48,095 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:42:48,095 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:42:48,206 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:42:53,206 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:42:58,207 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:43:03,094 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:43:03,095 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:43:04,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:43:09,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:43:14,201 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:43:16,389 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:43:18,095 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:43:18,095 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:43:20,195 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:43:25,196 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:43:30,197 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:43:33,136 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:43:33,136 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:43:35,208 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:43:40,209 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:43:45,210 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:43:46,390 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:43:48,216 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:43:48,216 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:43:50,348 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:43:55,349 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:44:00,350 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:44:03,216 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:44:03,216 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:44:06,313 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:44:11,314 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:44:16,315 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:44:16,391 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:44:18,318 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:44:18,319 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:44:21,472 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:44:26,472 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:44:31,473 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:44:33,318 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:44:33,319 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:44:36,478 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:44:41,479 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:44:46,391 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:44:47,392 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:44:48,319 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:44:48,319 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:44:52,431 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:44:57,432 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:45:02,432 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:45:03,319 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:45:03,319 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:45:07,481 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:45:12,481 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:45:16,392 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:45:17,750 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:45:18,319 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:45:18,319 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:45:23,444 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:45:28,446 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:45:33,319 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:45:33,319 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:45:33,453 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:45:38,454 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:45:43,455 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:45:46,393 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:45:48,319 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:45:48,320 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:45:49,372 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:45:54,373 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:45:59,374 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:46:03,328 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:46:03,329 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:46:04,482 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:46:09,483 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:46:14,484 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:46:16,394 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:46:18,328 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:46:18,329 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:46:20,418 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:46:25,419 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:46:30,420 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:46:33,329 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:46:33,329 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:46:35,495 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:46:40,496 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:46:45,497 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:46:46,394 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:46:48,329 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:46:48,329 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:46:51,426 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:46:56,427 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:47:01,428 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:47:03,329 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:47:03,329 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:47:06,482 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:47:11,483 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:47:16,395 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:47:17,396 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:47:18,329 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:47:18,329 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:47:22,479 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:47:27,480 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:47:32,437 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 21:47:32,439 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 21:47:32,439 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 21:47:32,439 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 21:47:32,858 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 21:47:33,175 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:47:33,351 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:47:33,351 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:47:38,442 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:47:43,443 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:47:46,396 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:47:48,340 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:47:48,340 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:47:48,481 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:47:53,482 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:47:58,483 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:48:03,340 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:48:03,340 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:48:03,495 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:48:08,496 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:48:13,497 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:48:16,397 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:48:18,434 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:48:18,434 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:48:18,579 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:48:23,580 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:48:28,581 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:48:33,434 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:48:33,434 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:48:33,593 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:48:38,594 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:48:43,595 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:48:46,398 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:48:48,434 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:48:48,434 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:48:49,547 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:48:54,548 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:48:59,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:49:03,434 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:49:03,435 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:49:05,504 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:49:10,505 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:49:15,506 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:49:16,398 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:49:18,435 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:49:18,435 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:49:20,576 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:49:25,578 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:49:30,578 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:49:33,489 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:49:33,490 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:49:35,637 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:49:40,639 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:49:45,640 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:49:46,399 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:49:48,487 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:49:48,488 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:49:50,653 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:49:55,654 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:50:00,656 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:50:03,508 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:50:03,509 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:50:06,606 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:50:11,607 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:50:16,400 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:50:17,401 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:50:18,507 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:50:18,507 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:50:22,658 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:50:27,659 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:50:32,660 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:50:33,507 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:50:33,507 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:50:38,628 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:50:43,629 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:50:46,401 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:50:48,507 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:50:48,507 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:50:48,657 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:50:53,658 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:50:58,659 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:51:03,507 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:51:03,508 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:51:04,582 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:51:09,583 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:51:14,584 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:51:16,402 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:51:18,507 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:51:18,508 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:51:19,605 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:51:24,605 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:51:29,606 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:51:33,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:51:33,568 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:51:34,723 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:51:39,724 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:51:44,725 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:51:46,402 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:51:48,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:51:48,568 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:51:50,678 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:51:55,679 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:52:00,680 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:52:03,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:52:03,568 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:52:06,649 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:52:11,651 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:52:16,403 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:52:17,404 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:52:18,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:52:18,568 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:52:22,702 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:52:27,703 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:52:32,704 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:52:33,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:52:33,568 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:52:38,703 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:52:43,704 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:52:46,404 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:52:48,614 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:52:48,614 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:52:48,786 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:52:53,788 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:52:58,789 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:53:03,734 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:53:03,734 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:53:03,798 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:53:08,799 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:53:13,800 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:53:16,405 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:53:18,733 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:53:18,734 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:53:18,880 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:53:23,881 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:53:28,881 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:53:33,734 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:53:33,734 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:53:34,875 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:53:39,876 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:53:44,876 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:53:46,406 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:53:48,734 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:53:48,734 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:53:50,850 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:53:55,850 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:54:00,851 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:54:03,760 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:54:03,761 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:54:05,916 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:54:10,917 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:54:15,917 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:54:16,406 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:54:18,760 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:54:18,760 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:54:21,908 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:54:26,909 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:54:31,910 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:54:33,760 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:54:33,760 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:54:37,825 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:54:42,827 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:54:46,408 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:54:47,113 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 21:54:47,114 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 21:54:47,114 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 21:54:47,114 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 21:54:48,013 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 21:54:48,115 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:54:48,760 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:54:48,761 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:54:53,840 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:54:58,841 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:55:03,842 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:55:03,912 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:55:03,913 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:55:09,011 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:55:14,012 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:55:16,408 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:55:19,028 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:55:19,028 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:55:19,029 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:55:24,150 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:55:29,151 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:55:34,028 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:55:34,028 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:55:35,094 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:55:40,094 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:55:45,095 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:55:46,409 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:55:49,028 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:55:49,028 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:55:50,179 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:55:55,180 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:56:00,181 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:56:04,028 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:56:04,028 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:56:06,087 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:56:11,088 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:56:16,089 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:56:16,409 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:56:19,028 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:56:19,029 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:56:21,123 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:56:26,123 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:56:31,125 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:56:34,028 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:56:34,029 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:56:36,179 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:56:41,180 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:56:46,180 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:56:46,410 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:56:49,029 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:56:49,029 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:56:52,104 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:56:57,105 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:57:02,106 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:57:04,069 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:57:04,070 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:57:07,177 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:57:12,178 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:57:16,411 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:57:17,412 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:57:19,069 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:57:19,069 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:57:23,134 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:57:28,135 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:57:33,136 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:57:34,105 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:57:34,105 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:57:38,172 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:57:43,173 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:57:46,412 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:57:48,412 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:57:49,103 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:57:49,103 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:57:54,176 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:57:59,177 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:58:04,178 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:58:04,308 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:58:04,309 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:58:09,467 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:58:14,468 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:58:16,412 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:58:19,308 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:58:19,309 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:58:20,417 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:58:25,418 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:58:30,419 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:58:34,309 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:58:34,309 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:58:35,425 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:58:40,426 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:58:45,427 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:58:46,413 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:58:49,309 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:58:49,309 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:58:51,386 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:58:56,387 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:59:01,388 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:59:04,440 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:59:04,440 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:59:06,554 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:59:11,555 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:59:16,414 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:59:17,415 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:59:19,449 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:59:19,450 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:59:22,556 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:59:27,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:59:32,558 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:59:34,450 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:59:34,450 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:59:38,542 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:59:43,543 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:59:46,415 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 21:59:49,416 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:59:49,449 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 21:59:49,450 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 21:59:54,540 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 21:59:59,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:00:04,449 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:00:04,450 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:00:04,570 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:00:09,571 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:00:14,572 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:00:16,416 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:00:19,450 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:00:19,450 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:00:20,509 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:00:25,510 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:00:30,512 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:00:34,450 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:00:34,450 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:00:35,526 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:00:40,527 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:00:45,528 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:00:46,417 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:00:49,450 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:00:49,450 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:00:50,566 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:00:55,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:01:00,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:01:04,508 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:01:04,508 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:01:05,629 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:01:10,630 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:01:15,631 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:01:16,417 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:01:19,519 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:01:19,519 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:01:21,578 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:01:26,579 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:01:31,580 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:01:34,517 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:01:34,517 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:01:36,697 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:01:41,698 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:01:46,418 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:01:47,419 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:01:49,738 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:01:49,739 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:01:52,889 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:01:57,890 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:02:02,891 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:02:03,981 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 22:02:03,982 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 22:02:03,982 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 22:02:03,982 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 22:02:04,183 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 22:02:04,737 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:02:04,738 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:02:08,866 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:02:13,867 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:02:16,423 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:02:19,424 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:02:19,738 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:02:19,738 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:02:24,901 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:02:29,902 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:02:34,868 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:02:34,869 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:02:35,005 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:02:40,006 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:02:45,007 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:02:46,420 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:02:49,896 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:02:49,897 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:02:50,990 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:02:55,992 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:03:00,993 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:03:04,897 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:03:04,897 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:03:06,974 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:03:11,976 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:03:16,422 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:03:17,423 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:03:19,896 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:03:19,897 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:03:22,997 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:03:27,998 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:03:32,999 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:03:34,896 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:03:34,897 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:03:38,022 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:03:43,023 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:03:46,421 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:03:48,422 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:03:49,896 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:03:49,897 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:03:54,049 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:03:59,050 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:04:04,051 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:04:04,938 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:04:04,938 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:04:09,093 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:04:14,094 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:04:16,422 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:04:19,423 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:04:20,060 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:04:20,061 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:04:25,127 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:04:30,128 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:04:35,059 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:04:35,060 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:04:35,211 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:04:40,212 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:04:45,213 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:04:46,423 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:04:50,060 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:04:50,061 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:04:51,200 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:04:56,201 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:05:01,202 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:05:05,060 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:05:05,060 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:05:07,142 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:05:12,143 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:05:16,424 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:05:17,424 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:05:20,244 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:05:20,245 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:05:23,393 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:05:28,394 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:05:33,395 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:05:35,245 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:05:35,245 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:05:39,385 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:05:44,386 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:05:46,424 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:05:49,425 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:05:50,244 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:05:50,245 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:05:55,364 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:06:00,365 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:06:05,245 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:06:05,245 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:06:05,373 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:06:10,373 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:06:15,374 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:06:16,425 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:06:20,273 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:06:20,273 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:06:20,403 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:06:25,404 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:06:30,405 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:06:35,405 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:06:35,490 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:06:35,491 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:06:40,643 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:06:45,644 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:06:46,426 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:06:50,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:06:50,557 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:06:51,639 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:06:56,640 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:07:01,640 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:07:05,556 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:07:05,557 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:07:06,687 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:07:11,688 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:07:16,427 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:07:17,427 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:07:20,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:07:20,557 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:07:22,671 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:07:27,672 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:07:32,673 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:07:35,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:07:35,557 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:07:38,647 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:07:43,647 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:07:46,427 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:07:49,428 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:07:50,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:07:50,557 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:07:54,659 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:07:59,660 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:08:04,661 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:08:05,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:08:05,557 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:08:09,666 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:08:14,667 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:08:16,428 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:08:20,429 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:08:20,605 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:08:20,605 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:08:25,766 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:08:30,767 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:08:35,583 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:08:35,583 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:08:36,704 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:08:41,705 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:08:46,429 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:08:47,429 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:08:50,791 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:08:50,791 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:08:52,929 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:08:57,929 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:09:02,930 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:09:05,791 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:09:05,791 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:09:07,943 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:09:12,945 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:09:16,430 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:09:18,430 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:09:20,791 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:09:20,792 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:09:21,970 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 22:09:21,971 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 22:09:21,971 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 22:09:21,972 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 22:09:22,334 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 22:09:23,972 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:09:28,973 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:09:33,974 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:09:35,791 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:09:35,792 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:09:39,902 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:09:44,902 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:09:46,430 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:09:50,431 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:09:50,791 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:09:50,792 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:09:55,896 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:10:00,897 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:10:05,791 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:10:05,792 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:10:05,913 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:10:10,914 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:10:15,915 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:10:16,431 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:10:20,853 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:10:20,854 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:10:20,966 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:10:25,968 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:10:30,968 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:10:35,853 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:10:35,853 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:10:35,975 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:10:40,976 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:10:45,977 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:10:46,432 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:10:51,032 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:10:51,033 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:10:51,033 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:10:56,195 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:11:01,196 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:11:06,032 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:11:06,033 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:11:07,152 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:11:12,153 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:11:16,433 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:11:17,433 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:11:21,032 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:11:21,033 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:11:23,117 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:11:28,118 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:11:33,119 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:11:36,032 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:11:36,033 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:11:39,093 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:11:44,094 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:11:46,433 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:11:49,434 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:11:51,033 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:11:51,033 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:11:55,096 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:12:00,097 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:12:05,098 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:12:06,033 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:12:06,033 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:12:10,187 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:12:15,188 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:12:16,434 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:12:20,435 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:12:21,033 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:12:21,033 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:12:26,138 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:12:31,139 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:12:36,033 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:12:36,033 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:12:36,188 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:12:41,188 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:12:46,190 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:12:46,435 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:12:51,033 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:12:51,034 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:12:52,095 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:12:57,097 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:13:02,098 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:13:06,033 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:13:06,034 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:13:07,132 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:13:12,133 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:13:16,435 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:13:17,436 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:13:21,034 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:13:21,034 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:13:23,139 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:13:28,140 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:13:33,141 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:13:36,254 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:13:36,254 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:13:38,330 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:13:43,331 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:13:46,436 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:13:48,437 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:13:51,254 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:13:51,254 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:13:54,342 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:13:59,343 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:14:04,344 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:14:06,299 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:14:06,299 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:14:09,452 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:14:14,453 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:14:16,437 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:14:20,438 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:14:21,276 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:14:21,277 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:14:25,452 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:14:30,453 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:14:35,454 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:14:36,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:14:36,355 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:14:40,507 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:14:45,508 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:14:46,437 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:14:51,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:14:51,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:14:51,354 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:14:56,454 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:15:01,455 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:15:06,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:15:06,354 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:15:06,498 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:15:11,499 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:15:16,438 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:15:17,439 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:15:21,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:15:21,355 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:15:22,492 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:15:27,493 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:15:32,493 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:15:36,355 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:15:36,355 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:15:37,784 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:15:42,785 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:15:46,439 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:15:48,440 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:15:51,355 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:15:51,356 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:15:53,517 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:15:58,518 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:16:03,519 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:16:06,593 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:16:06,594 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:16:08,675 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:16:13,676 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:16:16,440 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:16:19,441 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:16:21,593 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:16:21,594 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:16:24,742 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:16:29,743 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:16:34,743 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:16:36,594 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:16:36,594 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:16:40,710 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:16:45,711 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:16:46,440 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:16:51,441 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:16:51,593 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:16:51,594 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:16:56,742 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:17:01,743 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:17:06,594 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:17:06,594 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:17:06,751 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:17:11,752 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:17:16,441 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:17:17,442 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:17:21,594 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:17:21,594 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:17:22,749 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:17:27,750 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:17:32,751 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:17:36,603 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:17:36,604 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:17:37,764 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:17:42,765 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:17:46,442 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:17:48,443 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:17:51,602 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:17:51,603 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:17:53,767 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:17:58,768 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:18:03,769 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:18:06,625 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:18:06,626 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:18:09,729 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:18:14,730 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:18:16,443 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:18:20,444 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:18:21,625 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:18:21,625 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:18:25,757 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:18:30,759 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:18:35,760 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:18:36,625 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:18:36,625 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:18:40,790 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:18:45,791 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:18:46,443 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:18:51,444 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:18:51,625 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:18:51,625 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:18:56,780 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:18:57,678 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 22:18:57,679 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 22:18:57,680 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 22:18:57,680 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 22:18:58,552 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/output.log 2024-07-04 22:18:58,553 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 22:19:02,681 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:19:06,625 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:19:06,625 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:19:07,775 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:19:12,776 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:19:16,444 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:19:18,445 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:19:20,955 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 22:19:20,956 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 22:19:20,957 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 22:19:20,957 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 22:19:21,560 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 22:19:21,625 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:19:21,626 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:19:23,747 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:19:28,748 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:19:33,749 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:19:36,625 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:19:36,626 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:19:38,790 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:19:43,791 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:19:46,445 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:19:49,446 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:19:51,708 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:19:51,709 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:19:54,771 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:19:59,772 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:20:04,773 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:20:06,707 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:20:06,708 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:20:09,832 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:20:14,833 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:20:16,445 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:20:20,447 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:20:21,849 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:20:21,850 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:20:25,913 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:20:30,913 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:20:35,914 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:20:36,849 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:20:36,850 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:20:41,018 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:20:46,019 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:20:46,446 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:20:51,447 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:20:51,849 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:20:51,850 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:20:56,942 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:21:01,943 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:21:06,943 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:21:06,969 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:21:06,970 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:21:12,075 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:21:16,447 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:21:17,448 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:21:21,969 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:21:21,969 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:21:23,078 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:21:28,079 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:21:33,080 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:21:37,108 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:21:37,109 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:21:38,264 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:21:43,265 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:21:46,448 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:21:48,449 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:21:52,108 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:21:52,108 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:21:54,266 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:21:59,267 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:22:04,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:22:07,108 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:22:07,109 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:22:10,237 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:22:15,238 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:22:16,449 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:22:20,450 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:22:22,108 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:22:22,109 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:22:26,265 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:22:31,266 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:22:36,267 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:22:37,109 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:22:37,109 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:22:42,248 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:22:46,449 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:22:47,450 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:22:52,108 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:22:52,109 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:22:53,241 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:22:58,242 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:23:03,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:23:07,109 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:23:07,109 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:23:09,162 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:23:14,163 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:23:16,450 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:23:19,451 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:23:22,109 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:23:22,110 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:23:25,259 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:23:30,260 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:23:35,261 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:23:37,109 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:23:37,110 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:23:41,243 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:23:46,244 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:23:46,451 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:23:51,452 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:23:52,278 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:23:52,279 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:23:57,431 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:24:02,432 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:24:07,278 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:24:07,278 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:24:08,421 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:24:13,422 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:24:16,452 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:24:18,453 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:24:22,278 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:24:22,279 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:24:24,429 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:24:29,430 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:24:34,431 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:24:37,463 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:24:37,464 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:24:39,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:24:44,562 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:24:46,453 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:24:50,454 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:24:52,463 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:24:52,463 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:24:55,622 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:25:00,623 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:25:05,624 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:25:07,463 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:25:07,463 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:25:11,566 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:25:16,453 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:25:17,454 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:25:22,455 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:25:22,463 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:25:22,463 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:25:27,559 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:25:32,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:25:37,463 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:25:37,464 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:25:37,600 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:25:42,601 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:25:46,454 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:25:48,455 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:25:52,509 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:25:52,509 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:25:53,599 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:25:58,599 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:26:03,600 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:26:07,497 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:26:07,497 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:26:09,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:26:14,552 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:26:16,460 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:26:20,461 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:26:22,497 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:26:22,498 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:26:25,657 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:26:30,658 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:26:35,658 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:26:37,081 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 22:26:37,082 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 22:26:37,083 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 22:26:37,083 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 22:26:37,497 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:26:37,498 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:26:37,721 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 22:26:41,617 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:26:46,456 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:26:47,456 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:26:52,457 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:26:52,497 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:26:52,498 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:26:57,583 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:27:02,584 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:27:07,498 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:27:07,498 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:27:08,579 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:27:13,580 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:27:16,458 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:27:19,459 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:27:22,498 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:27:22,498 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:27:24,664 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:27:29,665 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:27:34,666 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:27:37,552 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:27:37,552 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:27:39,697 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:27:44,698 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:27:46,457 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:27:50,458 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:27:52,532 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:27:52,532 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:27:55,629 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:28:00,630 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:28:05,631 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:28:07,532 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:28:07,532 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:28:10,669 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:28:15,670 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:28:16,458 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:28:21,459 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:28:22,577 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:28:22,577 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:28:26,683 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:28:31,683 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:28:36,684 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:28:37,593 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:28:37,594 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:28:41,743 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:28:46,459 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:28:47,460 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:28:52,461 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:28:52,593 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:28:52,594 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:28:57,754 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:29:02,755 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:29:07,594 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:29:07,594 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:29:08,738 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:29:13,739 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:29:16,460 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:29:19,461 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:29:22,594 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:29:22,594 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:29:24,738 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:29:29,739 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:29:34,740 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:29:37,594 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:29:37,595 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:29:39,749 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:29:44,750 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:29:46,460 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:29:50,461 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:29:52,594 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:29:52,594 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:29:55,674 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:30:00,674 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:30:05,675 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:30:07,611 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:30:07,611 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:30:10,752 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:30:15,752 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:30:16,461 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:30:21,463 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:30:22,645 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:30:22,646 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:30:26,783 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:30:31,784 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:30:36,784 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:30:37,645 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:30:37,645 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:30:42,735 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:30:46,462 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:30:48,463 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:30:52,645 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:30:52,646 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:30:53,723 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:30:58,723 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:31:03,724 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:31:07,646 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:31:07,646 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:31:08,801 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:31:13,802 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:31:16,463 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:31:19,464 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:31:22,646 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:31:22,646 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:31:24,752 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:31:29,753 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:31:34,754 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:31:37,646 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:31:37,647 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:31:39,767 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:31:44,767 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:31:46,464 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:31:50,465 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:31:52,742 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:31:52,742 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:31:55,837 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:32:00,838 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:32:05,839 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:32:07,741 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:32:07,742 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:32:11,001 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:32:16,002 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:32:16,464 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:32:21,466 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:32:22,742 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:32:22,742 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:32:26,817 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:32:31,818 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:32:36,819 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:32:37,752 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:32:37,752 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:32:41,871 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:32:46,465 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:32:47,466 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:32:52,467 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:32:52,798 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:32:52,799 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:32:57,880 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:33:02,880 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:33:07,823 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:33:07,824 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:33:07,978 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:33:12,980 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:33:16,466 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:33:18,467 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:33:22,822 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:33:22,823 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:33:23,957 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:33:28,958 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:33:33,959 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:33:37,822 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:33:37,823 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:33:39,954 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:33:44,955 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:33:46,467 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:33:50,468 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:33:52,823 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:33:52,823 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:33:54,215 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 22:33:54,217 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 22:33:54,217 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 22:33:54,217 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 22:33:54,890 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 22:33:56,218 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:34:01,219 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:34:06,220 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:34:07,823 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:34:07,823 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:34:11,948 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:34:16,468 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:34:17,468 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:34:22,469 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:34:22,823 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:34:22,823 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:34:27,929 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:34:32,929 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:34:37,823 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:34:37,824 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:34:37,931 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:34:42,932 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:34:46,468 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:34:48,469 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:34:52,823 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:34:52,824 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:34:53,974 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:34:58,975 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:35:03,976 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:35:07,878 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:35:07,878 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:35:09,018 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:35:14,018 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:35:16,469 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:35:19,470 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:35:23,064 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:35:23,065 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:35:25,127 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:35:30,128 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:35:35,129 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:35:38,172 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:35:38,173 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:35:40,282 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:35:45,283 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:35:46,470 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:35:50,471 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:35:53,172 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:35:53,173 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:35:56,277 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:36:01,278 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:36:06,279 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:36:08,172 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:36:08,173 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:36:12,271 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:36:16,471 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:36:17,471 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:36:22,472 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:36:23,310 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:36:23,311 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:36:28,401 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:36:33,402 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:36:38,309 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:36:38,310 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:36:38,424 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:36:43,425 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:36:46,471 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:36:48,472 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:36:53,309 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:36:53,310 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:36:54,422 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:36:59,422 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:37:04,424 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:37:08,309 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:37:08,310 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:37:10,407 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:37:15,408 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:37:16,472 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:37:20,473 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:37:23,310 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:37:23,310 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:37:26,410 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:37:31,411 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:37:36,413 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:37:38,310 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:37:38,310 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:37:41,841 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:37:46,473 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:37:47,474 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:37:52,475 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:37:53,310 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:37:53,311 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:37:58,394 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:38:03,395 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:38:08,310 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:38:08,311 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:38:08,441 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:38:13,442 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:38:16,473 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:38:18,474 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:38:23,311 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:38:23,312 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:38:24,458 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:38:29,459 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:38:34,460 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:38:38,457 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:38:38,458 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:38:39,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:38:44,571 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:38:46,474 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:38:50,475 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:38:53,457 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:38:53,458 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:38:55,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:39:00,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:39:05,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:39:08,457 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:39:08,458 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:39:11,535 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:39:16,475 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:39:17,476 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:39:22,477 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:39:23,457 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:39:23,458 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:39:27,924 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:39:32,925 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:39:37,926 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:39:38,664 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:39:38,665 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:39:43,814 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:39:46,476 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:39:49,477 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:39:53,664 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:39:53,664 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:39:55,058 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:40:00,059 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:40:05,061 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:40:08,664 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:40:08,664 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:40:10,723 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:40:15,723 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:40:16,477 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:40:21,478 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:40:23,782 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:40:23,783 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:40:27,264 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:40:32,265 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:40:37,266 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:40:38,782 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:40:38,782 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:40:42,392 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:40:46,477 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:40:47,478 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:40:52,479 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:40:53,782 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:40:53,782 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:40:58,247 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:41:03,249 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:41:08,250 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:41:08,782 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:41:08,783 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:41:10,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 22:41:10,542 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 22:41:10,542 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 22:41:10,543 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 22:41:11,100 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 22:41:13,543 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:41:16,478 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:41:19,479 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:41:23,817 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:41:23,818 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:41:24,878 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:41:29,879 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:41:34,879 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:41:38,796 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:41:38,797 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:41:39,962 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:41:44,963 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:41:46,479 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:41:50,480 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:41:53,796 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:41:53,797 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:41:55,850 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:42:00,851 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:42:05,851 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:42:08,797 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:42:08,797 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:42:10,891 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:42:15,892 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:42:16,479 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:42:21,481 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:42:23,797 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:42:23,797 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:42:26,943 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:42:31,944 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:42:36,944 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:42:38,797 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:42:38,797 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:42:42,926 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:42:46,480 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:42:48,481 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:42:53,482 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:42:53,903 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:42:53,904 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:42:59,046 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:43:04,047 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:43:08,942 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:43:08,943 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:43:09,184 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:43:14,185 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:43:16,481 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:43:19,482 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:43:23,942 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:43:23,942 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:43:24,992 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:43:29,993 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:43:34,994 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:43:38,942 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:43:38,943 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:43:40,080 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:43:45,081 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:43:46,482 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:43:50,483 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:43:53,942 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:43:53,943 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:43:56,062 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:44:01,063 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:44:06,063 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:44:08,942 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:44:08,943 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:44:12,053 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:44:16,482 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:44:17,483 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:44:22,484 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:44:24,155 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:44:24,156 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:44:28,208 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:44:33,209 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:44:38,210 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:44:39,155 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:44:39,155 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:44:43,589 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:44:46,483 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:44:49,484 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:44:54,155 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:44:54,155 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:44:54,616 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:44:59,617 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:45:04,618 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:45:09,155 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:45:09,156 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:45:14,156 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:45:14,242 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:45:16,484 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:45:19,485 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:45:24,155 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:45:24,155 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:45:24,156 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:45:25,272 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:45:30,273 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:45:35,274 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:45:39,155 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:45:39,156 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:45:40,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:45:45,355 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:45:46,485 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:45:50,486 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:45:54,155 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:45:54,156 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:45:56,286 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:46:01,286 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:46:06,287 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:46:09,155 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:46:09,156 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:46:12,267 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:46:16,486 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:46:17,486 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:46:22,487 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:46:24,156 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:46:24,156 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:46:28,214 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:46:33,216 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:46:38,217 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:46:39,169 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:46:39,169 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:46:39,170 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:46:43,327 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:46:46,486 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:46:48,487 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:46:53,488 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:46:54,169 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:46:54,169 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:46:54,170 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:46:59,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:47:04,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:47:09,169 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:47:09,170 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:47:09,170 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:47:10,257 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:47:15,258 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:47:16,487 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:47:20,488 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:47:24,169 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:47:24,170 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:47:24,170 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:47:26,250 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:47:31,251 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:47:36,252 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:47:39,215 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:47:39,216 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:47:39,217 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:47:41,296 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:47:46,297 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:47:46,488 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:47:51,489 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:47:54,351 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:47:54,351 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:47:54,351 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:47:57,448 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:48:02,449 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:48:07,449 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:48:09,488 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:48:09,489 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:48:09,489 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:48:12,576 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:48:16,489 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:48:18,489 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:48:23,491 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:48:24,488 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:48:24,489 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:48:24,489 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:48:28,450 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 22:48:28,451 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 22:48:28,451 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 22:48:28,452 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 22:48:29,305 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 22:48:29,452 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:48:34,453 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:48:39,454 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:48:39,489 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:48:39,489 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:48:39,489 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:48:44,625 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:48:46,489 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:48:50,490 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:48:54,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:48:54,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:48:54,608 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:48:55,728 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:49:00,729 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:49:05,730 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:49:09,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:49:09,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:49:09,608 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:49:10,863 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:49:15,864 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:49:16,490 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:49:21,491 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:49:24,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:49:24,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:49:24,609 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:49:26,669 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:49:31,670 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:49:36,670 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:49:39,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:49:39,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:49:39,609 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:49:41,756 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:49:46,491 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:49:47,492 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:49:52,492 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:49:54,813 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:49:54,813 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:49:54,814 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:49:57,893 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:50:02,894 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:50:07,895 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:50:09,813 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:50:09,813 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:50:09,813 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:50:12,898 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:50:16,495 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:50:18,496 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:50:23,497 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:50:24,813 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:50:24,813 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:50:24,814 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:50:28,891 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:50:33,892 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:50:38,892 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:50:39,813 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:50:39,813 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:50:39,814 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:50:44,023 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:50:46,492 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:50:49,493 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:50:54,494 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:50:54,813 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:50:54,814 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:50:54,814 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:50:59,866 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:51:04,867 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:51:09,813 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:51:09,814 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:51:09,814 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:51:09,910 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:51:14,911 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:51:16,495 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:51:20,496 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:51:24,814 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:51:24,814 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:51:24,814 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:51:25,909 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:51:30,910 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:51:35,911 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:51:40,044 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:51:40,045 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:51:40,045 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:51:41,204 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:51:46,205 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:51:46,494 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:51:51,495 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:51:55,059 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:51:55,071 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:51:55,071 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:51:57,212 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:52:02,213 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:52:07,214 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:52:10,059 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:52:10,069 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:52:10,070 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:52:12,829 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:52:16,495 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:52:18,495 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:52:23,497 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:52:25,059 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:52:25,069 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:52:25,070 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:52:29,131 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:52:34,132 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:52:39,132 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:52:40,060 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:52:40,070 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:52:40,070 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:52:44,224 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:52:46,495 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:52:49,496 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:52:54,497 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:52:55,060 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:52:55,123 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:52:55,123 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:53:00,272 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:53:05,273 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:53:10,150 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:53:10,150 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:53:10,151 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:53:10,297 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:53:15,298 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:53:16,496 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:53:20,497 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:53:25,198 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:53:25,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:53:25,199 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:53:25,843 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:53:30,844 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:53:35,845 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:53:40,198 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:53:40,198 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:53:40,199 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:53:44,902 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:53:46,497 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:53:50,498 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:53:55,198 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:53:55,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:53:55,199 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:53:56,374 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:54:01,375 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:54:06,375 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:54:10,198 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:54:10,199 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:54:10,199 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:54:13,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:54:16,497 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:54:19,498 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:54:24,499 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:54:25,267 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:54:25,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:54:25,268 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:54:29,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:54:34,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:54:39,552 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:54:40,267 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:54:40,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:54:40,268 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:54:44,785 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:54:46,498 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:54:50,499 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:54:55,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:54:55,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:54:55,268 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:54:56,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:55:01,562 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:55:06,563 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:55:10,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:55:10,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:55:10,269 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:55:12,356 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:55:16,499 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:55:17,500 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:55:22,500 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:55:25,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:55:25,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:55:25,269 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:55:28,329 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:55:33,330 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:55:38,330 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:55:40,268 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:55:40,269 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:55:40,269 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:55:43,894 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:55:45,537 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 22:55:45,538 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 22:55:45,539 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 22:55:45,539 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 22:55:46,500 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:55:46,517 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 22:55:49,501 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:55:54,502 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:55:55,291 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:55:55,292 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:55:55,293 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:55:59,735 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:56:04,736 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:56:09,736 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:56:10,456 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:56:10,456 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:56:10,457 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:56:14,823 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:56:16,501 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:56:20,502 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:56:25,456 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:56:25,456 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:56:25,457 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:56:25,571 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:56:30,572 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:56:35,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:56:40,456 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:56:40,456 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:56:40,457 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:56:43,310 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:56:46,501 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:56:48,502 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:56:53,503 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:56:55,456 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:56:55,457 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:56:55,457 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:56:58,716 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:57:03,717 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:57:08,717 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:57:10,457 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:57:10,457 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:57:10,457 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:57:14,598 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:57:16,502 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:57:20,503 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:57:25,496 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:57:25,497 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:57:25,497 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:57:25,571 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:57:30,571 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:57:35,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:57:40,496 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:57:40,496 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:57:40,496 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:57:40,610 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:57:45,611 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:57:46,503 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:57:51,504 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:57:55,519 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:57:55,520 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:57:55,520 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:57:56,679 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:58:01,680 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:58:06,681 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:58:10,509 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:58:10,509 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:58:10,509 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:58:13,501 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:58:16,503 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:58:18,504 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:58:23,506 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:58:25,509 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:58:25,509 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:58:25,510 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:58:28,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:58:33,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:58:38,574 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:58:40,509 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:58:40,509 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:58:40,510 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:58:43,659 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:58:46,504 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:58:49,505 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:58:54,506 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:58:55,534 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:58:55,555 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:58:55,556 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:59:00,264 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:59:05,265 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:59:10,266 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:59:10,534 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:59:10,545 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:59:10,545 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:59:15,639 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:59:16,505 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:59:21,506 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:59:25,535 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:59:25,545 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:59:25,545 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:59:26,614 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:59:31,616 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:59:36,616 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:59:40,545 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:59:40,547 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:59:40,547 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:59:41,689 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:59:46,506 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 22:59:47,507 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:59:52,508 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 22:59:55,546 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 22:59:55,546 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 22:59:55,547 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 22:59:57,689 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:00:02,690 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:00:07,691 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:00:10,546 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:00:10,546 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 23:00:10,547 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 23:00:13,645 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:00:16,507 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:00:19,508 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:00:24,489 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 23:00:24,490 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 23:00:24,491 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:00:24,491 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:00:24,649 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 23:00:25,492 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:00:25,559 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:00:25,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 23:00:25,560 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 23:00:25,649 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/output.log 2024-07-04 23:00:30,742 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:00:35,743 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:00:40,559 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:00:40,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 23:00:40,560 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 23:00:41,649 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:00:46,507 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:00:47,508 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:00:52,509 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:00:55,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:00:55,608 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 23:00:55,609 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 23:00:57,670 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:01:02,671 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:01:07,671 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:01:10,645 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:01:10,656 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 23:01:10,656 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: stop_status 2024-07-04 23:01:15,894 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:01:21,066 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:01:26,227 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:01:31,393 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:01:36,481 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:01:41,649 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:01:46,810 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:01:51,920 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:01:56,995 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:02,300 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:07,469 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:11,621 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'complete': False, 'failed': False, 'dropped': 0, 'uploaded': []}} 2024-07-04 23:02:12,637 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:14,253 INFO SenderThread:136 [retry.py:__call__():172] Retry attempt failed: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request response = conn.getresponse() File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 461, in getresponse httplib_response = super().getresponse() File "/opt/conda/lib/python3.10/http/client.py", line 1375, in getresponse response.begin() File "/opt/conda/lib/python3.10/http/client.py", line 318, in begin version, status, reason = self._read_status() File "/opt/conda/lib/python3.10/http/client.py", line 279, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/opt/conda/lib/python3.10/socket.py", line 705, in readinto return self._sock.recv_into(b) File "/opt/conda/lib/python3.10/ssl.py", line 1307, in recv_into return self.read(nbytes, buffer) File "/opt/conda/lib/python3.10/ssl.py", line 1163, in read return self._sslobj.read(len, buffer) TimeoutError: The read operation timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen retries = retries.increment( File "/opt/conda/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment raise reraise(type(error), error, _stacktrace) File "/opt/conda/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise raise value File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen response = self._make_request( File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 538, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Read timed out. (read timeout=20) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 131, in __call__ result = self._call_fn(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 340, in execute return self.client.execute(*args, **kwargs) # type: ignore File "/opt/conda/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute result = self._get_result(document, *args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result return self.transport.execute(document, *args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/gql_request.py", line 58, in execute request = self.session.post(self.url, **post_args) File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, **kwargs) File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 713, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.wandb.ai', port=443): Read timed out. (read timeout=20) 2024-07-04 23:02:17,790 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:22,903 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:28,048 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:28,048 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:33,205 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:38,370 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:43,534 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:48,689 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:53,845 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:02:59,014 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:03,253 ERROR SenderThread:136 [internal_api.py:execute():344] 500 response executing GraphQL. 2024-07-04 23:03:03,253 ERROR SenderThread:136 [internal_api.py:execute():345] {"error":"An internal error occurred. Please contact support."} 2024-07-04 23:03:04,163 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:09,067 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'complete': False, 'failed': False, 'dropped': 0, 'uploaded': []}} 2024-07-04 23:03:09,261 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:14,285 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:19,511 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:21,955 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'complete': False, 'failed': False, 'dropped': 0, 'uploaded': []}} 2024-07-04 23:03:24,682 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:29,430 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:03:29,431 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:03:29,431 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:03:29,432 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:03:29,432 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:03:29,432 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:03:29,856 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:31,728 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/output.log 2024-07-04 23:03:34,434 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:03:35,022 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:39,435 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:03:40,114 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:44,435 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:03:45,284 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:45,284 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:46,512 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:03:49,513 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:03:50,452 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:54,514 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:03:55,603 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:03:59,515 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:04:00,674 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:04:04,516 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:04:05,775 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:04:09,516 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:04:10,942 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:04:14,517 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:04:16,113 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:04:16,513 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:04:20,515 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:04:21,279 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:04:25,516 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:04:26,442 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:04:30,517 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:04:31,612 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:04:35,518 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:04:36,768 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:04:40,519 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:04:41,929 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:04:45,520 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:04:46,514 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:04:47,100 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:04:51,515 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:04:52,284 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:04:56,516 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:04:57,376 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:05:01,516 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:05:02,586 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:05:06,517 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:05:07,894 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:05:11,518 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:05:11,554 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 352, 'content': ['{"system.network.sent": 538277871.07, "system.network.recv": 44293762.33, "system.cpu": 25.02, "system.cpu.0.cpu_percent": 1.9, "system.cpu.1.cpu_percent": 15.0, "system.cpu.2.cpu_percent": 3.84, "system.cpu.3.cpu_percent": 85.97, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27262.31, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3457.09, "system.gpu.0.memoryAllocated": 84.02, "system.gpu.0.memoryAllocatedBytes": 14433797188.27, "system.gpu.0.memory": 17.67, "system.gpu.0.gpu": 98.07, "system.gpu.0.temp": 72.93, "system.gpu.0.powerWatts": 131.51, "system.gpu.0.powerPercent": 52.58, "_wandb": true, "_timestamp": 1720134226.511635, "_runtime": 10841.506206}']}}, 'dropped': 0}} 2024-07-04 23:05:12,977 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:05:16,514 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:05:17,515 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:05:18,138 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:05:22,516 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:05:23,230 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:05:27,517 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:05:28,262 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:05:32,518 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:05:33,353 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:05:37,519 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:05:38,510 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:05:39,954 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 23:05:39,955 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 23:05:39,955 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:05:39,955 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:05:40,778 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 23:05:42,956 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:05:43,687 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:05:46,515 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:05:48,516 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:05:48,850 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:05:53,517 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:05:54,014 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:05:58,517 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:05:59,166 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:06:03,519 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:06:04,358 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:06:08,520 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:06:09,039 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 352, 'content': ['{"system.network.sent": 538277871.07, "system.network.recv": 44293762.33, "system.cpu": 25.02, "system.cpu.0.cpu_percent": 1.9, "system.cpu.1.cpu_percent": 15.0, "system.cpu.2.cpu_percent": 3.84, "system.cpu.3.cpu_percent": 85.97, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27262.31, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3457.09, "system.gpu.0.memoryAllocated": 84.02, "system.gpu.0.memoryAllocatedBytes": 14433797188.27, "system.gpu.0.memory": 17.67, "system.gpu.0.gpu": 98.07, "system.gpu.0.temp": 72.93, "system.gpu.0.powerWatts": 131.51, "system.gpu.0.powerPercent": 52.58, "_wandb": true, "_timestamp": 1720134226.511635, "_runtime": 10841.506206}']}}, 'dropped': 0}} 2024-07-04 23:06:09,442 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:06:13,520 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:06:14,543 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:06:16,516 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:06:19,517 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:06:19,709 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:06:24,517 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:06:24,876 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:06:29,518 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:06:30,039 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:06:34,519 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:06:35,129 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:06:39,520 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:06:40,282 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:06:44,521 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:06:45,444 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:06:46,517 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:06:50,518 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:06:50,612 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:06:55,519 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:06:55,692 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:00,520 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:07:00,859 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:05,521 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:07:06,034 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:10,522 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:07:11,188 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:15,523 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:07:16,205 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:16,517 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:07:21,368 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:21,519 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:07:26,469 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:26,520 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:07:31,521 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:07:31,675 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:36,521 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:07:36,848 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:41,522 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:07:42,016 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:46,518 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:07:47,053 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:47,520 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:07:52,138 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:52,521 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:07:57,311 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:07:57,522 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:08:02,484 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:08:02,523 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:08:07,524 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:08:07,656 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:08:12,525 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:08:12,824 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:08:16,519 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:08:17,980 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:08:18,520 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:08:23,097 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:08:23,521 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:08:28,250 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:08:28,522 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:08:33,421 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:08:33,523 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:08:38,524 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:08:38,583 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:08:43,525 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:08:43,610 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:08:46,520 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:08:48,919 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:08:49,521 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:08:54,240 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:08:54,522 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:08:59,396 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:08:59,522 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:09:04,523 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:09:04,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:09:09,525 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:09:09,725 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:09:14,526 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:09:14,811 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:09:16,521 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:09:19,988 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:09:20,522 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:09:25,145 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:09:25,522 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:09:30,318 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:09:30,523 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:09:33,769 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 360, 'content': ['{"system.network.sent": 539696783.8, "system.network.recv": 45241213.4, "system.proc.memory.availableMB": 27265.78, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.cpu": 25.03, "system.cpu.0.cpu_percent": 2.63, "system.cpu.1.cpu_percent": 54.11, "system.cpu.2.cpu_percent": 6.4, "system.cpu.3.cpu_percent": 43.48, "system.proc.cpu.threads": 38, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.36, "system.gpu.0.memoryAllocated": 61.51, "system.gpu.0.memoryAllocatedBytes": 10567767381.33, "system.gpu.0.memory": 14.13, "system.gpu.0.gpu": 89.6, "system.gpu.0.temp": 72.87, "system.gpu.0.powerWatts": 123.47, "system.gpu.0.powerPercent": 49.55, "_wandb": true, "_timestamp": 1720134466.517741, "_runtime": 11081.512312}', '{"system.network.sent": 539871680.0, "system.network.recv": 45358740.27, "system.cpu": 25.04, "system.cpu.0.cpu_percent": 9.04, "system.cpu.1.cpu_percent": 3.23, "system.cpu.2.cpu_percent": 63.39, "system.cpu.3.cpu_percent": 31.11, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27266.54, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.79, "system.gpu.0.memoryAllocated": 72.77, "system.gpu.0.memoryAllocatedBytes": 12501061905.07, "system.gpu.0.memory": 15.4, "system.gpu.0.gpu": 92.47, "system.gpu.0.temp": 73.07, "system.gpu.0.powerWatts": 140.76, "system.gpu.0.powerPercent": 56.93, "_wandb": true, "_timestamp": 1720134496.51853, "_runtime": 11111.513101}']}}, 'dropped': 0}} 2024-07-04 23:09:35,473 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:09:35,524 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:09:40,525 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:09:40,593 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:09:45,526 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:09:45,751 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:09:46,521 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:09:50,901 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:09:51,523 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:09:56,058 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:09:56,523 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:10:01,078 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:10:01,524 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:10:06,307 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:10:06,525 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:10:11,394 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:10:11,526 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:10:16,522 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:10:16,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:10:17,523 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:10:21,734 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:10:22,390 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 360, 'content': ['{"system.network.sent": 539696783.8, "system.network.recv": 45241213.4, "system.proc.memory.availableMB": 27265.78, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.cpu": 25.03, "system.cpu.0.cpu_percent": 2.63, "system.cpu.1.cpu_percent": 54.11, "system.cpu.2.cpu_percent": 6.4, "system.cpu.3.cpu_percent": 43.48, "system.proc.cpu.threads": 38, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.36, "system.gpu.0.memoryAllocated": 61.51, "system.gpu.0.memoryAllocatedBytes": 10567767381.33, "system.gpu.0.memory": 14.13, "system.gpu.0.gpu": 89.6, "system.gpu.0.temp": 72.87, "system.gpu.0.powerWatts": 123.47, "system.gpu.0.powerPercent": 49.55, "_wandb": true, "_timestamp": 1720134466.517741, "_runtime": 11081.512312}', '{"system.network.sent": 539871680.0, "system.network.recv": 45358740.27, "system.cpu": 25.04, "system.cpu.0.cpu_percent": 9.04, "system.cpu.1.cpu_percent": 3.23, "system.cpu.2.cpu_percent": 63.39, "system.cpu.3.cpu_percent": 31.11, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27266.54, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.79, "system.gpu.0.memoryAllocated": 72.77, "system.gpu.0.memoryAllocatedBytes": 12501061905.07, "system.gpu.0.memory": 15.4, "system.gpu.0.gpu": 92.47, "system.gpu.0.temp": 73.07, "system.gpu.0.powerWatts": 140.76, "system.gpu.0.powerPercent": 56.93, "_wandb": true, "_timestamp": 1720134496.51853, "_runtime": 11111.513101}']}}, 'dropped': 0}} 2024-07-04 23:10:22,524 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:10:26,900 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:10:27,524 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:10:29,648 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 360, 'content': ['{"system.network.sent": 539696783.8, "system.network.recv": 45241213.4, "system.proc.memory.availableMB": 27265.78, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.cpu": 25.03, "system.cpu.0.cpu_percent": 2.63, "system.cpu.1.cpu_percent": 54.11, "system.cpu.2.cpu_percent": 6.4, "system.cpu.3.cpu_percent": 43.48, "system.proc.cpu.threads": 38, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.36, "system.gpu.0.memoryAllocated": 61.51, "system.gpu.0.memoryAllocatedBytes": 10567767381.33, "system.gpu.0.memory": 14.13, "system.gpu.0.gpu": 89.6, "system.gpu.0.temp": 72.87, "system.gpu.0.powerWatts": 123.47, "system.gpu.0.powerPercent": 49.55, "_wandb": true, "_timestamp": 1720134466.517741, "_runtime": 11081.512312}', '{"system.network.sent": 539871680.0, "system.network.recv": 45358740.27, "system.cpu": 25.04, "system.cpu.0.cpu_percent": 9.04, "system.cpu.1.cpu_percent": 3.23, "system.cpu.2.cpu_percent": 63.39, "system.cpu.3.cpu_percent": 31.11, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27266.54, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.79, "system.gpu.0.memoryAllocated": 72.77, "system.gpu.0.memoryAllocatedBytes": 12501061905.07, "system.gpu.0.memory": 15.4, "system.gpu.0.gpu": 92.47, "system.gpu.0.temp": 73.07, "system.gpu.0.powerWatts": 140.76, "system.gpu.0.powerPercent": 56.93, "_wandb": true, "_timestamp": 1720134496.51853, "_runtime": 11111.513101}']}}, 'dropped': 0}} 2024-07-04 23:10:32,058 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:10:32,525 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:10:37,174 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:10:37,526 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:10:42,342 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:10:42,376 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 360, 'content': ['{"system.network.sent": 539696783.8, "system.network.recv": 45241213.4, "system.proc.memory.availableMB": 27265.78, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.cpu": 25.03, "system.cpu.0.cpu_percent": 2.63, "system.cpu.1.cpu_percent": 54.11, "system.cpu.2.cpu_percent": 6.4, "system.cpu.3.cpu_percent": 43.48, "system.proc.cpu.threads": 38, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.36, "system.gpu.0.memoryAllocated": 61.51, "system.gpu.0.memoryAllocatedBytes": 10567767381.33, "system.gpu.0.memory": 14.13, "system.gpu.0.gpu": 89.6, "system.gpu.0.temp": 72.87, "system.gpu.0.powerWatts": 123.47, "system.gpu.0.powerPercent": 49.55, "_wandb": true, "_timestamp": 1720134466.517741, "_runtime": 11081.512312}', '{"system.network.sent": 539871680.0, "system.network.recv": 45358740.27, "system.cpu": 25.04, "system.cpu.0.cpu_percent": 9.04, "system.cpu.1.cpu_percent": 3.23, "system.cpu.2.cpu_percent": 63.39, "system.cpu.3.cpu_percent": 31.11, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27266.54, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.79, "system.gpu.0.memoryAllocated": 72.77, "system.gpu.0.memoryAllocatedBytes": 12501061905.07, "system.gpu.0.memory": 15.4, "system.gpu.0.gpu": 92.47, "system.gpu.0.temp": 73.07, "system.gpu.0.powerWatts": 140.76, "system.gpu.0.powerPercent": 56.93, "_wandb": true, "_timestamp": 1720134496.51853, "_runtime": 11111.513101}']}}, 'dropped': 0}} 2024-07-04 23:10:42,527 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:10:46,523 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:10:47,506 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:10:48,524 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:10:52,678 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:10:53,525 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:10:57,841 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:10:58,525 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:11:00,826 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 360, 'content': ['{"system.network.sent": 539696783.8, "system.network.recv": 45241213.4, "system.proc.memory.availableMB": 27265.78, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.cpu": 25.03, "system.cpu.0.cpu_percent": 2.63, "system.cpu.1.cpu_percent": 54.11, "system.cpu.2.cpu_percent": 6.4, "system.cpu.3.cpu_percent": 43.48, "system.proc.cpu.threads": 38, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.36, "system.gpu.0.memoryAllocated": 61.51, "system.gpu.0.memoryAllocatedBytes": 10567767381.33, "system.gpu.0.memory": 14.13, "system.gpu.0.gpu": 89.6, "system.gpu.0.temp": 72.87, "system.gpu.0.powerWatts": 123.47, "system.gpu.0.powerPercent": 49.55, "_wandb": true, "_timestamp": 1720134466.517741, "_runtime": 11081.512312}', '{"system.network.sent": 539871680.0, "system.network.recv": 45358740.27, "system.cpu": 25.04, "system.cpu.0.cpu_percent": 9.04, "system.cpu.1.cpu_percent": 3.23, "system.cpu.2.cpu_percent": 63.39, "system.cpu.3.cpu_percent": 31.11, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27266.54, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.79, "system.gpu.0.memoryAllocated": 72.77, "system.gpu.0.memoryAllocatedBytes": 12501061905.07, "system.gpu.0.memory": 15.4, "system.gpu.0.gpu": 92.47, "system.gpu.0.temp": 73.07, "system.gpu.0.powerWatts": 140.76, "system.gpu.0.powerPercent": 56.93, "_wandb": true, "_timestamp": 1720134496.51853, "_runtime": 11111.513101}']}}, 'dropped': 0}} 2024-07-04 23:11:02,995 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:11:03,526 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:11:08,156 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:11:08,527 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:11:13,232 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:11:13,528 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:11:16,524 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:11:18,290 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:11:19,525 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:11:23,430 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:11:24,526 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:11:28,434 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:11:29,526 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:11:33,600 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:11:34,527 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:11:38,764 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:11:39,528 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:11:40,854 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 360, 'content': ['{"system.network.sent": 539696783.8, "system.network.recv": 45241213.4, "system.proc.memory.availableMB": 27265.78, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.cpu": 25.03, "system.cpu.0.cpu_percent": 2.63, "system.cpu.1.cpu_percent": 54.11, "system.cpu.2.cpu_percent": 6.4, "system.cpu.3.cpu_percent": 43.48, "system.proc.cpu.threads": 38, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.36, "system.gpu.0.memoryAllocated": 61.51, "system.gpu.0.memoryAllocatedBytes": 10567767381.33, "system.gpu.0.memory": 14.13, "system.gpu.0.gpu": 89.6, "system.gpu.0.temp": 72.87, "system.gpu.0.powerWatts": 123.47, "system.gpu.0.powerPercent": 49.55, "_wandb": true, "_timestamp": 1720134466.517741, "_runtime": 11081.512312}', '{"system.network.sent": 539871680.0, "system.network.recv": 45358740.27, "system.cpu": 25.04, "system.cpu.0.cpu_percent": 9.04, "system.cpu.1.cpu_percent": 3.23, "system.cpu.2.cpu_percent": 63.39, "system.cpu.3.cpu_percent": 31.11, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27266.54, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.79, "system.gpu.0.memoryAllocated": 72.77, "system.gpu.0.memoryAllocatedBytes": 12501061905.07, "system.gpu.0.memory": 15.4, "system.gpu.0.gpu": 92.47, "system.gpu.0.temp": 73.07, "system.gpu.0.powerWatts": 140.76, "system.gpu.0.powerPercent": 56.93, "_wandb": true, "_timestamp": 1720134496.51853, "_runtime": 11111.513101}']}}, 'dropped': 0}} 2024-07-04 23:11:43,931 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:11:44,529 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:11:46,524 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:11:49,113 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:11:50,525 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:11:54,270 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:11:55,526 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:11:59,432 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:12:00,527 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:12:04,588 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:12:05,528 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:12:09,751 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:12:10,529 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:12:14,914 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:12:15,530 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:12:16,525 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:12:20,072 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:12:21,526 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:12:25,270 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:12:26,527 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:12:30,336 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:12:31,528 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:12:35,646 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:12:36,529 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:12:40,803 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:12:41,529 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:12:45,972 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:12:46,526 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:12:47,130 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 23:12:47,132 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 23:12:47,132 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:12:47,132 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:12:47,133 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:12:47,955 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 23:12:51,147 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:12:52,134 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:12:56,230 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:12:57,135 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:13:01,344 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:01,344 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:02,136 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:13:06,547 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:07,136 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:13:11,635 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:12,138 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:13:16,527 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:13:16,803 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:17,527 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:13:21,965 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:22,529 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:13:26,966 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:27,530 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:13:32,135 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:32,531 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:13:37,302 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:37,532 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:13:42,464 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:42,533 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:13:46,532 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:13:47,476 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:48,533 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:13:50,915 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 360, 'content': ['{"system.network.sent": 539696783.8, "system.network.recv": 45241213.4, "system.proc.memory.availableMB": 27265.78, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.cpu": 25.03, "system.cpu.0.cpu_percent": 2.63, "system.cpu.1.cpu_percent": 54.11, "system.cpu.2.cpu_percent": 6.4, "system.cpu.3.cpu_percent": 43.48, "system.proc.cpu.threads": 38, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.36, "system.gpu.0.memoryAllocated": 61.51, "system.gpu.0.memoryAllocatedBytes": 10567767381.33, "system.gpu.0.memory": 14.13, "system.gpu.0.gpu": 89.6, "system.gpu.0.temp": 72.87, "system.gpu.0.powerWatts": 123.47, "system.gpu.0.powerPercent": 49.55, "_wandb": true, "_timestamp": 1720134466.517741, "_runtime": 11081.512312}', '{"system.network.sent": 539871680.0, "system.network.recv": 45358740.27, "system.cpu": 25.04, "system.cpu.0.cpu_percent": 9.04, "system.cpu.1.cpu_percent": 3.23, "system.cpu.2.cpu_percent": 63.39, "system.cpu.3.cpu_percent": 31.11, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27266.54, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.79, "system.gpu.0.memoryAllocated": 72.77, "system.gpu.0.memoryAllocatedBytes": 12501061905.07, "system.gpu.0.memory": 15.4, "system.gpu.0.gpu": 92.47, "system.gpu.0.temp": 73.07, "system.gpu.0.powerWatts": 140.76, "system.gpu.0.powerPercent": 56.93, "_wandb": true, "_timestamp": 1720134496.51853, "_runtime": 11111.513101}']}}, 'dropped': 0}} 2024-07-04 23:13:52,645 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:53,534 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:13:57,801 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:13:58,535 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:14:02,964 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:14:03,536 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:14:08,130 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:14:08,537 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:14:13,274 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:14:13,539 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:14:16,528 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:14:18,461 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:14:19,529 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:14:23,712 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:14:24,530 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:14:28,753 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:14:29,531 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:14:33,895 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:14:34,532 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:14:39,011 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:14:39,533 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:14:44,152 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:14:44,534 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:14:46,529 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:14:49,245 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:14:50,530 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:14:54,409 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:14:55,531 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:14:59,563 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:14:59,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:15:00,533 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:15:04,717 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:15:05,534 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:15:09,879 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:15:10,535 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:15:15,041 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:15:15,535 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:15:16,530 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:15:20,212 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:15:21,531 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:15:25,299 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:15:26,532 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:15:30,412 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:15:31,534 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:15:35,545 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:15:36,535 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:15:40,641 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:15:41,536 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:15:45,817 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:15:46,531 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:15:47,532 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:15:50,933 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:15:52,533 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:15:56,109 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:15:57,534 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:16:01,162 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 360, 'content': ['{"system.network.sent": 539696783.8, "system.network.recv": 45241213.4, "system.proc.memory.availableMB": 27265.78, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.cpu": 25.03, "system.cpu.0.cpu_percent": 2.63, "system.cpu.1.cpu_percent": 54.11, "system.cpu.2.cpu_percent": 6.4, "system.cpu.3.cpu_percent": 43.48, "system.proc.cpu.threads": 38, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.36, "system.gpu.0.memoryAllocated": 61.51, "system.gpu.0.memoryAllocatedBytes": 10567767381.33, "system.gpu.0.memory": 14.13, "system.gpu.0.gpu": 89.6, "system.gpu.0.temp": 72.87, "system.gpu.0.powerWatts": 123.47, "system.gpu.0.powerPercent": 49.55, "_wandb": true, "_timestamp": 1720134466.517741, "_runtime": 11081.512312}', '{"system.network.sent": 539871680.0, "system.network.recv": 45358740.27, "system.cpu": 25.04, "system.cpu.0.cpu_percent": 9.04, "system.cpu.1.cpu_percent": 3.23, "system.cpu.2.cpu_percent": 63.39, "system.cpu.3.cpu_percent": 31.11, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27266.54, "system.memory": 15.1, "system.proc.memory.rssMB": 4274.34, "system.proc.memory.percent": 13.31, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3460.79, "system.gpu.0.memoryAllocated": 72.77, "system.gpu.0.memoryAllocatedBytes": 12501061905.07, "system.gpu.0.memory": 15.4, "system.gpu.0.gpu": 92.47, "system.gpu.0.temp": 73.07, "system.gpu.0.powerWatts": 140.76, "system.gpu.0.powerPercent": 56.93, "_wandb": true, "_timestamp": 1720134496.51853, "_runtime": 11111.513101}']}}, 'dropped': 0}} 2024-07-04 23:16:01,277 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:02,535 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:16:06,390 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:07,536 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:16:11,464 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:12,537 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:16:16,532 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:16:16,613 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:16,613 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:18,533 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:16:21,721 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:23,533 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:16:26,888 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:28,535 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:16:32,047 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:33,536 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:16:37,152 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:38,537 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:16:42,242 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:43,538 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:16:46,533 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:16:47,387 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:49,534 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:16:52,491 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:54,535 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:16:57,583 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:57,584 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:16:59,536 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:17:02,683 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:17:04,537 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:17:07,879 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:17:09,538 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:17:13,052 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:17:14,539 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:17:16,533 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:17:18,217 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:17:20,535 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:17:23,320 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:17:25,535 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:17:28,475 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:17:30,536 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:17:33,641 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:17:35,537 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:17:38,794 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:17:40,538 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:17:43,966 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:17:45,540 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:17:46,534 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:17:49,056 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:17:51,535 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:17:54,225 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:17:56,536 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:17:59,395 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:01,537 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:18:04,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:06,538 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:18:09,726 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:11,539 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:18:14,746 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:16,535 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:18:17,536 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:18:19,974 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:22,537 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:18:25,060 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:27,537 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:18:30,210 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:32,538 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:18:35,382 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:37,539 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:18:40,547 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:42,540 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:18:45,713 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:45,714 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:46,536 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:18:48,536 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:18:50,873 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:53,537 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:18:56,014 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:18:58,538 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:19:01,062 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:19:03,539 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:19:06,146 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:19:08,540 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:19:11,465 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:19:13,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:19:16,537 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:19:16,626 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:19:19,538 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:19:21,797 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:19:24,539 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:19:26,968 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:19:29,540 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:19:32,166 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:19:34,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:19:37,328 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:19:39,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:19:42,421 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:19:44,542 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:19:46,537 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:19:47,584 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:19:50,538 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:19:52,746 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:19:54,180 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 23:19:54,182 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 23:19:54,182 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:19:54,182 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:19:55,116 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 23:19:56,183 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:19:57,904 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:19:57,905 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:20:01,184 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:20:03,067 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:20:06,185 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:20:08,241 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:20:11,186 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:20:13,409 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:20:16,187 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:20:16,538 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:20:18,580 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:20:21,540 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:20:23,746 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:20:26,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:20:28,915 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:20:31,542 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:20:34,077 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:20:36,542 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:20:39,239 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:20:41,543 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:20:44,403 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:20:46,539 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:20:47,539 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:20:49,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:20:52,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:20:54,728 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:20:57,542 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:20:59,888 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:21:02,543 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:21:05,049 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:21:07,544 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:21:10,209 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:21:12,544 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:21:15,236 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:21:16,540 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:21:18,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:21:20,532 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:21:23,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:21:25,609 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:21:28,542 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:21:30,698 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:21:33,543 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:21:35,862 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:21:38,544 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:21:41,021 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:21:43,545 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:21:46,184 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:21:46,542 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:21:49,543 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:21:51,350 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:21:54,545 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:21:56,510 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:21:59,546 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:22:01,674 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:22:04,547 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:22:06,827 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:22:09,548 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:22:12,005 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:22:14,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:22:16,541 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:22:17,175 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:22:17,175 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:22:20,543 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:22:22,177 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:22:25,544 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:22:27,279 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:22:30,545 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:22:32,445 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:22:35,546 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:22:37,613 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:22:40,547 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:22:42,704 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:22:45,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:22:46,542 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:22:47,877 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:22:51,544 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:22:52,968 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:22:56,545 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:22:58,132 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:23:01,546 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:23:03,298 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:23:06,547 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:23:08,415 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:23:11,548 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:23:13,579 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:23:16,543 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:23:17,544 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:23:18,746 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:23:22,545 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:23:23,840 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:23:27,546 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:23:28,875 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:23:32,548 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:23:34,024 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:23:37,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:23:39,057 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:23:42,550 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:23:44,210 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:23:46,544 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:23:48,545 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:23:49,380 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:23:53,546 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:23:54,493 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:23:58,547 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:23:59,518 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:24:03,548 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:24:04,815 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:24:08,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:24:09,979 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:24:13,550 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:24:15,148 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:24:16,545 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:24:19,546 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:24:20,321 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:24:24,547 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:24:25,483 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:24:29,548 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:24:30,642 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:24:30,642 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:24:34,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:24:35,802 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:24:39,550 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:24:40,892 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:24:44,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:24:46,014 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:24:46,546 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:24:50,547 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:24:51,156 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:24:55,547 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:24:56,315 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:25:00,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:25:01,472 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:25:05,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:25:06,559 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:25:10,550 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:25:11,717 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:25:15,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:25:16,547 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:25:16,743 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:25:21,548 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:25:21,970 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:25:26,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:25:27,126 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:25:31,550 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:25:32,289 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:25:36,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:25:37,369 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:25:41,552 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:25:42,525 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:25:46,547 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:25:47,548 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:25:47,575 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:25:52,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:25:52,732 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:25:57,550 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:25:57,891 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:26:02,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:26:03,053 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:26:07,552 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:26:08,136 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:26:12,553 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:26:13,238 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:26:16,548 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:26:18,446 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:26:18,549 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:26:23,550 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:26:23,615 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:26:28,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:26:28,642 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:26:33,552 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:26:33,948 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:26:38,552 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:26:39,109 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:26:43,553 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:26:44,272 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:26:46,549 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:26:49,429 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:26:49,550 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:26:54,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:26:54,597 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:26:59,552 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:26:59,707 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:27:02,284 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 23:27:02,285 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 23:27:02,285 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:27:02,286 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:27:03,280 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 23:27:04,842 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:27:05,286 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:27:10,010 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:27:10,287 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:27:15,184 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:27:15,288 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:27:16,550 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:27:20,267 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:27:20,551 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:27:25,421 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:27:25,552 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:27:30,554 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:27:30,590 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:27:35,555 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:27:35,677 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:27:40,555 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:27:40,835 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:27:45,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:27:46,011 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:27:46,551 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:27:51,192 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:27:51,552 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:27:56,399 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:27:56,553 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:28:01,554 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:28:01,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:06,555 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:28:06,732 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:11,556 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:28:11,846 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:16,552 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:28:17,006 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:17,006 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:17,552 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:28:22,176 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:22,553 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:28:27,347 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:27,554 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:28:32,513 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:32,555 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:28:37,556 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:28:37,593 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:42,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:28:42,619 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:46,552 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:28:47,764 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:48,553 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:28:52,921 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:53,554 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:28:58,089 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:28:58,555 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:29:03,124 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:29:03,556 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:29:08,436 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:29:08,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:29:13,558 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:29:13,585 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:29:16,553 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:29:18,743 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:29:19,554 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:29:23,913 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:29:24,555 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:29:29,079 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:29:29,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:29:34,236 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:29:34,558 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:29:39,397 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:29:39,559 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:29:44,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:29:44,570 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:29:46,554 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:29:49,658 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:29:50,555 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:29:54,833 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:29:55,556 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:30:00,003 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:30:00,558 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:30:05,161 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:30:05,559 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:30:10,325 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:30:10,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:30:15,360 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:30:15,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:30:16,555 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:30:20,665 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:30:21,556 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:30:25,769 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:30:26,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:30:30,775 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:30:31,558 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:30:35,870 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:30:36,559 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:30:41,046 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:30:41,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:30:46,128 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:30:46,556 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:30:47,556 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:30:51,285 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:30:52,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:30:56,449 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:30:57,558 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:31:01,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:31:02,559 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:31:06,761 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:31:07,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:31:11,916 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:31:12,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:31:16,556 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:31:17,072 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:31:18,557 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:31:22,170 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:31:23,559 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:31:27,340 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:31:28,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:31:32,506 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:31:33,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:31:37,670 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:31:38,562 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:31:42,820 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:31:43,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:31:46,557 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:31:47,914 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:31:49,558 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:31:53,072 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:31:54,559 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:31:58,245 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:31:59,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:32:03,404 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:32:04,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:32:08,572 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:32:09,562 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:32:13,667 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:32:14,563 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:32:16,558 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:32:18,694 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:32:18,695 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:32:20,559 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:32:23,860 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:32:25,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:32:28,966 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:32:30,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:32:34,101 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:32:35,562 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:32:39,255 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:32:40,563 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:32:44,412 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:32:45,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:32:46,558 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:32:49,596 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:32:51,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:32:54,766 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:32:56,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:32:59,933 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:01,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:33:05,101 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:06,562 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:33:10,416 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:11,563 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:33:15,575 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:16,559 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:33:17,560 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:33:20,584 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:22,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:33:25,732 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:27,562 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:33:30,891 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:30,891 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:32,563 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:33:36,044 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:37,563 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:33:41,211 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:42,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:33:46,385 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:46,560 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:33:48,561 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:33:51,494 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:53,562 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:33:56,700 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:33:58,563 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:34:01,873 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:34:03,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:34:07,029 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:34:08,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:34:09,910 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 23:34:09,912 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 23:34:09,912 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:34:09,912 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:34:10,480 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 23:34:12,194 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:34:13,913 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:34:16,561 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:34:17,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:34:19,562 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:34:22,523 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:34:24,563 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:34:27,683 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:34:29,563 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:34:32,854 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:34:34,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:34:38,021 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:34:39,565 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:34:43,176 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:34:44,566 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:34:46,562 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:34:48,263 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:34:50,563 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:34:53,273 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:34:55,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:34:58,436 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:00,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:35:03,602 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:05,565 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:35:08,775 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:10,566 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:35:13,942 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:15,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:35:16,563 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:35:19,107 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:19,107 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:21,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:35:24,269 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:26,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:35:29,354 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:31,565 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:35:34,525 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:36,566 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:35:39,678 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:41,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:35:44,847 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:46,563 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:35:47,564 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:35:50,012 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:52,565 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:35:55,178 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:35:57,566 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:36:00,284 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:36:02,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:36:05,492 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:36:07,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:36:10,581 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:36:12,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:36:15,745 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:36:16,564 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:36:18,565 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:36:20,914 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:36:23,566 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:36:26,068 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:36:28,566 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:36:31,083 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:36:33,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:36:36,172 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:36:38,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:36:41,336 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:36:41,604 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 416, 'content': ['{"system.network.sent": 549594342.0, "system.network.recv": 51810971.0, "system.cpu": 25.02, "system.cpu.0.cpu_percent": 4.32, "system.cpu.1.cpu_percent": 46.19, "system.cpu.2.cpu_percent": 2.09, "system.cpu.3.cpu_percent": 54.25, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27253.16, "system.memory": 15.1, "system.proc.memory.rssMB": 4275.62, "system.proc.memory.percent": 13.32, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3483.23, "system.gpu.0.memoryAllocated": 76.52, "system.gpu.0.memoryAllocatedBytes": 13145306999.47, "system.gpu.0.memory": 14.87, "system.gpu.0.gpu": 92.67, "system.gpu.0.temp": 72.93, "system.gpu.0.powerWatts": 120.98, "system.gpu.0.powerPercent": 48.39, "_wandb": true, "_timestamp": 1720136146.562833, "_runtime": 12761.557404}']}}, 'dropped': 0}} 2024-07-04 23:36:43,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:36:46,501 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:36:46,565 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:36:49,566 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:36:51,770 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:36:54,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:36:56,914 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:36:59,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:37:02,003 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:37:04,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:37:07,170 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:37:09,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:37:12,343 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:37:14,570 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:37:16,565 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:37:17,514 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:37:20,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:37:22,693 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:37:25,567 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:37:25,617 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 416, 'content': ['{"system.network.sent": 549594342.0, "system.network.recv": 51810971.0, "system.cpu": 25.02, "system.cpu.0.cpu_percent": 4.32, "system.cpu.1.cpu_percent": 46.19, "system.cpu.2.cpu_percent": 2.09, "system.cpu.3.cpu_percent": 54.25, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27253.16, "system.memory": 15.1, "system.proc.memory.rssMB": 4275.62, "system.proc.memory.percent": 13.32, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3483.23, "system.gpu.0.memoryAllocated": 76.52, "system.gpu.0.memoryAllocatedBytes": 13145306999.47, "system.gpu.0.memory": 14.87, "system.gpu.0.gpu": 92.67, "system.gpu.0.temp": 72.93, "system.gpu.0.powerWatts": 120.98, "system.gpu.0.powerPercent": 48.39, "_wandb": true, "_timestamp": 1720136146.562833, "_runtime": 12761.557404}']}}, 'dropped': 0}} 2024-07-04 23:37:27,857 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:37:30,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:37:33,016 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:37:35,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:37:38,200 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:37:40,570 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:37:43,363 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:37:45,571 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:37:46,571 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:37:48,386 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:37:50,572 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:37:53,484 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 416, 'content': ['{"system.network.sent": 549594342.0, "system.network.recv": 51810971.0, "system.cpu": 25.02, "system.cpu.0.cpu_percent": 4.32, "system.cpu.1.cpu_percent": 46.19, "system.cpu.2.cpu_percent": 2.09, "system.cpu.3.cpu_percent": 54.25, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27253.16, "system.memory": 15.1, "system.proc.memory.rssMB": 4275.62, "system.proc.memory.percent": 13.32, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3483.23, "system.gpu.0.memoryAllocated": 76.52, "system.gpu.0.memoryAllocatedBytes": 13145306999.47, "system.gpu.0.memory": 14.87, "system.gpu.0.gpu": 92.67, "system.gpu.0.temp": 72.93, "system.gpu.0.powerWatts": 120.98, "system.gpu.0.powerPercent": 48.39, "_wandb": true, "_timestamp": 1720136146.562833, "_runtime": 12761.557404}']}}, 'dropped': 0}} 2024-07-04 23:37:53,613 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:37:55,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:37:58,775 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:38:00,574 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:38:03,932 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:38:05,575 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:38:09,096 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:38:10,576 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:38:14,264 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:38:15,577 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:38:16,567 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:38:19,265 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:38:21,568 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:38:24,298 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:38:26,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:38:29,299 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:38:31,570 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:38:34,339 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:38:36,571 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:38:39,340 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:38:41,572 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:38:44,388 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:38:46,568 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:38:47,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:38:49,389 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:38:52,569 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:38:54,426 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:38:57,570 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:38:58,479 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 416, 'content': ['{"system.network.sent": 549594342.0, "system.network.recv": 51810971.0, "system.cpu": 25.02, "system.cpu.0.cpu_percent": 4.32, "system.cpu.1.cpu_percent": 46.19, "system.cpu.2.cpu_percent": 2.09, "system.cpu.3.cpu_percent": 54.25, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27253.16, "system.memory": 15.1, "system.proc.memory.rssMB": 4275.62, "system.proc.memory.percent": 13.32, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3483.23, "system.gpu.0.memoryAllocated": 76.52, "system.gpu.0.memoryAllocatedBytes": 13145306999.47, "system.gpu.0.memory": 14.87, "system.gpu.0.gpu": 92.67, "system.gpu.0.temp": 72.93, "system.gpu.0.powerWatts": 120.98, "system.gpu.0.powerPercent": 48.39, "_wandb": true, "_timestamp": 1720136146.562833, "_runtime": 12761.557404}']}}, 'dropped': 0}} 2024-07-04 23:38:59,427 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:39:02,571 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:39:04,451 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:39:07,572 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:39:09,452 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:39:12,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:39:14,485 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:39:16,569 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:39:18,570 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:39:19,496 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:39:23,571 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:39:24,498 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:39:28,572 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:39:29,540 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:39:33,572 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:39:34,541 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:39:38,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:39:39,579 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:39:43,574 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:39:44,580 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:39:46,569 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:39:49,570 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:39:49,581 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:39:54,572 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:39:54,613 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:39:59,572 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:39:59,614 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:40:04,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:40:04,615 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:40:09,574 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:40:09,626 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:40:14,575 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:40:14,627 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:40:16,572 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:40:19,665 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:40:20,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:40:21,614 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 424, 'content': ['{"system.network.sent": 551727137.13, "system.network.recv": 53236584.93, "system.cpu": 25.05, "system.cpu.0.cpu_percent": 11.35, "system.cpu.1.cpu_percent": 29.2, "system.cpu.2.cpu_percent": 62.95, "system.cpu.3.cpu_percent": 4.21, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27255.33, "system.memory": 15.15, "system.proc.memory.rssMB": 4275.88, "system.proc.memory.percent": 13.32, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3486.47, "system.gpu.0.memoryAllocated": 69.05, "system.gpu.0.memoryAllocatedBytes": 11862409216.0, "system.gpu.0.memory": 27.47, "system.gpu.0.gpu": 76.67, "system.gpu.0.temp": 72.93, "system.gpu.0.powerWatts": 145.73, "system.gpu.0.powerPercent": 58.77, "_wandb": true, "_timestamp": 1720136386.569012, "_runtime": 13001.563583}']}}, 'dropped': 0}} 2024-07-04 23:40:24,666 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:40:25,574 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:40:28,815 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 424, 'content': ['{"system.network.sent": 551727137.13, "system.network.recv": 53236584.93, "system.cpu": 25.05, "system.cpu.0.cpu_percent": 11.35, "system.cpu.1.cpu_percent": 29.2, "system.cpu.2.cpu_percent": 62.95, "system.cpu.3.cpu_percent": 4.21, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27255.33, "system.memory": 15.15, "system.proc.memory.rssMB": 4275.88, "system.proc.memory.percent": 13.32, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3486.47, "system.gpu.0.memoryAllocated": 69.05, "system.gpu.0.memoryAllocatedBytes": 11862409216.0, "system.gpu.0.memory": 27.47, "system.gpu.0.gpu": 76.67, "system.gpu.0.temp": 72.93, "system.gpu.0.powerWatts": 145.73, "system.gpu.0.powerPercent": 58.77, "_wandb": true, "_timestamp": 1720136386.569012, "_runtime": 13001.563583}']}}, 'dropped': 0}} 2024-07-04 23:40:29,667 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:40:30,575 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:40:34,703 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:40:35,576 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:40:38,403 WARNING FileStreamThread:136 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream. func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/bossfortr/huggingface/v5ofm505/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 424, 'content': ['{"system.network.sent": 551727137.13, "system.network.recv": 53236584.93, "system.cpu": 25.05, "system.cpu.0.cpu_percent": 11.35, "system.cpu.1.cpu_percent": 29.2, "system.cpu.2.cpu_percent": 62.95, "system.cpu.3.cpu_percent": 4.21, "system.proc.cpu.threads": 38, "system.proc.memory.availableMB": 27255.33, "system.memory": 15.15, "system.proc.memory.rssMB": 4275.88, "system.proc.memory.percent": 13.32, "system.disk.\\\\.usagePercent": 70.6, "system.disk.\\\\.usageGB": 5689.5, "system.disk.in": 447.9, "system.disk.out": 3486.47, "system.gpu.0.memoryAllocated": 69.05, "system.gpu.0.memoryAllocatedBytes": 11862409216.0, "system.gpu.0.memory": 27.47, "system.gpu.0.gpu": 76.67, "system.gpu.0.temp": 72.93, "system.gpu.0.powerWatts": 145.73, "system.gpu.0.powerPercent": 58.77, "_wandb": true, "_timestamp": 1720136386.569012, "_runtime": 13001.563583}']}}, 'dropped': 0}} 2024-07-04 23:40:39,704 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:40:40,577 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:40:44,706 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:40:45,578 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:40:46,571 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:40:49,744 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:40:51,572 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:40:54,770 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:40:56,573 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:40:59,806 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:41:00,935 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 23:41:00,937 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 23:41:00,937 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:41:00,937 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:41:01,662 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 23:41:01,666 DEBUG SenderThread:136 [sender.py:send():378] send: telemetry 2024-07-04 23:41:01,666 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:41:01,667 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:41:01,669 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: partial_history 2024-07-04 23:41:01,670 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:41:01,670 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:41:01,671 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:41:01,671 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:41:01,671 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:41:01,672 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: pause 2024-07-04 23:41:01,672 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:41:01,672 INFO HandlerThread:136 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-04 23:41:01,672 INFO HandlerThread:136 [system_monitor.py:finish():203] Stopping system monitor 2024-07-04 23:41:01,673 DEBUG SystemMonitor:136 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-04 23:41:01,673 INFO HandlerThread:136 [interfaces.py:finish():200] Joined cpu monitor 2024-07-04 23:41:01,673 DEBUG SystemMonitor:136 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-04 23:41:01,674 INFO HandlerThread:136 [interfaces.py:finish():200] Joined disk monitor 2024-07-04 23:41:01,675 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:41:01,675 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:41:01,676 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:41:01,676 DEBUG SenderThread:136 [sender.py:send():378] send: history 2024-07-04 23:41:01,676 DEBUG SenderThread:136 [sender.py:send_request():405] send_request: summary_record 2024-07-04 23:41:01,677 INFO SenderThread:136 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-07-04 23:41:01,681 INFO HandlerThread:136 [interfaces.py:finish():200] Joined gpu monitor 2024-07-04 23:41:01,681 INFO HandlerThread:136 [interfaces.py:finish():200] Joined memory monitor 2024-07-04 23:41:01,682 INFO HandlerThread:136 [interfaces.py:finish():200] Joined network monitor 2024-07-04 23:41:01,682 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: resume 2024-07-04 23:41:01,682 INFO HandlerThread:136 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-04 23:41:01,682 INFO HandlerThread:136 [system_monitor.py:start():194] Starting system monitor 2024-07-04 23:41:01,682 DEBUG SenderThread:136 [sender.py:send():378] send: stats 2024-07-04 23:41:01,683 INFO SystemMonitor:136 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-04 23:41:01,683 INFO SystemMonitor:136 [interfaces.py:start():188] Started cpu monitoring 2024-07-04 23:41:01,684 INFO SystemMonitor:136 [interfaces.py:start():188] Started disk monitoring 2024-07-04 23:41:01,686 INFO SystemMonitor:136 [interfaces.py:start():188] Started gpu monitoring 2024-07-04 23:41:01,687 INFO SystemMonitor:136 [interfaces.py:start():188] Started memory monitoring 2024-07-04 23:41:01,687 INFO SystemMonitor:136 [interfaces.py:start():188] Started network monitoring 2024-07-04 23:41:02,662 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/output.log 2024-07-04 23:41:02,663 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/wandb-summary.json 2024-07-04 23:41:04,663 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/output.log 2024-07-04 23:41:04,853 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:41:07,100 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:41:09,855 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:41:12,106 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 23:41:14,856 DEBUG HandlerThread:136 [handler.py:handle_request():158] handle_request: keepalive 2024-07-04 23:41:16,668 INFO Thread-12 :136 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240704_200304-v5ofm505/files/config.yaml