| 2024-01-18 17:18:29,334 INFO StreamThr :127 [internal.py:wandb_internal():86] W&B internal server running at pid: 127, started at: 2024-01-18 17:18:29.333879 | |
| 2024-01-18 17:18:29,336 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status | |
| 2024-01-18 17:18:30,012 INFO WriterThread:127 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240118_171829-uvan9htf/run-uvan9htf.wandb | |
| 2024-01-18 17:18:30,013 DEBUG SenderThread:127 [sender.py:send():382] send: header | |
| 2024-01-18 17:18:30,016 DEBUG SenderThread:127 [sender.py:send():382] send: run | |
| 2024-01-18 17:18:30,401 INFO SenderThread:127 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files | |
| 2024-01-18 17:18:30,401 INFO SenderThread:127 [sender.py:_start_run_threads():1136] run started: uvan9htf with start time 1705598309.334874 | |
| 2024-01-18 17:18:30,408 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-01-18 17:18:30,409 DEBUG SenderThread:127 [sender.py:send_request():409] send_request: check_version | |
| 2024-01-18 17:18:30,460 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-01-18 17:18:30,471 DEBUG HandlerThread:127 [system_info.py:__init__():27] System info init | |
| 2024-01-18 17:18:30,471 DEBUG HandlerThread:127 [system_info.py:__init__():42] System info init done | |
| 2024-01-18 17:18:30,471 INFO HandlerThread:127 [system_monitor.py:start():194] Starting system monitor | |
| 2024-01-18 17:18:30,471 INFO SystemMonitor:127 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-01-18 17:18:30,472 INFO SystemMonitor:127 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-01-18 17:18:30,472 INFO SystemMonitor:127 [interfaces.py:start():190] Started disk monitoring | |
| 2024-01-18 17:18:30,473 INFO HandlerThread:127 [system_monitor.py:probe():214] Collecting system info | |
| 2024-01-18 17:18:30,473 INFO SystemMonitor:127 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-01-18 17:18:30,474 INFO SystemMonitor:127 [interfaces.py:start():190] Started memory monitoring | |
| 2024-01-18 17:18:30,475 INFO SystemMonitor:127 [interfaces.py:start():190] Started network monitoring | |
| 2024-01-18 17:18:30,484 DEBUG HandlerThread:127 [system_info.py:probe():151] Probing system | |
| 2024-01-18 17:18:30,487 DEBUG HandlerThread:127 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-01-18 17:18:30,487 DEBUG HandlerThread:127 [system_info.py:probe():199] Probing system done | |
| 2024-01-18 17:18:30,487 DEBUG HandlerThread:127 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-01-18T17:18:30.484995', 'startedAt': '2024-01-18T17:18:29.326975', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'bddbbc3e305c', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.136, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.136, 'min': 0.0, 'max': 0.0}, {'current': 2000.136, 'min': 0.0, 'max': 0.0}, {'current': 2000.136, 'min': 0.0, 'max': 0.0}, {'current': 2000.136, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5359.688289642334}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}} | |
| 2024-01-18 17:18:30,487 INFO HandlerThread:127 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-01-18 17:18:30,487 INFO HandlerThread:127 [system_monitor.py:probe():227] Publishing system info | |
| 2024-01-18 17:18:30,487 DEBUG HandlerThread:127 [system_info.py:_save_conda():208] Saving list of conda packages installed into the current environment | |
| 2024-01-18 17:18:31,403 INFO Thread-12 :127 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/conda-environment.yaml | |
| 2024-01-18 17:19:21,419 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/conda-environment.yaml | |
| 2024-01-18 17:19:21,847 DEBUG HandlerThread:127 [system_info.py:_save_conda():220] Saving conda packages done | |
| 2024-01-18 17:19:21,848 INFO HandlerThread:127 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-01-18 17:19:21,856 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:21,856 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:21,857 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:21,857 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:21,857 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:21,857 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:21,857 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:21,857 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:21,857 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:21,858 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:21,858 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:21,858 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-01-18 17:19:21,858 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-01-18 17:19:21,858 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-01-18 17:19:21,858 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:21,858 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:21,859 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:21,859 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:21,860 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:21,860 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:21,860 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:21,860 DEBUG SenderThread:127 [sender.py:send_request():409] send_request: python_packages | |
| 2024-01-18 17:19:21,860 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:21,862 DEBUG SenderThread:127 [sender.py:send_request():409] send_request: stop_status | |
| 2024-01-18 17:19:22,024 DEBUG SenderThread:127 [sender.py:send():382] send: telemetry | |
| 2024-01-18 17:19:22,025 DEBUG SenderThread:127 [sender.py:send():382] send: config | |
| 2024-01-18 17:19:22,026 DEBUG SenderThread:127 [sender.py:send():382] send: metric | |
| 2024-01-18 17:19:22,026 DEBUG SenderThread:127 [sender.py:send():382] send: telemetry | |
| 2024-01-18 17:19:22,027 DEBUG SenderThread:127 [sender.py:send():382] send: metric | |
| 2024-01-18 17:19:22,027 WARNING SenderThread:127 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) | |
| 2024-01-18 17:19:22,027 DEBUG SenderThread:127 [sender.py:send():382] send: telemetry | |
| 2024-01-18 17:19:22,028 DEBUG SenderThread:127 [sender.py:send():382] send: files | |
| 2024-01-18 17:19:22,028 INFO SenderThread:127 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now | |
| 2024-01-18 17:19:22,419 INFO Thread-12 :127 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/wandb-metadata.json | |
| 2024-01-18 17:19:22,419 INFO Thread-12 :127 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/output.log | |
| 2024-01-18 17:19:22,420 INFO Thread-12 :127 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/requirements.txt | |
| 2024-01-18 17:19:22,553 INFO wandb-upload_0:127 [upload_job.py:push():131] Uploaded file /tmp/tmphd9dizo7wandb/g66ljhg3-wandb-metadata.json | |
| 2024-01-18 17:19:24,420 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/output.log | |
| 2024-01-18 17:19:25,029 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:26,505 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:30,034 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:30,422 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/config.yaml | |
| 2024-01-18 17:19:30,475 DEBUG SystemMonitor:127 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-01-18 17:19:30,476 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:19:31,506 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:35,477 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:36,517 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:40,478 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:41,518 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:45,479 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:46,519 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:50,480 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:51,520 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:19:55,481 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:19:56,521 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:00,477 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:20:01,478 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:20:01,522 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:06,479 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:20:06,522 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:11,479 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:20:11,524 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:16,480 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:20:16,525 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:21,481 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:20:21,525 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:21,714 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-01-18 17:20:21,716 DEBUG SenderThread:127 [sender.py:send():382] send: metric | |
| 2024-01-18 17:20:21,716 DEBUG SenderThread:127 [sender.py:send():382] send: metric | |
| 2024-01-18 17:20:21,717 DEBUG SenderThread:127 [sender.py:send():382] send: metric | |
| 2024-01-18 17:20:21,717 DEBUG SenderThread:127 [sender.py:send():382] send: history | |
| 2024-01-18 17:20:21,717 DEBUG SenderThread:127 [sender.py:send_request():409] send_request: summary_record | |
| 2024-01-18 17:20:21,718 INFO SenderThread:127 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-01-18 17:20:22,439 INFO Thread-12 :127 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/wandb-summary.json | |
| 2024-01-18 17:20:26,440 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/output.log | |
| 2024-01-18 17:20:26,580 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:26,893 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:20:30,479 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:20:31,581 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:32,484 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:20:33,443 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/config.yaml | |
| 2024-01-18 17:20:36,582 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:37,726 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:20:41,587 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:42,726 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:20:46,590 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:47,727 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:20:51,593 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:52,728 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:20:56,594 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:20:57,729 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:21:00,479 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:21:01,595 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:21:03,480 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:21:06,596 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:21:08,481 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:21:11,598 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:21:13,482 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:21:16,744 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:21:18,483 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:21:21,745 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:21:23,484 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:21:26,749 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:21:28,485 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:21:30,480 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:21:31,756 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:21:34,481 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:21:36,757 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:21:39,482 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:21:41,759 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:21:44,483 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:21:44,710 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-01-18 17:21:44,711 DEBUG SenderThread:127 [sender.py:send():382] send: history | |
| 2024-01-18 17:21:44,711 DEBUG SenderThread:127 [sender.py:send_request():409] send_request: summary_record | |
| 2024-01-18 17:21:44,713 INFO SenderThread:127 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-01-18 17:21:45,469 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/wandb-summary.json | |
| 2024-01-18 17:21:46,768 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:21:48,470 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/output.log | |
| 2024-01-18 17:21:50,051 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:21:51,772 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:21:55,051 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:21:56,776 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:22:00,052 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:22:00,481 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:22:01,357 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: pause | |
| 2024-01-18 17:22:01,357 INFO HandlerThread:127 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-01-18 17:22:01,357 INFO HandlerThread:127 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-01-18 17:22:01,358 DEBUG SystemMonitor:127 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-01-18 17:22:01,358 INFO HandlerThread:127 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-01-18 17:22:01,358 DEBUG SystemMonitor:127 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-01-18 17:22:01,359 INFO HandlerThread:127 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-01-18 17:22:01,372 INFO HandlerThread:127 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-01-18 17:22:01,372 INFO HandlerThread:127 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-01-18 17:22:01,372 INFO HandlerThread:127 [interfaces.py:finish():202] Joined network monitor | |
| 2024-01-18 17:22:01,373 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:22:01,787 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:22:05,027 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: resume | |
| 2024-01-18 17:22:05,027 INFO HandlerThread:127 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-01-18 17:22:05,028 INFO HandlerThread:127 [system_monitor.py:start():194] Starting system monitor | |
| 2024-01-18 17:22:05,028 INFO SystemMonitor:127 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-01-18 17:22:05,028 INFO SystemMonitor:127 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-01-18 17:22:05,029 INFO SystemMonitor:127 [interfaces.py:start():190] Started disk monitoring | |
| 2024-01-18 17:22:05,029 INFO SystemMonitor:127 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-01-18 17:22:05,030 INFO SystemMonitor:127 [interfaces.py:start():190] Started memory monitoring | |
| 2024-01-18 17:22:05,030 INFO SystemMonitor:127 [interfaces.py:start():190] Started network monitoring | |
| 2024-01-18 17:22:05,374 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:22:06,788 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:22:10,375 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:22:11,789 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:22:15,376 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:22:16,792 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:22:20,377 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:22:21,793 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:22:25,377 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:22:26,795 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:22:30,378 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:22:31,797 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:22:35,379 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:22:36,803 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:22:40,380 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:22:41,804 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:22:45,381 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:22:46,805 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:22:50,382 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:22:51,807 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:22:55,382 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:22:56,809 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:00,383 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:23:01,809 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:05,031 DEBUG SystemMonitor:127 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-01-18 17:23:05,032 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:23:06,033 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:23:06,813 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:11,034 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:23:11,814 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:16,034 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:23:16,818 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:21,035 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:23:21,819 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:26,036 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:23:26,750 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-01-18 17:23:26,752 DEBUG SenderThread:127 [sender.py:send():382] send: history | |
| 2024-01-18 17:23:26,752 DEBUG SenderThread:127 [sender.py:send_request():409] send_request: summary_record | |
| 2024-01-18 17:23:26,754 INFO SenderThread:127 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-01-18 17:23:26,820 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:27,506 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/wandb-summary.json | |
| 2024-01-18 17:23:28,506 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/output.log | |
| 2024-01-18 17:23:31,837 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:32,021 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:23:32,508 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/output.log | |
| 2024-01-18 17:23:35,033 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:23:36,839 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:37,034 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:23:41,839 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:42,035 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:23:46,846 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:47,036 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:23:51,851 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:52,037 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:23:56,852 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:23:57,038 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:24:01,853 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:24:02,038 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:24:05,034 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:24:06,858 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:24:08,036 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:24:11,859 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:24:13,037 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:24:16,861 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:24:18,038 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:24:21,862 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:24:23,038 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:24:26,864 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:24:28,039 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:24:31,866 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:24:33,040 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:24:35,035 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:24:36,867 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:24:39,036 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:24:41,868 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:24:44,037 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:24:46,869 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:24:49,038 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:24:50,869 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-01-18 17:24:50,871 DEBUG SenderThread:127 [sender.py:send():382] send: history | |
| 2024-01-18 17:24:50,871 DEBUG SenderThread:127 [sender.py:send_request():409] send_request: summary_record | |
| 2024-01-18 17:24:50,871 INFO SenderThread:127 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-01-18 17:24:51,536 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/wandb-summary.json | |
| 2024-01-18 17:24:51,940 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:24:52,537 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/output.log | |
| 2024-01-18 17:24:54,537 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/output.log | |
| 2024-01-18 17:24:54,930 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:24:56,954 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:24:59,931 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:25:01,956 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:25:04,932 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:25:05,036 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:25:06,957 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:25:10,037 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:25:11,963 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:25:15,038 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:25:16,964 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:25:20,038 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:25:21,970 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:25:25,039 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:25:26,971 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:25:30,040 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:25:31,972 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:25:35,037 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:25:36,037 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:25:36,973 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:25:41,038 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:25:41,974 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:25:46,039 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:25:46,975 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:25:51,040 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:25:51,976 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:25:56,041 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:25:56,977 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:01,041 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:26:01,979 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:05,038 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:26:06,979 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:07,039 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:26:11,980 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:12,039 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:26:14,696 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-01-18 17:26:14,697 DEBUG SenderThread:127 [sender.py:send():382] send: history | |
| 2024-01-18 17:26:14,697 DEBUG SenderThread:127 [sender.py:send_request():409] send_request: summary_record | |
| 2024-01-18 17:26:14,697 INFO SenderThread:127 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-01-18 17:26:15,564 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/wandb-summary.json | |
| 2024-01-18 17:26:17,113 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:17,879 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:26:18,566 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/output.log | |
| 2024-01-18 17:26:22,115 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:22,880 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:26:27,116 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:27,881 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:26:32,117 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:32,882 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:26:35,038 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:26:37,119 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:38,039 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:26:42,120 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:43,040 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:26:47,128 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:48,041 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:26:52,130 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:53,042 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:26:57,133 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:26:58,043 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:27:02,136 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:27:03,043 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:27:05,040 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:27:07,136 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:27:09,041 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:27:12,137 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:27:14,042 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:27:17,139 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:27:19,043 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:27:22,140 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:27:24,044 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:27:27,140 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:27:29,045 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:27:32,146 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:27:34,045 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:27:35,041 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:27:37,148 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:27:37,915 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-01-18 17:27:37,916 DEBUG SenderThread:127 [sender.py:send():382] send: history | |
| 2024-01-18 17:27:37,917 DEBUG SenderThread:127 [sender.py:send_request():409] send_request: summary_record | |
| 2024-01-18 17:27:37,917 INFO SenderThread:127 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-01-18 17:27:38,594 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/wandb-summary.json | |
| 2024-01-18 17:27:39,918 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:27:40,595 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/output.log | |
| 2024-01-18 17:27:42,326 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:27:45,101 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:27:47,327 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:27:50,102 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:27:52,328 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:27:55,102 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:27:57,329 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:28:00,103 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:28:02,330 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:28:05,042 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:28:06,043 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:28:07,331 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:28:11,044 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:28:12,335 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:28:16,045 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:28:17,336 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:28:21,046 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:28:22,337 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:28:26,047 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:28:27,337 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:28:31,047 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:28:32,343 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:28:35,043 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:28:37,044 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:28:37,344 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:28:42,045 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:28:42,345 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:28:47,045 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:28:47,348 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:28:52,046 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:28:52,605 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:28:57,047 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:28:57,606 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:01,075 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-01-18 17:29:01,076 DEBUG SenderThread:127 [sender.py:send():382] send: history | |
| 2024-01-18 17:29:01,076 DEBUG SenderThread:127 [sender.py:send_request():409] send_request: summary_record | |
| 2024-01-18 17:29:01,076 INFO SenderThread:127 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-01-18 17:29:01,624 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/wandb-summary.json | |
| 2024-01-18 17:29:02,077 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:29:02,805 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:02,805 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:05,044 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:29:05,625 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/output.log | |
| 2024-01-18 17:29:07,808 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:08,045 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:29:12,809 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:13,046 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:29:17,809 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:18,047 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:29:19,629 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-01-18 17:29:19,632 DEBUG SenderThread:127 [sender.py:send():382] send: metric | |
| 2024-01-18 17:29:19,632 DEBUG SenderThread:127 [sender.py:send():382] send: metric | |
| 2024-01-18 17:29:19,632 DEBUG SenderThread:127 [sender.py:send():382] send: metric | |
| 2024-01-18 17:29:19,632 DEBUG SenderThread:127 [sender.py:send():382] send: metric | |
| 2024-01-18 17:29:19,632 DEBUG SenderThread:127 [sender.py:send():382] send: metric | |
| 2024-01-18 17:29:19,632 DEBUG SenderThread:127 [sender.py:send():382] send: history | |
| 2024-01-18 17:29:19,632 DEBUG SenderThread:127 [sender.py:send_request():409] send_request: summary_record | |
| 2024-01-18 17:29:19,633 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/wandb-summary.json | |
| 2024-01-18 17:29:19,634 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: pause | |
| 2024-01-18 17:29:19,634 INFO HandlerThread:127 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-01-18 17:29:19,635 INFO HandlerThread:127 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-01-18 17:29:19,635 DEBUG SystemMonitor:127 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-01-18 17:29:19,636 DEBUG SystemMonitor:127 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-01-18 17:29:19,637 INFO HandlerThread:127 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-01-18 17:29:19,637 INFO HandlerThread:127 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-01-18 17:29:19,647 INFO HandlerThread:127 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-01-18 17:29:19,647 INFO HandlerThread:127 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-01-18 17:29:19,648 INFO HandlerThread:127 [interfaces.py:finish():202] Joined network monitor | |
| 2024-01-18 17:29:19,670 INFO SenderThread:127 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-01-18 17:29:19,670 DEBUG SenderThread:127 [sender.py:send():382] send: stats | |
| 2024-01-18 17:29:22,810 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:23,672 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:29:27,811 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:28,673 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:29:32,812 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:33,673 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:29:37,813 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:38,674 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:29:42,814 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:43,675 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:29:47,815 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:48,681 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:29:49,643 INFO Thread-12 :127 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240118_171829-uvan9htf/files/config.yaml | |
| 2024-01-18 17:29:52,816 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:54,126 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:29:57,817 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:29:59,127 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:30:02,818 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:30:04,127 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:30:07,819 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:30:09,128 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:30:12,820 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:30:14,129 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:30:17,820 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:30:19,130 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:30:22,821 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:30:24,131 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:30:26,098 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: resume | |
| 2024-01-18 17:30:26,098 INFO HandlerThread:127 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-01-18 17:30:26,098 INFO HandlerThread:127 [system_monitor.py:start():194] Starting system monitor | |
| 2024-01-18 17:30:26,099 INFO SystemMonitor:127 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-01-18 17:30:26,099 INFO SystemMonitor:127 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-01-18 17:30:26,100 INFO SystemMonitor:127 [interfaces.py:start():190] Started disk monitoring | |
| 2024-01-18 17:30:26,101 INFO SystemMonitor:127 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-01-18 17:30:26,101 INFO SystemMonitor:127 [interfaces.py:start():190] Started memory monitoring | |
| 2024-01-18 17:30:26,103 INFO SystemMonitor:127 [interfaces.py:start():190] Started network monitoring | |
| 2024-01-18 17:30:28,115 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:30:29,132 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:30:33,116 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:30:34,132 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-01-18 17:30:38,116 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-01-18 17:30:39,133 DEBUG HandlerThread:127 [handler.py:handle_request():146] handle_request: status_report | |