File size: 18,820 Bytes
5cfeff5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 |
2024-02-28 16:47:28,218 INFO StreamThr :175 [internal.py:wandb_internal():86] W&B internal server running at pid: 175, started at: 2024-02-28 16:47:28.218262
2024-02-28 16:47:28,221 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status
2024-02-28 16:47:28,517 INFO WriterThread:175 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240228_164728-a7eaa078/run-a7eaa078.wandb
2024-02-28 16:47:28,518 DEBUG SenderThread:175 [sender.py:send():382] send: header
2024-02-28 16:47:28,520 DEBUG SenderThread:175 [sender.py:send():382] send: run
2024-02-28 16:47:28,717 INFO SenderThread:175 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files
2024-02-28 16:47:28,718 INFO SenderThread:175 [sender.py:_start_run_threads():1136] run started: a7eaa078 with start time 1709138848.222988
2024-02-28 16:47:28,726 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: check_version
2024-02-28 16:47:28,726 DEBUG SenderThread:175 [sender.py:send_request():409] send_request: check_version
2024-02-28 16:47:28,811 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: run_start
2024-02-28 16:47:28,822 DEBUG HandlerThread:175 [system_info.py:__init__():27] System info init
2024-02-28 16:47:28,822 DEBUG HandlerThread:175 [system_info.py:__init__():42] System info init done
2024-02-28 16:47:28,822 INFO HandlerThread:175 [system_monitor.py:start():194] Starting system monitor
2024-02-28 16:47:28,823 INFO SystemMonitor:175 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-02-28 16:47:28,823 INFO HandlerThread:175 [system_monitor.py:probe():214] Collecting system info
2024-02-28 16:47:28,823 INFO SystemMonitor:175 [interfaces.py:start():190] Started cpu monitoring
2024-02-28 16:47:28,824 INFO SystemMonitor:175 [interfaces.py:start():190] Started disk monitoring
2024-02-28 16:47:28,825 INFO SystemMonitor:175 [interfaces.py:start():190] Started gpu monitoring
2024-02-28 16:47:28,825 INFO SystemMonitor:175 [interfaces.py:start():190] Started memory monitoring
2024-02-28 16:47:28,828 INFO SystemMonitor:175 [interfaces.py:start():190] Started network monitoring
2024-02-28 16:47:28,837 DEBUG HandlerThread:175 [system_info.py:probe():151] Probing system
2024-02-28 16:47:28,839 DEBUG HandlerThread:175 [gitlib.py:_init_repo():56] git repository is invalid
2024-02-28 16:47:28,839 DEBUG HandlerThread:175 [system_info.py:probe():199] Probing system done
2024-02-28 16:47:28,839 DEBUG HandlerThread:175 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-02-28T16:47:28.837546', 'startedAt': '2024-02-28T16:47:28.210756', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'bafa38fbdbe9', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.198, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.198, 'min': 0.0, 'max': 0.0}, {'current': 2000.198, 'min': 0.0, 'max': 0.0}, {'current': 2000.198, 'min': 0.0, 'max': 0.0}, {'current': 2000.198, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5520.956398010254}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}}
2024-02-28 16:47:28,839 INFO HandlerThread:175 [system_monitor.py:probe():224] Finished collecting system info
2024-02-28 16:47:28,839 INFO HandlerThread:175 [system_monitor.py:probe():227] Publishing system info
2024-02-28 16:47:28,839 DEBUG HandlerThread:175 [system_info.py:_save_conda():208] Saving list of conda packages installed into the current environment
2024-02-28 16:47:29,719 INFO Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/conda-environment.yaml
2024-02-28 16:48:23,617 DEBUG HandlerThread:175 [system_info.py:_save_conda():220] Saving conda packages done
2024-02-28 16:48:23,618 INFO HandlerThread:175 [system_monitor.py:probe():229] Finished publishing system info
2024-02-28 16:48:23,632 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,632 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,633 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,633 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,633 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,633 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,633 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,634 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,634 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,634 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,634 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,634 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: python_packages
2024-02-28 16:48:23,634 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: stop_status
2024-02-28 16:48:23,635 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-28 16:48:23,635 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,635 DEBUG SenderThread:175 [sender.py:send_request():409] send_request: python_packages
2024-02-28 16:48:23,635 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,637 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,637 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,638 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,638 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,638 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,638 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,638 DEBUG SenderThread:175 [sender.py:send():382] send: telemetry
2024-02-28 16:48:23,639 DEBUG SenderThread:175 [sender.py:send_request():409] send_request: stop_status
2024-02-28 16:48:23,695 DEBUG SenderThread:175 [sender.py:send():382] send: config
2024-02-28 16:48:23,696 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:48:23,697 DEBUG SenderThread:175 [sender.py:send():382] send: telemetry
2024-02-28 16:48:23,697 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:48:23,697 WARNING SenderThread:175 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen)
2024-02-28 16:48:23,697 DEBUG SenderThread:175 [sender.py:send():382] send: telemetry
2024-02-28 16:48:23,697 DEBUG SenderThread:175 [sender.py:send():382] send: files
2024-02-28 16:48:23,697 INFO SenderThread:175 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now
2024-02-28 16:48:23,734 INFO Thread-12 :175 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/conda-environment.yaml
2024-02-28 16:48:23,734 INFO Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/wandb-metadata.json
2024-02-28 16:48:23,734 INFO Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/requirements.txt
2024-02-28 16:48:23,939 INFO wandb-upload_0:175 [upload_job.py:push():131] Uploaded file /tmp/tmpfgry5sqywandb/n1d5pray-wandb-metadata.json
2024-02-28 16:48:24,698 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:24,856 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:28,828 DEBUG SystemMonitor:175 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-02-28 16:48:28,830 DEBUG SenderThread:175 [sender.py:send():382] send: stats
2024-02-28 16:48:29,836 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:29,857 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:30,736 INFO Thread-12 :175 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/config.yaml
2024-02-28 16:48:34,858 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:34,966 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:39,859 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:39,967 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:44,859 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:44,968 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:49,860 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:49,968 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:54,861 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:54,969 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:58,831 DEBUG SenderThread:175 [sender.py:send():382] send: stats
2024-02-28 16:48:59,862 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:00,832 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:04,863 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:05,832 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:09,863 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:10,833 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:14,864 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:15,834 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:19,865 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:20,834 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:24,866 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:25,835 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:28,832 DEBUG SenderThread:175 [sender.py:send():382] send: stats
2024-02-28 16:49:29,867 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:31,833 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:34,868 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:36,833 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:39,875 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:41,834 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:44,876 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:46,835 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:49,877 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:51,835 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:54,878 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:56,836 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:58,833 DEBUG SenderThread:175 [sender.py:send():382] send: stats
2024-02-28 16:49:59,879 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:02,834 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:04,879 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:07,834 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:09,883 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:12,835 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:14,884 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:17,836 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:19,885 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:22,837 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:24,886 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:27,837 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:28,833 DEBUG SenderThread:175 [sender.py:send():382] send: stats
2024-02-28 16:50:29,914 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:33,834 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:35,220 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:37,773 INFO Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/output.log
2024-02-28 16:50:39,441 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:39,774 INFO Thread-12 :175 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/output.log
2024-02-28 16:50:40,252 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:44,442 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:45,208 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: partial_history
2024-02-28 16:50:45,211 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,211 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,213 DEBUG SenderThread:175 [sender.py:send():382] send: history
2024-02-28 16:50:45,213 DEBUG SenderThread:175 [sender.py:send_request():409] send_request: summary_record
2024-02-28 16:50:45,213 INFO SenderThread:175 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-02-28 16:50:45,279 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:45,776 INFO Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/wandb-summary.json
2024-02-28 16:50:46,908 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: partial_history
2024-02-28 16:50:46,910 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:46,910 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:46,910 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:46,910 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:46,910 DEBUG SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:46,910 DEBUG SenderThread:175 [sender.py:send():382] send: history
2024-02-28 16:50:46,911 DEBUG SenderThread:175 [sender.py:send_request():409] send_request: summary_record
2024-02-28 16:50:46,911 INFO SenderThread:175 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-02-28 16:50:47,776 INFO Thread-12 :175 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/wandb-summary.json
2024-02-28 16:50:49,912 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:50,299 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
|