File size: 18,820 Bytes
5cfeff5
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
2024-02-28 16:47:28,218 INFO    StreamThr :175 [internal.py:wandb_internal():86] W&B internal server running at pid: 175, started at: 2024-02-28 16:47:28.218262
2024-02-28 16:47:28,221 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status
2024-02-28 16:47:28,517 INFO    WriterThread:175 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240228_164728-a7eaa078/run-a7eaa078.wandb
2024-02-28 16:47:28,518 DEBUG   SenderThread:175 [sender.py:send():382] send: header
2024-02-28 16:47:28,520 DEBUG   SenderThread:175 [sender.py:send():382] send: run
2024-02-28 16:47:28,717 INFO    SenderThread:175 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files
2024-02-28 16:47:28,718 INFO    SenderThread:175 [sender.py:_start_run_threads():1136] run started: a7eaa078 with start time 1709138848.222988
2024-02-28 16:47:28,726 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: check_version
2024-02-28 16:47:28,726 DEBUG   SenderThread:175 [sender.py:send_request():409] send_request: check_version
2024-02-28 16:47:28,811 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: run_start
2024-02-28 16:47:28,822 DEBUG   HandlerThread:175 [system_info.py:__init__():27] System info init
2024-02-28 16:47:28,822 DEBUG   HandlerThread:175 [system_info.py:__init__():42] System info init done
2024-02-28 16:47:28,822 INFO    HandlerThread:175 [system_monitor.py:start():194] Starting system monitor
2024-02-28 16:47:28,823 INFO    SystemMonitor:175 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-02-28 16:47:28,823 INFO    HandlerThread:175 [system_monitor.py:probe():214] Collecting system info
2024-02-28 16:47:28,823 INFO    SystemMonitor:175 [interfaces.py:start():190] Started cpu monitoring
2024-02-28 16:47:28,824 INFO    SystemMonitor:175 [interfaces.py:start():190] Started disk monitoring
2024-02-28 16:47:28,825 INFO    SystemMonitor:175 [interfaces.py:start():190] Started gpu monitoring
2024-02-28 16:47:28,825 INFO    SystemMonitor:175 [interfaces.py:start():190] Started memory monitoring
2024-02-28 16:47:28,828 INFO    SystemMonitor:175 [interfaces.py:start():190] Started network monitoring
2024-02-28 16:47:28,837 DEBUG   HandlerThread:175 [system_info.py:probe():151] Probing system
2024-02-28 16:47:28,839 DEBUG   HandlerThread:175 [gitlib.py:_init_repo():56] git repository is invalid
2024-02-28 16:47:28,839 DEBUG   HandlerThread:175 [system_info.py:probe():199] Probing system done
2024-02-28 16:47:28,839 DEBUG   HandlerThread:175 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-02-28T16:47:28.837546', 'startedAt': '2024-02-28T16:47:28.210756', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'bafa38fbdbe9', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.198, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.198, 'min': 0.0, 'max': 0.0}, {'current': 2000.198, 'min': 0.0, 'max': 0.0}, {'current': 2000.198, 'min': 0.0, 'max': 0.0}, {'current': 2000.198, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5520.956398010254}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}}
2024-02-28 16:47:28,839 INFO    HandlerThread:175 [system_monitor.py:probe():224] Finished collecting system info
2024-02-28 16:47:28,839 INFO    HandlerThread:175 [system_monitor.py:probe():227] Publishing system info
2024-02-28 16:47:28,839 DEBUG   HandlerThread:175 [system_info.py:_save_conda():208] Saving list of conda packages installed into the current environment
2024-02-28 16:47:29,719 INFO    Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/conda-environment.yaml
2024-02-28 16:48:23,617 DEBUG   HandlerThread:175 [system_info.py:_save_conda():220] Saving conda packages done
2024-02-28 16:48:23,618 INFO    HandlerThread:175 [system_monitor.py:probe():229] Finished publishing system info
2024-02-28 16:48:23,632 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,632 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,633 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,633 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,633 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,633 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,633 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,634 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,634 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,634 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,634 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,634 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: python_packages
2024-02-28 16:48:23,634 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: stop_status
2024-02-28 16:48:23,635 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-28 16:48:23,635 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,635 DEBUG   SenderThread:175 [sender.py:send_request():409] send_request: python_packages
2024-02-28 16:48:23,635 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,637 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,637 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,638 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,638 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,638 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:23,638 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:23,638 DEBUG   SenderThread:175 [sender.py:send():382] send: telemetry
2024-02-28 16:48:23,639 DEBUG   SenderThread:175 [sender.py:send_request():409] send_request: stop_status
2024-02-28 16:48:23,695 DEBUG   SenderThread:175 [sender.py:send():382] send: config
2024-02-28 16:48:23,696 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:48:23,697 DEBUG   SenderThread:175 [sender.py:send():382] send: telemetry
2024-02-28 16:48:23,697 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:48:23,697 WARNING SenderThread:175 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen)
2024-02-28 16:48:23,697 DEBUG   SenderThread:175 [sender.py:send():382] send: telemetry
2024-02-28 16:48:23,697 DEBUG   SenderThread:175 [sender.py:send():382] send: files
2024-02-28 16:48:23,697 INFO    SenderThread:175 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now
2024-02-28 16:48:23,734 INFO    Thread-12 :175 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/conda-environment.yaml
2024-02-28 16:48:23,734 INFO    Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/wandb-metadata.json
2024-02-28 16:48:23,734 INFO    Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/requirements.txt
2024-02-28 16:48:23,939 INFO    wandb-upload_0:175 [upload_job.py:push():131] Uploaded file /tmp/tmpfgry5sqywandb/n1d5pray-wandb-metadata.json
2024-02-28 16:48:24,698 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:24,856 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:28,828 DEBUG   SystemMonitor:175 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-02-28 16:48:28,830 DEBUG   SenderThread:175 [sender.py:send():382] send: stats
2024-02-28 16:48:29,836 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:29,857 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:30,736 INFO    Thread-12 :175 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/config.yaml
2024-02-28 16:48:34,858 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:34,966 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:39,859 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:39,967 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:44,859 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:44,968 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:49,860 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:49,968 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:54,861 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:48:54,969 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:48:58,831 DEBUG   SenderThread:175 [sender.py:send():382] send: stats
2024-02-28 16:48:59,862 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:00,832 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:04,863 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:05,832 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:09,863 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:10,833 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:14,864 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:15,834 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:19,865 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:20,834 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:24,866 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:25,835 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:28,832 DEBUG   SenderThread:175 [sender.py:send():382] send: stats
2024-02-28 16:49:29,867 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:31,833 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:34,868 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:36,833 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:39,875 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:41,834 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:44,876 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:46,835 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:49,877 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:51,835 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:54,878 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:49:56,836 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:49:58,833 DEBUG   SenderThread:175 [sender.py:send():382] send: stats
2024-02-28 16:49:59,879 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:02,834 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:04,879 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:07,834 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:09,883 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:12,835 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:14,884 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:17,836 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:19,885 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:22,837 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:24,886 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:27,837 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:28,833 DEBUG   SenderThread:175 [sender.py:send():382] send: stats
2024-02-28 16:50:29,914 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:33,834 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:35,220 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:37,773 INFO    Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/output.log
2024-02-28 16:50:39,441 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:39,774 INFO    Thread-12 :175 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/output.log
2024-02-28 16:50:40,252 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:44,442 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:45,208 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: partial_history
2024-02-28 16:50:45,211 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,211 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,212 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:45,213 DEBUG   SenderThread:175 [sender.py:send():382] send: history
2024-02-28 16:50:45,213 DEBUG   SenderThread:175 [sender.py:send_request():409] send_request: summary_record
2024-02-28 16:50:45,213 INFO    SenderThread:175 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-02-28 16:50:45,279 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive
2024-02-28 16:50:45,776 INFO    Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/wandb-summary.json
2024-02-28 16:50:46,908 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: partial_history
2024-02-28 16:50:46,910 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:46,910 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:46,910 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:46,910 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:46,910 DEBUG   SenderThread:175 [sender.py:send():382] send: metric
2024-02-28 16:50:46,910 DEBUG   SenderThread:175 [sender.py:send():382] send: history
2024-02-28 16:50:46,911 DEBUG   SenderThread:175 [sender.py:send_request():409] send_request: summary_record
2024-02-28 16:50:46,911 INFO    SenderThread:175 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-02-28 16:50:47,776 INFO    Thread-12 :175 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/wandb-summary.json
2024-02-28 16:50:49,912 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report
2024-02-28 16:50:50,299 DEBUG   HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive