Hugging Face's logo

Gordon119
/
qa_test

Question Answering
Transformers
TensorBoard
Safetensors
bert
Generated from Trainer
Model card Files
xet
Metrics Community
qa_test
File size: 45,482 Bytes 
a921a3b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
 
2024-03-27 19:04:01,650 INFO    StreamThr :1855917 [internal.py:wandb_internal():86] W&B internal server running at pid: 1855917, started at: 2024-03-27 19:04:01.649845
2024-03-27 19:04:01,650 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status
2024-03-27 19:04:01,652 INFO    WriterThread:1855917 [datastore.py:open_for_write():87] open: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/run-h708adzm.wandb
2024-03-27 19:04:01,652 DEBUG   SenderThread:1855917 [sender.py:send():379] send: header
2024-03-27 19:04:01,652 DEBUG   SenderThread:1855917 [sender.py:send():379] send: run
2024-03-27 19:04:02,097 INFO    SenderThread:1855917 [dir_watcher.py:__init__():211] watching files in: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files
2024-03-27 19:04:02,097 INFO    SenderThread:1855917 [sender.py:_start_run_threads():1124] run started: h708adzm with start time 1711566241.648942
2024-03-27 19:04:02,100 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: check_version
2024-03-27 19:04:02,100 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: check_version
2024-03-27 19:04:02,267 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: run_start
2024-03-27 19:04:02,297 DEBUG   HandlerThread:1855917 [system_info.py:__init__():26] System info init
2024-03-27 19:04:02,297 DEBUG   HandlerThread:1855917 [system_info.py:__init__():41] System info init done
2024-03-27 19:04:02,297 INFO    HandlerThread:1855917 [system_monitor.py:start():194] Starting system monitor
2024-03-27 19:04:02,297 INFO    SystemMonitor:1855917 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-03-27 19:04:02,297 INFO    SystemMonitor:1855917 [interfaces.py:start():190] Started cpu monitoring
2024-03-27 19:04:02,298 INFO    HandlerThread:1855917 [system_monitor.py:probe():214] Collecting system info
2024-03-27 19:04:02,298 INFO    SystemMonitor:1855917 [interfaces.py:start():190] Started disk monitoring
2024-03-27 19:04:02,298 INFO    SystemMonitor:1855917 [interfaces.py:start():190] Started gpu monitoring
2024-03-27 19:04:02,299 INFO    SystemMonitor:1855917 [interfaces.py:start():190] Started memory monitoring
2024-03-27 19:04:02,299 INFO    SystemMonitor:1855917 [interfaces.py:start():190] Started network monitoring
2024-03-27 19:04:02,321 DEBUG   HandlerThread:1855917 [system_info.py:probe():150] Probing system
2024-03-27 19:04:02,322 DEBUG   HandlerThread:1855917 [gitlib.py:_init_repo():56] git repository is invalid
2024-03-27 19:04:02,322 DEBUG   HandlerThread:1855917 [system_info.py:probe():198] Probing system done
2024-03-27 19:04:02,322 DEBUG   HandlerThread:1855917 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-89-generic-x86_64-with-glibc2.35', 'python': '3.10.14', 'heartbeatAt': '2024-03-27T19:04:02.321789', 'startedAt': '2024-03-27T19:04:01.647083', 'docker': None, 'cuda': None, 'args': ('--model_name_or_path', 'google-bert/bert-base-uncased', '--output_dir', './run', '--do_train', '--do_eval', '--do_predict'), 'state': 'running', 'program': '/home/gordon1109/qa_test/run_qa.py', 'codePathLocal': 'run_qa.py', 'codePath': 'run_qa.py', 'host': 'sally', 'username': 'gordon1109', 'executable': '/home/gordon1109/.miniconda3/envs/qa_tune/bin/python', 'cpu_count': 16, 'cpu_count_logical': 32, 'cpu_freq': {'current': 4424.5814375, 'min': 3000.0, 'max': 4500.0}, 'cpu_freq_per_core': [{'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3586.606, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}], 'disk': {'/': {'total': 788.7955780029297, 'used': 47.308345794677734}}, 'gpu': 'NVIDIA GeForce RTX 4090', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4090', 'memory_total': 25757220864}], 'memory': {'total': 125.00867462158203}}
2024-03-27 19:04:02,322 INFO    HandlerThread:1855917 [system_monitor.py:probe():224] Finished collecting system info
2024-03-27 19:04:02,322 INFO    HandlerThread:1855917 [system_monitor.py:probe():227] Publishing system info
2024-03-27 19:04:02,322 DEBUG   HandlerThread:1855917 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-03-27 19:04:03,098 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/conda-environment.yaml
2024-03-27 19:04:03,338 DEBUG   HandlerThread:1855917 [system_info.py:_save_conda():222] Saving conda packages done
2024-03-27 19:04:03,339 INFO    HandlerThread:1855917 [system_monitor.py:probe():229] Finished publishing system info
2024-03-27 19:04:03,340 DEBUG   SenderThread:1855917 [sender.py:send():379] send: files
2024-03-27 19:04:03,340 INFO    SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-03-27 19:04:03,375 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: python_packages
2024-03-27 19:04:03,375 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: python_packages
2024-03-27 19:04:03,375 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:04:03,376 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:04:03,377 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:04:03,691 DEBUG   SenderThread:1855917 [sender.py:send():379] send: telemetry
2024-03-27 19:04:04,071 INFO    wandb-upload_0:1855917 [upload_job.py:push():131] Uploaded file /tmp/tmp1psp4pcbwandb/cmzt5n6v-wandb-metadata.json
2024-03-27 19:04:04,098 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/conda-environment.yaml
2024-03-27 19:04:04,098 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/requirements.txt
2024-03-27 19:04:04,098 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/wandb-metadata.json
2024-03-27 19:04:06,693 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:11,693 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:16,695 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:18,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:04:18,376 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:04:18,418 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:04:22,570 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:26,105 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:04:27,583 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:28,105 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:04:30,106 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:04:32,106 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:04:32,801 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:33,106 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/config.yaml
2024-03-27 19:04:33,338 INFO    StreamThr :1855917 [internal.py:wandb_internal():86] W&B internal server running at pid: 1855917, started at: 2024-03-27 19:04:33.338087
2024-03-27 19:04:33,339 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status
2024-03-27 19:04:33,340 INFO    WriterThread:1855917 [datastore.py:open_for_write():87] open: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/run-tszlk6iz.wandb
2024-03-27 19:04:33,340 DEBUG   SenderThread:1855917 [sender.py:send():379] send: header
2024-03-27 19:04:33,340 DEBUG   SenderThread:1855917 [sender.py:send():379] send: run
2024-03-27 19:04:33,375 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:04:33,376 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:04:33,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:04:33,742 INFO    SenderThread:1855917 [dir_watcher.py:__init__():211] watching files in: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files
2024-03-27 19:04:33,742 INFO    SenderThread:1855917 [sender.py:_start_run_threads():1124] run started: tszlk6iz with start time 1711566273.337732
2024-03-27 19:04:33,749 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: check_version
2024-03-27 19:04:33,749 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: check_version
2024-03-27 19:04:33,921 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: run_start
2024-03-27 19:04:33,922 DEBUG   HandlerThread:1855917 [system_info.py:__init__():26] System info init
2024-03-27 19:04:33,922 DEBUG   HandlerThread:1855917 [system_info.py:__init__():41] System info init done
2024-03-27 19:04:33,922 INFO    HandlerThread:1855917 [system_monitor.py:start():194] Starting system monitor
2024-03-27 19:04:33,922 INFO    SystemMonitor:1855917 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-03-27 19:04:33,923 INFO    HandlerThread:1855917 [system_monitor.py:probe():214] Collecting system info
2024-03-27 19:04:33,923 INFO    SystemMonitor:1855917 [interfaces.py:start():190] Started cpu monitoring
2024-03-27 19:04:33,923 INFO    SystemMonitor:1855917 [interfaces.py:start():190] Started disk monitoring
2024-03-27 19:04:33,924 INFO    SystemMonitor:1855917 [interfaces.py:start():190] Started gpu monitoring
2024-03-27 19:04:33,924 INFO    SystemMonitor:1855917 [interfaces.py:start():190] Started memory monitoring
2024-03-27 19:04:33,924 INFO    SystemMonitor:1855917 [interfaces.py:start():190] Started network monitoring
2024-03-27 19:04:33,967 DEBUG   HandlerThread:1855917 [system_info.py:probe():150] Probing system
2024-03-27 19:04:33,967 DEBUG   HandlerThread:1855917 [gitlib.py:_init_repo():56] git repository is invalid
2024-03-27 19:04:33,967 DEBUG   HandlerThread:1855917 [system_info.py:probe():198] Probing system done
2024-03-27 19:04:33,967 DEBUG   HandlerThread:1855917 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-89-generic-x86_64-with-glibc2.35', 'python': '3.10.14', 'heartbeatAt': '2024-03-27T19:04:33.967587', 'startedAt': '2024-03-27T19:04:33.335983', 'docker': None, 'cuda': None, 'args': ('--node-ip-address=140.112.90.72', '--node-manager-port=33181', '--object-store-name=/tmp/ray/session_2024-03-27_19-04-27_986745_1855858/sockets/plasma_store', '--raylet-name=/tmp/ray/session_2024-03-27_19-04-27_986745_1855858/sockets/raylet', '--redis-address=None', '--metrics-agent-port=52536', '--runtime-env-agent-port=54815', '--logging-rotate-bytes=536870912', '--logging-rotate-backup-count=5', '--runtime-env-agent-port=54815', '--gcs-address=140.112.90.72:44817', '--session-name=session_2024-03-27_19-04-27_986745_1855858', '--temp-dir=/tmp/ray', '--webui=127.0.0.1:8265', '--cluster-id=8366c6b584aa9ce726c1927d2af9c014ef82eaaddd946cb356a7cba8', '--startup-token=32', '--worker-launch-time-ms=1711566270449', '--runtime-env-hash=-1413791333'), 'state': 'running', 'program': '/home/gordon1109/.miniconda3/envs/qa_tune/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', 'codePathLocal': None, 'host': 'sally', 'username': 'gordon1109', 'executable': '/home/gordon1109/.miniconda3/envs/qa_tune/bin/python', 'cpu_count': 16, 'cpu_count_logical': 32, 'cpu_freq': {'current': 3542.56859375, 'min': 3000.0, 'max': 4500.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4964.736, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}], 'disk': {'/': {'total': 788.7955780029297, 'used': 47.309295654296875}}, 'gpu': 'NVIDIA GeForce RTX 4090', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4090', 'memory_total': 25757220864}], 'memory': {'total': 125.00867462158203}}
2024-03-27 19:04:33,967 INFO    HandlerThread:1855917 [system_monitor.py:probe():224] Finished collecting system info
2024-03-27 19:04:33,967 INFO    HandlerThread:1855917 [system_monitor.py:probe():227] Publishing system info
2024-03-27 19:04:33,967 DEBUG   HandlerThread:1855917 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-03-27 19:04:34,107 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:04:34,744 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/conda-environment.yaml
2024-03-27 19:04:34,983 DEBUG   HandlerThread:1855917 [system_info.py:_save_conda():222] Saving conda packages done
2024-03-27 19:04:34,983 INFO    HandlerThread:1855917 [system_monitor.py:probe():229] Finished publishing system info
2024-03-27 19:04:34,983 DEBUG   SenderThread:1855917 [sender.py:send():379] send: files
2024-03-27 19:04:34,984 INFO    SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-03-27 19:04:35,017 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: python_packages
2024-03-27 19:04:35,017 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: python_packages
2024-03-27 19:04:35,018 DEBUG   SenderThread:1855917 [sender.py:send():379] send: telemetry
2024-03-27 19:04:35,066 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:04:35,066 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:04:35,066 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:04:35,351 DEBUG   SenderThread:1855917 [sender.py:send():379] send: config
2024-03-27 19:04:35,352 DEBUG   SenderThread:1855917 [sender.py:send():379] send: metric
2024-03-27 19:04:35,352 DEBUG   SenderThread:1855917 [sender.py:send():379] send: telemetry
2024-03-27 19:04:35,352 DEBUG   SenderThread:1855917 [sender.py:send():379] send: metric
2024-03-27 19:04:35,352 WARNING SenderThread:1855917 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-03-27 19:04:35,352 DEBUG   SenderThread:1855917 [sender.py:send():379] send: telemetry
2024-03-27 19:04:35,649 INFO    wandb-upload_0:1855917 [upload_job.py:push():131] Uploaded file /tmp/tmpmtmt7bb2wandb/gxma4yqp-wandb-metadata.json
2024-03-27 19:04:35,742 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/conda-environment.yaml
2024-03-27 19:04:35,742 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/wandb-metadata.json
2024-03-27 19:04:35,743 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/requirements.txt
2024-03-27 19:04:36,107 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:04:38,025 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:38,353 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:43,025 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:43,353 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:48,026 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:48,354 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:48,375 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:04:48,376 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:04:48,377 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:04:50,017 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:04:50,018 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:04:50,058 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:04:53,614 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:54,076 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:58,615 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:04:59,077 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:02,112 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:02,299 DEBUG   SystemMonitor:1855917 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-03-27 19:05:02,300 DEBUG   SenderThread:1855917 [sender.py:send():379] send: stats
2024-03-27 19:05:03,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:05:03,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:05:03,376 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:05:03,644 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:04,080 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:04,748 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/config.yaml
2024-03-27 19:05:05,019 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:05:05,019 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:05:05,062 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:05:08,644 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:10,299 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:13,717 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:13,749 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log
2024-03-27 19:05:14,114 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:15,365 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:15,750 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log
2024-03-27 19:05:16,114 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:17,750 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log
2024-03-27 19:05:18,115 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:18,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:05:18,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:05:18,376 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:05:18,722 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:19,751 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log
2024-03-27 19:05:20,021 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:05:20,021 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:05:20,062 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:05:20,115 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:20,457 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:21,751 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log
2024-03-27 19:05:22,115 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:23,731 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:23,752 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log
2024-03-27 19:05:24,116 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:25,752 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log
2024-03-27 19:05:25,771 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:25,787 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: partial_history
2024-03-27 19:05:25,788 DEBUG   SenderThread:1855917 [sender.py:send():379] send: metric
2024-03-27 19:05:25,788 DEBUG   SenderThread:1855917 [sender.py:send():379] send: metric
2024-03-27 19:05:25,788 DEBUG   SenderThread:1855917 [sender.py:send():379] send: metric
2024-03-27 19:05:25,788 DEBUG   SenderThread:1855917 [sender.py:send():379] send: metric
2024-03-27 19:05:25,788 DEBUG   SenderThread:1855917 [sender.py:send():379] send: metric
2024-03-27 19:05:25,788 DEBUG   SenderThread:1855917 [sender.py:send():379] send: metric
2024-03-27 19:05:25,788 DEBUG   SenderThread:1855917 [sender.py:send():379] send: history
2024-03-27 19:05:25,788 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record
2024-03-27 19:05:25,788 INFO    SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-03-27 19:05:26,116 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:26,752 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/wandb-summary.json
2024-03-27 19:05:28,116 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:28,263 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: partial_history
2024-03-27 19:05:28,263 DEBUG   SenderThread:1855917 [sender.py:send():379] send: telemetry
2024-03-27 19:05:28,263 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record
2024-03-27 19:05:28,264 INFO    SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-03-27 19:05:28,264 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record
2024-03-27 19:05:28,265 INFO    SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-03-27 19:05:28,265 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record
2024-03-27 19:05:28,265 INFO    SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-03-27 19:05:28,265 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record
2024-03-27 19:05:28,265 INFO    SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-03-27 19:05:28,265 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record
2024-03-27 19:05:28,265 INFO    SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-03-27 19:05:28,266 DEBUG   SenderThread:1855917 [sender.py:send():379] send: history
2024-03-27 19:05:28,266 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record
2024-03-27 19:05:28,266 INFO    SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-03-27 19:05:28,753 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/wandb-summary.json
2024-03-27 19:05:28,753 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log
2024-03-27 19:05:29,378 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:29,753 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log
2024-03-27 19:05:29,935 INFO    memory    :1855917 [interfaces.py:monitor():140] Process proc.memory.rssMB has exited.
2024-03-27 19:05:29,936 DEBUG   SystemMonitor:1855917 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-03-27 19:05:29,936 DEBUG   SystemMonitor:1855917 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-03-27 19:05:29,936 DEBUG   SystemMonitor:1855917 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-03-27 19:05:29,937 DEBUG   SenderThread:1855917 [sender.py:send():379] send: stats
2024-03-27 19:05:30,117 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:30,753 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log
2024-03-27 19:05:30,938 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:32,301 DEBUG   SenderThread:1855917 [sender.py:send():379] send: stats
2024-03-27 19:05:33,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:05:33,376 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:05:33,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:05:34,562 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:35,942 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:36,755 INFO    Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/config.yaml
2024-03-27 19:05:39,562 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:41,180 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:44,563 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:46,180 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:48,120 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:48,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:05:48,376 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:05:48,378 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:05:49,571 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:50,121 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:51,181 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:52,121 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:54,121 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:54,837 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:56,122 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:56,181 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:05:58,122 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:05:59,876 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:01,123 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:01,182 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:02,301 DEBUG   SenderThread:1855917 [sender.py:send():379] send: stats
2024-03-27 19:06:03,123 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:03,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:06:03,376 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:06:03,377 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:06:04,906 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:05,124 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:06,182 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:07,124 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:09,124 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:09,922 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:11,125 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:11,183 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:13,125 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:14,950 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:15,126 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:16,183 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:17,126 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:18,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:06:18,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:06:18,376 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:06:19,126 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:20,012 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:21,127 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:21,184 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:23,127 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:25,014 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:25,128 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:26,184 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:27,128 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:29,129 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:30,102 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:31,129 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:31,185 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:32,302 DEBUG   SenderThread:1855917 [sender.py:send():379] send: stats
2024-03-27 19:06:33,129 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:33,376 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status
2024-03-27 19:06:33,376 DEBUG   SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status
2024-03-27 19:06:33,378 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-27 19:06:35,112 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:35,130 INFO    Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log
2024-03-27 19:06:36,185 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:40,150 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report
2024-03-27 19:06:41,186 DEBUG   HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report