nornor02 commited on
Commit
c91af39
·
1 Parent(s): 4e17ad5

Training in progress, step 1500

Browse files
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:3937a677c8267caa7345b10575877df18e7170cc530e9a5b8354c4b6bc41badf
3
  size 328693404
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:93d24e9540ec52b3e2bf0001cb7dc98f438fd7ff1c46683971fbc8e09649f060
3
  size 328693404
runs/Dec21_19-44-12_fa65ad7b7892/events.out.tfevents.1703187853.fa65ad7b7892.130.1 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:ab84a4efd36b16fe6fe05ac5833e95d759763dc2a407bd402186d8068d8dc668
3
- size 4714
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:722ab7d5d11304fcac8d61563f00e7adcc5c68672fb3906b263a245951d45d15
3
+ size 5225
wandb/debug-internal.log CHANGED
@@ -308,3 +308,68 @@
308
  2023-12-21 19:48:39,816 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
309
  2023-12-21 19:48:42,084 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
310
  2023-12-21 19:48:42,857 DEBUG SenderThread:204 [sender.py:send():382] send: stats
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
308
  2023-12-21 19:48:39,816 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
309
  2023-12-21 19:48:42,084 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
310
  2023-12-21 19:48:42,857 DEBUG SenderThread:204 [sender.py:send():382] send: stats
311
+ 2023-12-21 19:48:44,817 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
312
+ 2023-12-21 19:48:47,858 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
313
+ 2023-12-21 19:48:49,818 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
314
+ 2023-12-21 19:48:52,859 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
315
+ 2023-12-21 19:48:54,819 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
316
+ 2023-12-21 19:48:57,860 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
317
+ 2023-12-21 19:48:59,820 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
318
+ 2023-12-21 19:49:02,860 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
319
+ 2023-12-21 19:49:04,821 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
320
+ 2023-12-21 19:49:07,861 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
321
+ 2023-12-21 19:49:09,823 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
322
+ 2023-12-21 19:49:12,858 DEBUG SenderThread:204 [sender.py:send():382] send: stats
323
+ 2023-12-21 19:49:13,859 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
324
+ 2023-12-21 19:49:14,830 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
325
+ 2023-12-21 19:49:18,860 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
326
+ 2023-12-21 19:49:19,831 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
327
+ 2023-12-21 19:49:23,860 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
328
+ 2023-12-21 19:49:24,832 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
329
+ 2023-12-21 19:49:28,861 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
330
+ 2023-12-21 19:49:29,833 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
331
+ 2023-12-21 19:49:33,862 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
332
+ 2023-12-21 19:49:34,834 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
333
+ 2023-12-21 19:49:38,863 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
334
+ 2023-12-21 19:49:39,846 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
335
+ 2023-12-21 19:49:42,859 DEBUG SenderThread:204 [sender.py:send():382] send: stats
336
+ 2023-12-21 19:49:44,847 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
337
+ 2023-12-21 19:49:44,859 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
338
+ 2023-12-21 19:49:49,848 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
339
+ 2023-12-21 19:49:49,860 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
340
+ 2023-12-21 19:49:54,850 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
341
+ 2023-12-21 19:49:54,861 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
342
+ 2023-12-21 19:49:59,856 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
343
+ 2023-12-21 19:49:59,862 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
344
+ 2023-12-21 19:50:04,857 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
345
+ 2023-12-21 19:50:04,863 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
346
+ 2023-12-21 19:50:09,858 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
347
+ 2023-12-21 19:50:09,863 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
348
+ 2023-12-21 19:50:12,860 DEBUG SenderThread:204 [sender.py:send():382] send: stats
349
+ 2023-12-21 19:50:14,859 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
350
+ 2023-12-21 19:50:15,861 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
351
+ 2023-12-21 19:50:19,865 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
352
+ 2023-12-21 19:50:20,862 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
353
+ 2023-12-21 19:50:24,867 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
354
+ 2023-12-21 19:50:25,863 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
355
+ 2023-12-21 19:50:29,871 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
356
+ 2023-12-21 19:50:30,864 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
357
+ 2023-12-21 19:50:34,872 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
358
+ 2023-12-21 19:50:35,865 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
359
+ 2023-12-21 19:50:39,873 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
360
+ 2023-12-21 19:50:40,265 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: partial_history
361
+ 2023-12-21 19:50:40,267 DEBUG SenderThread:204 [sender.py:send():382] send: history
362
+ 2023-12-21 19:50:40,267 DEBUG SenderThread:204 [sender.py:send_request():409] send_request: summary_record
363
+ 2023-12-21 19:50:40,267 INFO SenderThread:204 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
364
+ 2023-12-21 19:50:41,209 INFO Thread-12 :204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20231221_194223-lb3081d4/files/wandb-summary.json
365
+ 2023-12-21 19:50:41,268 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
366
+ 2023-12-21 19:50:42,476 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: partial_history
367
+ 2023-12-21 19:50:42,477 DEBUG SenderThread:204 [sender.py:send():382] send: history
368
+ 2023-12-21 19:50:42,477 DEBUG SenderThread:204 [sender.py:send_request():409] send_request: summary_record
369
+ 2023-12-21 19:50:42,478 INFO SenderThread:204 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
370
+ 2023-12-21 19:50:42,861 DEBUG SenderThread:204 [sender.py:send():382] send: stats
371
+ 2023-12-21 19:50:43,210 INFO Thread-12 :204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20231221_194223-lb3081d4/files/wandb-summary.json
372
+ 2023-12-21 19:50:44,992 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
373
+ 2023-12-21 19:50:46,862 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
374
+ 2023-12-21 19:50:49,993 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
375
+ 2023-12-21 19:50:51,863 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20231221_194223-lb3081d4/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/train_runtime": 65.2496, "train/train_samples_per_second": 5.517, "train/train_steps_per_second": 0.383, "train/total_flos": 11935962593280.0, "train/train_loss": 0.013434079885482788, "train/epoch": 200.0, "train/global_step": 1000, "_timestamp": 1703188108.7855973, "_runtime": 365.4836673736572, "_step": 2, "train/loss": 0.0, "train/learning_rate": 6.666666666666667e-06}
 
1
+ {"train/train_runtime": 389.1313, "train/train_samples_per_second": 55.508, "train/train_steps_per_second": 3.855, "train/total_flos": 716157755596800.0, "train/train_loss": 8.66718401084654e-07, "train/epoch": 300.0, "train/global_step": 1500, "_timestamp": 1703188242.4755788, "_runtime": 499.1736488342285, "_step": 4, "train/loss": 0.0, "train/learning_rate": 0.0}
wandb/run-20231221_194223-lb3081d4/logs/debug-internal.log CHANGED
@@ -308,3 +308,68 @@
308
  2023-12-21 19:48:39,816 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
309
  2023-12-21 19:48:42,084 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
310
  2023-12-21 19:48:42,857 DEBUG SenderThread:204 [sender.py:send():382] send: stats
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
308
  2023-12-21 19:48:39,816 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
309
  2023-12-21 19:48:42,084 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
310
  2023-12-21 19:48:42,857 DEBUG SenderThread:204 [sender.py:send():382] send: stats
311
+ 2023-12-21 19:48:44,817 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
312
+ 2023-12-21 19:48:47,858 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
313
+ 2023-12-21 19:48:49,818 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
314
+ 2023-12-21 19:48:52,859 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
315
+ 2023-12-21 19:48:54,819 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
316
+ 2023-12-21 19:48:57,860 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
317
+ 2023-12-21 19:48:59,820 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
318
+ 2023-12-21 19:49:02,860 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
319
+ 2023-12-21 19:49:04,821 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
320
+ 2023-12-21 19:49:07,861 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
321
+ 2023-12-21 19:49:09,823 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
322
+ 2023-12-21 19:49:12,858 DEBUG SenderThread:204 [sender.py:send():382] send: stats
323
+ 2023-12-21 19:49:13,859 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
324
+ 2023-12-21 19:49:14,830 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
325
+ 2023-12-21 19:49:18,860 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
326
+ 2023-12-21 19:49:19,831 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
327
+ 2023-12-21 19:49:23,860 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
328
+ 2023-12-21 19:49:24,832 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
329
+ 2023-12-21 19:49:28,861 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
330
+ 2023-12-21 19:49:29,833 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
331
+ 2023-12-21 19:49:33,862 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
332
+ 2023-12-21 19:49:34,834 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
333
+ 2023-12-21 19:49:38,863 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
334
+ 2023-12-21 19:49:39,846 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
335
+ 2023-12-21 19:49:42,859 DEBUG SenderThread:204 [sender.py:send():382] send: stats
336
+ 2023-12-21 19:49:44,847 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
337
+ 2023-12-21 19:49:44,859 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
338
+ 2023-12-21 19:49:49,848 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
339
+ 2023-12-21 19:49:49,860 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
340
+ 2023-12-21 19:49:54,850 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
341
+ 2023-12-21 19:49:54,861 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
342
+ 2023-12-21 19:49:59,856 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
343
+ 2023-12-21 19:49:59,862 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
344
+ 2023-12-21 19:50:04,857 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
345
+ 2023-12-21 19:50:04,863 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
346
+ 2023-12-21 19:50:09,858 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
347
+ 2023-12-21 19:50:09,863 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
348
+ 2023-12-21 19:50:12,860 DEBUG SenderThread:204 [sender.py:send():382] send: stats
349
+ 2023-12-21 19:50:14,859 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
350
+ 2023-12-21 19:50:15,861 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
351
+ 2023-12-21 19:50:19,865 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
352
+ 2023-12-21 19:50:20,862 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
353
+ 2023-12-21 19:50:24,867 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
354
+ 2023-12-21 19:50:25,863 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
355
+ 2023-12-21 19:50:29,871 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
356
+ 2023-12-21 19:50:30,864 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
357
+ 2023-12-21 19:50:34,872 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
358
+ 2023-12-21 19:50:35,865 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
359
+ 2023-12-21 19:50:39,873 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
360
+ 2023-12-21 19:50:40,265 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: partial_history
361
+ 2023-12-21 19:50:40,267 DEBUG SenderThread:204 [sender.py:send():382] send: history
362
+ 2023-12-21 19:50:40,267 DEBUG SenderThread:204 [sender.py:send_request():409] send_request: summary_record
363
+ 2023-12-21 19:50:40,267 INFO SenderThread:204 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
364
+ 2023-12-21 19:50:41,209 INFO Thread-12 :204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20231221_194223-lb3081d4/files/wandb-summary.json
365
+ 2023-12-21 19:50:41,268 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
366
+ 2023-12-21 19:50:42,476 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: partial_history
367
+ 2023-12-21 19:50:42,477 DEBUG SenderThread:204 [sender.py:send():382] send: history
368
+ 2023-12-21 19:50:42,477 DEBUG SenderThread:204 [sender.py:send_request():409] send_request: summary_record
369
+ 2023-12-21 19:50:42,478 INFO SenderThread:204 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
370
+ 2023-12-21 19:50:42,861 DEBUG SenderThread:204 [sender.py:send():382] send: stats
371
+ 2023-12-21 19:50:43,210 INFO Thread-12 :204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20231221_194223-lb3081d4/files/wandb-summary.json
372
+ 2023-12-21 19:50:44,992 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
373
+ 2023-12-21 19:50:46,862 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
374
+ 2023-12-21 19:50:49,993 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
375
+ 2023-12-21 19:50:51,863 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20231221_194223-lb3081d4/run-lb3081d4.wandb CHANGED
Binary files a/wandb/run-20231221_194223-lb3081d4/run-lb3081d4.wandb and b/wandb/run-20231221_194223-lb3081d4/run-lb3081d4.wandb differ