nornor02 commited on
Commit
b880bcb
·
verified ·
1 Parent(s): 54f1ebe

Training in progress, step 4500

Browse files
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:dc81d7660fc71a769834227c71601030cdd1445903a5c8bbadf85ae1146e71bf
3
  size 328693404
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4b6c29fbfce895408d6996c09ce561e957c5cb99ce1218f1ff1ab122ce7b6255
3
  size 328693404
runs/Jan12_02-28-34_ea19ce76d6cc/events.out.tfevents.1705026515.ea19ce76d6cc.10822.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:3f8a50bb8a70a34e43d10755db4a358b5f20c661f3b1b27a8180d547734f937e
3
- size 5621
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5ba8ef601ea4a898435fee2dd11f0c9d9f63325fdd161c49f5d9669c1ca578e0
3
+ size 5778
wandb/debug-internal.log CHANGED
@@ -410,3 +410,48 @@
410
  2024-01-12 02:40:22,588 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
411
  2024-01-12 02:40:27,125 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
412
  2024-01-12 02:40:27,589 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
410
  2024-01-12 02:40:22,588 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
411
  2024-01-12 02:40:27,125 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
412
  2024-01-12 02:40:27,589 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
413
+ 2024-01-12 02:40:32,126 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
414
+ 2024-01-12 02:40:32,590 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
415
+ 2024-01-12 02:40:37,136 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
416
+ 2024-01-12 02:40:37,590 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
417
+ 2024-01-12 02:40:38,311 DEBUG SenderThread:10874 [sender.py:send():382] send: stats
418
+ 2024-01-12 02:40:42,137 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
419
+ 2024-01-12 02:40:43,313 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
420
+ 2024-01-12 02:40:47,139 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
421
+ 2024-01-12 02:40:48,314 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
422
+ 2024-01-12 02:40:52,140 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
423
+ 2024-01-12 02:40:53,314 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
424
+ 2024-01-12 02:40:57,141 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
425
+ 2024-01-12 02:40:58,315 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
426
+ 2024-01-12 02:41:02,147 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
427
+ 2024-01-12 02:41:03,316 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
428
+ 2024-01-12 02:41:07,148 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
429
+ 2024-01-12 02:41:08,308 DEBUG SenderThread:10874 [sender.py:send():382] send: stats
430
+ 2024-01-12 02:41:09,309 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
431
+ 2024-01-12 02:41:12,149 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
432
+ 2024-01-12 02:41:14,310 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
433
+ 2024-01-12 02:41:17,157 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
434
+ 2024-01-12 02:41:19,311 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
435
+ 2024-01-12 02:41:22,163 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
436
+ 2024-01-12 02:41:24,312 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
437
+ 2024-01-12 02:41:27,164 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
438
+ 2024-01-12 02:41:29,312 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
439
+ 2024-01-12 02:41:32,165 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
440
+ 2024-01-12 02:41:34,313 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
441
+ 2024-01-12 02:41:37,166 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
442
+ 2024-01-12 02:41:38,310 DEBUG SenderThread:10874 [sender.py:send():382] send: stats
443
+ 2024-01-12 02:41:38,666 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: partial_history
444
+ 2024-01-12 02:41:38,667 DEBUG SenderThread:10874 [sender.py:send():382] send: history
445
+ 2024-01-12 02:41:38,668 DEBUG SenderThread:10874 [sender.py:send_request():409] send_request: summary_record
446
+ 2024-01-12 02:41:38,668 INFO SenderThread:10874 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
447
+ 2024-01-12 02:41:39,473 INFO Thread-12 :10874 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240112_022837-twhkj0lf/files/wandb-summary.json
448
+ 2024-01-12 02:41:39,669 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
449
+ 2024-01-12 02:41:40,473 INFO Thread-12 :10874 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240112_022837-twhkj0lf/files/output.log
450
+ 2024-01-12 02:41:42,474 INFO Thread-12 :10874 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240112_022837-twhkj0lf/files/output.log
451
+ 2024-01-12 02:41:42,578 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
452
+ 2024-01-12 02:41:42,578 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
453
+ 2024-01-12 02:41:45,452 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
454
+ 2024-01-12 02:41:47,579 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
455
+ 2024-01-12 02:41:50,453 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
456
+ 2024-01-12 02:41:52,581 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
457
+ 2024-01-12 02:41:55,454 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240112_022837-twhkj0lf/files/output.log CHANGED
@@ -23,4 +23,7 @@ Checkpoint destination directory /kaggle/working/checkpoint-3500 already exists
23
  /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
24
  warnings.warn('Was asked to gather along dimension 0, but all '
25
  Checkpoint destination directory /kaggle/working/checkpoint-4000 already exists and is non-empty.Saving will proceed but saved results may be invalid.
 
 
 
26
  /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
 
23
  /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
24
  warnings.warn('Was asked to gather along dimension 0, but all '
25
  Checkpoint destination directory /kaggle/working/checkpoint-4000 already exists and is non-empty.Saving will proceed but saved results may be invalid.
26
+ /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
27
+ warnings.warn('Was asked to gather along dimension 0, but all '
28
+ Checkpoint destination directory /kaggle/working/checkpoint-4500 already exists and is non-empty.Saving will proceed but saved results may be invalid.
29
  /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
wandb/run-20240112_022837-twhkj0lf/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 0.5064, "train/learning_rate": 4.615384615384616e-06, "train/epoch": 307.69, "train/global_step": 4000, "_timestamp": 1705027214.829182, "_runtime": 697.1702299118042, "_step": 7}
 
1
+ {"train/loss": 0.4679, "train/learning_rate": 2.6923076923076923e-06, "train/epoch": 346.15, "train/global_step": 4500, "_timestamp": 1705027298.665949, "_runtime": 781.0069971084595, "_step": 8}
wandb/run-20240112_022837-twhkj0lf/logs/debug-internal.log CHANGED
@@ -410,3 +410,48 @@
410
  2024-01-12 02:40:22,588 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
411
  2024-01-12 02:40:27,125 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
412
  2024-01-12 02:40:27,589 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
410
  2024-01-12 02:40:22,588 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
411
  2024-01-12 02:40:27,125 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
412
  2024-01-12 02:40:27,589 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
413
+ 2024-01-12 02:40:32,126 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
414
+ 2024-01-12 02:40:32,590 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
415
+ 2024-01-12 02:40:37,136 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
416
+ 2024-01-12 02:40:37,590 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
417
+ 2024-01-12 02:40:38,311 DEBUG SenderThread:10874 [sender.py:send():382] send: stats
418
+ 2024-01-12 02:40:42,137 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
419
+ 2024-01-12 02:40:43,313 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
420
+ 2024-01-12 02:40:47,139 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
421
+ 2024-01-12 02:40:48,314 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
422
+ 2024-01-12 02:40:52,140 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
423
+ 2024-01-12 02:40:53,314 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
424
+ 2024-01-12 02:40:57,141 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
425
+ 2024-01-12 02:40:58,315 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
426
+ 2024-01-12 02:41:02,147 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
427
+ 2024-01-12 02:41:03,316 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
428
+ 2024-01-12 02:41:07,148 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
429
+ 2024-01-12 02:41:08,308 DEBUG SenderThread:10874 [sender.py:send():382] send: stats
430
+ 2024-01-12 02:41:09,309 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
431
+ 2024-01-12 02:41:12,149 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
432
+ 2024-01-12 02:41:14,310 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
433
+ 2024-01-12 02:41:17,157 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
434
+ 2024-01-12 02:41:19,311 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
435
+ 2024-01-12 02:41:22,163 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
436
+ 2024-01-12 02:41:24,312 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
437
+ 2024-01-12 02:41:27,164 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
438
+ 2024-01-12 02:41:29,312 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
439
+ 2024-01-12 02:41:32,165 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
440
+ 2024-01-12 02:41:34,313 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
441
+ 2024-01-12 02:41:37,166 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
442
+ 2024-01-12 02:41:38,310 DEBUG SenderThread:10874 [sender.py:send():382] send: stats
443
+ 2024-01-12 02:41:38,666 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: partial_history
444
+ 2024-01-12 02:41:38,667 DEBUG SenderThread:10874 [sender.py:send():382] send: history
445
+ 2024-01-12 02:41:38,668 DEBUG SenderThread:10874 [sender.py:send_request():409] send_request: summary_record
446
+ 2024-01-12 02:41:38,668 INFO SenderThread:10874 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
447
+ 2024-01-12 02:41:39,473 INFO Thread-12 :10874 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240112_022837-twhkj0lf/files/wandb-summary.json
448
+ 2024-01-12 02:41:39,669 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
449
+ 2024-01-12 02:41:40,473 INFO Thread-12 :10874 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240112_022837-twhkj0lf/files/output.log
450
+ 2024-01-12 02:41:42,474 INFO Thread-12 :10874 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240112_022837-twhkj0lf/files/output.log
451
+ 2024-01-12 02:41:42,578 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
452
+ 2024-01-12 02:41:42,578 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
453
+ 2024-01-12 02:41:45,452 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
454
+ 2024-01-12 02:41:47,579 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
455
+ 2024-01-12 02:41:50,453 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
456
+ 2024-01-12 02:41:52,581 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
457
+ 2024-01-12 02:41:55,454 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240112_022837-twhkj0lf/run-twhkj0lf.wandb CHANGED
Binary files a/wandb/run-20240112_022837-twhkj0lf/run-twhkj0lf.wandb and b/wandb/run-20240112_022837-twhkj0lf/run-twhkj0lf.wandb differ