nornor02 commited on
Commit
32705d5
·
1 Parent(s): 30cb1fb

Training in progress, step 5500

Browse files
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:79fe0ec08ccbafeb04414cf137a945100bf4a3fbba37922ad952ec27881ab157
3
  size 328693404
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d15eb5a22e9bad5867a31548b33c5d75172f29bf799776312c37c6c1d845504b
3
  size 328693404
runs/Dec21_19-57-36_fa65ad7b7892/events.out.tfevents.1703188656.fa65ad7b7892.130.2 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:13b847ad8de5987f2a1b3e06d566e8fc116fb40cfcfd3079ea0bf0567d243f19
3
- size 5942
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:222995c53d257ff3a417efa1a477a4a55d17ce5c957810421cd357760bc7cb47
3
+ size 6099
wandb/debug-internal.log CHANGED
@@ -1410,3 +1410,43 @@
1410
  2023-12-21 20:10:53,592 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1411
  2023-12-21 20:10:57,000 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1412
  2023-12-21 20:10:58,593 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1410
  2023-12-21 20:10:53,592 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1411
  2023-12-21 20:10:57,000 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1412
  2023-12-21 20:10:58,593 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1413
+ 2023-12-21 20:11:02,001 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1414
+ 2023-12-21 20:11:03,593 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1415
+ 2023-12-21 20:11:06,106 DEBUG SenderThread:204 [sender.py:send():382] send: stats
1416
+ 2023-12-21 20:11:07,003 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1417
+ 2023-12-21 20:11:09,107 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1418
+ 2023-12-21 20:11:12,006 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1419
+ 2023-12-21 20:11:14,108 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1420
+ 2023-12-21 20:11:17,007 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1421
+ 2023-12-21 20:11:19,109 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1422
+ 2023-12-21 20:11:22,012 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1423
+ 2023-12-21 20:11:24,110 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1424
+ 2023-12-21 20:11:27,013 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1425
+ 2023-12-21 20:11:29,111 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1426
+ 2023-12-21 20:11:32,014 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1427
+ 2023-12-21 20:11:34,111 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1428
+ 2023-12-21 20:11:36,107 DEBUG SenderThread:204 [sender.py:send():382] send: stats
1429
+ 2023-12-21 20:11:37,015 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1430
+ 2023-12-21 20:11:40,108 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1431
+ 2023-12-21 20:11:42,016 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1432
+ 2023-12-21 20:11:45,109 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1433
+ 2023-12-21 20:11:47,017 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1434
+ 2023-12-21 20:11:50,110 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1435
+ 2023-12-21 20:11:52,019 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1436
+ 2023-12-21 20:11:55,111 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1437
+ 2023-12-21 20:11:57,020 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1438
+ 2023-12-21 20:12:00,111 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1439
+ 2023-12-21 20:12:02,022 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1440
+ 2023-12-21 20:12:05,112 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1441
+ 2023-12-21 20:12:05,233 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: partial_history
1442
+ 2023-12-21 20:12:05,234 DEBUG SenderThread:204 [sender.py:send():382] send: history
1443
+ 2023-12-21 20:12:05,234 DEBUG SenderThread:204 [sender.py:send_request():409] send_request: summary_record
1444
+ 2023-12-21 20:12:05,234 INFO SenderThread:204 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1445
+ 2023-12-21 20:12:05,691 INFO Thread-12 :204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20231221_194223-lb3081d4/files/wandb-summary.json
1446
+ 2023-12-21 20:12:06,108 DEBUG SenderThread:204 [sender.py:send():382] send: stats
1447
+ 2023-12-21 20:12:07,190 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1448
+ 2023-12-21 20:12:10,498 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1449
+ 2023-12-21 20:12:10,693 INFO Thread-12 :204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20231221_194223-lb3081d4/files/output.log
1450
+ 2023-12-21 20:12:12,191 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1451
+ 2023-12-21 20:12:15,498 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1452
+ 2023-12-21 20:12:17,194 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
wandb/run-20231221_194223-lb3081d4/files/output.log CHANGED
@@ -35,4 +35,6 @@ Checkpoint destination directory /kaggle/working/checkpoint-1500 already exists
35
  warnings.warn('Was asked to gather along dimension 0, but all '
36
  /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
37
  warnings.warn('Was asked to gather along dimension 0, but all '
 
 
38
  /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
 
35
  warnings.warn('Was asked to gather along dimension 0, but all '
36
  /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
37
  warnings.warn('Was asked to gather along dimension 0, but all '
38
+ /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
39
+ warnings.warn('Was asked to gather along dimension 0, but all '
40
  /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
wandb/run-20231221_194223-lb3081d4/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/train_runtime": 389.1313, "train/train_samples_per_second": 55.508, "train/train_steps_per_second": 3.855, "train/total_flos": 716157755596800.0, "train/train_loss": 8.66718401084654e-07, "train/epoch": 250.0, "train/global_step": 5000, "_timestamp": 1703189446.3344004, "_runtime": 1703.0324704647064, "_step": 14, "train/loss": 0.0, "train/learning_rate": 3.3333333333333333e-06}
 
1
+ {"train/train_runtime": 389.1313, "train/train_samples_per_second": 55.508, "train/train_steps_per_second": 3.855, "train/total_flos": 716157755596800.0, "train/train_loss": 8.66718401084654e-07, "train/epoch": 275.0, "train/global_step": 5500, "_timestamp": 1703189525.2322974, "_runtime": 1781.9303674697876, "_step": 15, "train/loss": 0.0, "train/learning_rate": 1.6666666666666667e-06}
wandb/run-20231221_194223-lb3081d4/logs/debug-internal.log CHANGED
@@ -1410,3 +1410,43 @@
1410
  2023-12-21 20:10:53,592 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1411
  2023-12-21 20:10:57,000 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1412
  2023-12-21 20:10:58,593 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1410
  2023-12-21 20:10:53,592 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1411
  2023-12-21 20:10:57,000 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1412
  2023-12-21 20:10:58,593 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1413
+ 2023-12-21 20:11:02,001 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1414
+ 2023-12-21 20:11:03,593 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1415
+ 2023-12-21 20:11:06,106 DEBUG SenderThread:204 [sender.py:send():382] send: stats
1416
+ 2023-12-21 20:11:07,003 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1417
+ 2023-12-21 20:11:09,107 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1418
+ 2023-12-21 20:11:12,006 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1419
+ 2023-12-21 20:11:14,108 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1420
+ 2023-12-21 20:11:17,007 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1421
+ 2023-12-21 20:11:19,109 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1422
+ 2023-12-21 20:11:22,012 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1423
+ 2023-12-21 20:11:24,110 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1424
+ 2023-12-21 20:11:27,013 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1425
+ 2023-12-21 20:11:29,111 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1426
+ 2023-12-21 20:11:32,014 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1427
+ 2023-12-21 20:11:34,111 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1428
+ 2023-12-21 20:11:36,107 DEBUG SenderThread:204 [sender.py:send():382] send: stats
1429
+ 2023-12-21 20:11:37,015 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1430
+ 2023-12-21 20:11:40,108 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1431
+ 2023-12-21 20:11:42,016 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1432
+ 2023-12-21 20:11:45,109 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1433
+ 2023-12-21 20:11:47,017 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1434
+ 2023-12-21 20:11:50,110 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1435
+ 2023-12-21 20:11:52,019 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1436
+ 2023-12-21 20:11:55,111 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1437
+ 2023-12-21 20:11:57,020 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1438
+ 2023-12-21 20:12:00,111 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1439
+ 2023-12-21 20:12:02,022 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1440
+ 2023-12-21 20:12:05,112 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1441
+ 2023-12-21 20:12:05,233 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: partial_history
1442
+ 2023-12-21 20:12:05,234 DEBUG SenderThread:204 [sender.py:send():382] send: history
1443
+ 2023-12-21 20:12:05,234 DEBUG SenderThread:204 [sender.py:send_request():409] send_request: summary_record
1444
+ 2023-12-21 20:12:05,234 INFO SenderThread:204 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1445
+ 2023-12-21 20:12:05,691 INFO Thread-12 :204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20231221_194223-lb3081d4/files/wandb-summary.json
1446
+ 2023-12-21 20:12:06,108 DEBUG SenderThread:204 [sender.py:send():382] send: stats
1447
+ 2023-12-21 20:12:07,190 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1448
+ 2023-12-21 20:12:10,498 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1449
+ 2023-12-21 20:12:10,693 INFO Thread-12 :204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20231221_194223-lb3081d4/files/output.log
1450
+ 2023-12-21 20:12:12,191 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
1451
+ 2023-12-21 20:12:15,498 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: status_report
1452
+ 2023-12-21 20:12:17,194 DEBUG HandlerThread:204 [handler.py:handle_request():146] handle_request: keepalive
wandb/run-20231221_194223-lb3081d4/run-lb3081d4.wandb CHANGED
Binary files a/wandb/run-20231221_194223-lb3081d4/run-lb3081d4.wandb and b/wandb/run-20231221_194223-lb3081d4/run-lb3081d4.wandb differ